2023-10-12T00:01:23.436 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-12T00:01:23.437 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157 --verbose --name yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/orig.config.yaml 2023-10-12T00:01:23.500 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T00:01:23.585 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157 branch: wip-yuri-testing-2023-10-10-0840-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '7422157' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-10-10-0840-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-qa-63004-reef sha1: 4305a5835ec2627a352020742c8983c32bac9341 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 1482 sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-qa-63004-reef suite_path: /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 4305a5835ec2627a352020742c8983c32bac9341 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2JLp2+vfzHVyxL6Acrw4wk1jHYH8MuiL4DWD9tM4S7ijSMYlbEw5+h08dafiwWGRp9Fogc9JG7tebz2ws0vSE= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZJXmqjfBSt/G/tZUHbg1vA/GOsb80hVRDsm9AHTjNE85QX13Lf8BegOL5Bji68NC+JX6KnRItmilTlMA9TtVA= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-11_18:32:33 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-12T00:01:23.585 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa; will attempt to use it 2023-10-12T00:01:23.687 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks 2023-10-12T00:01:23.687 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-12T00:01:23.690 INFO:teuthology.task.internal:Checking packages... 2023-10-12T00:01:23.929 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22' 2023-10-12T00:01:23.929 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-12T00:01:23.930 INFO:teuthology.packaging:ref: None 2023-10-12T00:01:23.930 INFO:teuthology.packaging:tag: None 2023-10-12T00:01:23.930 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-10-0840-reef 2023-10-12T00:01:23.931 INFO:teuthology.packaging:sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-12T00:01:23.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-10-0840-reef 2023-10-12T00:01:24.147 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-666.gaa3250ce 2023-10-12T00:01:24.147 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-12T00:01:24.175 INFO:teuthology.task.internal:no buildpackages task found 2023-10-12T00:01:24.175 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-12T00:01:24.184 INFO:teuthology.task.internal:Saving configuration 2023-10-12T00:01:24.203 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-12T00:01:24.217 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T00:01:24.244 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 23:55:30.444168', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2JLp2+vfzHVyxL6Acrw4wk1jHYH8MuiL4DWD9tM4S7ijSMYlbEw5+h08dafiwWGRp9Fogc9JG7tebz2ws0vSE='} 2023-10-12T00:01:24.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 23:55:30.442643', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZJXmqjfBSt/G/tZUHbg1vA/GOsb80hVRDsm9AHTjNE85QX13Lf8BegOL5Bji68NC+JX6KnRItmilTlMA9TtVA='} 2023-10-12T00:01:24.266 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-12T00:01:24.289 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-10-12T00:01:24.290 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-10-12T00:01:24.290 INFO:teuthology.run_tasks:Running task console_log... 2023-10-12T00:01:24.362 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd034fb5a60>, signals=[15]) 2023-10-12T00:01:24.364 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-12T00:01:24.396 INFO:teuthology.task.internal:Opening connections... 2023-10-12T00:01:24.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:01:24.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:01:24.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:01:24.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:01:24.536 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-12T00:01:24.542 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2023-10-12T00:01:24.565 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-10-12T00:01:24.565 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2023-10-12T00:01:24.626 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2023-10-12T00:01:24.627 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8" 2023-10-12T00:01:24.627 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2023-10-12T00:01:24.627 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2023-10-12T00:01:24.627 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8" 2023-10-12T00:01:24.627 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2023-10-12T00:01:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-12T00:01:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2023-10-12T00:01:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-12T00:01:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2023-10-12T00:01:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T00:01:24.629 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T00:01:24.629 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-12T00:01:24.630 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-10-12T00:01:24.653 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2023-10-12T00:01:24.676 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2023-10-12T00:01:24.677 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2023-10-12T00:01:24.738 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2023-10-12T00:01:24.739 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8" 2023-10-12T00:01:24.739 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2023-10-12T00:01:24.739 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2023-10-12T00:01:24.739 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2023-10-12T00:01:24.740 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2023-10-12T00:01:24.740 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-12T00:01:24.740 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2023-10-12T00:01:24.740 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-12T00:01:24.740 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2023-10-12T00:01:24.741 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T00:01:24.741 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T00:01:24.741 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-12T00:01:24.741 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2023-10-12T00:01:24.761 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-12T00:01:24.770 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-12T00:01:24.777 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-12T00:01:24.778 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2023-10-12T00:01:24.781 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2023-10-12T00:01:24.803 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-12T00:01:24.809 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-12T00:01:24.810 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2023-10-12T00:01:24.842 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2023-10-12T00:01:24.875 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-12T00:01:24.958 INFO:teuthology.run_tasks:Running task kernel... 2023-10-12T00:01:24.980 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-12T00:01:24.981 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-12T00:01:24.981 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T00:01:24.982 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T00:01:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T00:01:25.003 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-10-12T00:01:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T00:01:25.005 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-12T00:01:25.066 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-489.el8.x86_64 2023-10-12T00:01:25.066 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-489.el8.x86_64 2023-10-12T00:01:25.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2023-10-12T00:01:25.068 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-489.el8.x86_64 2023-10-12T00:01:25.068 INFO:teuthology.task.kernel:Running kernel on smithi118: 4.18.0-489.el8.x86_64 2023-10-12T00:01:25.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2023-10-12T00:01:28.856 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 27 MB/s | 33 MB 00:01 2023-10-12T00:01:29.189 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - AppStream 25 MB/s | 33 MB 00:01 2023-10-12T00:01:36.798 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 49 MB 00:01 2023-10-12T00:01:37.806 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 49 MB 00:02 2023-10-12T00:01:44.107 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 59 kB/s | 18 kB 00:00 2023-10-12T00:01:44.582 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2023-10-12T00:01:44.925 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-10-12T00:01:45.272 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-10-12T00:01:45.914 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-10-12T00:01:45.928 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-10-12T00:01:46.191 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-12T00:01:49.785 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-10-12T00:01:51.234 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 16 MB 00:04 2023-10-12T00:01:53.275 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-12T00:01:53.276 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-12T00:01:53.541 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:01:53.542 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:01:53.542 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-10-12T00:01:53.542 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:01:53.542 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:01:53.543 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-12T00:01:53.543 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:01:53.543 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-12T00:01:53.543 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-12T00:01:53.543 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:01:53.544 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:01:53.544 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:01:53.544 INFO:teuthology.orchestra.run.smithi112.stdout:Install 3 Packages 2023-10-12T00:01:53.544 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:01:53.545 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 89 M 2023-10-12T00:01:53.545 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 96 M 2023-10-12T00:01:53.545 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:01:54.344 INFO:teuthology.orchestra.run.smithi112.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2023-10-12T00:01:55.016 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 334 kB/s | 24 kB 00:00 2023-10-12T00:01:55.544 INFO:teuthology.orchestra.run.smithi112.stdout:(2/3): kernel-core-4.18.0-516.el8.x86_64.rpm 22 MB/s | 43 MB 00:01 2023-10-12T00:01:55.970 INFO:teuthology.orchestra.run.smithi112.stdout:(3/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2023-10-12T00:01:56.302 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:01:56.302 INFO:teuthology.orchestra.run.smithi112.stdout:Total 37 MB/s | 89 MB 00:02 2023-10-12T00:01:56.648 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:01:57.060 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:01:57.061 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:01:58.295 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:01:58.330 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:01:58.582 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-12T00:01:58.582 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-12T00:01:58.864 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:01:58.865 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:01:58.865 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-12T00:01:58.865 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:01:58.866 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:01:58.866 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-12T00:01:58.866 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:01:58.866 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-12T00:01:58.866 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-12T00:01:58.867 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:01:58.867 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:01:58.867 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:01:58.867 INFO:teuthology.orchestra.run.smithi118.stdout:Install 3 Packages 2023-10-12T00:01:58.867 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:01:58.868 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 89 M 2023-10-12T00:01:58.868 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 96 M 2023-10-12T00:01:58.868 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:01:59.842 INFO:teuthology.orchestra.run.smithi118.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 11 MB/s | 10 MB 00:00 2023-10-12T00:02:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:(2/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2023-10-12T00:02:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:02:01.226 INFO:teuthology.orchestra.run.smithi118.stdout:(3/3): kernel-core-4.18.0-516.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-10-12T00:02:01.226 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:02:01.227 INFO:teuthology.orchestra.run.smithi118.stdout:Total 38 MB/s | 89 MB 00:02 2023-10-12T00:02:01.691 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:01.901 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:02:02.325 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:02:02.326 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:02:03.604 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:02:03.620 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:06.374 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:02:06.902 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:09.205 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:09.486 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:10.976 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:14.355 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:14.621 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:31.382 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:31.406 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:32.728 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:32.728 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:02:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:02:32.728 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-12T00:02:32.729 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-12T00:02:32.729 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:02:32.729 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:02:32.875 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T00:02:33.099 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T00:02:33.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T00:02:33.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-12T00:02:33.101 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T00:02:33.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T00:02:33.101 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-10-12T00:02:33.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T00:02:33.102 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T00:02:33.310 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T00:02:33.311 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T00:02:33.311 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2023-10-12T00:02:33.345 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2023-10-12T00:02:33.345 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2023-10-12T00:02:33.346 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.02 2023-10-12T00:02:33.346 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 129.el8 2023-10-12T00:02:33.346 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2023-10-12T00:02:33.347 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-12T00:02:33.347 INFO:teuthology.orchestra.run.smithi112.stdout:Group : System Environment/Base 2023-10-12T00:02:33.347 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 9265586 2023-10-12T00:02:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2023-10-12T00:02:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-12T00:02:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-12T00:02:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-12T00:02:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-12T00:02:33.349 INFO:teuthology.orchestra.run.smithi112.stdout:Relocations : (not relocatable) 2023-10-12T00:02:33.349 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : CentOS BuildSystem 2023-10-12T00:02:33.349 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2023-10-12T00:02:33.349 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T00:02:33.349 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2023-10-12T00:02:33.350 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2023-10-12T00:02:33.350 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:02:33.350 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T00:02:33.350 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T00:02:33.351 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T00:02:33.351 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2023-10-12T00:02:33.351 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:02:33.351 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T00:02:33.353 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T00:02:33.353 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 4.18.0-516.el8.x86_64 2023-10-12T00:02:33.353 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T00:02:34.724 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2023-10-12T00:02:36.370 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:36.393 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-12T00:02:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-12T00:02:36.887 INFO:teuthology.orchestra.run.smithi112.stderr:done 2023-10-12T00:02:36.889 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-10-12T00:02:36.926 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.mBBbqkWWMj 2023-10-12T00:02:36.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mBBbqkWWMj 2023-10-12T00:02:37.000 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.mBBbqkWWMj 2023-10-12T00:02:37.141 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.mBBbqkWWMj is 6KB 2023-10-12T00:02:37.189 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.mBBbqkWWMj 2023-10-12T00:02:37.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T00:02:37.282 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-12T00:02:37.283 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-12T00:02:37.283 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-12T00:02:37.283 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-12T00:02:37.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-12T00:02:37.474 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2023-10-12T00:02:37.486 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T00:02:37.486 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:02:37.487 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:02:37.487 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-12T00:02:37.487 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-12T00:02:37.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:02:37.487 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:02:37.632 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T00:02:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T00:02:37.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T00:02:37.851 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-12T00:02:37.851 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T00:02:37.852 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T00:02:37.852 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-10-12T00:02:37.852 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T00:02:37.852 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T00:02:38.065 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T00:02:38.065 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T00:02:38.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2023-10-12T00:02:38.099 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2023-10-12T00:02:38.100 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2023-10-12T00:02:38.100 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.02 2023-10-12T00:02:38.100 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 129.el8 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:Group : System Environment/Base 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 9265586 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2023-10-12T00:02:38.101 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-12T00:02:38.102 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-12T00:02:38.102 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-12T00:02:38.102 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-12T00:02:38.102 INFO:teuthology.orchestra.run.smithi118.stdout:Relocations : (not relocatable) 2023-10-12T00:02:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : CentOS BuildSystem 2023-10-12T00:02:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2023-10-12T00:02:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T00:02:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2023-10-12T00:02:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2023-10-12T00:02:38.104 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:02:38.104 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T00:02:38.104 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T00:02:38.104 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T00:02:38.105 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2023-10-12T00:02:38.105 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:02:38.105 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T00:02:38.106 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T00:02:38.106 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 4.18.0-516.el8.x86_64 2023-10-12T00:02:38.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T00:02:40.645 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2023-10-12T00:02:42.606 INFO:teuthology.orchestra.run.smithi118.stderr:done 2023-10-12T00:02:42.608 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-12T00:02:42.631 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.HqIfU1ZPb9 2023-10-12T00:02:42.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HqIfU1ZPb9 2023-10-12T00:02:42.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.HqIfU1ZPb9 2023-10-12T00:02:42.833 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.HqIfU1ZPb9 is 6KB 2023-10-12T00:02:42.881 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.HqIfU1ZPb9 2023-10-12T00:02:42.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T00:02:42.971 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-12T00:02:42.971 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-12T00:02:42.971 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-12T00:02:42.972 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-12T00:02:42.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-12T00:02:43.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2023-10-12T00:03:07.498 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-10-12T00:03:07.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:03:07.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:13.142 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-10-12T00:03:13.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:03:13.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:25.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-10-12T00:03:31.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-12T00:03:34.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:03:34.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:38.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-10-12T00:03:40.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:03:40.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:43.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-12T00:03:50.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:03:50.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:55.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:03:55.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:03:58.714 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-12T00:04:08.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-10-12T00:04:13.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:04:13.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:04:23.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:04:23.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:04:24.371 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-10-12T00:04:25.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:04:25.184 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-516.el8.x86_64"... 2023-10-12T00:04:25.184 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-10-12T00:04:25.203 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-516.el8.x86_64 2023-10-12T00:04:25.203 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-12T00:04:25.203 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T00:04:25.203 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-12T00:04:26.205 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-10-12T00:04:26.205 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T00:04:26.282 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2023-10-12T00:04:26.296 DEBUG:teuthology.parallel:result is None 2023-10-12T00:04:45.691 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-12T00:04:45.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:04:45.962 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-516.el8.x86_64"... 2023-10-12T00:04:45.963 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-12T00:04:45.980 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-516.el8.x86_64 2023-10-12T00:04:45.981 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-12T00:04:45.981 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T00:04:45.981 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-12T00:04:46.982 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-10-12T00:04:46.983 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T00:04:47.067 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2023-10-12T00:04:47.079 DEBUG:teuthology.parallel:result is None 2023-10-12T00:04:47.079 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-12T00:04:47.287 INFO:teuthology.task.internal:Creating test directory... 2023-10-12T00:04:47.287 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T00:04:47.290 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T00:04:47.312 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-12T00:04:47.327 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-12T00:04:47.335 INFO:teuthology.task.internal:Creating archive directory... 2023-10-12T00:04:47.335 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T00:04:47.353 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T00:04:47.418 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-12T00:04:47.432 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-12T00:04:47.433 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T00:04:47.436 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T00:04:47.477 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T00:04:47.483 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T00:04:47.488 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T00:04:47.495 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T00:04:47.496 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-12T00:04:47.520 INFO:teuthology.task.internal:Configuring sudo... 2023-10-12T00:04:47.521 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T00:04:47.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T00:04:47.555 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-12T00:04:47.572 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-12T00:04:47.572 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T00:04:47.597 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T00:04:47.617 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T00:04:47.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T00:04:47.767 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T00:04:47.813 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T00:04:47.859 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:04:47.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T00:04:47.931 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T00:04:47.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T00:04:48.040 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T00:04:48.090 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T00:04:48.138 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:04:48.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T00:04:48.212 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2023-10-12T00:04:48.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2023-10-12T00:04:48.292 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T00:04:48.327 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T00:04:48.794 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-12T00:04:49.008 INFO:teuthology.task.internal:Starting timer... 2023-10-12T00:04:49.009 INFO:teuthology.run_tasks:Running task pcp... 2023-10-12T00:04:49.041 INFO:teuthology.run_tasks:Running task selinux... 2023-10-12T00:04:49.054 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-12T00:04:49.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2023-10-12T00:04:49.134 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2023-10-12T00:04:49.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2023-10-12T00:04:49.216 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2023-10-12T00:04:49.218 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-12T00:04:49.218 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2023-10-12T00:04:49.303 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2023-10-12T00:04:49.303 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2023-10-12T00:04:49.347 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2023-10-12T00:04:49.347 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2023-10-12T00:04:49.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T00:04:49.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T00:04:49.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T00:04:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T00:04:49.417 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-12T00:04:49.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2023-10-12T00:04:49.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2023-10-12T00:04:49.485 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-12T00:04:49.658 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-12T00:04:49.664 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-12T00:04:49.786 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-12T00:04:49.802 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-12T00:04:49.803 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-12T00:12:18.069 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2023-10-12T00:12:18.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:12:18.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:12:18.139 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-10-12T00:12:18.208 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-10-12T00:12:18.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:12:18.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T00:12:18.278 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-12T00:12:18.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-12T00:12:18.350 INFO:teuthology.run_tasks:Running task clock... 2023-10-12T00:12:18.370 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-12T00:12:18.371 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T00:12:18.372 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T00:12:18.376 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T00:12:18.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T00:12:18.405 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T00:12:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T00:12:18.421 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T00:12:18.435 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T00:12:18.485 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2023-10-12T00:12:18.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2023-10-12T00:12:18.497 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2023-10-12T00:12:18.508 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2023-10-12T00:12:18.511 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T00:12:18.521 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T00:12:18.524 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T00:12:18.535 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T00:12:18.598 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-10-12T00:12:18.602 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T00:12:18.602 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-10-12T00:12:18.602 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.602 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.603 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.603 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.605 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2023-10-12T00:12:18.610 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T00:12:18.610 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2023-10-12T00:12:18.610 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.610 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.611 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.611 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T00:12:18.611 INFO:teuthology.run_tasks:Running task install... 2023-10-12T00:12:18.620 DEBUG:teuthology.task.install:project ceph 2023-10-12T00:12:18.621 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}} 2023-10-12T00:12:18.621 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'} 2023-10-12T00:12:18.621 INFO:teuthology.task.install:Using flavor: default 2023-10-12T00:12:18.627 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-12T00:12:18.628 INFO:teuthology.task.install:extra packages: [] 2023-10-12T00:12:18.628 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'tag': None, 'wait_for_package': False} 2023-10-12T00:12:18.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-12T00:12:18.631 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'tag': None, 'wait_for_package': False} 2023-10-12T00:12:18.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-12T00:12:18.776 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/ 2023-10-12T00:12:18.777 INFO:teuthology.task.install.rpm:Package version is 18.2.0-666.gaa3250ce 2023-10-12T00:12:18.788 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/ 2023-10-12T00:12:18.788 INFO:teuthology.task.install.rpm:Package version is 18.2.0-666.gaa3250ce 2023-10-12T00:12:18.900 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T00:12:18.901 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:12:18.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T00:12:18.930 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T00:12:18.931 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T00:12:18.952 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T00:12:18.952 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:12:18.952 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T00:12:18.982 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T00:12:18.982 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T00:12:18.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T00:12:19.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T00:12:19.077 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T00:12:19.131 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T00:12:19.143 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2023-10-12T00:12:19.144 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2023-10-12T00:12:19.198 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2023-10-12T00:12:19.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2023-10-12T00:12:19.419 INFO:teuthology.orchestra.run.smithi118.stdout:76 files removed 2023-10-12T00:12:19.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2023-10-12T00:12:19.472 INFO:teuthology.orchestra.run.smithi112.stdout:76 files removed 2023-10-12T00:12:19.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2023-10-12T00:12:20.214 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 221 kB/s | 80 kB 00:00 2023-10-12T00:12:20.267 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 222 kB/s | 80 kB 00:00 2023-10-12T00:12:20.962 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-10-12T00:12:20.968 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-10-12T00:12:21.269 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2023-10-12T00:12:21.273 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 5.3 kB/s | 1.5 kB 00:00 2023-10-12T00:12:22.377 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 31 MB/s | 33 MB 00:01 2023-10-12T00:12:22.591 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - AppStream 26 MB/s | 33 MB 00:01 2023-10-12T00:12:30.508 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - BaseOS 27 MB/s | 49 MB 00:01 2023-10-12T00:12:30.651 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 49 MB 00:01 2023-10-12T00:12:37.966 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-10-12T00:12:38.337 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 37 kB/s | 18 kB 00:00 2023-10-12T00:12:38.600 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-10-12T00:12:38.846 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-10-12T00:12:39.293 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 6.5 MB 00:00 2023-10-12T00:12:39.722 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - PowerTools 8.9 MB/s | 6.5 MB 00:00 2023-10-12T00:12:40.720 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-10-12T00:12:41.198 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-10-12T00:12:42.346 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-12T00:12:42.802 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-12T00:12:46.229 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-10-12T00:12:46.707 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-10-12T00:12:50.674 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:12:50.676 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:12:50.676 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:12:50.677 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:12:50.677 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:12:50.677 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 13 M 2023-10-12T00:12:50.677 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2023-10-12T00:12:50.677 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.7 M 2023-10-12T00:12:50.678 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.6 M 2023-10-12T00:12:50.678 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:12:50.678 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 5.4 M 2023-10-12T00:12:50.678 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 22 M 2023-10-12T00:12:50.678 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 24 k 2023-10-12T00:12:50.679 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T00:12:50.679 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 770 k 2023-10-12T00:12:50.679 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T00:12:50.679 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-12T00:12:50.679 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 474 k 2023-10-12T00:12:50.680 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-12T00:12:50.680 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.5 M 2023-10-12T00:12:50.680 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T00:12:50.680 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-12T00:12:50.680 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 45 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 123 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 190 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 371 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 364 k 2023-10-12T00:12:50.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 107 k 2023-10-12T00:12:50.682 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-12T00:12:50.682 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:12:50.682 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:12:50.682 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:12:50.682 INFO:teuthology.orchestra.run.smithi118.stdout:Install 21 Packages 2023-10-12T00:12:50.683 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2023-10-12T00:12:50.683 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:12:50.684 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 59 M 2023-10-12T00:12:50.684 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:12:51.184 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:12:51.187 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:12:51.187 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:12:51.187 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:12:51.187 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:12:51.188 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 13 M 2023-10-12T00:12:51.188 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2023-10-12T00:12:51.188 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.7 M 2023-10-12T00:12:51.188 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.6 M 2023-10-12T00:12:51.188 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:12:51.189 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 5.4 M 2023-10-12T00:12:51.189 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 22 M 2023-10-12T00:12:51.189 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 24 k 2023-10-12T00:12:51.189 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T00:12:51.189 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 770 k 2023-10-12T00:12:51.190 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T00:12:51.190 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-12T00:12:51.190 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 474 k 2023-10-12T00:12:51.190 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-12T00:12:51.190 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.5 M 2023-10-12T00:12:51.191 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T00:12:51.191 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-12T00:12:51.191 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 45 k 2023-10-12T00:12:51.191 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 123 k 2023-10-12T00:12:51.191 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 190 k 2023-10-12T00:12:51.192 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-12T00:12:51.192 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 371 k 2023-10-12T00:12:51.192 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 364 k 2023-10-12T00:12:51.192 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 107 k 2023-10-12T00:12:51.192 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout:Install 21 Packages 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2023-10-12T00:12:51.193 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:12:51.194 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 59 M 2023-10-12T00:12:51.194 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:12:52.214 INFO:teuthology.orchestra.run.smithi118.stdout:(1/23): ceph-base-18.2.0-666.gaa3250ce.el8.x86_ 5.6 MB/s | 5.4 MB 00:00 2023-10-12T00:12:52.348 INFO:teuthology.orchestra.run.smithi118.stdout:(2/23): ceph-selinux-18.2.0-666.gaa3250ce.el8.x 182 kB/s | 24 kB 00:00 2023-10-12T00:12:52.606 INFO:teuthology.orchestra.run.smithi118.stdout:(3/23): ceph-radosgw-18.2.0-666.gaa3250ce.el8.x 9.6 MB/s | 13 MB 00:01 2023-10-12T00:12:52.640 INFO:teuthology.orchestra.run.smithi118.stdout:(4/23): libcephfs2-18.2.0-666.gaa3250ce.el8.x86 2.6 MB/s | 770 kB 00:00 2023-10-12T00:12:52.782 INFO:teuthology.orchestra.run.smithi118.stdout:(5/23): libradosstriper1-18.2.0-666.gaa3250ce.e 2.6 MB/s | 474 kB 00:00 2023-10-12T00:12:52.924 INFO:teuthology.orchestra.run.smithi118.stdout:(6/23): python3-ceph-argparse-18.2.0-666.gaa325 314 kB/s | 45 kB 00:00 2023-10-12T00:12:53.033 INFO:teuthology.orchestra.run.smithi118.stdout:(7/23): python3-ceph-common-18.2.0-666.gaa3250c 1.1 MB/s | 123 kB 00:00 2023-10-12T00:12:53.116 INFO:teuthology.orchestra.run.smithi118.stdout:(8/23): python3-cephfs-18.2.0-666.gaa3250ce.el8 2.2 MB/s | 190 kB 00:00 2023-10-12T00:12:53.458 INFO:teuthology.orchestra.run.smithi118.stdout:(9/23): librgw2-18.2.0-666.gaa3250ce.el8.x86_64 8.0 MB/s | 6.5 MB 00:00 2023-10-12T00:12:53.483 INFO:teuthology.orchestra.run.smithi118.stdout:(10/23): python3-rados-18.2.0-666.gaa3250ce.el8 1.0 MB/s | 371 kB 00:00 2023-10-12T00:12:53.542 INFO:teuthology.orchestra.run.smithi112.stdout:(1/23): ceph-base-18.2.0-666.gaa3250ce.el8.x86_ 2.6 MB/s | 5.4 MB 00:02 2023-10-12T00:12:53.550 INFO:teuthology.orchestra.run.smithi118.stdout:(11/23): python3-rbd-18.2.0-666.gaa3250ce.el8.x 3.9 MB/s | 364 kB 00:00 2023-10-12T00:12:53.576 INFO:teuthology.orchestra.run.smithi118.stdout:(12/23): python3-rgw-18.2.0-666.gaa3250ce.el8.x 1.1 MB/s | 107 kB 00:00 2023-10-12T00:12:53.601 INFO:teuthology.orchestra.run.smithi112.stdout:(2/23): ceph-selinux-18.2.0-666.gaa3250ce.el8.x 418 kB/s | 24 kB 00:00 2023-10-12T00:12:53.751 INFO:teuthology.orchestra.run.smithi112.stdout:(3/23): libcephfs2-18.2.0-666.gaa3250ce.el8.x86 5.0 MB/s | 770 kB 00:00 2023-10-12T00:12:53.777 INFO:teuthology.orchestra.run.smithi118.stdout:(13/23): ceph-common-18.2.0-666.gaa3250ce.el8.x 8.6 MB/s | 22 MB 00:02 2023-10-12T00:12:53.902 INFO:teuthology.orchestra.run.smithi112.stdout:(4/23): libradosstriper1-18.2.0-666.gaa3250ce.e 3.1 MB/s | 474 kB 00:00 2023-10-12T00:12:53.943 INFO:teuthology.orchestra.run.smithi118.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 264 kB/s | 44 kB 00:00 2023-10-12T00:12:54.002 INFO:teuthology.orchestra.run.smithi118.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 610 kB/s | 259 kB 00:00 2023-10-12T00:12:54.027 INFO:teuthology.orchestra.run.smithi118.stdout:(16/23): librdkafka-0.11.4-3.el8.x86_64.rpm 743 kB/s | 354 kB 00:00 2023-10-12T00:12:54.144 INFO:teuthology.orchestra.run.smithi118.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2023-10-12T00:12:54.186 INFO:teuthology.orchestra.run.smithi118.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 371 kB/s | 59 kB 00:00 2023-10-12T00:12:54.286 INFO:teuthology.orchestra.run.smithi118.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-10-12T00:12:54.345 INFO:teuthology.orchestra.run.smithi118.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 374 kB/s | 75 kB 00:00 2023-10-12T00:12:54.463 INFO:teuthology.orchestra.run.smithi118.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2023-10-12T00:12:54.493 INFO:teuthology.orchestra.run.smithi112.stdout:(5/23): librgw2-18.2.0-666.gaa3250ce.el8.x86_64 11 MB/s | 6.5 MB 00:00 2023-10-12T00:12:54.569 INFO:teuthology.orchestra.run.smithi112.stdout:(6/23): python3-ceph-argparse-18.2.0-666.gaa325 597 kB/s | 45 kB 00:00 2023-10-12T00:12:54.660 INFO:teuthology.orchestra.run.smithi112.stdout:(7/23): python3-ceph-common-18.2.0-666.gaa3250c 1.3 MB/s | 123 kB 00:00 2023-10-12T00:12:54.707 INFO:teuthology.orchestra.run.smithi118.stdout:(22/23): librados2-18.2.0-666.gaa3250ce.el8.x86 8.8 MB/s | 3.7 MB 00:00 2023-10-12T00:12:54.728 INFO:teuthology.orchestra.run.smithi112.stdout:(8/23): python3-cephfs-18.2.0-666.gaa3250ce.el8 2.8 MB/s | 190 kB 00:00 2023-10-12T00:12:54.804 INFO:teuthology.orchestra.run.smithi118.stdout:(23/23): librbd1-18.2.0-666.gaa3250ce.el8.x86_6 7.8 MB/s | 3.6 MB 00:00 2023-10-12T00:12:54.806 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:12:54.807 INFO:teuthology.orchestra.run.smithi118.stdout:Total 14 MB/s | 59 MB 00:04 2023-10-12T00:12:54.811 INFO:teuthology.orchestra.run.smithi112.stdout:(9/23): python3-rados-18.2.0-666.gaa3250ce.el8. 4.3 MB/s | 371 kB 00:00 2023-10-12T00:12:54.848 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:12:54.878 INFO:teuthology.orchestra.run.smithi112.stdout:(10/23): python3-rbd-18.2.0-666.gaa3250ce.el8.x 5.4 MB/s | 364 kB 00:00 2023-10-12T00:12:54.893 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:12:54.894 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:12:54.937 INFO:teuthology.orchestra.run.smithi112.stdout:(11/23): python3-rgw-18.2.0-666.gaa3250ce.el8.x 1.8 MB/s | 107 kB 00:00 2023-10-12T00:12:55.206 INFO:teuthology.orchestra.run.smithi112.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2023-10-12T00:12:55.249 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:12:55.252 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:12:55.279 INFO:teuthology.orchestra.run.smithi112.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.5 MB/s | 259 kB 00:00 2023-10-12T00:12:55.337 INFO:teuthology.orchestra.run.smithi112.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 755 kB/s | 44 kB 00:00 2023-10-12T00:12:55.521 INFO:teuthology.orchestra.run.smithi112.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 256 kB/s | 47 kB 00:00 2023-10-12T00:12:55.997 INFO:teuthology.orchestra.run.smithi112.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 645 kB/s | 306 kB 00:00 2023-10-12T00:12:56.097 INFO:teuthology.orchestra.run.smithi112.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 588 kB/s | 59 kB 00:00 2023-10-12T00:12:56.214 INFO:teuthology.orchestra.run.smithi112.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 644 kB/s | 75 kB 00:00 2023-10-12T00:12:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:12:56.631 INFO:teuthology.orchestra.run.smithi112.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2023-10-12T00:12:56.688 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-12T00:12:56.893 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-12T00:12:57.098 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-12T00:12:57.115 INFO:teuthology.orchestra.run.smithi112.stdout:(20/23): librados2-18.2.0-666.gaa3250ce.el8.x86 7.6 MB/s | 3.7 MB 00:00 2023-10-12T00:12:57.390 INFO:teuthology.orchestra.run.smithi112.stdout:(21/23): librbd1-18.2.0-666.gaa3250ce.el8.x86_6 13 MB/s | 3.6 MB 00:00 2023-10-12T00:12:57.484 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-12T00:12:57.681 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:12:57.715 INFO:teuthology.orchestra.run.smithi112.stdout:(22/23): ceph-radosgw-18.2.0-666.gaa3250ce.el8. 2.1 MB/s | 13 MB 00:06 2023-10-12T00:12:57.798 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:12:58.079 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-12T00:12:58.256 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:12:58.656 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:12:58.873 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:12:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:12:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-12T00:12:59.331 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-12T00:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-12T00:12:59.549 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 10/25 2023-10-12T00:12:59.741 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-12T00:12:59.890 INFO:teuthology.orchestra.run.smithi112.stdout:(23/23): ceph-common-18.2.0-666.gaa3250ce.el8.x 2.6 MB/s | 22 MB 00:08 2023-10-12T00:12:59.892 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:12:59.892 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.8 MB/s | 59 MB 00:08 2023-10-12T00:12:59.933 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:12:59.973 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-12T00:12:59.979 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:12:59.980 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:13:00.114 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:00.328 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:13:00.332 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:13:00.390 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-12T00:13:01.137 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-12T00:13:01.518 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-12T00:13:01.592 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-12T00:13:01.723 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:13:01.797 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 17/25 2023-10-12T00:13:01.931 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-12T00:13:01.994 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-12T00:13:02.151 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-12T00:13:02.181 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 19/25 2023-10-12T00:13:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-12T00:13:03.146 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-12T00:13:03.375 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:13:03.473 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:13:03.780 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-12T00:13:03.983 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:13:04.373 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:13:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:13:04.646 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:13:04.708 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:04.847 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-12T00:13:04.955 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-12T00:13:05.092 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-12T00:13:05.285 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 10/25 2023-10-12T00:13:05.519 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-12T00:13:05.582 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:05.759 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-12T00:13:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-12T00:13:05.911 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-12T00:13:05.916 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:06.088 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:06.226 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-12T00:13:07.012 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-12T00:13:07.149 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-12T00:13:07.252 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-12T00:13:07.474 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 17/25 2023-10-12T00:13:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-12T00:13:07.908 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 19/25 2023-10-12T00:13:10.802 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:11.656 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-12T00:13:11.975 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-12T00:13:12.074 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-12T00:13:12.257 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:24.632 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2023-10-12T00:13:24.632 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2023-10-12T00:13:24.633 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2023-10-12T00:13:24.633 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /run 2023-10-12T00:13:24.633 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2023-10-12T00:13:24.633 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2023-10-12T00:13:24.634 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2023-10-12T00:13:24.634 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2023-10-12T00:13:24.634 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2023-10-12T00:13:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:26.044 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-12T00:13:26.065 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-12T00:13:26.066 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T00:13:26.270 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T00:13:26.271 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 1/25 2023-10-12T00:13:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2/25 2023-10-12T00:13:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 3/25 2023-10-12T00:13:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:13:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-12T00:13:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:13:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:13:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 8/25 2023-10-12T00:13:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 9/25 2023-10-12T00:13:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 10/25 2023-10-12T00:13:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-12T00:13:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-12T00:13:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-12T00:13:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-12T00:13:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-12T00:13:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-12T00:13:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 24/25 2023-10-12T00:13:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:28.081 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2023-10-12T00:13:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.082 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:13:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T00:13:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T00:13:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-12T00:13:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-12T00:13:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T00:13:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T00:13:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T00:13:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-12T00:13:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:28.088 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:13:28.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2023-10-12T00:13:29.257 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:43 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:13:30.856 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:13:30.857 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:30.857 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:13:30.857 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:30.857 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:13:30.858 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 47 M 2023-10-12T00:13:30.858 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:13:30.858 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-12T00:13:30.858 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 137 k 2023-10-12T00:13:30.858 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-12T00:13:30.859 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-12T00:13:30.859 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-12T00:13:30.859 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:30.859 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:13:30.859 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:30.860 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2023-10-12T00:13:30.860 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:30.861 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 48 M 2023-10-12T00:13:30.861 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 211 M 2023-10-12T00:13:30.861 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:13:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:31.083 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2023-10-12T00:13:31.083 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2023-10-12T00:13:31.083 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-10-12T00:13:31.084 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2023-10-12T00:13:31.084 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2023-10-12T00:13:31.084 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2023-10-12T00:13:31.084 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2023-10-12T00:13:31.084 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2023-10-12T00:13:31.085 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-10-12T00:13:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:31.138 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 851 kB/s | 202 kB 00:00 2023-10-12T00:13:31.188 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): libcephsqlite-18.2.0-666.gaa3250ce.el8.x 475 kB/s | 137 kB 00:00 2023-10-12T00:13:31.214 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2023-10-12T00:13:31.272 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-10-12T00:13:31.414 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2023-10-12T00:13:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-12T00:13:32.493 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-12T00:13:32.493 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T00:13:32.740 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-12T00:13:32.740 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:33.231 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-18.2.0-666.gaa3250ce.el8.x86_6 20 MB/s | 47 MB 00:02 2023-10-12T00:13:33.233 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:13:33.233 INFO:teuthology.orchestra.run.smithi118.stdout:Total 20 MB/s | 48 MB 00:02 2023-10-12T00:13:33.279 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:13:33.300 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:13:33.301 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:13:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 1/25 2023-10-12T00:13:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2/25 2023-10-12T00:13:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 3/25 2023-10-12T00:13:33.482 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-12T00:13:33.482 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-12T00:13:33.482 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-12T00:13:33.482 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-12T00:13:33.482 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 8/25 2023-10-12T00:13:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 9/25 2023-10-12T00:13:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 10/25 2023-10-12T00:13:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-12T00:13:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-12T00:13:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-12T00:13:33.484 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-12T00:13:33.484 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-12T00:13:33.484 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-12T00:13:33.484 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-12T00:13:33.484 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-12T00:13:33.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 24/25 2023-10-12T00:13:33.640 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:13:33.643 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:13:34.121 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:13:34.378 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-12T00:13:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-12T00:13:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:34.611 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2023-10-12T00:13:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.612 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:13:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T00:13:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T00:13:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-12T00:13:34.614 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.614 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-12T00:13:34.614 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.614 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T00:13:34.614 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.615 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.616 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:34.616 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-12T00:13:34.616 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:34.616 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:13:34.617 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-12T00:13:34.758 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T00:13:34.829 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T00:13:34.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2023-10-12T00:13:35.004 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-12T00:13:35.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-12T00:13:35.824 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:49 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:13:37.439 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:13:37.440 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:13:37.440 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:37.441 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:13:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 47 M 2023-10-12T00:13:37.441 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:13:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-12T00:13:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 137 k 2023-10-12T00:13:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-12T00:13:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-12T00:13:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-12T00:13:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:37.443 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:13:37.443 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:37.443 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-10-12T00:13:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:37.444 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 48 M 2023-10-12T00:13:37.444 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 211 M 2023-10-12T00:13:37.445 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:13:37.875 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 840 kB/s | 202 kB 00:00 2023-10-12T00:13:37.942 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2023-10-12T00:13:37.967 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): libcephsqlite-18.2.0-666.gaa3250ce.el8.x 411 kB/s | 137 kB 00:00 2023-10-12T00:13:38.018 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-10-12T00:13:38.159 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 368 kB/s | 71 kB 00:00 2023-10-12T00:13:39.725 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-12T00:13:39.918 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-test-18.2.0-666.gaa3250ce.el8.x86_6 20 MB/s | 47 MB 00:02 2023-10-12T00:13:39.920 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:13:39.920 INFO:teuthology.orchestra.run.smithi112.stdout:Total 19 MB/s | 48 MB 00:02 2023-10-12T00:13:39.962 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-12T00:13:39.966 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:13:39.984 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:13:39.984 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:13:40.247 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-12T00:13:40.248 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 1/6 2023-10-12T00:13:40.248 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2/6 2023-10-12T00:13:40.248 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-12T00:13:40.248 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-12T00:13:40.248 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-12T00:13:40.299 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:13:40.302 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:13:40.778 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:13:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-12T00:13:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:40.924 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:13:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-7.el8.x86_64 2023-10-12T00:13:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-12T00:13:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-12T00:13:40.926 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T00:13:40.926 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:40.926 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:13:41.026 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-12T00:13:41.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2023-10-12T00:13:41.239 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-12T00:13:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T00:13:41.507 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T00:13:41.707 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-12T00:13:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-12T00:13:42.031 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:13:43.638 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:13:43.641 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:43.641 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:13:43.642 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:43.642 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:13:43.642 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.4 k 2023-10-12T00:13:43.642 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:13:43.642 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 2.3 M 2023-10-12T00:13:43.643 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 1.5 M 2023-10-12T00:13:43.643 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 242 k 2023-10-12T00:13:43.643 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 4.6 M 2023-10-12T00:13:43.643 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 18 M 2023-10-12T00:13:43.643 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T00:13:43.644 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T00:13:43.644 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-12T00:13:43.644 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T00:13:43.644 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T00:13:43.644 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-12T00:13:43.645 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T00:13:43.645 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T00:13:43.645 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T00:13:43.645 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-12T00:13:43.645 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-12T00:13:43.646 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T00:13:43.646 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T00:13:43.646 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T00:13:43.646 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-12T00:13:43.646 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-12T00:13:43.647 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-12T00:13:43.647 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T00:13:43.647 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T00:13:43.647 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T00:13:43.647 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T00:13:43.648 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T00:13:43.648 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T00:13:43.648 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T00:13:43.648 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-12T00:13:43.648 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T00:13:43.649 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:43.649 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:13:43.649 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:13:43.649 INFO:teuthology.orchestra.run.smithi118.stdout:Install 32 Packages 2023-10-12T00:13:43.649 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:13:43.650 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 M 2023-10-12T00:13:43.650 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 119 M 2023-10-12T00:13:43.651 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:13:44.332 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-18.2.0-666.gaa3250ce.el8.x86_64.rp 26 kB/s | 6.4 kB 00:00 2023-10-12T00:13:44.551 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-mgr-18.2.0-666.gaa3250ce.el8.x86_6 3.4 MB/s | 1.5 MB 00:00 2023-10-12T00:13:44.609 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): ceph-mds-18.2.0-666.gaa3250ce.el8.x86_6 4.4 MB/s | 2.3 MB 00:00 2023-10-12T00:13:44.726 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): ceph-mgr-modules-core-18.2.0-666.gaa325 2.0 MB/s | 242 kB 00:00 2023-10-12T00:13:44.935 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 754 kB/s | 157 kB 00:00 2023-10-12T00:13:45.010 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 519 kB/s | 39 kB 00:00 2023-10-12T00:13:45.094 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.2 MB/s | 103 kB 00:00 2023-10-12T00:13:45.153 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 976 kB/s | 57 kB 00:00 2023-10-12T00:13:45.253 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.5 MB/s | 457 kB 00:00 2023-10-12T00:13:45.437 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): ceph-osd-18.2.0-666.gaa3250ce.el8.x86_ 20 MB/s | 18 MB 00:00 2023-10-12T00:13:45.712 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 542 kB/s | 249 kB 00:00 2023-10-12T00:13:45.787 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): ceph-mon-18.2.0-666.gaa3250ce.el8.x86_ 3.2 MB/s | 4.6 MB 00:01 2023-10-12T00:13:45.812 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.1 MB/s | 109 kB 00:00 2023-10-12T00:13:46.004 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-10-12T00:13:46.122 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 131 kB/s | 44 kB 00:00 2023-10-12T00:13:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-12T00:13:46.472 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 281 kB/s | 185 kB 00:00 2023-10-12T00:13:46.597 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2023-10-12T00:13:46.672 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 698 kB/s | 384 kB 00:00 2023-10-12T00:13:46.682 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-12T00:13:46.723 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2023-10-12T00:13:46.848 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 278 kB/s | 49 kB 00:00 2023-10-12T00:13:46.907 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 320 kB/s | 59 kB 00:00 2023-10-12T00:13:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-12T00:13:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 1/6 2023-10-12T00:13:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2/6 2023-10-12T00:13:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-12T00:13:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-12T00:13:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-12T00:13:47.015 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-10-12T00:13:47.095 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-10-12T00:13:47.133 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 153 kB/s | 173 kB 00:01 2023-10-12T00:13:47.158 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2023-10-12T00:13:47.199 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 230 kB/s | 24 kB 00:00 2023-10-12T00:13:47.241 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2023-10-12T00:13:47.283 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2023-10-12T00:13:47.362 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.6 MB/s | 255 kB 00:00 2023-10-12T00:13:47.426 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 601 kB/s | 85 kB 00:00 2023-10-12T00:13:47.459 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-10-12T00:13:47.484 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2023-10-12T00:13:47.486 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:13:47.486 INFO:teuthology.orchestra.run.smithi118.stdout:Total 7.8 MB/s | 30 MB 00:03 2023-10-12T00:13:47.569 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:13:47.607 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:13:47.607 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:13:47.732 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-12T00:13:47.733 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:47.733 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:13:47.733 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:47.733 INFO:teuthology.orchestra.run.smithi112.stdout: jq-1.6-7.el8.x86_64 2023-10-12T00:13:47.734 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:13:47.734 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-12T00:13:47.734 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-12T00:13:47.734 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T00:13:47.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:47.735 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:13:47.915 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2023-10-12T00:13:47.925 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:13:47.929 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:13:48.772 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:02 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:13:49.033 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:13:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-12T00:13:49.432 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-12T00:13:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-12T00:13:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-12T00:13:50.044 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-12T00:13:50.259 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-12T00:13:50.395 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:13:50.398 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:50.398 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:13:50.399 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:50.399 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:13:50.399 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.4 k 2023-10-12T00:13:50.399 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:13:50.399 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 2.3 M 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 1.5 M 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 242 k 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 4.6 M 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 18 M 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T00:13:50.400 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T00:13:50.401 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-12T00:13:50.401 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T00:13:50.401 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T00:13:50.401 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-12T00:13:50.401 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T00:13:50.402 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T00:13:50.402 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T00:13:50.402 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-12T00:13:50.403 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-12T00:13:50.404 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-12T00:13:50.404 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T00:13:50.404 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T00:13:50.404 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T00:13:50.404 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T00:13:50.405 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:50.406 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:13:50.406 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:13:50.406 INFO:teuthology.orchestra.run.smithi112.stdout:Install 32 Packages 2023-10-12T00:13:50.407 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:13:50.408 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 M 2023-10-12T00:13:50.408 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 119 M 2023-10-12T00:13:50.408 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:13:50.459 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-12T00:13:50.687 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-12T00:13:50.923 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-12T00:13:51.160 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-12T00:13:51.194 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-18.2.0-666.gaa3250ce.el8.x86_64.rp 37 kB/s | 6.4 kB 00:00 2023-10-12T00:13:51.319 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-mds-18.2.0-666.gaa3250ce.el8.x86_6 7.6 MB/s | 2.3 MB 00:00 2023-10-12T00:13:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-12T00:13:51.613 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-12T00:13:51.998 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-12T00:13:52.244 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-osd-18.2.0-666.gaa3250ce.el8.x86_6 19 MB/s | 18 MB 00:00 2023-10-12T00:13:52.286 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): ceph-mgr-18.2.0-666.gaa3250ce.el8.x86_6 1.2 MB/s | 1.5 MB 00:01 2023-10-12T00:13:52.335 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-12T00:13:52.335 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-mgr-modules-core-18.2.0-666.gaa325 2.6 MB/s | 242 kB 00:00 2023-10-12T00:13:52.487 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 257 kB/s | 39 kB 00:00 2023-10-12T00:13:52.512 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 697 kB/s | 157 kB 00:00 2023-10-12T00:13:52.531 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-12T00:13:52.587 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 761 kB/s | 57 kB 00:00 2023-10-12T00:13:52.613 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 824 kB/s | 103 kB 00:00 2023-10-12T00:13:52.680 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): ceph-mon-18.2.0-666.gaa3250ce.el8.x86_ 3.1 MB/s | 4.6 MB 00:01 2023-10-12T00:13:52.768 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2023-10-12T00:13:52.801 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-12T00:13:52.947 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 745 kB/s | 249 kB 00:00 2023-10-12T00:13:53.007 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T00:13:53.022 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 430 kB/s | 109 kB 00:00 2023-10-12T00:13:53.231 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-10-12T00:13:53.340 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-12T00:13:53.348 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 109 kB/s | 44 kB 00:00 2023-10-12T00:13:53.570 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-12T00:13:53.643 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 299 kB/s | 185 kB 00:00 2023-10-12T00:13:53.715 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2023-10-12T00:13:53.740 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 114 kB/s | 11 kB 00:00 2023-10-12T00:13:53.750 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-12T00:13:53.765 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 324 kB/s | 173 kB 00:00 2023-10-12T00:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-10-12T00:13:53.883 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 343 kB/s | 49 kB 00:00 2023-10-12T00:13:53.908 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 412 kB/s | 59 kB 00:00 2023-10-12T00:13:53.919 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-12T00:13:53.958 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-10-12T00:13:54.000 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-10-12T00:13:54.025 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2023-10-12T00:13:54.101 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 168 kB/s | 24 kB 00:00 2023-10-12T00:13:54.126 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-10-12T00:13:54.151 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2023-10-12T00:13:54.177 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-12T00:13:54.252 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2023-10-12T00:13:54.310 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 539 kB/s | 85 kB 00:00 2023-10-12T00:13:54.335 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2023-10-12T00:13:54.360 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2023-10-12T00:13:54.365 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:13:54.365 INFO:teuthology.orchestra.run.smithi112.stdout:Total 7.6 MB/s | 30 MB 00:03 2023-10-12T00:13:54.410 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-12T00:13:54.450 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:13:54.487 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:13:54.487 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:13:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-12T00:13:54.820 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:13:54.823 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:13:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-12T00:13:55.283 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-12T00:13:55.665 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 27/32 2023-10-12T00:13:55.855 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-12T00:13:56.069 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:13:56.340 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-12T00:13:56.609 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-12T00:13:56.851 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-12T00:13:57.082 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-12T00:13:57.297 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-12T00:13:57.434 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-12T00:13:57.526 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-12T00:13:57.597 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-12T00:13:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-12T00:13:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-12T00:13:58.057 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-12T00:13:58.190 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-12T00:13:58.238 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-12T00:13:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-12T00:13:58.454 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-12T00:13:58.597 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-12T00:13:58.615 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-12T00:13:58.692 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-12T00:13:58.721 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-12T00:13:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-12T00:13:59.123 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-12T00:13:59.123 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 1/32 2023-10-12T00:13:59.123 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2/32 2023-10-12T00:13:59.123 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 3/32 2023-10-12T00:13:59.124 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 4/32 2023-10-12T00:13:59.124 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 5/32 2023-10-12T00:13:59.124 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 6/32 2023-10-12T00:13:59.124 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-12T00:13:59.124 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-12T00:13:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-12T00:13:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-12T00:13:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-12T00:13:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-12T00:13:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-12T00:13:59.126 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-12T00:13:59.127 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-12T00:13:59.127 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-12T00:13:59.127 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-12T00:13:59.127 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-12T00:13:59.127 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-12T00:13:59.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-12T00:13:59.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-12T00:13:59.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-12T00:13:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-12T00:13:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-12T00:13:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-12T00:13:59.131 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-12T00:13:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-12T00:13:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-12T00:13:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-12T00:14:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-12T00:14:00.327 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T00:14:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-12T00:14:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-12T00:14:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:00.851 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T00:14:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T00:14:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-12T00:14:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T00:14:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T00:14:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-12T00:14:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T00:14:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T00:14:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T00:14:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-12T00:14:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T00:14:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T00:14:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T00:14:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T00:14:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-12T00:14:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T00:14:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T00:14:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T00:14:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T00:14:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T00:14:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T00:14:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:00.858 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:00.940 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-12T00:14:01.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2023-10-12T00:14:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-12T00:14:01.422 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-12T00:14:01.719 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-12T00:14:01.957 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:15 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:01.971 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-12T00:14:02.232 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-12T00:14:02.581 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-12T00:14:02.932 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-12T00:14:03.359 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:14:03.366 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 27/32 2023-10-12T00:14:03.553 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:03.554 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:14:03.554 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:03.643 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-12T00:14:03.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2023-10-12T00:14:04.519 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:18 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-12T00:14:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-12T00:14:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-12T00:14:06.156 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:06.157 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:06.157 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:06.157 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 169 k 2023-10-12T00:14:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:06.158 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:06.158 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:06.158 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:14:06.159 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:06.159 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 169 k 2023-10-12T00:14:06.159 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 671 k 2023-10-12T00:14:06.159 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-12T00:14:06.833 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-18.2.0-666.gaa3250ce.el8.noarch.rpm 250 kB/s | 169 kB 00:00 2023-10-12T00:14:06.834 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:06.834 INFO:teuthology.orchestra.run.smithi118.stdout:Total 249 kB/s | 169 kB 00:00 2023-10-12T00:14:06.835 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:06.843 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:06.843 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:06.845 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:06.849 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:06.896 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:06.996 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-12T00:14:07.184 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-12T00:14:07.202 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-12T00:14:07.358 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-12T00:14:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:07.591 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:07.752 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-12T00:14:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 1/32 2023-10-12T00:14:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2/32 2023-10-12T00:14:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 3/32 2023-10-12T00:14:07.753 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 4/32 2023-10-12T00:14:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 5/32 2023-10-12T00:14:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 6/32 2023-10-12T00:14:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-12T00:14:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-12T00:14:07.754 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-12T00:14:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-12T00:14:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-12T00:14:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-12T00:14:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-12T00:14:07.755 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-12T00:14:07.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-12T00:14:07.757 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-12T00:14:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-12T00:14:07.758 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-12T00:14:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-12T00:14:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-12T00:14:07.759 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-12T00:14:07.821 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:08.443 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:08.443 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:08.444 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:08.444 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:08.672 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T00:14:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-12T00:14:08.966 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:08.966 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:08.967 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T00:14:08.968 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T00:14:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-12T00:14:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T00:14:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T00:14:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T00:14:08.969 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-12T00:14:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T00:14:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T00:14:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T00:14:08.970 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T00:14:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-12T00:14:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-12T00:14:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-12T00:14:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T00:14:08.971 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T00:14:08.972 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T00:14:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T00:14:08.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:08.973 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:09.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2023-10-12T00:14:09.532 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:23 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:10.149 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:24 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:11.136 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:11.137 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:11.138 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-12T00:14:11.138 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:11.138 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:11.138 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 165 k 2023-10-12T00:14:11.138 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:11.139 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:11.139 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:11.139 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:14:11.139 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:11.139 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 165 k 2023-10-12T00:14:11.140 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 503 k 2023-10-12T00:14:11.140 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:11.543 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:14:11.738 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:11.738 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:14:11.738 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:11.780 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-18.2.0-666.gaa3250c 258 kB/s | 165 kB 00:00 2023-10-12T00:14:11.780 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:11.781 INFO:teuthology.orchestra.run.smithi118.stdout:Total 257 kB/s | 165 kB 00:00 2023-10-12T00:14:11.781 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:11.793 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:11.793 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:11.825 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2023-10-12T00:14:11.841 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:11.844 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:11.953 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:12.136 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:12.556 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:12.673 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:26 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:13.184 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:13.184 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:13.185 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:13.185 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:13.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:13.185 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:13.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2023-10-12T00:14:14.250 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:28 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:14.251 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:14.252 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:14.252 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:14:14.253 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:14.253 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:14.253 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 169 k 2023-10-12T00:14:14.253 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:14.254 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:14.254 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:14.254 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:14:14.254 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:14.255 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 169 k 2023-10-12T00:14:14.255 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 671 k 2023-10-12T00:14:14.255 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:14.432 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-18.2.0-666.gaa3250ce.el8.noarch.rpm 947 kB/s | 169 kB 00:00 2023-10-12T00:14:14.433 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:14.433 INFO:teuthology.orchestra.run.smithi112.stdout:Total 941 kB/s | 169 kB 00:00 2023-10-12T00:14:14.433 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:14:14.440 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:14:14.440 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:14:14.442 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:14:14.446 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:14:14.517 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:14:15.087 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:15.263 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:15.487 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:15.676 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:14:15.871 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:15.873 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:14:15.873 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:15.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T00:14:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:14:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:16.168 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:16.169 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:16.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T00:14:16.815 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:30 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:17.263 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:31 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:18.474 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:18.476 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:18.476 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:18.476 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:18.476 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:18.476 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 1.8 M 2023-10-12T00:14:18.477 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:14:18.477 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 23 k 2023-10-12T00:14:18.477 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 15 k 2023-10-12T00:14:18.477 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-12T00:14:18.477 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2023-10-12T00:14:18.478 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:18.479 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.1 M 2023-10-12T00:14:18.479 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 8.5 M 2023-10-12T00:14:18.480 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:18.868 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:18.868 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-10-12T00:14:18.868 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:18.869 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 165 k 2023-10-12T00:14:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:18.869 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:18.869 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:18.870 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:14:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:18.870 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 165 k 2023-10-12T00:14:18.870 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 503 k 2023-10-12T00:14:18.870 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:18.877 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): ceph-prometheus-alerts-18.2.0-666.gaa325 104 kB/s | 15 kB 00:00 2023-10-12T00:14:18.894 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): ceph-grafana-dashboards-18.2.0-666.gaa32 147 kB/s | 23 kB 00:00 2023-10-12T00:14:19.086 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-18.2.0-666.gaa3250c 757 kB/s | 165 kB 00:00 2023-10-12T00:14:19.088 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:19.088 INFO:teuthology.orchestra.run.smithi112.stdout:Total 753 kB/s | 165 kB 00:00 2023-10-12T00:14:19.088 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:14:19.098 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:14:19.099 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:14:19.111 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2023-10-12T00:14:19.137 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2023-10-12T00:14:19.145 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:14:19.148 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:14:19.286 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:14:19.498 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:19.499 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 508 kB/s | 196 kB 00:00 2023-10-12T00:14:19.629 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-mgr-dashboard-18.2.0-666.gaa3250ce. 2.0 MB/s | 1.8 MB 00:00 2023-10-12T00:14:19.632 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:19.633 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.8 MB/s | 2.1 MB 00:01 2023-10-12T00:14:19.644 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:19.654 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:19.654 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:19.733 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:19.738 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:19.908 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:19.993 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:20.224 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-12T00:14:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-12T00:14:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-12T00:14:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:20.558 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:14:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:20.559 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:20.571 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-12T00:14:20.690 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 4/6 2023-10-12T00:14:20.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2023-10-12T00:14:21.029 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 5/6 2023-10-12T00:14:21.201 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-12T00:14:21.478 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-12T00:14:21.478 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 1/6 2023-10-12T00:14:21.478 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2/6 2023-10-12T00:14:21.478 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 3/6 2023-10-12T00:14:21.479 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-12T00:14:21.479 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-12T00:14:21.629 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:35 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:22.337 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-12T00:14:22.338 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:22.338 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:22.338 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:22.339 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:22.339 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:22.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T00:14:22.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T00:14:22.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T00:14:22.340 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:22.341 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:22.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T00:14:23.048 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:14:23.243 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:23.244 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:14:23.244 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:23.327 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:23.343 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T00:14:24.195 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:38 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:24.914 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:24.916 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================================ 2023-10-12T00:14:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:24.916 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================================ 2023-10-12T00:14:24.916 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 7.4 M 2023-10-12T00:14:24.917 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:14:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-12T00:14:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-12T00:14:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-12T00:14:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-12T00:14:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-12T00:14:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-12T00:14:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-12T00:14:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:24.918 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:24.919 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================================ 2023-10-12T00:14:24.919 INFO:teuthology.orchestra.run.smithi118.stdout:Install 8 Packages 2023-10-12T00:14:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:24.920 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 35 M 2023-10-12T00:14:24.920 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 210 M 2023-10-12T00:14:24.920 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:25.586 INFO:teuthology.orchestra.run.smithi118.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-10-12T00:14:25.661 INFO:teuthology.orchestra.run.smithi118.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.7 MB/s | 4.9 MB 00:00 2023-10-12T00:14:25.753 INFO:teuthology.orchestra.run.smithi118.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2023-10-12T00:14:25.778 INFO:teuthology.orchestra.run.smithi118.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-10-12T00:14:25.786 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:25.788 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:25.788 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:14:25.788 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:25.788 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:25.788 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 1.8 M 2023-10-12T00:14:25.789 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:14:25.789 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 23 k 2023-10-12T00:14:25.789 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 15 k 2023-10-12T00:14:25.789 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-12T00:14:25.789 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T00:14:25.790 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T00:14:25.790 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:25.790 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:25.790 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:25.790 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-10-12T00:14:25.791 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:25.791 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.1 M 2023-10-12T00:14:25.792 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 8.5 M 2023-10-12T00:14:25.792 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:26.034 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-grafana-dashboards-18.2.0-666.gaa32 133 kB/s | 23 kB 00:00 2023-10-12T00:14:26.079 INFO:teuthology.orchestra.run.smithi118.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 42 MB/s | 14 MB 00:00 2023-10-12T00:14:26.147 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-prometheus-alerts-18.2.0-666.gaa325 51 kB/s | 15 kB 00:00 2023-10-12T00:14:26.204 INFO:teuthology.orchestra.run.smithi118.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2023-10-12T00:14:26.206 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 575 kB/s | 34 kB 00:00 2023-10-12T00:14:26.253 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 194 kB/s | 43 kB 00:00 2023-10-12T00:14:26.298 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.1 MB/s | 196 kB 00:00 2023-10-12T00:14:26.329 INFO:teuthology.orchestra.run.smithi118.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 690 kB/s | 172 kB 00:00 2023-10-12T00:14:26.429 INFO:teuthology.orchestra.run.smithi118.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.0-666 5.8 MB/s | 7.4 MB 00:01 2023-10-12T00:14:26.430 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:26.431 INFO:teuthology.orchestra.run.smithi118.stdout:Total 23 MB/s | 35 MB 00:01 2023-10-12T00:14:26.661 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:26.679 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:26.679 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:27.007 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-mgr-dashboard-18.2.0-666.gaa3250ce. 1.5 MB/s | 1.8 MB 00:01 2023-10-12T00:14:27.009 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:27.009 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.7 MB/s | 2.1 MB 00:01 2023-10-12T00:14:27.021 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:14:27.030 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:14:27.031 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:14:27.071 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:27.074 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:27.114 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:14:27.117 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:14:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:14:27.594 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-12T00:14:27.672 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:27.856 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-12T00:14:28.066 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-12T00:14:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 4/6 2023-10-12T00:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T00:14:28.482 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T00:14:28.544 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 5/6 2023-10-12T00:14:28.633 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T00:14:28.765 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-12T00:14:29.163 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-12T00:14:29.164 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 1/6 2023-10-12T00:14:29.164 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2/6 2023-10-12T00:14:29.164 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 3/6 2023-10-12T00:14:29.164 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-12T00:14:29.164 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-12T00:14:29.231 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T00:14:29.408 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:29.824 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-12T00:14:29.824 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:29.824 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:29.824 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:29.825 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:29.825 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:29.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T00:14:29.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T00:14:29.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T00:14:29.826 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:29.826 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:29.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T00:14:30.064 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:30.241 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T00:14:30.842 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:44 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:30.946 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T00:14:31.270 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T00:14:31.524 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T00:14:32.430 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:32.432 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-10-12T00:14:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:14:32.432 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-10-12T00:14:32.432 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 7.4 M 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-12T00:14:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-12T00:14:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-12T00:14:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-12T00:14:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-12T00:14:32.434 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:32.434 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:32.435 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-10-12T00:14:32.435 INFO:teuthology.orchestra.run.smithi112.stdout:Install 8 Packages 2023-10-12T00:14:32.435 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:32.436 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 35 M 2023-10-12T00:14:32.437 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 210 M 2023-10-12T00:14:32.437 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:33.063 INFO:teuthology.orchestra.run.smithi112.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-10-12T00:14:33.172 INFO:teuthology.orchestra.run.smithi112.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.5 MB/s | 4.9 MB 00:00 2023-10-12T00:14:33.271 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T00:14:33.272 INFO:teuthology.orchestra.run.smithi112.stdout:(3/8): ceph-mgr-diskprediction-local-18.2.0-666 12 MB/s | 7.4 MB 00:00 2023-10-12T00:14:33.297 INFO:teuthology.orchestra.run.smithi112.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2023-10-12T00:14:33.381 INFO:teuthology.orchestra.run.smithi112.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2023-10-12T00:14:33.606 INFO:teuthology.orchestra.run.smithi112.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 767 kB/s | 172 kB 00:00 2023-10-12T00:14:33.648 INFO:teuthology.orchestra.run.smithi112.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2023-10-12T00:14:33.989 INFO:teuthology.orchestra.run.smithi112.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 MB/s | 14 MB 00:00 2023-10-12T00:14:33.989 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:33.989 INFO:teuthology.orchestra.run.smithi112.stdout:Total 23 MB/s | 35 MB 00:01 2023-10-12T00:14:34.221 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:14:34.239 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:14:34.239 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:14:34.668 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-12T00:14:34.874 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-12T00:14:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:14:34.883 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:14:35.189 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-12T00:14:35.189 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 1/8 2023-10-12T00:14:35.190 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-12T00:14:35.190 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:35.190 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-12T00:14:35.191 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T00:14:35.191 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-12T00:14:35.191 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-12T00:14:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-12T00:14:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:35.848 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-12T00:14:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-12T00:14:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-12T00:14:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-12T00:14:35.850 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-12T00:14:35.850 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-12T00:14:35.850 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-12T00:14:35.850 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:35.850 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:35.939 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:14:35.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2023-10-12T00:14:36.079 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T00:14:36.238 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T00:14:36.421 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T00:14:36.829 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:50 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:37.015 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T00:14:37.246 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:37.870 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:38.021 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T00:14:38.422 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:38.423 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:38.423 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:38.424 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:38.424 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:38.424 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 47 k 2023-10-12T00:14:38.424 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:14:38.424 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T00:14:38.425 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T00:14:38.425 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T00:14:38.425 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-12T00:14:38.425 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-12T00:14:38.425 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T00:14:38.426 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-12T00:14:38.426 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-12T00:14:38.426 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-12T00:14:38.426 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-12T00:14:38.426 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T00:14:38.427 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T00:14:38.427 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:38.427 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:38.427 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:38.427 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2023-10-12T00:14:38.428 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:38.428 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 1.9 M 2023-10-12T00:14:38.428 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 25 M 2023-10-12T00:14:38.429 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:38.745 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T00:14:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T00:14:39.130 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 142 kB/s | 20 kB 00:00 2023-10-12T00:14:39.147 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 168 kB/s | 27 kB 00:00 2023-10-12T00:14:39.164 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): ceph-mgr-rook-18.2.0-666.gaa3250ce.el8. 271 kB/s | 47 kB 00:00 2023-10-12T00:14:39.265 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 942 kB/s | 126 kB 00:00 2023-10-12T00:14:39.331 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 601 kB/s | 110 kB 00:00 2023-10-12T00:14:39.445 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T00:14:39.449 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 546 kB/s | 155 kB 00:00 2023-10-12T00:14:39.474 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 204 kB/s | 43 kB 00:00 2023-10-12T00:14:39.499 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 196 kB/s | 33 kB 00:00 2023-10-12T00:14:39.583 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-10-12T00:14:39.600 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 646 kB/s | 81 kB 00:00 2023-10-12T00:14:39.650 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 962 kB/s | 64 kB 00:00 2023-10-12T00:14:39.709 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2023-10-12T00:14:39.801 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2023-10-12T00:14:39.801 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:39.802 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-10-12T00:14:39.861 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:39.876 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:39.877 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:39.962 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:39.965 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:40.481 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:40.771 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T00:14:40.947 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-12T00:14:41.191 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-12T00:14:41.205 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T00:14:41.444 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-12T00:14:41.662 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-12T00:14:41.901 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-12T00:14:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-12T00:14:42.443 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-12T00:14:42.461 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-12T00:14:42.661 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-12T00:14:42.982 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-12T00:14:42.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 1/8 2023-10-12T00:14:42.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-12T00:14:42.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-12T00:14:42.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-12T00:14:42.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T00:14:42.984 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-12T00:14:42.984 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-12T00:14:43.442 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-12T00:14:43.651 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-12T00:14:43.651 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:43.652 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:43.652 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:43.652 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-12T00:14:43.653 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-12T00:14:43.653 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-12T00:14:43.654 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-12T00:14:43.654 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-12T00:14:43.654 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-12T00:14:43.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-12T00:14:43.655 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:43.655 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:43.762 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-12T00:14:43.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2023-10-12T00:14:43.971 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-12T00:14:44.173 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-12T00:14:44.376 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-12T00:14:44.626 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:58 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-12T00:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 1/13 2023-10-12T00:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-12T00:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-12T00:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-12T00:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-12T00:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-12T00:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-12T00:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-12T00:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T00:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-12T00:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-12T00:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-12T00:14:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-12T00:14:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:45.377 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T00:14:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T00:14:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T00:14:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T00:14:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T00:14:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T00:14:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T00:14:45.381 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T00:14:45.381 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T00:14:45.381 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T00:14:45.382 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T00:14:45.382 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T00:14:45.382 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:45.383 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:45.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T00:14:46.203 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:46.205 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:46.205 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:14:46.206 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:46.206 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:46.206 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 47 k 2023-10-12T00:14:46.207 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:14:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T00:14:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T00:14:46.207 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T00:14:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-12T00:14:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-12T00:14:46.208 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T00:14:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-12T00:14:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-12T00:14:46.209 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-12T00:14:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-12T00:14:46.210 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T00:14:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T00:14:46.211 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:46.211 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:46.212 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:46.212 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2023-10-12T00:14:46.212 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:46.214 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.9 M 2023-10-12T00:14:46.214 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 25 M 2023-10-12T00:14:46.214 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:00 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:46.764 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 154 kB/s | 20 kB 00:00 2023-10-12T00:14:46.829 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 137 kB/s | 27 kB 00:00 2023-10-12T00:14:46.906 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): ceph-mgr-rook-18.2.0-666.gaa3250ce.el8. 174 kB/s | 47 kB 00:00 2023-10-12T00:14:46.931 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 756 kB/s | 126 kB 00:00 2023-10-12T00:14:46.956 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 863 kB/s | 110 kB 00:00 2023-10-12T00:14:47.123 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 198 kB/s | 33 kB 00:00 2023-10-12T00:14:47.149 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-10-12T00:14:47.174 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 578 kB/s | 155 kB 00:00 2023-10-12T00:14:47.199 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 216 kB/s | 16 kB 00:00 2023-10-12T00:14:47.316 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 551 kB/s | 64 kB 00:00 2023-10-12T00:14:47.366 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 373 kB/s | 81 kB 00:00 2023-10-12T00:14:47.400 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 733 kB/s | 61 kB 00:00 2023-10-12T00:14:47.611 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2023-10-12T00:14:47.613 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:47.613 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-10-12T00:14:47.672 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:14:47.688 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:14:47.688 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:14:47.775 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:14:47.778 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:14:47.965 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:47.966 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:47.966 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:47.967 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:47.967 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:47.967 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 132 k 2023-10-12T00:14:47.967 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:14:47.967 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-12T00:14:47.968 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-12T00:14:47.968 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-12T00:14:47.968 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-12T00:14:47.968 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:47.968 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:47.969 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:47.969 INFO:teuthology.orchestra.run.smithi118.stdout:Install 5 Packages 2023-10-12T00:14:47.969 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:47.970 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.0 M 2023-10-12T00:14:47.970 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2023-10-12T00:14:47.970 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:14:48.420 INFO:teuthology.orchestra.run.smithi118.stdout:(1/5): ceph-mgr-cephadm-18.2.0-666.gaa3250ce.el 675 kB/s | 132 kB 00:00 2023-10-12T00:14:48.554 INFO:teuthology.orchestra.run.smithi118.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2023-10-12T00:14:48.639 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T00:14:48.738 INFO:teuthology.orchestra.run.smithi118.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 9.3 MB/s | 4.8 MB 00:00 2023-10-12T00:14:48.861 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-12T00:14:49.071 INFO:teuthology.orchestra.run.smithi118.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 731 kB/s | 476 kB 00:00 2023-10-12T00:14:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-12T00:14:49.384 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-12T00:14:49.617 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-12T00:14:49.924 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-12T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 40 kB/s | 60 kB 00:01 2023-10-12T00:14:50.064 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:50.064 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.8 MB/s | 6.0 MB 00:02 2023-10-12T00:14:50.118 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:50.129 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:50.129 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:50.233 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-12T00:14:50.240 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:50.244 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:50.499 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-12T00:14:50.545 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:50.909 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-12T00:14:51.572 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-12T00:14:51.874 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-12T00:14:51.892 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-12T00:14:52.109 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-12T00:14:52.274 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-12T00:14:52.336 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-12T00:14:52.536 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-12T00:14:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-12T00:14:52.721 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-12T00:14:52.857 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-12T00:14:52.857 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 1/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-12T00:14:52.858 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-12T00:14:52.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-12T00:14:52.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T00:14:52.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-12T00:14:52.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-12T00:14:52.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-12T00:14:53.024 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-12T00:14:53.025 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/5 2023-10-12T00:14:53.025 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-12T00:14:53.025 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-12T00:14:53.026 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-12T00:14:53.595 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-12T00:14:53.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:53.595 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:14:53.595 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:53.596 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T00:14:53.596 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T00:14:53.596 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T00:14:53.596 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T00:14:53.596 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T00:14:53.597 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T00:14:53.597 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T00:14:53.597 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T00:14:53.597 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T00:14:53.597 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T00:14:53.598 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T00:14:53.598 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T00:14:53.598 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:53.598 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:14:53.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-12T00:14:53.665 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:53.665 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:53.665 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:14:53.665 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-12T00:14:53.666 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-12T00:14:53.666 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-12T00:14:53.666 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-12T00:14:53.666 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:53.666 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:53.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T00:14:53.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2023-10-12T00:14:54.571 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:08 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:54.641 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:08 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:14:56.173 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:14:56.174 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:56.174 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:14:56.175 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:56.175 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:14:56.175 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 132 k 2023-10-12T00:14:56.175 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:14:56.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-12T00:14:56.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-12T00:14:56.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-12T00:14:56.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-12T00:14:56.177 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:56.177 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:14:56.177 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:14:56.177 INFO:teuthology.orchestra.run.smithi112.stdout:Install 5 Packages 2023-10-12T00:14:56.177 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:14:56.178 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 6.0 M 2023-10-12T00:14:56.178 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2023-10-12T00:14:56.178 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:14:56.234 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:14:56.235 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:56.235 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:14:56.235 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:56.235 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:14:56.235 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 847 k 2023-10-12T00:14:56.236 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-12T00:14:56.236 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-12T00:14:56.236 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:56.236 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:14:56.237 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:14:56.237 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2023-10-12T00:14:56.237 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:56.237 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 930 k 2023-10-12T00:14:56.237 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 3.0 M 2023-10-12T00:14:56.238 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:14:56.721 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): ceph-fuse-18.2.0-666.gaa3250ce.el8.x86_6 3.1 MB/s | 847 kB 00:00 2023-10-12T00:14:56.746 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 288 kB/s | 83 kB 00:00 2023-10-12T00:14:56.748 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:14:56.748 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.8 MB/s | 930 kB 00:00 2023-10-12T00:14:56.758 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:14:56.769 INFO:teuthology.orchestra.run.smithi112.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2023-10-12T00:14:56.779 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:14:56.780 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:14:56.866 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:14:56.869 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:14:56.877 INFO:teuthology.orchestra.run.smithi112.stdout:(2/5): ceph-mgr-cephadm-18.2.0-666.gaa3250ce.el 352 kB/s | 132 kB 00:00 2023-10-12T00:14:56.952 INFO:teuthology.orchestra.run.smithi112.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2023-10-12T00:14:57.057 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:14:57.303 INFO:teuthology.orchestra.run.smithi112.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 142 kB/s | 60 kB 00:00 2023-10-12T00:14:57.305 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-12T00:14:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-12T00:14:57.892 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-12T00:14:57.893 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/2 2023-10-12T00:14:58.529 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-12T00:14:58.529 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:58.530 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:14:58.530 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-12T00:14:58.530 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:14:58.530 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:14:58.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2023-10-12T00:14:59.591 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:13 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:00.994 INFO:teuthology.orchestra.run.smithi112.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 113 kB/s | 476 kB 00:04 2023-10-12T00:15:00.995 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:00.995 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.2 MB/s | 6.0 MB 00:04 2023-10-12T00:15:01.050 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:01.062 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:01.062 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:01.194 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:01.195 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:01.195 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-12T00:15:01.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:01.196 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:01.196 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 261 k 2023-10-12T00:15:01.196 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:01.196 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:01.197 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:01.197 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:01.197 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:01.197 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 261 k 2023-10-12T00:15:01.197 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.3 M 2023-10-12T00:15:01.198 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:01.530 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:01.534 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-volume-18.2.0-666.gaa3250ce.el8.noarch.rpm 773 kB/s | 261 kB 00:00 2023-10-12T00:15:01.535 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:01.535 INFO:teuthology.orchestra.run.smithi118.stdout:Total 770 kB/s | 261 kB 00:00 2023-10-12T00:15:01.536 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:01.546 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:01.546 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:01.573 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:01.578 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:02.066 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-12T00:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:03.086 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-12T00:15:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:03.109 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:15:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:03.109 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:03.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2023-10-12T00:15:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-12T00:15:03.774 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-12T00:15:04.008 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-12T00:15:04.198 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:18 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:04.326 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-12T00:15:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/5 2023-10-12T00:15:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-12T00:15:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-12T00:15:04.327 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-12T00:15:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-12T00:15:04.955 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-12T00:15:04.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-12T00:15:04.957 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:04.957 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:05.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2023-10-12T00:15:05.791 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:05.791 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 139 k 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:05.792 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:05.793 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:05.793 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:05.794 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 139 k 2023-10-12T00:15:05.794 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 513 k 2023-10-12T00:15:05.794 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:05.928 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:19 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:06.612 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-18.2.0-666.gaa3250ce.el8.x86_64. 170 kB/s | 139 kB 00:00 2023-10-12T00:15:06.613 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:06.613 INFO:teuthology.orchestra.run.smithi118.stdout:Total 169 kB/s | 139 kB 00:00 2023-10-12T00:15:06.614 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:06.620 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:06.621 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:06.675 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:06.678 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:06.808 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:07.536 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:07.536 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:07.537 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:15:07.537 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:07.537 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:07.537 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 847 k 2023-10-12T00:15:07.537 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-10-12T00:15:07.538 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-12T00:15:07.538 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:07.538 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:07.538 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:07.538 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2023-10-12T00:15:07.539 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:07.539 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 930 k 2023-10-12T00:15:07.539 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 3.0 M 2023-10-12T00:15:07.540 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:07.941 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:07.942 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:07.942 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:07.942 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:07.943 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:07.943 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:08.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2023-10-12T00:15:08.952 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 394 kB/s | 83 kB 00:00 2023-10-12T00:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:22 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:09.219 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-18.2.0-666.gaa3250ce.el8.x86_6 1.7 MB/s | 847 kB 00:00 2023-10-12T00:15:09.220 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:09.220 INFO:teuthology.orchestra.run.smithi112.stdout:Total 553 kB/s | 930 kB 00:01 2023-10-12T00:15:09.232 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:09.255 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:09.255 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:09.347 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:09.350 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:09.523 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:09.824 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-12T00:15:10.050 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-12T00:15:10.366 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:10.441 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-12T00:15:10.441 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/2 2023-10-12T00:15:10.561 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:10.563 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:10.563 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:10.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2023-10-12T00:15:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-12T00:15:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:11.115 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-12T00:15:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:11.115 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:11.311 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-volume 2023-10-12T00:15:11.500 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:25 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:12.167 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:26 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:13.082 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:13.083 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:13.084 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-12T00:15:13.084 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:13.084 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:13.084 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 30 k 2023-10-12T00:15:13.085 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:13.085 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:13.085 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:13.085 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:13.085 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:13.086 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 k 2023-10-12T00:15:13.086 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 137 k 2023-10-12T00:15:13.086 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:13.664 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-18.2.0-666.gaa3250ce.el8.x86_64 52 kB/s | 30 kB 00:00 2023-10-12T00:15:13.665 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:13.665 INFO:teuthology.orchestra.run.smithi118.stdout:Total 52 kB/s | 30 kB 00:00 2023-10-12T00:15:13.666 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:13.669 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:13.669 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:13.698 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:13.701 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:13.761 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:13.775 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:13.776 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:13.776 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-10-12T00:15:13.776 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:13.777 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 261 k 2023-10-12T00:15:13.777 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:13.777 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:13.777 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:13.778 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:13.778 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:13.778 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 261 k 2023-10-12T00:15:13.778 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.3 M 2023-10-12T00:15:13.778 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:13.991 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:14.267 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:14.456 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-volume-18.2.0-666.gaa3250ce.el8.noarch.rpm 385 kB/s | 261 kB 00:00 2023-10-12T00:15:14.457 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:14.457 INFO:teuthology.orchestra.run.smithi112.stdout:Total 384 kB/s | 261 kB 00:00 2023-10-12T00:15:14.458 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:14.468 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:14.468 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:14.513 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:14.692 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:14.888 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:14.888 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:14.888 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:14.889 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:14.889 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:14.889 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:14.966 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:15.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2023-10-12T00:15:15.380 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:15.870 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:29 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-12T00:15:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:16.042 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-12T00:15:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:16.043 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:16.221 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2023-10-12T00:15:17.085 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:31 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:17.272 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:17.464 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:17.465 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:17.466 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:17.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2023-10-12T00:15:18.418 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:32 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:18.677 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:18.678 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:18.678 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-10-12T00:15:18.678 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:18.678 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 139 k 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:18.679 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:18.680 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 139 k 2023-10-12T00:15:18.680 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 513 k 2023-10-12T00:15:18.681 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:18.979 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-18.2.0-666.gaa3250ce.el8.x86_64. 463 kB/s | 139 kB 00:00 2023-10-12T00:15:18.980 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:18.980 INFO:teuthology.orchestra.run.smithi112.stdout:Total 461 kB/s | 139 kB 00:00 2023-10-12T00:15:18.981 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:18.986 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:18.986 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:19.036 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:19.039 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:19.193 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:19.464 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:19.788 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:19.814 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:20.007 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:20.008 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:20.009 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:20.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2023-10-12T00:15:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:20.382 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:20.382 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:20.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2023-10-12T00:15:20.967 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:34 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:21.451 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:35 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:22.370 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:22.565 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:22.566 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:22.567 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:22.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2023-10-12T00:15:22.861 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:23.054 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:23.055 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:23.056 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:23.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2023-10-12T00:15:23.511 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:37 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:23.998 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:37 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:24.920 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:25.112 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:25.114 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:25.114 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:25.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2023-10-12T00:15:25.602 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:25.603 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:25.603 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-10-12T00:15:25.604 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:25.604 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:25.604 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 30 k 2023-10-12T00:15:25.604 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:25.604 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:25.605 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:25.605 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:25.605 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:25.605 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 k 2023-10-12T00:15:25.605 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 137 k 2023-10-12T00:15:25.606 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:25.760 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-18.2.0-666.gaa3250ce.el8.x86_64 196 kB/s | 30 kB 00:00 2023-10-12T00:15:25.761 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:25.761 INFO:teuthology.orchestra.run.smithi112.stdout:Total 194 kB/s | 30 kB 00:00 2023-10-12T00:15:25.761 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:25.764 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:25.764 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:25.788 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:25.791 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:25.853 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:26.065 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:40 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:26.097 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:26.359 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:26.955 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:26.956 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:27.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2023-10-12T00:15:27.449 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:27.644 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:27.645 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:27.645 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:27.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2023-10-12T00:15:27.916 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:41 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:28.585 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:42 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:29.317 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:29.508 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:29.509 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:29.509 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:29.603 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2023-10-12T00:15:29.993 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:30.185 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:30.186 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-12T00:15:30.186 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:30.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2023-10-12T00:15:30.462 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:44 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:31.144 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:45 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:31.868 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:32.061 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:32.062 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:32.063 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:32.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2023-10-12T00:15:32.746 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:32.747 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:32.747 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-12T00:15:32.747 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:32.747 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:32.748 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 95 k 2023-10-12T00:15:32.748 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:32.748 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:32.748 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:32.748 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:32.749 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:32.749 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 95 k 2023-10-12T00:15:32.749 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 280 k 2023-10-12T00:15:32.749 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:33.012 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:47 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-18.2.0-666.gaa3250ce.el8.x86_64.rpm 336 kB/s | 95 kB 00:00 2023-10-12T00:15:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:Total 334 kB/s | 95 kB 00:00 2023-10-12T00:15:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:33.041 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:33.041 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:33.090 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:33.093 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:33.192 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:33.389 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:33.660 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:34.275 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:34.275 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:34.414 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:34.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2023-10-12T00:15:34.606 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:34.607 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:34.607 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:34.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2023-10-12T00:15:35.353 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:49 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:35.565 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:49 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:36.953 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:36.954 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:36.954 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-12T00:15:36.955 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:36.955 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:36.955 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.4 M 2023-10-12T00:15:36.955 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:36.955 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:36.956 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:36.956 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:36.956 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:36.956 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.4 M 2023-10-12T00:15:36.956 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 16 M 2023-10-12T00:15:36.957 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:36.962 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:37.156 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:37.157 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:37.157 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:37.250 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2023-10-12T00:15:38.103 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:52 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:38.340 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-18.2.0-666.gaa3250ce.el8.x86_64.rpm 2.5 MB/s | 3.4 MB 00:01 2023-10-12T00:15:38.341 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:38.342 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.5 MB/s | 3.4 MB 00:01 2023-10-12T00:15:38.342 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:38.420 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:38.423 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:38.865 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:39.064 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:39.486 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:39.524 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:39.734 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:39.735 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:39.735 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:39.819 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2023-10-12T00:15:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:40.096 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:40.097 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:40.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2023-10-12T00:15:40.673 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:54 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:41.143 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:55 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:42.071 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-12T00:15:42.269 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:42.270 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-10-12T00:15:42.271 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:42.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2023-10-12T00:15:42.752 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-12T00:15:42.753 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-12T00:15:42.753 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:42.754 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-12T00:15:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 179 k 2023-10-12T00:15:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:42.754 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-12T00:15:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-12T00:15:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-12T00:15:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 179 k 2023-10-12T00:15:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 568 k 2023-10-12T00:15:42.756 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-12T00:15:43.221 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:57 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:43.236 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-18.2.0-666.gaa3250ce.el8.x86_64.rpm 372 kB/s | 179 kB 00:00 2023-10-12T00:15:43.236 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:43.237 INFO:teuthology.orchestra.run.smithi118.stdout:Total 371 kB/s | 179 kB 00:00 2023-10-12T00:15:43.237 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-12T00:15:43.248 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-12T00:15:43.249 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-12T00:15:43.300 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-12T00:15:43.302 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-12T00:15:43.428 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-12T00:15:43.604 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:43.902 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:44.565 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:44.565 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:44.566 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-12T00:15:44.566 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:44.566 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:44.566 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-12T00:15:44.807 DEBUG:teuthology.parallel:result is None 2023-10-12T00:15:44.820 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:44.821 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:44.821 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-10-12T00:15:44.821 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:44.821 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:44.822 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 95 k 2023-10-12T00:15:44.822 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:44.822 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:44.822 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:44.822 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:44.823 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:44.823 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 95 k 2023-10-12T00:15:44.824 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 280 k 2023-10-12T00:15:44.824 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:45.372 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-18.2.0-666.gaa3250ce.el8.x86_64.rpm 172 kB/s | 95 kB 00:00 2023-10-12T00:15:45.373 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:45.373 INFO:teuthology.orchestra.run.smithi112.stdout:Total 172 kB/s | 95 kB 00:00 2023-10-12T00:15:45.373 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:45.384 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:45.384 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:45.435 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:45.438 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:46.127 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:46.399 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:47.078 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:47.079 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:47.080 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:47.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2023-10-12T00:15:48.160 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:02 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:49.741 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:49.742 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-10-12T00:15:49.742 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:49.742 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.4 M 2023-10-12T00:15:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:49.743 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:49.743 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:49.743 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:49.744 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.4 M 2023-10-12T00:15:49.744 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 16 M 2023-10-12T00:15:49.744 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:50.524 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-18.2.0-666.gaa3250ce.el8.x86_64.rpm 4.4 MB/s | 3.4 MB 00:00 2023-10-12T00:15:50.525 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:50.526 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.4 MB/s | 3.4 MB 00:00 2023-10-12T00:15:50.526 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:50.538 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:50.538 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:50.601 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:50.604 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:51.049 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:51.286 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:51.716 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:52.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:52.361 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:52.361 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:52.362 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:52.362 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:52.362 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:52.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2023-10-12T00:15:53.399 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:07 ago on Thu 12 Oct 2023 12:12:46 AM UTC. 2023-10-12T00:15:54.989 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-10-12T00:15:54.990 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-10-12T00:15:54.990 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:54.991 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-10-12T00:15:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 179 k 2023-10-12T00:15:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 179 k 2023-10-12T00:15:54.992 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 568 k 2023-10-12T00:15:54.993 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-10-12T00:15:55.202 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-18.2.0-666.gaa3250ce.el8.x86_64.rpm 849 kB/s | 179 kB 00:00 2023-10-12T00:15:55.203 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-10-12T00:15:55.203 INFO:teuthology.orchestra.run.smithi112.stdout:Total 844 kB/s | 179 kB 00:00 2023-10-12T00:15:55.204 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-10-12T00:15:55.214 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-10-12T00:15:55.214 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-10-12T00:15:55.264 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-10-12T00:15:55.267 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-10-12T00:15:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-10-12T00:15:55.616 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-12T00:15:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:56.534 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-10-12T00:15:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-12T00:15:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:56.535 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-10-12T00:15:56.721 DEBUG:teuthology.parallel:result is None 2023-10-12T00:15:56.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-12T00:15:56.873 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T00:15:56.896 INFO:teuthology.orchestra.run.smithi112.stdout:18.2.0-666.gaa3250ce.el8 2023-10-12T00:15:56.897 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-666.gaa3250ce.el8 2023-10-12T00:15:56.897 INFO:teuthology.task.install:The correct ceph version 18.2.0-666.gaa3250ce is installed. 2023-10-12T00:15:56.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-12T00:15:57.033 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T00:15:57.061 INFO:teuthology.orchestra.run.smithi118.stdout:18.2.0-666.gaa3250ce.el8 2023-10-12T00:15:57.061 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-666.gaa3250ce.el8 2023-10-12T00:15:57.062 INFO:teuthology.task.install:The correct ceph version 18.2.0-666.gaa3250ce is installed. 2023-10-12T00:15:57.062 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-12T00:15:57.063 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:57.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T00:15:57.093 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:57.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T00:15:57.136 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-12T00:15:57.137 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:57.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T00:15:57.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T00:15:57.231 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:57.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T00:15:57.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T00:15:57.340 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-12T00:15:57.341 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:57.341 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T00:15:57.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T00:15:57.435 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:57.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T00:15:57.470 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T00:15:57.543 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-12T00:15:57.545 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:57.546 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T00:15:57.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T00:15:57.640 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:57.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T00:15:57.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T00:15:57.750 INFO:teuthology.run_tasks:Running task ceph... 2023-10-12T00:15:57.812 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-12T00:15:57.813 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2023-10-12T00:15:57.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2023-10-12T00:15:57.849 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-12T00:15:57.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-12T00:15:57.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-12T00:15:57.923 INFO:tasks.ceph:Creating extra log directories... 2023-10-12T00:15:57.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-12T00:15:57.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-12T00:15:58.007 INFO:tasks.ceph:Creating ceph cluster c1... 2023-10-12T00:15:58.007 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-12T00:15:58.008 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'branch': 'wip-yuri-testing-2023-10-10-0840-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7422157', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-10-0840-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-reef', 'sha1': '4305a5835ec2627a352020742c8983c32bac9341'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1482, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-reef', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '4305a5835ec2627a352020742c8983c32bac9341', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2JLp2+vfzHVyxL6Acrw4wk1jHYH8MuiL4DWD9tM4S7ijSMYlbEw5+h08dafiwWGRp9Fogc9JG7tebz2ws0vSE=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZJXmqjfBSt/G/tZUHbg1vA/GOsb80hVRDsm9AHTjNE85QX13Lf8BegOL5Bji68NC+JX6KnRItmilTlMA9TtVA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-11_18:32:33', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-12T00:15:58.008 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-12T00:15:58.017 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-12T00:15:58.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T00:15:58.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T00:15:58.133 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:58.133 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2023-10-12T00:15:58.158 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-12T00:15:58.161 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2023-10-12T00:15:58.216 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-12T00:15:58.217 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:15:58.217 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 397476 Links: 1 2023-10-12T00:15:58.217 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:15:58.217 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:15:58.218 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-10-12 00:15:51.464354504 +0000 2023-10-12T00:15:58.218 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-10-12 00:11:40.620057263 +0000 2023-10-12T00:15:58.218 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-10-12 00:11:40.620057263 +0000 2023-10-12T00:15:58.218 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-10-12T00:15:58.219 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-12T00:15:58.282 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-10-12T00:15:58.282 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-10-12T00:15:58.283 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000263738 s, 1.9 MB/s 2023-10-12T00:15:58.284 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-12T00:15:58.342 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2023-10-12T00:15:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-12T00:15:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:15:58.401 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 390083 Links: 1 2023-10-12T00:15:58.401 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:15:58.401 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:15:58.401 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-10-12 00:15:51.464354504 +0000 2023-10-12T00:15:58.402 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-10-12 00:11:41.142043666 +0000 2023-10-12T00:15:58.402 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-10-12 00:11:41.142043666 +0000 2023-10-12T00:15:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-10-12T00:15:58.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-12T00:15:58.467 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-10-12T00:15:58.467 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-10-12T00:15:58.467 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000219203 s, 2.3 MB/s 2023-10-12T00:15:58.468 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-12T00:15:58.527 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2023-10-12T00:15:58.587 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-12T00:15:58.587 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:15:58.587 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 397064 Links: 1 2023-10-12T00:15:58.587 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:15:58.588 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:15:58.588 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-10-12 00:15:51.465354478 +0000 2023-10-12T00:15:58.588 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-10-12 00:11:41.887024262 +0000 2023-10-12T00:15:58.588 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-10-12 00:11:41.887024262 +0000 2023-10-12T00:15:58.588 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-10-12T00:15:58.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-12T00:15:58.652 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-10-12T00:15:58.653 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-10-12T00:15:58.653 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000219185 s, 2.3 MB/s 2023-10-12T00:15:58.655 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-12T00:15:58.713 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2023-10-12T00:15:58.771 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-12T00:15:58.772 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:15:58.772 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 400455 Links: 1 2023-10-12T00:15:58.772 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:15:58.772 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:15:58.773 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-10-12 00:15:51.465354478 +0000 2023-10-12T00:15:58.773 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-10-12 00:11:42.407010718 +0000 2023-10-12T00:15:58.773 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-10-12 00:11:42.407010718 +0000 2023-10-12T00:15:58.773 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-10-12T00:15:58.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-12T00:15:58.837 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-10-12T00:15:58.838 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-10-12T00:15:58.838 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000347934 s, 1.5 MB/s 2023-10-12T00:15:58.839 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-12T00:15:58.898 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:15:58.899 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T00:15:58.899 INFO:tasks.ceph:Generating config... 2023-10-12T00:15:58.900 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-12T00:15:58.900 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-12T00:15:58.901 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-12T00:15:58.901 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-12T00:15:58.901 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:15:58.901 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-12T00:15:58.902 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-12T00:15:58.902 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-12T00:15:58.902 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-12T00:15:58.902 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-12T00:15:58.903 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-12T00:15:58.903 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-12T00:15:58.903 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-12T00:15:58.903 INFO:tasks.ceph:[client] setuser = ceph 2023-10-12T00:15:58.904 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-12T00:15:58.904 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-12T00:15:58.904 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-12T00:15:58.904 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-12T00:15:58.905 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-12T00:15:58.905 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-12T00:15:58.905 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-12T00:15:58.905 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-12T00:15:58.906 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-12T00:15:58.906 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-12T00:15:58.906 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-12T00:15:58.906 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-12T00:15:58.907 INFO:tasks.ceph:Setting up c1.mon.a... 2023-10-12T00:15:58.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-10-12T00:15:58.994 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.keyring 2023-10-12T00:15:58.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-10-12T00:15:59.041 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T00:15:59.068 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.112')] 2023-10-12T00:15:59.069 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-12T00:15:59.070 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:59.070 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T00:15:59.126 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --print /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:15:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:15:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:15:59.210 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:15:59.211 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid 0c81838e-35f0-4b18-94d7-e1eb95e2a375 2023-10-12T00:15:59.211 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = pacific 2023-10-12T00:15:59.211 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2023-10-12T00:15:59.211 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 0c81838e-35f0-4b18-94d7-e1eb95e2a375 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2023-10-12T00:15:59.210224+0000 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:created 2023-10-12T00:15:59.210224+0000 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 16 (pacific) 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2023-10-12T00:15:59.212 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-10-12T00:15:59.213 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T00:15:59.268 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 0c81838e-35f0-4b18-94d7-e1eb95e2a375... 2023-10-12T00:15:59.269 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-12T00:15:59.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-12T00:15:59.354 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-10-12T00:15:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-10-12T00:15:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T00:15:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-10-12T00:15:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-10-12T00:15:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-10-12T00:15:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-10-12T00:15:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-10-12T00:15:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-10-12T00:15:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-10-12T00:15:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T00:15:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-10-12T00:15:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T00:15:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T00:15:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-10-12T00:15:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-10-12T00:15:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-10-12T00:15:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-10-12T00:15:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-10-12T00:15:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-10-12T00:15:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-10-12T00:15:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-10-12T00:15:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T00:15:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T00:15:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-10-12T00:15:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-12T00:15:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-10-12T00:15:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-10-12T00:15:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-10-12T00:15:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-10-12T00:15:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-10-12T00:15:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.112 2023-10-12T00:15:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 0c81838e-35f0-4b18-94d7-e1eb95e2a375 2023-10-12T00:15:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.364 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-10-12T00:15:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-10-12T00:15:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-10-12T00:15:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-10-12T00:15:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-10-12T00:15:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-10-12T00:15:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-10-12T00:15:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-10-12T00:15:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-10-12T00:15:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-10-12T00:15:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-10-12T00:15:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-10-12T00:15:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-10-12T00:15:59.368 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.368 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-10-12T00:15:59.368 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.368 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-10-12T00:15:59.369 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.369 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-10-12T00:15:59.369 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-10-12T00:15:59.369 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.370 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-10-12T00:15:59.370 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-10-12T00:15:59.370 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:15:59.370 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-10-12T00:15:59.371 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-10-12T00:15:59.371 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.371 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-10-12T00:15:59.371 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:15:59.371 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-10-12T00:15:59.372 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-10-12T00:15:59.372 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-10-12T00:15:59.372 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:15:59.372 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-10-12T00:15:59.373 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-10-12T00:15:59.374 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-10-12T00:15:59.374 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-10-12T00:15:59.374 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-10-12T00:15:59.374 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:15:59.374 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-10-12T00:15:59.375 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-10-12T00:15:59.375 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:15:59.375 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.375 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T00:15:59.376 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T00:15:59.376 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T00:15:59.376 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.376 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T00:15:59.376 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T00:15:59.377 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T00:15:59.377 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.377 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-10-12T00:15:59.377 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-10-12T00:15:59.378 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.378 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-10-12T00:15:59.378 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:15:59.378 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-10-12T00:15:59.378 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-10-12T00:15:59.379 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-10-12T00:15:59.379 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-10-12T00:15:59.379 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T00:15:59.380 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T00:15:59.380 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-10-12T00:15:59.380 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-10-12T00:15:59.380 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-10-12T00:15:59.380 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-10-12T00:15:59.381 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:15:59.381 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-10-12T00:15:59.381 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-10-12T00:15:59.381 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-10-12T00:15:59.381 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-10-12T00:15:59.382 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-10-12T00:15:59.382 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T00:15:59.382 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-10-12T00:15:59.382 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-10-12T00:15:59.382 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-10-12T00:15:59.383 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-10-12T00:15:59.383 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-10-12T00:15:59.383 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-10-12T00:15:59.383 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-10-12T00:15:59.383 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T00:15:59.384 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-10-12T00:15:59.384 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T00:15:59.384 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-10-12T00:15:59.384 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T00:15:59.386 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2023-10-12T00:15:59.386 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2023-10-12T00:15:59.386 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T00:15:59.387 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2023-10-12T00:15:59.387 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.387 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2023-10-12T00:15:59.387 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.387 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2023-10-12T00:15:59.388 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.388 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2023-10-12T00:15:59.388 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2023-10-12T00:15:59.388 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.388 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2023-10-12T00:15:59.389 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2023-10-12T00:15:59.389 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.389 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T00:15:59.389 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2023-10-12T00:15:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T00:15:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T00:15:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2023-10-12T00:15:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2023-10-12T00:15:59.392 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2023-10-12T00:15:59.392 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2023-10-12T00:15:59.392 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T00:15:59.392 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T00:15:59.392 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.393 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2023-10-12T00:15:59.394 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.394 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2023-10-12T00:15:59.394 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2023-10-12T00:15:59.394 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2023-10-12T00:15:59.394 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.112 2023-10-12T00:15:59.395 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 0c81838e-35f0-4b18-94d7-e1eb95e2a375 2023-10-12T00:15:59.395 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.395 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2023-10-12T00:15:59.395 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2023-10-12T00:15:59.395 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.397 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2023-10-12T00:15:59.397 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2023-10-12T00:15:59.397 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2023-10-12T00:15:59.398 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.398 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2023-10-12T00:15:59.398 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2023-10-12T00:15:59.398 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.398 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2023-10-12T00:15:59.399 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2023-10-12T00:15:59.399 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2023-10-12T00:15:59.399 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.399 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2023-10-12T00:15:59.400 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2023-10-12T00:15:59.400 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.400 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2023-10-12T00:15:59.400 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.400 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2023-10-12T00:15:59.401 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.401 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2023-10-12T00:15:59.401 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.401 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2023-10-12T00:15:59.401 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2023-10-12T00:15:59.402 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.402 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2023-10-12T00:15:59.402 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2023-10-12T00:15:59.402 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:15:59.402 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2023-10-12T00:15:59.403 INFO:teuthology.orchestra.run.smithi118.stdout: rocksdb delete range threshold = 0 2023-10-12T00:15:59.403 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.403 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2023-10-12T00:15:59.403 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:15:59.403 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2023-10-12T00:15:59.404 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.405 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2023-10-12T00:15:59.405 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:15:59.405 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-12T00:15:59.405 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2023-10-12T00:15:59.405 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-12T00:15:59.406 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2023-10-12T00:15:59.406 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2023-10-12T00:15:59.406 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:15:59.406 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2023-10-12T00:15:59.406 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2023-10-12T00:15:59.407 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:15:59.407 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.407 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T00:15:59.407 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T00:15:59.407 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T00:15:59.408 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.408 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T00:15:59.408 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T00:15:59.408 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T00:15:59.408 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.409 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2023-10-12T00:15:59.409 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2023-10-12T00:15:59.409 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.409 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2023-10-12T00:15:59.409 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:15:59.410 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2023-10-12T00:15:59.410 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2023-10-12T00:15:59.410 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2023-10-12T00:15:59.410 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2023-10-12T00:15:59.410 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T00:15:59.411 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T00:15:59.411 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw = 20 2023-10-12T00:15:59.411 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw sync = 20 2023-10-12T00:15:59.411 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt require ssl = False 2023-10-12T00:15:59.411 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms backend = testing 2023-10-12T00:15:59.412 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:15:59.412 INFO:teuthology.orchestra.run.smithi118.stdout: rgw curl low speed time = 300 2023-10-12T00:15:59.412 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data log num shards = 4 2023-10-12T00:15:59.412 INFO:teuthology.orchestra.run.smithi118.stdout: rgw md log max shards = 4 2023-10-12T00:15:59.412 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync data inject err probability = 0.1 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync log trim interval = 0 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync obj etag verify = True 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout: setgroup = ceph 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout: setuser = ceph 2023-10-12T00:15:59.413 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2023-10-12T00:15:59.414 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2023-10-12T00:15:59.414 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data notify interval msec = 0 2023-10-12T00:15:59.414 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2023-10-12T00:15:59.414 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T00:15:59.414 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2023-10-12T00:15:59.415 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T00:15:59.415 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2023-10-12T00:15:59.415 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T00:15:59.415 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-10-12T00:15:59.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-10-12T00:15:59.484 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-12T00:15:59.484 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:59.484 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-10-12T00:15:59.503 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:59.503 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-10-12T00:15:59.560 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:15:59.561 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:59.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c1.keyring 2023-10-12T00:15:59.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T00:15:59.639 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:15:59.639 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-12T00:15:59.696 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:15:59.697 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:59.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/c1.keyring 2023-10-12T00:15:59.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-12T00:15:59.740 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:15:59.741 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-12T00:15:59.800 INFO:tasks.ceph:Setting up mon nodes... 2023-10-12T00:15:59.800 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-12T00:15:59.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-10-12T00:15:59.854 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-10-12T00:15:59.856 INFO:tasks.ceph:Setting up mds nodes... 2023-10-12T00:15:59.857 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-12T00:15:59.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-10-12T00:15:59.902 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.0.keyring 2023-10-12T00:15:59.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-10-12T00:15:59.958 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.1.keyring 2023-10-12T00:15:59.970 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-12T00:15:59.971 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T00:15:59.971 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-10-12T00:15:59.998 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:15:59.998 INFO:tasks.ceph:role: c1.osd.0 2023-10-12T00:15:59.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:15:59.999 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-12T00:16:00.071 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:00.072 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:00.072 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:00.072 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:00.073 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:00.073 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:00.073 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:00.073 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:00.073 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:00.074 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:00.508 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-10-12T00:16:00.509 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-10-12T00:16:00.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-10-12T00:16:00.684 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-10-12T00:16:00.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-10-12T00:16:00.786 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:00.787 INFO:tasks.ceph:role: c1.osd.1 2023-10-12T00:16:00.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:16:00.788 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-12T00:16:00.860 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:00.863 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:01.285 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-10-12T00:16:01.286 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-10-12T00:16:01.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-10-12T00:16:01.392 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-10-12T00:16:01.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-10-12T00:16:01.491 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:01.491 INFO:tasks.ceph:role: c1.osd.2 2023-10-12T00:16:01.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:16:01.492 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-12T00:16:01.566 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:01.567 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:01.567 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:01.567 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:01.568 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:02.011 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-10-12T00:16:02.013 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-10-12T00:16:02.013 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-10-12T00:16:02.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-10-12T00:16:02.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:16:02.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.315+0000 7f3b7372e640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-10-12T00:16:02.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.316+0000 7f3b7372e640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-10-12T00:16:02.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.316+0000 7f3b7372e640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T00:16:02.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.316+0000 7f3b7372e640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T00:16:02.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.316+0000 7f3b7372e640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-12T00:16:02.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:02.322+0000 7f3b7372e640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-10-12T00:16:04.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-10-12T00:16:04.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:16:04.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.532+0000 7f0f1e4d1640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-10-12T00:16:04.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.532+0000 7f0f1e4d1640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-10-12T00:16:04.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.532+0000 7f0f1e4d1640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T00:16:04.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.532+0000 7f0f1e4d1640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T00:16:04.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.532+0000 7f0f1e4d1640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-12T00:16:04.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:04.533+0000 7f0f1e4d1640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-10-12T00:16:06.614 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-10-12T00:16:06.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:16:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.738+0000 7f483395a640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-10-12T00:16:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.738+0000 7f483395a640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-10-12T00:16:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.738+0000 7f483395a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T00:16:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.738+0000 7f483395a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T00:16:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.738+0000 7f483395a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-12T00:16:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:16:06.739+0000 7f483395a640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-10-12T00:16:08.823 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-10-12T00:16:08.854 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-12T00:16:08.855 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:08.855 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-10-12T00:16:08.924 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:08.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-10-12T00:16:08.993 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:08.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-10-12T00:16:09.062 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:09.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-10-12T00:16:09.129 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:09.130 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-10-12T00:16:09.189 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:09.189 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-10-12T00:16:09.246 INFO:tasks.ceph:Adding keys to all mons... 2023-10-12T00:16:09.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/c1.keyring 2023-10-12T00:16:09.312 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2023-10-12T00:16:09.312 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC/OidlwQz7MhAA/zWvZO4g2mcBUn4bqZVFbQ== 2023-10-12T00:16:09.313 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-10-12T00:16:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDCOidlzUvUEhAA8WXlR3FnE33EFS5/mB59Sg== 2023-10-12T00:16:09.313 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-10-12T00:16:09.313 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDEOidl3m/CHxAAPSnhI7YCXF0g7pn7jMS4Ng== 2023-10-12T00:16:09.313 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-10-12T00:16:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDGOidlxxQZLBAAJxoTsxaO1+a6uItZnXesRg== 2023-10-12T00:16:09.314 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-10-12T00:16:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC/OidlPqfUNRAAv8e5lb9kVhSz2ecmbjdThQ== 2023-10-12T00:16:09.314 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2023-10-12T00:16:09.314 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC/OidlQR0qORAAlfSSk051jLdSuPsAsZ0S8w== 2023-10-12T00:16:09.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-12T00:16:09.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:09.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:09.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:09.571 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T00:16:09.628 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T00:16:09.685 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-12T00:16:09.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-10-12T00:16:09.713 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-10-12T00:16:09.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-10-12T00:16:09.839 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-10-12T00:16:09.897 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-10-12T00:16:09.898 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-10-12T00:16:09.898 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-10-12T00:16:09.940 INFO:tasks.ceph.c1.mon.a:Started 2023-10-12T00:16:09.940 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-10-12T00:16:09.941 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-10-12T00:16:09.941 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-10-12T00:16:09.943 INFO:tasks.ceph.c1.mgr.x:Started 2023-10-12T00:16:09.943 INFO:tasks.ceph:Setting crush tunables to default 2023-10-12T00:16:09.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd crush tunables default 2023-10-12T00:16:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2023-10-12T00:16:10.162 INFO:tasks.ceph:check_enable_crimson: False 2023-10-12T00:16:10.163 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-10-12T00:16:10.163 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:10.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-10-12T00:16:10.190 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:10.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-10-12T00:16:10.257 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:10.257 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-10-12T00:16:10.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new f2bbcec7-8e86-49a7-bb96-caa35a67030e 0 2023-10-12T00:16:10.342 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:10.341+0000 7f34c8aaa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-12T00:16:10.526 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:10.525+0000 7f34c8aaa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-12T00:16:10.584 INFO:teuthology.orchestra.run.smithi112.stdout:0 2023-10-12T00:16:10.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new 7a936562-d306-42b7-8afc-11ff11acc7af 1 2023-10-12T00:16:10.812 INFO:teuthology.orchestra.run.smithi112.stdout:1 2023-10-12T00:16:10.824 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new 9f57cff1-ad5c-441c-a6f1-8d3d569abd03 2 2023-10-12T00:16:10.883 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:10.883+0000 7f34c8aaa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-12T00:16:11.041 INFO:teuthology.orchestra.run.smithi112.stdout:2 2023-10-12T00:16:11.053 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-10-12T00:16:11.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-10-12T00:16:11.055 INFO:tasks.ceph.c1.osd.0:Started 2023-10-12T00:16:11.056 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-10-12T00:16:11.056 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-10-12T00:16:11.058 INFO:tasks.ceph.c1.osd.1:Started 2023-10-12T00:16:11.059 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-10-12T00:16:11.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-10-12T00:16:11.061 INFO:tasks.ceph.c1.osd.2:Started 2023-10-12T00:16:11.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T00:16:11.280 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:11.280 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"fsid":"0c81838e-35f0-4b18-94d7-e1eb95e2a375","created":"2023-10-12T00:16:10.058439+0000","modified":"2023-10-12T00:16:11.040009+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T00:16:11.040009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f2bbcec7-8e86-49a7-bb96-caa35a67030e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a936562-d306-42b7-8afc-11ff11acc7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f57cff1-ad5c-441c-a6f1-8d3d569abd03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:11.292 INFO:tasks.ceph.ceph_manager.c1:[] 2023-10-12T00:16:11.292 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-12T00:16:11.433 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-10-12T00:16:11.432+0000 7f3b909f1640 -1 Falling back to public interface 2023-10-12T00:16:11.448 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:11.448+0000 7f34c8aaa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-12T00:16:11.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-12T00:16:11.645 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:11.644+0000 7f34c8aaa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-12T00:16:11.677 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-10-12T00:16:11.676+0000 7f1c5e3f0640 -1 Falling back to public interface 2023-10-12T00:16:11.686 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-10-12T00:16:11.685+0000 7fde93c55640 -1 Falling back to public interface 2023-10-12T00:16:11.790 INFO:teuthology.misc.health.smithi112.stdout: 2023-10-12T00:16:11.790 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":5,"fsid":"0c81838e-35f0-4b18-94d7-e1eb95e2a375","created":"2023-10-12T00:16:10.058439+0000","modified":"2023-10-12T00:16:11.040009+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T00:16:11.040009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f2bbcec7-8e86-49a7-bb96-caa35a67030e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a936562-d306-42b7-8afc-11ff11acc7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f57cff1-ad5c-441c-a6f1-8d3d569abd03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:11.801 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-12T00:16:12.022 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:12.022+0000 7f34c8aaa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-12T00:16:12.687 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:12.686+0000 7f34c8aaa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-12T00:16:12.759 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:12.759+0000 7f34c8aaa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-12T00:16:12.895 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:12.894+0000 7f34c8aaa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-12T00:16:12.964 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:12.963+0000 7f34c8aaa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-12T00:16:13.146 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.145+0000 7f34c8aaa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-12T00:16:13.225 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.224+0000 7f34c8aaa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-12T00:16:13.302 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.301+0000 7f34c8aaa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-12T00:16:13.406 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.406+0000 7f34c8aaa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-12T00:16:13.589 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.588+0000 7f34c8aaa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-12T00:16:13.672 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.671+0000 7f34c8aaa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-12T00:16:13.963 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:13.961+0000 7f34c8aaa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-12T00:16:14.032 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:14.031+0000 7f34c8aaa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-12T00:16:14.186 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-10-12T00:16:14.186+0000 7f1c5e3f0640 -1 osd.1 0 log_to_monitors true 2023-10-12T00:16:14.230 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-10-12T00:16:14.229+0000 7f3b909f1640 -1 osd.2 0 log_to_monitors true 2023-10-12T00:16:14.436 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-10-12T00:16:14.436+0000 7fde93c55640 -1 osd.0 0 log_to_monitors true 2023-10-12T00:16:14.561 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:14.560+0000 7f34c8aaa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-12T00:16:14.876 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:14.875+0000 7f34c8aaa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-12T00:16:15.018 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:15.017+0000 7f34c8aaa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-12T00:16:15.449 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:15.448+0000 7f34c8aaa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-12T00:16:15.622 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:15.621+0000 7f34c8aaa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-12T00:16:15.794 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:15.792+0000 7f34c8aaa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-12T00:16:16.104 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-10-12T00:16:16.103+0000 7f3b84d92700 -1 osd.2 0 waiting for initial osdmap 2023-10-12T00:16:16.130 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:16.129+0000 7f34c8aaa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-12T00:16:16.158 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-10-12T00:16:16.158+0000 7f3b7c96c700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:16.627 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-10-12T00:16:16.626+0000 7f1c52791700 -1 osd.1 0 waiting for initial osdmap 2023-10-12T00:16:16.665 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-10-12T00:16:16.665+0000 7f1c4a36b700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:16.766 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-10-12T00:16:16.765+0000 7fde87ff6700 -1 osd.0 0 waiting for initial osdmap 2023-10-12T00:16:16.775 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-10-12T00:16:16.775+0000 7fde7fbd0700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:17.149 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-10-12T00:16:17.149+0000 7f3489ec3700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-12T00:16:18.104 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-12T00:16:18.428 INFO:teuthology.misc.health.smithi112.stdout: 2023-10-12T00:16:18.429 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":9,"fsid":"0c81838e-35f0-4b18-94d7-e1eb95e2a375","created":"2023-10-12T00:16:10.058439+0000","modified":"2023-10-12T00:16:17.630089+0000","last_up_change":"2023-10-12T00:16:17.630089+0000","last_in_change":"2023-10-12T00:16:11.040009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f2bbcec7-8e86-49a7-bb96-caa35a67030e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":114222},{"type":"v1","addr":"172.21.15.112:6817","nonce":114222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":114222},{"type":"v1","addr":"172.21.15.112:6819","nonce":114222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":114222},{"type":"v1","addr":"172.21.15.112:6823","nonce":114222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":114222},{"type":"v1","addr":"172.21.15.112:6821","nonce":114222}]},"public_addr":"172.21.15.112:6817/114222","cluster_addr":"172.21.15.112:6819/114222","heartbeat_back_addr":"172.21.15.112:6823/114222","heartbeat_front_addr":"172.21.15.112:6821/114222","state":["exists","up"]},{"osd":1,"uuid":"7a936562-d306-42b7-8afc-11ff11acc7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":114221},{"type":"v1","addr":"172.21.15.112:6809","nonce":114221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":114221},{"type":"v1","addr":"172.21.15.112:6811","nonce":114221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":114221},{"type":"v1","addr":"172.21.15.112:6815","nonce":114221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":114221},{"type":"v1","addr":"172.21.15.112:6813","nonce":114221}]},"public_addr":"172.21.15.112:6809/114221","cluster_addr":"172.21.15.112:6811/114221","heartbeat_back_addr":"172.21.15.112:6815/114221","heartbeat_front_addr":"172.21.15.112:6813/114221","state":["exists","up"]},{"osd":2,"uuid":"9f57cff1-ad5c-441c-a6f1-8d3d569abd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":114233},{"type":"v1","addr":"172.21.15.112:6801","nonce":114233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":114233},{"type":"v1","addr":"172.21.15.112:6803","nonce":114233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":114233},{"type":"v1","addr":"172.21.15.112:6807","nonce":114233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":114233},{"type":"v1","addr":"172.21.15.112:6805","nonce":114233}]},"public_addr":"172.21.15.112:6801/114233","cluster_addr":"172.21.15.112:6803/114233","heartbeat_back_addr":"172.21.15.112:6807/114233","heartbeat_front_addr":"172.21.15.112:6805/114233","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.191064+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:18.441 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-12T00:16:18.442 INFO:tasks.ceph:Creating RBD pool 2023-10-12T00:16:18.442 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool create rbd 8 2023-10-12T00:16:19.641 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2023-10-12T00:16:19.657 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-12T00:16:20.645 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-12T00:16:20.657 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-10-12T00:16:20.658 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-10-12T00:16:20.658 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-10-12T00:16:20.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-10-12T00:16:20.659 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-12T00:16:21.036 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:21.048 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":113809},{"type":"v1","addr":"172.21.15.112:6825","nonce":113809}]},"active_addr":"172.21.15.112:6825/113809","active_change":"2023-10-12T00:16:16.143339+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3871190497}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1538984376}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2797495776}]}]} 2023-10-12T00:16:21.051 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-10-12T00:16:21.051 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-10-12T00:16:21.052 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T00:16:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:21.371 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":12,"fsid":"0c81838e-35f0-4b18-94d7-e1eb95e2a375","created":"2023-10-12T00:16:10.058439+0000","modified":"2023-10-12T00:16:20.634982+0000","last_up_change":"2023-10-12T00:16:17.630089+0000","last_in_change":"2023-10-12T00:16:11.040009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-12T00:16:18.188473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T00:16:18.779953+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2bbcec7-8e86-49a7-bb96-caa35a67030e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":114222},{"type":"v1","addr":"172.21.15.112:6817","nonce":114222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":114222},{"type":"v1","addr":"172.21.15.112:6819","nonce":114222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":114222},{"type":"v1","addr":"172.21.15.112:6823","nonce":114222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":114222},{"type":"v1","addr":"172.21.15.112:6821","nonce":114222}]},"public_addr":"172.21.15.112:6817/114222","cluster_addr":"172.21.15.112:6819/114222","heartbeat_back_addr":"172.21.15.112:6823/114222","heartbeat_front_addr":"172.21.15.112:6821/114222","state":["exists","up"]},{"osd":1,"uuid":"7a936562-d306-42b7-8afc-11ff11acc7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":114221},{"type":"v1","addr":"172.21.15.112:6809","nonce":114221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":114221},{"type":"v1","addr":"172.21.15.112:6811","nonce":114221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":114221},{"type":"v1","addr":"172.21.15.112:6815","nonce":114221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":114221},{"type":"v1","addr":"172.21.15.112:6813","nonce":114221}]},"public_addr":"172.21.15.112:6809/114221","cluster_addr":"172.21.15.112:6811/114221","heartbeat_back_addr":"172.21.15.112:6815/114221","heartbeat_front_addr":"172.21.15.112:6813/114221","state":["exists","up"]},{"osd":2,"uuid":"9f57cff1-ad5c-441c-a6f1-8d3d569abd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":114233},{"type":"v1","addr":"172.21.15.112:6801","nonce":114233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":114233},{"type":"v1","addr":"172.21.15.112:6803","nonce":114233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":114233},{"type":"v1","addr":"172.21.15.112:6807","nonce":114233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":114233},{"type":"v1","addr":"172.21.15.112:6805","nonce":114233}]},"public_addr":"172.21.15.112:6801/114233","cluster_addr":"172.21.15.112:6803/114233","heartbeat_back_addr":"172.21.15.112:6807/114233","heartbeat_front_addr":"172.21.15.112:6805/114233","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.459811+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.216414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.191064+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:21.384 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-10-12T00:16:21.385 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-12T00:16:21.705 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:21.705 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":13,"fsid":"0c81838e-35f0-4b18-94d7-e1eb95e2a375","created":"2023-10-12T00:16:10.058439+0000","modified":"2023-10-12T00:16:21.638540+0000","last_up_change":"2023-10-12T00:16:17.630089+0000","last_in_change":"2023-10-12T00:16:11.040009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-12T00:16:18.188473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T00:16:18.779953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2bbcec7-8e86-49a7-bb96-caa35a67030e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":114222},{"type":"v1","addr":"172.21.15.112:6817","nonce":114222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":114222},{"type":"v1","addr":"172.21.15.112:6819","nonce":114222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":114222},{"type":"v1","addr":"172.21.15.112:6823","nonce":114222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":114222},{"type":"v1","addr":"172.21.15.112:6821","nonce":114222}]},"public_addr":"172.21.15.112:6817/114222","cluster_addr":"172.21.15.112:6819/114222","heartbeat_back_addr":"172.21.15.112:6823/114222","heartbeat_front_addr":"172.21.15.112:6821/114222","state":["exists","up"]},{"osd":1,"uuid":"7a936562-d306-42b7-8afc-11ff11acc7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":114221},{"type":"v1","addr":"172.21.15.112:6809","nonce":114221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":114221},{"type":"v1","addr":"172.21.15.112:6811","nonce":114221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":114221},{"type":"v1","addr":"172.21.15.112:6815","nonce":114221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":114221},{"type":"v1","addr":"172.21.15.112:6813","nonce":114221}]},"public_addr":"172.21.15.112:6809/114221","cluster_addr":"172.21.15.112:6811/114221","heartbeat_back_addr":"172.21.15.112:6815/114221","heartbeat_front_addr":"172.21.15.112:6813/114221","state":["exists","up"]},{"osd":2,"uuid":"9f57cff1-ad5c-441c-a6f1-8d3d569abd03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":114233},{"type":"v1","addr":"172.21.15.112:6801","nonce":114233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":114233},{"type":"v1","addr":"172.21.15.112:6803","nonce":114233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":114233},{"type":"v1","addr":"172.21.15.112:6807","nonce":114233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":114233},{"type":"v1","addr":"172.21.15.112:6805","nonce":114233}]},"public_addr":"172.21.15.112:6801/114233","cluster_addr":"172.21.15.112:6803/114233","heartbeat_back_addr":"172.21.15.112:6807/114233","heartbeat_front_addr":"172.21.15.112:6805/114233","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.459811+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.216414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:15.191064+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:21.718 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-10-12T00:16:21.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-10-12T00:16:21.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-10-12T00:16:21.891 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-10-12T00:16:21.892 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-12T00:16:21.896 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-10-12T00:16:21.896 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-12T00:16:21.904 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-10-12T00:16:21.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-12T00:16:22.230 INFO:teuthology.orchestra.run.smithi112.stdout:34359738369 2023-10-12T00:16:22.244 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-10-12T00:16:22.284 INFO:teuthology.orchestra.run.smithi112.stdout:38654705665 2023-10-12T00:16:22.287 INFO:teuthology.orchestra.run.smithi112.stdout:38654705665 2023-10-12T00:16:22.300 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.1 2023-10-12T00:16:22.302 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.0 2023-10-12T00:16:23.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-12T00:16:23.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-12T00:16:23.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-12T00:16:23.609 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-10-12T00:16:23.622 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-10-12T00:16:23.623 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:23.661 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-10-12T00:16:23.673 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-10-12T00:16:23.675 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.1 2023-10-12T00:16:23.675 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:23.688 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.0 2023-10-12T00:16:23.688 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:23.688 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-10-12T00:16:23.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-12T00:16:24.041 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:24.043 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-10-12T00:16:24.054 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-12T00:16:22.147987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82452,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490348,"statfs":{"total":322122547200,"available":322038116352,"internally_reserved":0,"allocated":1503232,"data_stored":1264237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641105+0000","last_change":"2023-10-12T00:16:20.644276+0000","last_active":"2023-10-12T00:16:21.641105+0000","last_peered":"2023-10-12T00:16:21.641105+0000","last_clean":"2023-10-12T00:16:21.641105+0000","last_became_active":"2023-10-12T00:16:20.644000+0000","last_became_peered":"2023-10-12T00:16:20.644000+0000","last_unstale":"2023-10-12T00:16:21.641105+0000","last_undegraded":"2023-10-12T00:16:21.641105+0000","last_fullsized":"2023-10-12T00:16:21.641105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T09:40:52.980150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641070+0000","last_change":"2023-10-12T00:16:20.642703+0000","last_active":"2023-10-12T00:16:21.641070+0000","last_peered":"2023-10-12T00:16:21.641070+0000","last_clean":"2023-10-12T00:16:21.641070+0000","last_became_active":"2023-10-12T00:16:20.641992+0000","last_became_peered":"2023-10-12T00:16:20.641992+0000","last_unstale":"2023-10-12T00:16:21.641070+0000","last_undegraded":"2023-10-12T00:16:21.641070+0000","last_fullsized":"2023-10-12T00:16:21.641070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T10:04:31.866459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640994+0000","last_change":"2023-10-12T00:16:20.642361+0000","last_active":"2023-10-12T00:16:21.640994+0000","last_peered":"2023-10-12T00:16:21.640994+0000","last_clean":"2023-10-12T00:16:21.640994+0000","last_became_active":"2023-10-12T00:16:20.642093+0000","last_became_peered":"2023-10-12T00:16:20.642093+0000","last_unstale":"2023-10-12T00:16:21.640994+0000","last_undegraded":"2023-10-12T00:16:21.640994+0000","last_fullsized":"2023-10-12T00:16:21.640994+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:04:32.391787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:20.648833+0000","last_change":"2023-10-12T00:16:20.642753+0000","last_active":"2023-10-12T00:16:20.648833+0000","last_peered":"2023-10-12T00:16:20.648833+0000","last_clean":"2023-10-12T00:16:20.648833+0000","last_became_active":"2023-10-12T00:16:20.642472+0000","last_became_peered":"2023-10-12T00:16:20.642472+0000","last_unstale":"2023-10-12T00:16:20.648833+0000","last_undegraded":"2023-10-12T00:16:20.648833+0000","last_fullsized":"2023-10-12T00:16:20.648833+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T07:30:12.836713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640916+0000","last_change":"2023-10-12T00:16:19.644380+0000","last_active":"2023-10-12T00:16:21.640916+0000","last_peered":"2023-10-12T00:16:21.640916+0000","last_clean":"2023-10-12T00:16:21.640916+0000","last_became_active":"2023-10-12T00:16:19.643515+0000","last_became_peered":"2023-10-12T00:16:19.643515+0000","last_unstale":"2023-10-12T00:16:21.640916+0000","last_undegraded":"2023-10-12T00:16:21.640916+0000","last_fullsized":"2023-10-12T00:16:21.640916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:18.631335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:18.631335+0000","last_clean_scrub_stamp":"2023-10-12T00:16:18.631335+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T05:07:57.449726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640950+0000","last_change":"2023-10-12T00:16:20.643719+0000","last_active":"2023-10-12T00:16:21.640950+0000","last_peered":"2023-10-12T00:16:21.640950+0000","last_clean":"2023-10-12T00:16:21.640950+0000","last_became_active":"2023-10-12T00:16:20.643482+0000","last_became_peered":"2023-10-12T00:16:20.643482+0000","last_unstale":"2023-10-12T00:16:21.640950+0000","last_undegraded":"2023-10-12T00:16:21.640950+0000","last_fullsized":"2023-10-12T00:16:21.640950+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:49:15.400292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:20.645852+0000","last_change":"2023-10-12T00:16:20.641761+0000","last_active":"2023-10-12T00:16:20.645852+0000","last_peered":"2023-10-12T00:16:20.645852+0000","last_clean":"2023-10-12T00:16:20.645852+0000","last_became_active":"2023-10-12T00:16:20.641188+0000","last_became_peered":"2023-10-12T00:16:20.641188+0000","last_unstale":"2023-10-12T00:16:20.645852+0000","last_undegraded":"2023-10-12T00:16:20.645852+0000","last_fullsized":"2023-10-12T00:16:20.645852+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T11:00:04.791714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:20.646009+0000","last_change":"2023-10-12T00:16:20.642670+0000","last_active":"2023-10-12T00:16:20.646009+0000","last_peered":"2023-10-12T00:16:20.646009+0000","last_clean":"2023-10-12T00:16:20.646009+0000","last_became_active":"2023-10-12T00:16:20.641958+0000","last_became_peered":"2023-10-12T00:16:20.641958+0000","last_unstale":"2023-10-12T00:16:20.646009+0000","last_undegraded":"2023-10-12T00:16:20.646009+0000","last_fullsized":"2023-10-12T00:16:20.646009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:24:49.625640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641013+0000","last_change":"2023-10-12T00:16:20.643631+0000","last_active":"2023-10-12T00:16:21.641013+0000","last_peered":"2023-10-12T00:16:21.641013+0000","last_clean":"2023-10-12T00:16:21.641013+0000","last_became_active":"2023-10-12T00:16:20.643292+0000","last_became_peered":"2023-10-12T00:16:20.643292+0000","last_unstale":"2023-10-12T00:16:21.641013+0000","last_undegraded":"2023-10-12T00:16:21.641013+0000","last_fullsized":"2023-10-12T00:16:21.641013+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:11:06.527231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27764,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829836,"statfs":{"total":107374182400,"available":107345752064,"internally_reserved":0,"allocated":700416,"data_stored":618523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":27191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829964,"statfs":{"total":107374182400,"available":107345883136,"internally_reserved":0,"allocated":700416,"data_stored":618523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T00:16:24.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-12T00:16:24.370 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:24.373 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-10-12T00:16:24.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-12T00:16:24.148370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82460,"kb_used_data":1476,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490340,"statfs":{"total":322122547200,"available":322038108160,"internally_reserved":0,"allocated":1511424,"data_stored":1267715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000383"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641105+0000","last_change":"2023-10-12T00:16:20.644276+0000","last_active":"2023-10-12T00:16:21.641105+0000","last_peered":"2023-10-12T00:16:21.641105+0000","last_clean":"2023-10-12T00:16:21.641105+0000","last_became_active":"2023-10-12T00:16:20.644000+0000","last_became_peered":"2023-10-12T00:16:20.644000+0000","last_unstale":"2023-10-12T00:16:21.641105+0000","last_undegraded":"2023-10-12T00:16:21.641105+0000","last_fullsized":"2023-10-12T00:16:21.641105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T09:40:52.980150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641070+0000","last_change":"2023-10-12T00:16:20.642703+0000","last_active":"2023-10-12T00:16:21.641070+0000","last_peered":"2023-10-12T00:16:21.641070+0000","last_clean":"2023-10-12T00:16:21.641070+0000","last_became_active":"2023-10-12T00:16:20.641992+0000","last_became_peered":"2023-10-12T00:16:20.641992+0000","last_unstale":"2023-10-12T00:16:21.641070+0000","last_undegraded":"2023-10-12T00:16:21.641070+0000","last_fullsized":"2023-10-12T00:16:21.641070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T10:04:31.866459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640994+0000","last_change":"2023-10-12T00:16:20.642361+0000","last_active":"2023-10-12T00:16:21.640994+0000","last_peered":"2023-10-12T00:16:21.640994+0000","last_clean":"2023-10-12T00:16:21.640994+0000","last_became_active":"2023-10-12T00:16:20.642093+0000","last_became_peered":"2023-10-12T00:16:20.642093+0000","last_unstale":"2023-10-12T00:16:21.640994+0000","last_undegraded":"2023-10-12T00:16:21.640994+0000","last_fullsized":"2023-10-12T00:16:21.640994+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:04:32.391787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:20.648833+0000","last_change":"2023-10-12T00:16:20.642753+0000","last_active":"2023-10-12T00:16:20.648833+0000","last_peered":"2023-10-12T00:16:20.648833+0000","last_clean":"2023-10-12T00:16:20.648833+0000","last_became_active":"2023-10-12T00:16:20.642472+0000","last_became_peered":"2023-10-12T00:16:20.642472+0000","last_unstale":"2023-10-12T00:16:20.648833+0000","last_undegraded":"2023-10-12T00:16:20.648833+0000","last_fullsized":"2023-10-12T00:16:20.648833+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T07:30:12.836713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640916+0000","last_change":"2023-10-12T00:16:19.644380+0000","last_active":"2023-10-12T00:16:21.640916+0000","last_peered":"2023-10-12T00:16:21.640916+0000","last_clean":"2023-10-12T00:16:21.640916+0000","last_became_active":"2023-10-12T00:16:19.643515+0000","last_became_peered":"2023-10-12T00:16:19.643515+0000","last_unstale":"2023-10-12T00:16:21.640916+0000","last_undegraded":"2023-10-12T00:16:21.640916+0000","last_fullsized":"2023-10-12T00:16:21.640916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:18.631335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:18.631335+0000","last_clean_scrub_stamp":"2023-10-12T00:16:18.631335+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T05:07:57.449726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.640950+0000","last_change":"2023-10-12T00:16:20.643719+0000","last_active":"2023-10-12T00:16:21.640950+0000","last_peered":"2023-10-12T00:16:21.640950+0000","last_clean":"2023-10-12T00:16:21.640950+0000","last_became_active":"2023-10-12T00:16:20.643482+0000","last_became_peered":"2023-10-12T00:16:20.643482+0000","last_unstale":"2023-10-12T00:16:21.640950+0000","last_undegraded":"2023-10-12T00:16:21.640950+0000","last_fullsized":"2023-10-12T00:16:21.640950+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:49:15.400292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:20.645852+0000","last_change":"2023-10-12T00:16:20.641761+0000","last_active":"2023-10-12T00:16:20.645852+0000","last_peered":"2023-10-12T00:16:20.645852+0000","last_clean":"2023-10-12T00:16:20.645852+0000","last_became_active":"2023-10-12T00:16:20.641188+0000","last_became_peered":"2023-10-12T00:16:20.641188+0000","last_unstale":"2023-10-12T00:16:20.645852+0000","last_undegraded":"2023-10-12T00:16:20.645852+0000","last_fullsized":"2023-10-12T00:16:20.645852+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T11:00:04.791714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.879452+0000","last_change":"2023-10-12T00:16:20.642670+0000","last_active":"2023-10-12T00:16:21.879452+0000","last_peered":"2023-10-12T00:16:21.879452+0000","last_clean":"2023-10-12T00:16:21.879452+0000","last_became_active":"2023-10-12T00:16:20.641958+0000","last_became_peered":"2023-10-12T00:16:20.641958+0000","last_unstale":"2023-10-12T00:16:21.879452+0000","last_undegraded":"2023-10-12T00:16:21.879452+0000","last_fullsized":"2023-10-12T00:16:21.879452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:24:49.625640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:21.641013+0000","last_change":"2023-10-12T00:16:20.643631+0000","last_active":"2023-10-12T00:16:21.641013+0000","last_peered":"2023-10-12T00:16:21.641013+0000","last_clean":"2023-10-12T00:16:21.641013+0000","last_became_active":"2023-10-12T00:16:20.643292+0000","last_became_peered":"2023-10-12T00:16:20.643292+0000","last_unstale":"2023-10-12T00:16:21.641013+0000","last_undegraded":"2023-10-12T00:16:21.641013+0000","last_fullsized":"2023-10-12T00:16:21.641013+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:19.634032+0000","last_clean_scrub_stamp":"2023-10-12T00:16:19.634032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:11:06.527231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27772,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":708608,"data_stored":622001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":27191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829964,"statfs":{"total":107374182400,"available":107345883136,"internally_reserved":0,"allocated":700416,"data_stored":618523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T00:16:24.385 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-10-12T00:16:24.385 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-10-12T00:16:24.385 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-10-12T00:16:24.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-10-12T00:16:24.746 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:24.746 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-12T00:16:24.760 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-10-12T00:16:24.760 INFO:teuthology.run_tasks:Running task ceph... 2023-10-12T00:16:24.771 INFO:tasks.ceph:Creating ceph cluster c2... 2023-10-12T00:16:24.771 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-12T00:16:24.771 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'branch': 'wip-yuri-testing-2023-10-10-0840-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7422157', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-10-0840-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-reef', 'sha1': '4305a5835ec2627a352020742c8983c32bac9341'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1482, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-reef', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '4305a5835ec2627a352020742c8983c32bac9341', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2JLp2+vfzHVyxL6Acrw4wk1jHYH8MuiL4DWD9tM4S7ijSMYlbEw5+h08dafiwWGRp9Fogc9JG7tebz2ws0vSE=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZJXmqjfBSt/G/tZUHbg1vA/GOsb80hVRDsm9AHTjNE85QX13Lf8BegOL5Bji68NC+JX6KnRItmilTlMA9TtVA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-11_18:32:33', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-12T00:16:24.772 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-12T00:16:24.774 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-12T00:16:24.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T00:16:24.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2023-10-12T00:16:24.872 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:24.873 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2023-10-12T00:16:24.935 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-12T00:16:24.935 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2023-10-12T00:16:24.997 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-12T00:16:24.997 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:16:24.997 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 395595 Links: 1 2023-10-12T00:16:24.997 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:16:24.998 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:16:24.998 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-12 00:15:39.235715534 +0000 2023-10-12T00:16:24.998 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-12 00:11:40.616190580 +0000 2023-10-12T00:16:24.998 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-12 00:11:40.616190580 +0000 2023-10-12T00:16:24.998 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-12T00:16:24.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-12T00:16:25.068 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-12T00:16:25.069 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-12T00:16:25.069 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000134942 s, 3.8 MB/s 2023-10-12T00:16:25.070 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-12T00:16:25.134 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2023-10-12T00:16:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-12T00:16:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:16:25.200 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 398420 Links: 1 2023-10-12T00:16:25.200 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:16:25.200 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:16:25.200 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-12 00:15:39.235715534 +0000 2023-10-12T00:16:25.201 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-12 00:11:41.138179398 +0000 2023-10-12T00:16:25.201 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-12 00:11:41.138179398 +0000 2023-10-12T00:16:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-12T00:16:25.202 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-12T00:16:25.271 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-12T00:16:25.271 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-12T00:16:25.271 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000124209 s, 4.1 MB/s 2023-10-12T00:16:25.272 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-12T00:16:25.336 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2023-10-12T00:16:25.397 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-12T00:16:25.397 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:16:25.397 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 398532 Links: 1 2023-10-12T00:16:25.398 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:16:25.398 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:16:25.398 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-12 00:15:39.235715534 +0000 2023-10-12T00:16:25.398 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-12 00:11:41.655168323 +0000 2023-10-12T00:16:25.399 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-12 00:11:41.655168323 +0000 2023-10-12T00:16:25.399 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-12T00:16:25.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-12T00:16:25.467 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-12T00:16:25.468 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-12T00:16:25.468 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.7426e-05 s, 5.3 MB/s 2023-10-12T00:16:25.469 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-12T00:16:25.531 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2023-10-12T00:16:25.591 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-12T00:16:25.591 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-12T00:16:25.591 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 391735 Links: 1 2023-10-12T00:16:25.591 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-12T00:16:25.592 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-12T00:16:25.592 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-12 00:15:39.236715512 +0000 2023-10-12T00:16:25.592 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-12 00:11:42.193156799 +0000 2023-10-12T00:16:25.592 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-12 00:11:42.193156799 +0000 2023-10-12T00:16:25.592 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-12T00:16:25.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-12T00:16:25.659 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-12T00:16:25.659 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-12T00:16:25.660 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000117459 s, 4.4 MB/s 2023-10-12T00:16:25.661 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-12T00:16:25.721 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:25.721 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T00:16:25.721 INFO:tasks.ceph:Generating config... 2023-10-12T00:16:25.723 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-12T00:16:25.723 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-12T00:16:25.723 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-12T00:16:25.723 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-12T00:16:25.724 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:16:25.724 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-12T00:16:25.724 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-12T00:16:25.724 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-12T00:16:25.725 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-12T00:16:25.725 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-12T00:16:25.725 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-12T00:16:25.725 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-12T00:16:25.726 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-12T00:16:25.726 INFO:tasks.ceph:[client] setuser = ceph 2023-10-12T00:16:25.726 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-12T00:16:25.727 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-12T00:16:25.727 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-12T00:16:25.727 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-12T00:16:25.727 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-12T00:16:25.728 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-12T00:16:25.728 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-12T00:16:25.728 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-12T00:16:25.728 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-12T00:16:25.729 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-12T00:16:25.729 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-12T00:16:25.729 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-12T00:16:25.730 INFO:tasks.ceph:Setting up c2.mon.a... 2023-10-12T00:16:25.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-10-12T00:16:25.809 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.keyring 2023-10-12T00:16:25.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-10-12T00:16:25.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T00:16:25.888 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.118')] 2023-10-12T00:16:25.889 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.118'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-12T00:16:25.890 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:25.890 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T00:16:25.946 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.118 --print /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:26.029 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:16:26.029 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:16:26.030 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:26.030 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: generated fsid eb47bf95-bcad-4e64-bdca-484ec2af758b 2023-10-12T00:16:26.030 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = pacific 2023-10-12T00:16:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 0 2023-10-12T00:16:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:fsid eb47bf95-bcad-4e64-bdca-484ec2af758b 2023-10-12T00:16:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:last_changed 2023-10-12T00:16:26.030606+0000 2023-10-12T00:16:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:created 2023-10-12T00:16:26.030606+0000 2023-10-12T00:16:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:min_mon_release 16 (pacific) 2023-10-12T00:16:26.032 INFO:teuthology.orchestra.run.smithi118.stdout:election_strategy: 1 2023-10-12T00:16:26.032 INFO:teuthology.orchestra.run.smithi118.stdout:0: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.a 2023-10-12T00:16:26.032 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-10-12T00:16:26.033 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-12T00:16:26.091 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID eb47bf95-bcad-4e64-bdca-484ec2af758b... 2023-10-12T00:16:26.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-12T00:16:26.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-12T00:16:26.146 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-10-12T00:16:26.147 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-10-12T00:16:26.147 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T00:16:26.147 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-10-12T00:16:26.147 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.147 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-10-12T00:16:26.148 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.148 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-10-12T00:16:26.148 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.148 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-10-12T00:16:26.148 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-10-12T00:16:26.149 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.149 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-10-12T00:16:26.149 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-10-12T00:16:26.149 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.149 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T00:16:26.150 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-10-12T00:16:26.150 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.150 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T00:16:26.150 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T00:16:26.150 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.151 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-10-12T00:16:26.151 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.151 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-10-12T00:16:26.151 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-10-12T00:16:26.151 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-10-12T00:16:26.152 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-10-12T00:16:26.152 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-10-12T00:16:26.152 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-10-12T00:16:26.152 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-10-12T00:16:26.152 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T00:16:26.153 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T00:16:26.153 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-10-12T00:16:26.153 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.153 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-12T00:16:26.153 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.154 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-10-12T00:16:26.154 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.154 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-10-12T00:16:26.154 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.154 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.118 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = eb47bf95-bcad-4e64-bdca-484ec2af758b 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.155 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-10-12T00:16:26.156 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-10-12T00:16:26.156 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.156 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-10-12T00:16:26.156 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-10-12T00:16:26.156 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-10-12T00:16:26.157 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.157 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-10-12T00:16:26.157 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-10-12T00:16:26.157 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.157 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-10-12T00:16:26.158 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.159 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-10-12T00:16:26.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.159 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-10-12T00:16:26.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.159 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-10-12T00:16:26.160 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-10-12T00:16:26.161 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:16:26.161 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-10-12T00:16:26.161 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-10-12T00:16:26.161 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.161 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-10-12T00:16:26.162 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:16:26.162 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-10-12T00:16:26.162 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-10-12T00:16:26.162 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-10-12T00:16:26.162 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:16:26.163 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:16:26.163 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-10-12T00:16:26.163 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.163 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-10-12T00:16:26.163 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-10-12T00:16:26.164 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:16:26.165 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-10-12T00:16:26.165 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-10-12T00:16:26.165 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:16:26.165 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.165 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T00:16:26.166 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T00:16:26.166 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T00:16:26.166 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.166 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T00:16:26.166 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T00:16:26.167 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T00:16:26.167 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.167 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-10-12T00:16:26.167 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-10-12T00:16:26.167 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-10-12T00:16:26.168 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-10-12T00:16:26.169 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T00:16:26.169 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T00:16:26.169 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-10-12T00:16:26.169 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-10-12T00:16:26.169 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-10-12T00:16:26.170 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-10-12T00:16:26.170 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:16:26.170 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-10-12T00:16:26.170 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-10-12T00:16:26.170 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-10-12T00:16:26.171 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-10-12T00:16:26.172 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-10-12T00:16:26.172 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-10-12T00:16:26.172 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-10-12T00:16:26.172 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-10-12T00:16:26.172 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T00:16:26.173 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-10-12T00:16:26.173 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T00:16:26.173 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-10-12T00:16:26.173 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T00:16:26.176 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2023-10-12T00:16:26.176 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2023-10-12T00:16:26.176 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-12T00:16:26.176 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2023-10-12T00:16:26.177 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.177 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2023-10-12T00:16:26.177 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.177 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2023-10-12T00:16:26.177 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.178 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2023-10-12T00:16:26.178 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2023-10-12T00:16:26.178 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.178 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2023-10-12T00:16:26.178 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2023-10-12T00:16:26.179 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.179 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-12T00:16:26.179 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2023-10-12T00:16:26.179 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.179 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2023-10-12T00:16:26.180 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-12T00:16:26.180 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.180 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2023-10-12T00:16:26.180 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.180 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2023-10-12T00:16:26.181 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2023-10-12T00:16:26.181 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2023-10-12T00:16:26.181 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2023-10-12T00:16:26.181 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2023-10-12T00:16:26.181 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2023-10-12T00:16:26.182 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2023-10-12T00:16:26.182 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-12T00:16:26.182 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-12T00:16:26.182 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2023-10-12T00:16:26.182 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2023-10-12T00:16:26.183 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.184 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2023-10-12T00:16:26.184 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2023-10-12T00:16:26.184 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2023-10-12T00:16:26.184 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.118 2023-10-12T00:16:26.184 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = eb47bf95-bcad-4e64-bdca-484ec2af758b 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2023-10-12T00:16:26.185 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2023-10-12T00:16:26.186 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2023-10-12T00:16:26.186 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.186 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2023-10-12T00:16:26.186 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2023-10-12T00:16:26.186 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.187 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2023-10-12T00:16:26.187 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2023-10-12T00:16:26.187 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2023-10-12T00:16:26.187 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.187 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2023-10-12T00:16:26.188 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.189 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2023-10-12T00:16:26.189 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.189 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2023-10-12T00:16:26.189 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2023-10-12T00:16:26.190 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.190 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2023-10-12T00:16:26.190 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2023-10-12T00:16:26.190 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:16:26.191 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2023-10-12T00:16:26.191 INFO:teuthology.orchestra.run.smithi118.stdout: rocksdb delete range threshold = 0 2023-10-12T00:16:26.191 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.191 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2023-10-12T00:16:26.191 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:16:26.192 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2023-10-12T00:16:26.192 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-12T00:16:26.192 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-12T00:16:26.192 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:16:26.192 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-12T00:16:26.193 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2023-10-12T00:16:26.194 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-12T00:16:26.194 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2023-10-12T00:16:26.194 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2023-10-12T00:16:26.194 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-12T00:16:26.194 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2023-10-12T00:16:26.195 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2023-10-12T00:16:26.196 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.196 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2023-10-12T00:16:26.196 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-12T00:16:26.196 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-12T00:16:26.196 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.197 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2023-10-12T00:16:26.197 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2023-10-12T00:16:26.197 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.197 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2023-10-12T00:16:26.197 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:26.198 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2023-10-12T00:16:26.198 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2023-10-12T00:16:26.198 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2023-10-12T00:16:26.198 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2023-10-12T00:16:26.198 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw = 20 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw sync = 20 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt require ssl = False 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms backend = testing 2023-10-12T00:16:26.199 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-12T00:16:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: rgw curl low speed time = 300 2023-10-12T00:16:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data log num shards = 4 2023-10-12T00:16:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: rgw md log max shards = 4 2023-10-12T00:16:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync data inject err probability = 0.1 2023-10-12T00:16:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync log trim interval = 0 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync meta inject err probability = 0.1 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync obj etag verify = True 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: setgroup = ceph 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: setuser = ceph 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2023-10-12T00:16:26.201 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2023-10-12T00:16:26.202 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data notify interval msec = 0 2023-10-12T00:16:26.202 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2023-10-12T00:16:26.202 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10 2023-10-12T00:16:26.202 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2023-10-12T00:16:26.202 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 1000 2023-10-12T00:16:26.203 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2023-10-12T00:16:26.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10000 2023-10-12T00:16:26.204 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-10-12T00:16:26.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-10-12T00:16:26.302 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-12T00:16:26.303 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:26.303 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-10-12T00:16:26.321 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:26.322 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-10-12T00:16:26.379 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T00:16:26.380 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:26.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c2.keyring 2023-10-12T00:16:26.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T00:16:26.424 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-10-12T00:16:26.424 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:26.484 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:16:26.484 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:26.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/c2.keyring 2023-10-12T00:16:26.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-12T00:16:26.521 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:26.521 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:26.579 INFO:tasks.ceph:Setting up mon nodes... 2023-10-12T00:16:26.580 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-12T00:16:26.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-10-12T00:16:26.673 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-10-12T00:16:26.675 INFO:tasks.ceph:Setting up mds nodes... 2023-10-12T00:16:26.676 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-12T00:16:26.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-10-12T00:16:26.720 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.client.0.keyring 2023-10-12T00:16:26.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-10-12T00:16:26.776 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.client.1.keyring 2023-10-12T00:16:26.788 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-12T00:16:26.789 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-12T00:16:26.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-10-12T00:16:26.856 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:26.857 INFO:tasks.ceph:role: c2.osd.0 2023-10-12T00:16:26.857 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:16:26.858 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-12T00:16:26.923 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:26.925 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:26.925 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:26.946 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-12T00:16:26.947 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-12T00:16:26.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-10-12T00:16:27.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-10-12T00:16:27.100 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-10-12T00:16:27.168 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:27.168 INFO:tasks.ceph:role: c2.osd.1 2023-10-12T00:16:27.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:16:27.169 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-12T00:16:27.234 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:27.234 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:27.235 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:27.235 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:27.236 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:27.237 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:27.237 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:27.257 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-12T00:16:27.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-12T00:16:27.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-10-12T00:16:27.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-10-12T00:16:27.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-10-12T00:16:27.477 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-12T00:16:27.477 INFO:tasks.ceph:role: c2.osd.2 2023-10-12T00:16:27.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T00:16:27.478 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-12T00:16:27.543 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-12T00:16:27.544 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-12T00:16:27.544 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-12T00:16:27.544 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-12T00:16:27.544 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-12T00:16:27.544 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-12T00:16:27.545 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-12T00:16:27.545 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-12T00:16:27.545 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-12T00:16:27.545 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-12T00:16:27.567 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-12T00:16:27.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-12T00:16:27.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-10-12T00:16:27.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-10-12T00:16:27.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:27.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.811+0000 7f6f08ea6640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-10-12T00:16:27.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.811+0000 7f6f08ea6640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-10-12T00:16:27.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.811+0000 7f6f08ea6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T00:16:27.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.811+0000 7f6f08ea6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T00:16:27.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.811+0000 7f6f08ea6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-12T00:16:27.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:27.812+0000 7f6f08ea6640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-10-12T00:16:29.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-10-12T00:16:29.927 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:30.023 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-10-12T00:16:30.023 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-10-12T00:16:30.024 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T00:16:30.024 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T00:16:30.024 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-12T00:16:30.024 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:30.022+0000 7efe2a9c6640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-10-12T00:16:32.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-10-12T00:16:32.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:32.231 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.229+0000 7f3f892bf640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-10-12T00:16:32.231 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.229+0000 7f3f892bf640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-10-12T00:16:32.231 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.229+0000 7f3f892bf640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T00:16:32.231 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.229+0000 7f3f892bf640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T00:16:32.231 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.229+0000 7f3f892bf640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-12T00:16:32.232 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:16:32.230+0000 7f3f892bf640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-10-12T00:16:34.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-10-12T00:16:34.351 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-12T00:16:34.351 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-10-12T00:16:34.424 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-10-12T00:16:34.497 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-10-12T00:16:34.569 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-10-12T00:16:34.640 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.641 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-10-12T00:16:34.701 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:34.702 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-10-12T00:16:34.762 INFO:tasks.ceph:Adding keys to all mons... 2023-10-12T00:16:34.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/c2.keyring 2023-10-12T00:16:34.832 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2023-10-12T00:16:34.832 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDaOidlL4cvKBAAve5fYj7DzEFao3sJptGEwg== 2023-10-12T00:16:34.833 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2023-10-12T00:16:34.833 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDbOidlufl1MBAAD12qGzghtzhKtC6W9oZEOA== 2023-10-12T00:16:34.833 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2023-10-12T00:16:34.833 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDeOidlvCdqARAAZVJyBwRs8qrIyMNpPLiYKQ== 2023-10-12T00:16:34.833 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2023-10-12T00:16:34.834 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDgOidlFybHDRAAJVXB3sbifXpujp9YrfdxUA== 2023-10-12T00:16:34.834 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2023-10-12T00:16:34.834 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDaOidlE276KhAACiRCMPrTK4EmsY2EKKDd1A== 2023-10-12T00:16:34.834 INFO:teuthology.orchestra.run.smithi118.stdout:[client.1] 2023-10-12T00:16:34.834 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDaOidlznNSLhAAAbf29nxWIATRtdgCwifI+g== 2023-10-12T00:16:34.835 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-12T00:16:34.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:34.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:35.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-12T00:16:35.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T00:16:35.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-12T00:16:35.210 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-12T00:16:35.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-10-12T00:16:35.238 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-10-12T00:16:35.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-10-12T00:16:35.365 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-10-12T00:16:35.423 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-10-12T00:16:35.424 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-10-12T00:16:35.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-10-12T00:16:35.466 INFO:tasks.ceph.c2.mon.a:Started 2023-10-12T00:16:35.466 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-10-12T00:16:35.467 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-10-12T00:16:35.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-10-12T00:16:35.469 INFO:tasks.ceph.c2.mgr.x:Started 2023-10-12T00:16:35.469 INFO:tasks.ceph:Setting crush tunables to default 2023-10-12T00:16:35.470 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd crush tunables default 2023-10-12T00:16:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2023-10-12T00:16:35.687 INFO:tasks.ceph:check_enable_crimson: False 2023-10-12T00:16:35.688 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-10-12T00:16:35.688 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:35.688 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-10-12T00:16:35.715 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:35.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-10-12T00:16:35.782 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-12T00:16:35.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-10-12T00:16:35.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new 58018925-196b-4b31-b4f1-5b48d39fc915 0 2023-10-12T00:16:35.881 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:35.880+0000 7f15c9156200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-12T00:16:36.070 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:36.067+0000 7f15c9156200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-12T00:16:36.123 INFO:teuthology.orchestra.run.smithi118.stdout:0 2023-10-12T00:16:36.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new 3f22d63b-b712-43a2-8dac-9a6be7475717 1 2023-10-12T00:16:36.350 INFO:teuthology.orchestra.run.smithi118.stdout:1 2023-10-12T00:16:36.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new 88e36e20-d3dc-49d0-b65a-92de728dc172 2 2023-10-12T00:16:36.425 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:36.425+0000 7f15c9156200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-12T00:16:36.578 INFO:teuthology.orchestra.run.smithi118.stdout:2 2023-10-12T00:16:36.591 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-10-12T00:16:36.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-10-12T00:16:36.594 INFO:tasks.ceph.c2.osd.0:Started 2023-10-12T00:16:36.595 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-10-12T00:16:36.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-10-12T00:16:36.598 INFO:tasks.ceph.c2.osd.1:Started 2023-10-12T00:16:36.598 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-10-12T00:16:36.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-10-12T00:16:36.601 INFO:tasks.ceph.c2.osd.2:Started 2023-10-12T00:16:36.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T00:16:36.828 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:36.829 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"fsid":"eb47bf95-bcad-4e64-bdca-484ec2af758b","created":"2023-10-12T00:16:35.584570+0000","modified":"2023-10-12T00:16:36.577172+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T00:16:36.577172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"58018925-196b-4b31-b4f1-5b48d39fc915","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3f22d63b-b712-43a2-8dac-9a6be7475717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88e36e20-d3dc-49d0-b65a-92de728dc172","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:36.841 INFO:tasks.ceph.ceph_manager.c2:[] 2023-10-12T00:16:36.841 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-12T00:16:36.985 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:36.984+0000 7f15c9156200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-12T00:16:37.144 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-12T00:16:37.180 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:37.179+0000 7f15c9156200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-12T00:16:37.201 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2023-10-12T00:16:37.201+0000 7fa0aa37d640 -1 Falling back to public interface 2023-10-12T00:16:37.215 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2023-10-12T00:16:37.214+0000 7fc161bbc640 -1 Falling back to public interface 2023-10-12T00:16:37.224 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2023-10-12T00:16:37.223+0000 7f32aa0fb640 -1 Falling back to public interface 2023-10-12T00:16:37.340 INFO:teuthology.misc.health.smithi118.stdout: 2023-10-12T00:16:37.340 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":5,"fsid":"eb47bf95-bcad-4e64-bdca-484ec2af758b","created":"2023-10-12T00:16:35.584570+0000","modified":"2023-10-12T00:16:36.577172+0000","last_up_change":"0.000000","last_in_change":"2023-10-12T00:16:36.577172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"58018925-196b-4b31-b4f1-5b48d39fc915","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3f22d63b-b712-43a2-8dac-9a6be7475717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88e36e20-d3dc-49d0-b65a-92de728dc172","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:37.351 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-12T00:16:37.572 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:37.572+0000 7f15c9156200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-12T00:16:38.236 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.235+0000 7f15c9156200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-12T00:16:38.310 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.310+0000 7f15c9156200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-12T00:16:38.446 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.445+0000 7f15c9156200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-12T00:16:38.517 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.516+0000 7f15c9156200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-12T00:16:38.713 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.712+0000 7f15c9156200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-12T00:16:38.792 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.791+0000 7f15c9156200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-12T00:16:38.869 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.868+0000 7f15c9156200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-12T00:16:38.972 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:38.971+0000 7f15c9156200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-12T00:16:39.155 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:39.156+0000 7f15c9156200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-12T00:16:39.237 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:39.236+0000 7f15c9156200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-12T00:16:39.544 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:39.543+0000 7f15c9156200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-12T00:16:39.614 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:39.614+0000 7f15c9156200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-12T00:16:39.700 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2023-10-12T00:16:39.699+0000 7fa0aa37d640 -1 osd.0 0 log_to_monitors true 2023-10-12T00:16:39.940 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2023-10-12T00:16:39.939+0000 7fc161bbc640 -1 osd.2 0 log_to_monitors true 2023-10-12T00:16:39.972 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2023-10-12T00:16:39.971+0000 7f32aa0fb640 -1 osd.1 0 log_to_monitors true 2023-10-12T00:16:40.147 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:40.147+0000 7f15c9156200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-12T00:16:40.463 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:40.463+0000 7f15c9156200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-12T00:16:40.606 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:40.607+0000 7f15c9156200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-12T00:16:41.034 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:41.034+0000 7f15c9156200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-12T00:16:41.094 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2023-10-12T00:16:41.094+0000 7fa09e71e700 -1 osd.0 0 waiting for initial osdmap 2023-10-12T00:16:41.106 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2023-10-12T00:16:41.107+0000 7fa0962f8700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:41.191 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:41.191+0000 7f15c9156200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-12T00:16:41.342 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:41.343+0000 7f15c9156200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-12T00:16:41.646 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:41.647+0000 7f15c9156200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-12T00:16:42.303 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2023-10-12T00:16:42.303+0000 7f329e49c700 -1 osd.1 0 waiting for initial osdmap 2023-10-12T00:16:42.311 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2023-10-12T00:16:42.312+0000 7f3296076700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:42.313 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2023-10-12T00:16:42.314+0000 7fc155f5d700 -1 osd.2 0 waiting for initial osdmap 2023-10-12T00:16:42.323 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2023-10-12T00:16:42.323+0000 7fc14db37700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-12T00:16:42.667 INFO:tasks.ceph.c2.mgr.x.smithi118.stderr:2023-10-12T00:16:42.667+0000 7f158a56f700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-12T00:16:43.655 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-12T00:16:43.988 INFO:teuthology.misc.health.smithi118.stdout: 2023-10-12T00:16:43.989 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":9,"fsid":"eb47bf95-bcad-4e64-bdca-484ec2af758b","created":"2023-10-12T00:16:35.584570+0000","modified":"2023-10-12T00:16:43.315297+0000","last_up_change":"2023-10-12T00:16:43.315297+0000","last_in_change":"2023-10-12T00:16:36.577172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"58018925-196b-4b31-b4f1-5b48d39fc915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":114327},{"type":"v1","addr":"172.21.15.118:6801","nonce":114327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":114327},{"type":"v1","addr":"172.21.15.118:6803","nonce":114327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":114327},{"type":"v1","addr":"172.21.15.118:6807","nonce":114327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":114327},{"type":"v1","addr":"172.21.15.118:6805","nonce":114327}]},"public_addr":"172.21.15.118:6801/114327","cluster_addr":"172.21.15.118:6803/114327","heartbeat_back_addr":"172.21.15.118:6807/114327","heartbeat_front_addr":"172.21.15.118:6805/114327","state":["exists","up"]},{"osd":1,"uuid":"3f22d63b-b712-43a2-8dac-9a6be7475717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":114339},{"type":"v1","addr":"172.21.15.118:6817","nonce":114339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":114339},{"type":"v1","addr":"172.21.15.118:6819","nonce":114339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":114339},{"type":"v1","addr":"172.21.15.118:6823","nonce":114339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":114339},{"type":"v1","addr":"172.21.15.118:6821","nonce":114339}]},"public_addr":"172.21.15.118:6817/114339","cluster_addr":"172.21.15.118:6819/114339","heartbeat_back_addr":"172.21.15.118:6823/114339","heartbeat_front_addr":"172.21.15.118:6821/114339","state":["exists","up"]},{"osd":2,"uuid":"88e36e20-d3dc-49d0-b65a-92de728dc172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":114328},{"type":"v1","addr":"172.21.15.118:6809","nonce":114328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":114328},{"type":"v1","addr":"172.21.15.118:6811","nonce":114328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":114328},{"type":"v1","addr":"172.21.15.118:6815","nonce":114328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":114328},{"type":"v1","addr":"172.21.15.118:6813","nonce":114328}]},"public_addr":"172.21.15.118:6809/114328","cluster_addr":"172.21.15.118:6811/114328","heartbeat_back_addr":"172.21.15.118:6815/114328","heartbeat_front_addr":"172.21.15.118:6813/114328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.728766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:44.001 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-12T00:16:44.002 INFO:tasks.ceph:Creating RBD pool 2023-10-12T00:16:44.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd pool create rbd 8 2023-10-12T00:16:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'rbd' created 2023-10-12T00:16:45.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-12T00:16:46.329 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-12T00:16:46.341 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-10-12T00:16:46.342 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-10-12T00:16:46.342 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-10-12T00:16:46.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-10-12T00:16:46.343 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-12T00:16:46.719 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:46.732 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":113915},{"type":"v1","addr":"172.21.15.118:6825","nonce":113915}]},"active_addr":"172.21.15.118:6825/113915","active_change":"2023-10-12T00:16:41.660518+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":168804626}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3472608738}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1146433360}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3414891465}]}]} 2023-10-12T00:16:46.736 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-10-12T00:16:46.736 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-10-12T00:16:46.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T00:16:47.053 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:47.054 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":12,"fsid":"eb47bf95-bcad-4e64-bdca-484ec2af758b","created":"2023-10-12T00:16:35.584570+0000","modified":"2023-10-12T00:16:46.319965+0000","last_up_change":"2023-10-12T00:16:43.315297+0000","last_in_change":"2023-10-12T00:16:36.577172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-12T00:16:43.705146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T00:16:44.340930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"58018925-196b-4b31-b4f1-5b48d39fc915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":114327},{"type":"v1","addr":"172.21.15.118:6801","nonce":114327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":114327},{"type":"v1","addr":"172.21.15.118:6803","nonce":114327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":114327},{"type":"v1","addr":"172.21.15.118:6807","nonce":114327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":114327},{"type":"v1","addr":"172.21.15.118:6805","nonce":114327}]},"public_addr":"172.21.15.118:6801/114327","cluster_addr":"172.21.15.118:6803/114327","heartbeat_back_addr":"172.21.15.118:6807/114327","heartbeat_front_addr":"172.21.15.118:6805/114327","state":["exists","up"]},{"osd":1,"uuid":"3f22d63b-b712-43a2-8dac-9a6be7475717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":114339},{"type":"v1","addr":"172.21.15.118:6817","nonce":114339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":114339},{"type":"v1","addr":"172.21.15.118:6819","nonce":114339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":114339},{"type":"v1","addr":"172.21.15.118:6823","nonce":114339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":114339},{"type":"v1","addr":"172.21.15.118:6821","nonce":114339}]},"public_addr":"172.21.15.118:6817/114339","cluster_addr":"172.21.15.118:6819/114339","heartbeat_back_addr":"172.21.15.118:6823/114339","heartbeat_front_addr":"172.21.15.118:6821/114339","state":["exists","up"]},{"osd":2,"uuid":"88e36e20-d3dc-49d0-b65a-92de728dc172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":114328},{"type":"v1","addr":"172.21.15.118:6809","nonce":114328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":114328},{"type":"v1","addr":"172.21.15.118:6811","nonce":114328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":114328},{"type":"v1","addr":"172.21.15.118:6815","nonce":114328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":114328},{"type":"v1","addr":"172.21.15.118:6813","nonce":114328}]},"public_addr":"172.21.15.118:6809/114328","cluster_addr":"172.21.15.118:6811/114328","heartbeat_back_addr":"172.21.15.118:6815/114328","heartbeat_front_addr":"172.21.15.118:6813/114328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.728766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.985562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.899453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:47.067 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-10-12T00:16:47.067 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-12T00:16:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:47.388 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":13,"fsid":"eb47bf95-bcad-4e64-bdca-484ec2af758b","created":"2023-10-12T00:16:35.584570+0000","modified":"2023-10-12T00:16:47.323324+0000","last_up_change":"2023-10-12T00:16:43.315297+0000","last_in_change":"2023-10-12T00:16:36.577172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-12T00:16:43.705146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-12T00:16:44.340930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"58018925-196b-4b31-b4f1-5b48d39fc915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":114327},{"type":"v1","addr":"172.21.15.118:6801","nonce":114327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":114327},{"type":"v1","addr":"172.21.15.118:6803","nonce":114327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":114327},{"type":"v1","addr":"172.21.15.118:6807","nonce":114327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":114327},{"type":"v1","addr":"172.21.15.118:6805","nonce":114327}]},"public_addr":"172.21.15.118:6801/114327","cluster_addr":"172.21.15.118:6803/114327","heartbeat_back_addr":"172.21.15.118:6807/114327","heartbeat_front_addr":"172.21.15.118:6805/114327","state":["exists","up"]},{"osd":1,"uuid":"3f22d63b-b712-43a2-8dac-9a6be7475717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":114339},{"type":"v1","addr":"172.21.15.118:6817","nonce":114339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":114339},{"type":"v1","addr":"172.21.15.118:6819","nonce":114339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":114339},{"type":"v1","addr":"172.21.15.118:6823","nonce":114339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":114339},{"type":"v1","addr":"172.21.15.118:6821","nonce":114339}]},"public_addr":"172.21.15.118:6817/114339","cluster_addr":"172.21.15.118:6819/114339","heartbeat_back_addr":"172.21.15.118:6823/114339","heartbeat_front_addr":"172.21.15.118:6821/114339","state":["exists","up"]},{"osd":2,"uuid":"88e36e20-d3dc-49d0-b65a-92de728dc172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":114328},{"type":"v1","addr":"172.21.15.118:6809","nonce":114328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":114328},{"type":"v1","addr":"172.21.15.118:6811","nonce":114328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":114328},{"type":"v1","addr":"172.21.15.118:6815","nonce":114328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":114328},{"type":"v1","addr":"172.21.15.118:6813","nonce":114328}]},"public_addr":"172.21.15.118:6809/114328","cluster_addr":"172.21.15.118:6811/114328","heartbeat_back_addr":"172.21.15.118:6815/114328","heartbeat_front_addr":"172.21.15.118:6813/114328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.728766+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.985562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-12T00:16:40.899453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-12T00:16:47.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-10-12T00:16:47.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-10-12T00:16:47.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-10-12T00:16:47.565 INFO:teuthology.orchestra.run.smithi118.stdout:38654705667 2023-10-12T00:16:47.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-12T00:16:47.584 INFO:teuthology.orchestra.run.smithi118.stdout:38654705666 2023-10-12T00:16:47.584 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-12T00:16:47.585 INFO:teuthology.orchestra.run.smithi118.stdout:34359738370 2023-10-12T00:16:47.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-12T00:16:47.924 INFO:teuthology.orchestra.run.smithi118.stdout:38654705666 2023-10-12T00:16:47.938 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.2 2023-10-12T00:16:47.963 INFO:teuthology.orchestra.run.smithi118.stdout:34359738369 2023-10-12T00:16:47.977 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2023-10-12T00:16:47.978 INFO:teuthology.orchestra.run.smithi118.stdout:38654705665 2023-10-12T00:16:47.992 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.1 2023-10-12T00:16:48.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-12T00:16:48.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-12T00:16:48.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-12T00:16:49.322 INFO:teuthology.orchestra.run.smithi118.stdout:38654705667 2023-10-12T00:16:49.337 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.2 2023-10-12T00:16:49.337 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:49.338 INFO:teuthology.orchestra.run.smithi118.stdout:34359738370 2023-10-12T00:16:49.352 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2023-10-12T00:16:49.353 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:49.387 INFO:teuthology.orchestra.run.smithi118.stdout:38654705666 2023-10-12T00:16:49.401 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.1 2023-10-12T00:16:49.401 DEBUG:teuthology.parallel:result is None 2023-10-12T00:16:49.401 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-10-12T00:16:49.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-12T00:16:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:49.761 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2023-10-12T00:16:49.772 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-12T00:16:49.665548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82592,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490208,"statfs":{"total":322122547200,"available":322037972992,"internally_reserved":0,"allocated":1515520,"data_stored":1268457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.325021+0000","last_change":"2023-10-12T00:16:46.329135+0000","last_active":"2023-10-12T00:16:47.325021+0000","last_peered":"2023-10-12T00:16:47.325021+0000","last_clean":"2023-10-12T00:16:47.325021+0000","last_became_active":"2023-10-12T00:16:46.328861+0000","last_became_peered":"2023-10-12T00:16:46.328861+0000","last_unstale":"2023-10-12T00:16:47.325021+0000","last_undegraded":"2023-10-12T00:16:47.325021+0000","last_fullsized":"2023-10-12T00:16:47.325021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:49:47.944161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.325031+0000","last_change":"2023-10-12T00:16:46.329078+0000","last_active":"2023-10-12T00:16:47.325031+0000","last_peered":"2023-10-12T00:16:47.325031+0000","last_clean":"2023-10-12T00:16:47.325031+0000","last_became_active":"2023-10-12T00:16:46.328751+0000","last_became_peered":"2023-10-12T00:16:46.328751+0000","last_unstale":"2023-10-12T00:16:47.325031+0000","last_undegraded":"2023-10-12T00:16:47.325031+0000","last_fullsized":"2023-10-12T00:16:47.325031+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:36:17.239548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324984+0000","last_change":"2023-10-12T00:16:46.329418+0000","last_active":"2023-10-12T00:16:47.324984+0000","last_peered":"2023-10-12T00:16:47.324984+0000","last_clean":"2023-10-12T00:16:47.324984+0000","last_became_active":"2023-10-12T00:16:46.328978+0000","last_became_peered":"2023-10-12T00:16:46.328978+0000","last_unstale":"2023-10-12T00:16:47.324984+0000","last_undegraded":"2023-10-12T00:16:47.324984+0000","last_fullsized":"2023-10-12T00:16:47.324984+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:38:46.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.554578+0000","last_change":"2023-10-12T00:16:46.326148+0000","last_active":"2023-10-12T00:16:47.554578+0000","last_peered":"2023-10-12T00:16:47.554578+0000","last_clean":"2023-10-12T00:16:47.554578+0000","last_became_active":"2023-10-12T00:16:46.325624+0000","last_became_peered":"2023-10-12T00:16:46.325624+0000","last_unstale":"2023-10-12T00:16:47.554578+0000","last_undegraded":"2023-10-12T00:16:47.554578+0000","last_fullsized":"2023-10-12T00:16:47.554578+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T09:20:20.956429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324916+0000","last_change":"2023-10-12T00:16:45.327285+0000","last_active":"2023-10-12T00:16:47.324916+0000","last_peered":"2023-10-12T00:16:47.324916+0000","last_clean":"2023-10-12T00:16:47.324916+0000","last_became_active":"2023-10-12T00:16:45.326973+0000","last_became_peered":"2023-10-12T00:16:45.326973+0000","last_unstale":"2023-10-12T00:16:47.324916+0000","last_undegraded":"2023-10-12T00:16:47.324916+0000","last_fullsized":"2023-10-12T00:16:47.324916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:44.316446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:44.316446+0000","last_clean_scrub_stamp":"2023-10-12T00:16:44.316446+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T11:02:57.717702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324929+0000","last_change":"2023-10-12T00:16:46.325742+0000","last_active":"2023-10-12T00:16:47.324929+0000","last_peered":"2023-10-12T00:16:47.324929+0000","last_clean":"2023-10-12T00:16:47.324929+0000","last_became_active":"2023-10-12T00:16:46.325477+0000","last_became_peered":"2023-10-12T00:16:46.325477+0000","last_unstale":"2023-10-12T00:16:47.324929+0000","last_undegraded":"2023-10-12T00:16:47.324929+0000","last_fullsized":"2023-10-12T00:16:47.324929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:56:45.026249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.554600+0000","last_change":"2023-10-12T00:16:46.326001+0000","last_active":"2023-10-12T00:16:47.554600+0000","last_peered":"2023-10-12T00:16:47.554600+0000","last_clean":"2023-10-12T00:16:47.554600+0000","last_became_active":"2023-10-12T00:16:46.325764+0000","last_became_peered":"2023-10-12T00:16:46.325764+0000","last_unstale":"2023-10-12T00:16:47.554600+0000","last_undegraded":"2023-10-12T00:16:47.554600+0000","last_fullsized":"2023-10-12T00:16:47.554600+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T07:49:52.552215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:46.329675+0000","last_change":"2023-10-12T00:16:46.328907+0000","last_active":"2023-10-12T00:16:46.329675+0000","last_peered":"2023-10-12T00:16:46.329675+0000","last_clean":"2023-10-12T00:16:46.329675+0000","last_became_active":"2023-10-12T00:16:46.328646+0000","last_became_peered":"2023-10-12T00:16:46.328646+0000","last_unstale":"2023-10-12T00:16:46.329675+0000","last_undegraded":"2023-10-12T00:16:46.329675+0000","last_fullsized":"2023-10-12T00:16:46.329675+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:42:56.619982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324991+0000","last_change":"2023-10-12T00:16:46.328983+0000","last_active":"2023-10-12T00:16:47.324991+0000","last_peered":"2023-10-12T00:16:47.324991+0000","last_clean":"2023-10-12T00:16:47.324991+0000","last_became_active":"2023-10-12T00:16:46.328772+0000","last_became_peered":"2023-10-12T00:16:46.328772+0000","last_unstale":"2023-10-12T00:16:47.324991+0000","last_undegraded":"2023-10-12T00:16:47.324991+0000","last_fullsized":"2023-10-12T00:16:47.324991+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:45:38.612543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27764,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829836,"statfs":{"total":107374182400,"available":107345752064,"internally_reserved":0,"allocated":700416,"data_stored":618449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27772,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":708608,"data_stored":621927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T00:16:49.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-12T00:16:50.087 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:50.088 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2023-10-12T00:16:50.100 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-12T00:16:49.665548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82592,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490208,"statfs":{"total":322122547200,"available":322037972992,"internally_reserved":0,"allocated":1515520,"data_stored":1268457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.325021+0000","last_change":"2023-10-12T00:16:46.329135+0000","last_active":"2023-10-12T00:16:47.325021+0000","last_peered":"2023-10-12T00:16:47.325021+0000","last_clean":"2023-10-12T00:16:47.325021+0000","last_became_active":"2023-10-12T00:16:46.328861+0000","last_became_peered":"2023-10-12T00:16:46.328861+0000","last_unstale":"2023-10-12T00:16:47.325021+0000","last_undegraded":"2023-10-12T00:16:47.325021+0000","last_fullsized":"2023-10-12T00:16:47.325021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:49:47.944161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.325031+0000","last_change":"2023-10-12T00:16:46.329078+0000","last_active":"2023-10-12T00:16:47.325031+0000","last_peered":"2023-10-12T00:16:47.325031+0000","last_clean":"2023-10-12T00:16:47.325031+0000","last_became_active":"2023-10-12T00:16:46.328751+0000","last_became_peered":"2023-10-12T00:16:46.328751+0000","last_unstale":"2023-10-12T00:16:47.325031+0000","last_undegraded":"2023-10-12T00:16:47.325031+0000","last_fullsized":"2023-10-12T00:16:47.325031+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:36:17.239548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324984+0000","last_change":"2023-10-12T00:16:46.329418+0000","last_active":"2023-10-12T00:16:47.324984+0000","last_peered":"2023-10-12T00:16:47.324984+0000","last_clean":"2023-10-12T00:16:47.324984+0000","last_became_active":"2023-10-12T00:16:46.328978+0000","last_became_peered":"2023-10-12T00:16:46.328978+0000","last_unstale":"2023-10-12T00:16:47.324984+0000","last_undegraded":"2023-10-12T00:16:47.324984+0000","last_fullsized":"2023-10-12T00:16:47.324984+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:38:46.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.554578+0000","last_change":"2023-10-12T00:16:46.326148+0000","last_active":"2023-10-12T00:16:47.554578+0000","last_peered":"2023-10-12T00:16:47.554578+0000","last_clean":"2023-10-12T00:16:47.554578+0000","last_became_active":"2023-10-12T00:16:46.325624+0000","last_became_peered":"2023-10-12T00:16:46.325624+0000","last_unstale":"2023-10-12T00:16:47.554578+0000","last_undegraded":"2023-10-12T00:16:47.554578+0000","last_fullsized":"2023-10-12T00:16:47.554578+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T09:20:20.956429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324916+0000","last_change":"2023-10-12T00:16:45.327285+0000","last_active":"2023-10-12T00:16:47.324916+0000","last_peered":"2023-10-12T00:16:47.324916+0000","last_clean":"2023-10-12T00:16:47.324916+0000","last_became_active":"2023-10-12T00:16:45.326973+0000","last_became_peered":"2023-10-12T00:16:45.326973+0000","last_unstale":"2023-10-12T00:16:47.324916+0000","last_undegraded":"2023-10-12T00:16:47.324916+0000","last_fullsized":"2023-10-12T00:16:47.324916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:44.316446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:44.316446+0000","last_clean_scrub_stamp":"2023-10-12T00:16:44.316446+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T11:02:57.717702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324929+0000","last_change":"2023-10-12T00:16:46.325742+0000","last_active":"2023-10-12T00:16:47.324929+0000","last_peered":"2023-10-12T00:16:47.324929+0000","last_clean":"2023-10-12T00:16:47.324929+0000","last_became_active":"2023-10-12T00:16:46.325477+0000","last_became_peered":"2023-10-12T00:16:46.325477+0000","last_unstale":"2023-10-12T00:16:47.324929+0000","last_undegraded":"2023-10-12T00:16:47.324929+0000","last_fullsized":"2023-10-12T00:16:47.324929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:56:45.026249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.554600+0000","last_change":"2023-10-12T00:16:46.326001+0000","last_active":"2023-10-12T00:16:47.554600+0000","last_peered":"2023-10-12T00:16:47.554600+0000","last_clean":"2023-10-12T00:16:47.554600+0000","last_became_active":"2023-10-12T00:16:46.325764+0000","last_became_peered":"2023-10-12T00:16:46.325764+0000","last_unstale":"2023-10-12T00:16:47.554600+0000","last_undegraded":"2023-10-12T00:16:47.554600+0000","last_fullsized":"2023-10-12T00:16:47.554600+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T07:49:52.552215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-12T00:16:46.329675+0000","last_change":"2023-10-12T00:16:46.328907+0000","last_active":"2023-10-12T00:16:46.329675+0000","last_peered":"2023-10-12T00:16:46.329675+0000","last_clean":"2023-10-12T00:16:46.329675+0000","last_became_active":"2023-10-12T00:16:46.328646+0000","last_became_peered":"2023-10-12T00:16:46.328646+0000","last_unstale":"2023-10-12T00:16:46.329675+0000","last_undegraded":"2023-10-12T00:16:46.329675+0000","last_fullsized":"2023-10-12T00:16:46.329675+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T06:42:56.619982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-12T00:16:47.324991+0000","last_change":"2023-10-12T00:16:46.328983+0000","last_active":"2023-10-12T00:16:47.324991+0000","last_peered":"2023-10-12T00:16:47.324991+0000","last_clean":"2023-10-12T00:16:47.324991+0000","last_became_active":"2023-10-12T00:16:46.328772+0000","last_became_peered":"2023-10-12T00:16:46.328772+0000","last_unstale":"2023-10-12T00:16:47.324991+0000","last_undegraded":"2023-10-12T00:16:47.324991+0000","last_fullsized":"2023-10-12T00:16:47.324991+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-12T00:16:45.319181+0000","last_clean_scrub_stamp":"2023-10-12T00:16:45.319181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T08:45:38.612543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27764,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829836,"statfs":{"total":107374182400,"available":107345752064,"internally_reserved":0,"allocated":700416,"data_stored":618449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27772,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":708608,"data_stored":621927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-12T00:16:50.101 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-10-12T00:16:50.101 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-10-12T00:16:50.101 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-10-12T00:16:50.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-10-12T00:16:50.462 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-12T00:16:50.462 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-12T00:16:50.476 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-10-12T00:16:50.477 INFO:teuthology.run_tasks:Running task rgw... 2023-10-12T00:16:50.492 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-10-12T00:16:50.492 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-10-12T00:16:50.493 INFO:tasks.rgw:Creating data pools 2023-10-12T00:16:50.493 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-10-12T00:16:50.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-12T00:16:51.081 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' created 2023-10-12T00:16:51.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-12T00:16:52.088 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T00:16:52.107 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-12T00:16:53.089 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' created 2023-10-12T00:16:53.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-12T00:16:54.098 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T00:16:54.118 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-10-12T00:16:54.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-12T00:16:54.446 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-12T00:16:54.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-12T00:16:55.093 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T00:16:55.104 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-12T00:16:55.427 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-12T00:16:55.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-12T00:16:56.096 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T00:16:56.108 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-10-12T00:16:56.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-12T00:16:56.573 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.data' created 2023-10-12T00:16:56.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-12T00:16:57.577 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T00:16:57.598 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-12T00:16:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.index' created 2023-10-12T00:16:58.592 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-12T00:16:59.583 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T00:16:59.604 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-10-12T00:16:59.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-12T00:16:59.931 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-12T00:16:59.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-12T00:17:00.576 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-12T00:17:00.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-12T00:17:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-12T00:17:00.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-12T00:17:01.577 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-12T00:17:01.589 DEBUG:tasks.rgw:Pools created 2023-10-12T00:17:01.589 INFO:tasks.rgw:Configuring compression type = random 2023-10-12T00:17:01.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-10-12T00:17:01.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-10-12T00:17:01.591 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-10-12T00:17:01.707 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:01.708 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:01.717+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.087+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.087+0000 7f5d4b078780 20 realm 2023-10-12T00:17:03.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.087+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.089+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.089+0000 7f5d4b078780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-12T00:17:03.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.089+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.091+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.092+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.092+0000 7f5d4b078780 10 Creating default zonegroup 2023-10-12T00:17:03.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.092+0000 7f5d4b078780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-12T00:17:03.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.102+0000 7f5d4b078780 10 WARNING: driver->list_zones() returned r=-2 2023-10-12T00:17:03.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.102+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.102+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.106+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.107+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.107+0000 7f5d4b078780 10 could not read realm id: (2) No such file or directory 2023-10-12T00:17:03.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.107+0000 7f5d4b078780 10 WARNING: failed to set zone as default, r=-22 2023-10-12T00:17:03.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.107+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.107+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.110+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.111+0000 7f5d4b078780 20 rados_obj.operate() r=0 bl.length=874 2023-10-12T00:17:03.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.111+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.112+0000 7f5d4b078780 20 rados_obj.operate() r=0 bl.length=420 2023-10-12T00:17:03.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.112+0000 7f5d4b078780 10 Using default name default 2023-10-12T00:17:03.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.112+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.112+0000 7f5d4b078780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:03.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.112+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 rados_obj.operate() r=0 bl.length=874 2023-10-12T00:17:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 zone default found 2023-10-12T00:17:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 4 Realm: () 2023-10-12T00:17:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 4 ZoneGroup: default (3421d9bc-2a1f-4fb5-b8f5-a5cced3c6604) 2023-10-12T00:17:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 4 Zone: default (0386b4fd-406d-43fe-9775-337ed9f153ee) 2023-10-12T00:17:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-12T00:17:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 zonegroup default 2023-10-12T00:17:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:03.113+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:05.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:05.106+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:05.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:05.106+0000 7f5d4b078780 20 rados->read ofs=0 len=0 2023-10-12T00:17:05.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:05.107+0000 7f5d4b078780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:05.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:05.107+0000 7f5d4b078780 20 started sync module instance, tier type = 2023-10-12T00:17:05.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:05.107+0000 7f5d4b078780 20 started zone id=0386b4fd-406d-43fe-9775-337ed9f153ee (name=default) with tier type = 2023-10-12T00:17:07.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=0 2023-10-12T00:17:07.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=1 2023-10-12T00:17:07.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=2 2023-10-12T00:17:07.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=3 2023-10-12T00:17:07.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=4 2023-10-12T00:17:07.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=5 2023-10-12T00:17:07.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.133+0000 7f5d4b078780 20 add_watcher() i=6 2023-10-12T00:17:07.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.134+0000 7f5d4b078780 20 add_watcher() i=7 2023-10-12T00:17:07.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.134+0000 7f5d4b078780 2 all 8 watchers are set, enabling cache 2023-10-12T00:17:07.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.134+0000 7f5d4b078780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-12T00:17:07.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.134+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-12T00:17:07.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.134+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:07.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-12T00:17:07.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-12T00:17:07.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:07.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-12T00:17:07.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-12T00:17:07.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:07.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-12T00:17:07.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-12T00:17:07.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:07.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-12T00:17:07.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.135+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-12T00:17:07.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.136+0000 7f5d4b078780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:07.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.139+0000 7f5d2fff4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:07.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.139+0000 7f5d4b078780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:07.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:07.139+0000 7f5d4b078780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:10.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.108+0000 7f5d4b078780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-12T00:17:10.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.109+0000 7f5d4b078780 5 note: GC not initialized 2023-10-12T00:17:10.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.110+0000 7f5d2b7eb700 20 reqs_thread_entry: start 2023-10-12T00:17:10.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.110+0000 7f5d4b078780 20 init_complete bucket index max shards: 11 2023-10-12T00:17:10.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.110+0000 7f5d4b078780 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T00:17:10.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.110+0000 7f5d297e7700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 00:17:40 2023 2023-10-12T00:17:10.110 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-10-12T00:17:10.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.110+0000 7f5d28fc5700 20 reqs_thread_entry: start 2023-10-12T00:17:10.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.130+0000 7f5d2fff4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:10.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.132+0000 7f5d4b078780 20 remove_watcher() i=0 2023-10-12T00:17:10.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.132+0000 7f5d4b078780 2 removed watcher, disabling cache 2023-10-12T00:17:10.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.133+0000 7f5d4b078780 20 remove_watcher() i=1 2023-10-12T00:17:10.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.135+0000 7f5d4b078780 20 remove_watcher() i=2 2023-10-12T00:17:10.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.138+0000 7f5d4b078780 20 remove_watcher() i=3 2023-10-12T00:17:10.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.139+0000 7f5d4b078780 20 remove_watcher() i=4 2023-10-12T00:17:10.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.141+0000 7f5d4b078780 20 remove_watcher() i=5 2023-10-12T00:17:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.144+0000 7f5d4b078780 20 remove_watcher() i=6 2023-10-12T00:17:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.145+0000 7f5d4b078780 20 remove_watcher() i=7 2023-10-12T00:17:10.147 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:17:10.149 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T00:17:10.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:10.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:10.150 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T00:17:10.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:10.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:10.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.215+0000 7f20f3d2f780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:10.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.215+0000 7f20f3d2f780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:10.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.219+0000 7f20d54a4700 20 reqs_thread_entry: start 2023-10-12T00:17:10.224 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:10.224 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "0386b4fd-406d-43fe-9775-337ed9f153ee", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T00:17:10.225 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-10-12T00:17:10.226 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T00:17:10.226 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T00:17:10.226 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T00:17:10.226 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T00:17:10.227 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:10.228 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-10-12T00:17:10.228 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-10-12T00:17:10.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:10.228 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:10.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:10.229 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:10.229 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:10.229 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T00:17:10.229 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:10.230 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:10.230 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T00:17:10.230 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:17:10.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:10.231 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T00:17:10.231 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:10.231 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:10.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.232 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:10.232 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-10-12T00:17:10.232 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T00:17:10.232 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:10.233 DEBUG:tasks.util.rgw: json result: {'id': '0386b4fd-406d-43fe-9775-337ed9f153ee', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T00:17:10.233 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-10-12T00:17:10.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-10-12T00:17:10.234 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-10-12T00:17:10.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:10.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:10.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.287+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 20 realm 2023-10-12T00:17:10.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-12T00:17:10.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.288+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:10.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 rados_obj.operate() r=0 bl.length=890 2023-10-12T00:17:10.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.289+0000 7feae3521780 20 searching for the correct realm 2023-10-12T00:17:10.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 RGWRados::pool_iterate: got zone_info.0386b4fd-406d-43fe-9775-337ed9f153ee 2023-10-12T00:17:10.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 RGWRados::pool_iterate: got zonegroup_info.3421d9bc-2a1f-4fb5-b8f5-a5cced3c6604 2023-10-12T00:17:10.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-12T00:17:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-12T00:17:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.296+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:10.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados_obj.operate() r=0 bl.length=420 2023-10-12T00:17:10.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 zone default found 2023-10-12T00:17:10.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 4 Realm: () 2023-10-12T00:17:10.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 4 ZoneGroup: default (3421d9bc-2a1f-4fb5-b8f5-a5cced3c6604) 2023-10-12T00:17:10.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 4 Zone: default (0386b4fd-406d-43fe-9775-337ed9f153ee) 2023-10-12T00:17:10.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-12T00:17:10.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 zonegroup default 2023-10-12T00:17:10.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados->read ofs=0 len=0 2023-10-12T00:17:10.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:10.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 started sync module instance, tier type = 2023-10-12T00:17:10.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.297+0000 7feae3521780 20 started zone id=0386b4fd-406d-43fe-9775-337ed9f153ee (name=default) with tier type = 2023-10-12T00:17:10.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=0 2023-10-12T00:17:10.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=1 2023-10-12T00:17:10.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=2 2023-10-12T00:17:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=3 2023-10-12T00:17:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=4 2023-10-12T00:17:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=5 2023-10-12T00:17:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.323+0000 7feae3521780 20 add_watcher() i=6 2023-10-12T00:17:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.329+0000 7feae3521780 20 add_watcher() i=7 2023-10-12T00:17:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.329+0000 7feae3521780 2 all 8 watchers are set, enabling cache 2023-10-12T00:17:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.330+0000 7feac849d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.330+0000 7feae3521780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.330+0000 7feae3521780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.334+0000 7feae3521780 5 note: GC not initialized 2023-10-12T00:17:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.334+0000 7feac3c94700 20 reqs_thread_entry: start 2023-10-12T00:17:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.335+0000 7feae3521780 20 init_complete bucket index max shards: 11 2023-10-12T00:17:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.335+0000 7feae3521780 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T00:17:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.335+0000 7feac1c90700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 00:17:40 2023 2023-10-12T00:17:10.335 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-10-12T00:17:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.335+0000 7feac146e700 20 reqs_thread_entry: start 2023-10-12T00:17:10.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.342+0000 7feac849d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:10.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.346+0000 7feae3521780 20 remove_watcher() i=0 2023-10-12T00:17:10.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.346+0000 7feae3521780 2 removed watcher, disabling cache 2023-10-12T00:17:10.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.347+0000 7feae3521780 20 remove_watcher() i=1 2023-10-12T00:17:10.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.347+0000 7feae3521780 20 remove_watcher() i=2 2023-10-12T00:17:10.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.348+0000 7feae3521780 20 remove_watcher() i=3 2023-10-12T00:17:10.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.349+0000 7feae3521780 20 remove_watcher() i=4 2023-10-12T00:17:10.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.352+0000 7feae3521780 20 remove_watcher() i=5 2023-10-12T00:17:10.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.353+0000 7feae3521780 20 remove_watcher() i=6 2023-10-12T00:17:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.355+0000 7feae3521780 20 remove_watcher() i=7 2023-10-12T00:17:10.356 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:17:10.358 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T00:17:10.358 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:10.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:10.359 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T00:17:10.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:10.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:10.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.417+0000 7fab7b8f5780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:10.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.417+0000 7fab7b8f5780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:10.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:10.420+0000 7fab5d06a700 20 reqs_thread_entry: start 2023-10-12T00:17:10.425 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:10.425 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "0386b4fd-406d-43fe-9775-337ed9f153ee", 2023-10-12T00:17:10.426 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-10-12T00:17:10.426 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T00:17:10.426 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-10-12T00:17:10.426 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T00:17:10.426 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T00:17:10.427 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-10-12T00:17:10.427 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T00:17:10.427 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T00:17:10.427 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T00:17:10.427 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T00:17:10.428 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:10.429 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-10-12T00:17:10.429 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-10-12T00:17:10.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:10.429 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:10.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:10.430 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:10.430 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:10.430 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T00:17:10.430 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:10.430 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T00:17:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:17:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-10-12T00:17:10.433 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T00:17:10.433 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:10.434 DEBUG:tasks.util.rgw: json result: {'id': '0386b4fd-406d-43fe-9775-337ed9f153ee', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T00:17:10.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-10-12T00:17:10.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-10-12T00:17:10.435 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-10-12T00:17:10.548 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:10.549 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:10.560 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:10.559+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.576 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.574+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.577 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.574+0000 7fd9cd1ef780 20 realm 2023-10-12T00:17:12.577 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.577 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.577 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-12T00:17:12.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.575+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.576+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.576+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.577+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.577+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.577+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.577+0000 7fd9cd1ef780 10 Creating default zonegroup 2023-10-12T00:17:12.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.577+0000 7fd9cd1ef780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-12T00:17:12.594 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.593+0000 7fd9cd1ef780 10 WARNING: driver->list_zones() returned r=-2 2023-10-12T00:17:12.594 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.593+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.595 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.593+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 10 could not read realm id: (2) No such file or directory 2023-10-12T00:17:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 10 WARNING: failed to set zone as default, r=-22 2023-10-12T00:17:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.598 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.596+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.599 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.598+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.600 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.598+0000 7fd9cd1ef780 20 rados_obj.operate() r=0 bl.length=874 2023-10-12T00:17:12.600 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.598+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.600 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados_obj.operate() r=0 bl.length=420 2023-10-12T00:17:12.600 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 10 Using default name default 2023-10-12T00:17:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:12.601 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.602 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados_obj.operate() r=0 bl.length=874 2023-10-12T00:17:12.602 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 zone default found 2023-10-12T00:17:12.602 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 4 Realm: () 2023-10-12T00:17:12.603 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 4 ZoneGroup: default (6986cff7-c771-4b7d-af24-29b924610377) 2023-10-12T00:17:12.603 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 4 Zone: default (8669d6aa-a676-4c9b-955e-b4e2f447fb20) 2023-10-12T00:17:12.603 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-12T00:17:12.603 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 zonegroup default 2023-10-12T00:17:12.603 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.599+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:12.604 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.600+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:12.604 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:12.600+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:14.589 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:14.588+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:14.589 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:14.588+0000 7fd9cd1ef780 20 rados->read ofs=0 len=0 2023-10-12T00:17:14.589 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:14.588+0000 7fd9cd1ef780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:14.590 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:14.588+0000 7fd9cd1ef780 20 started sync module instance, tier type = 2023-10-12T00:17:14.590 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:14.588+0000 7fd9cd1ef780 20 started zone id=8669d6aa-a676-4c9b-955e-b4e2f447fb20 (name=default) with tier type = 2023-10-12T00:17:16.607 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=0 2023-10-12T00:17:16.607 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=1 2023-10-12T00:17:16.607 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=2 2023-10-12T00:17:16.607 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=3 2023-10-12T00:17:16.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=4 2023-10-12T00:17:16.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=5 2023-10-12T00:17:16.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.606+0000 7fd9cd1ef780 20 add_watcher() i=6 2023-10-12T00:17:16.609 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 add_watcher() i=7 2023-10-12T00:17:16.609 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 2 all 8 watchers are set, enabling cache 2023-10-12T00:17:16.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-12T00:17:16.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-12T00:17:16.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:16.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-12T00:17:16.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-12T00:17:16.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.607+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:16.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-12T00:17:16.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-12T00:17:16.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:16.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-12T00:17:16.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-12T00:17:16.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:16.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-12T00:17:16.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.608+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-12T00:17:16.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.609+0000 7fd9cd1ef780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-12T00:17:16.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.612+0000 7fd9b216b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:16.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.612+0000 7fd9cd1ef780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:16.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:16.612+0000 7fd9cd1ef780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:19.584 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.583+0000 7fd9cd1ef780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-12T00:17:19.584 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.583+0000 7fd9cd1ef780 5 note: GC not initialized 2023-10-12T00:17:19.585 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.585+0000 7fd9cd1ef780 20 init_complete bucket index max shards: 11 2023-10-12T00:17:19.587 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.585+0000 7fd9cd1ef780 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T00:17:19.587 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.585+0000 7fd9ad962700 20 reqs_thread_entry: start 2023-10-12T00:17:19.587 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.585+0000 7fd9ab95e700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 00:17:49 2023 2023-10-12T00:17:19.588 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-12T00:17:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.585+0000 7fd9ab15d700 20 reqs_thread_entry: start 2023-10-12T00:17:19.606 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.606+0000 7fd9b216b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:19.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.608+0000 7fd9cd1ef780 20 remove_watcher() i=0 2023-10-12T00:17:19.609 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.608+0000 7fd9cd1ef780 2 removed watcher, disabling cache 2023-10-12T00:17:19.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.609+0000 7fd9cd1ef780 20 remove_watcher() i=1 2023-10-12T00:17:19.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.610+0000 7fd9cd1ef780 20 remove_watcher() i=2 2023-10-12T00:17:19.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.611+0000 7fd9cd1ef780 20 remove_watcher() i=3 2023-10-12T00:17:19.613 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.612+0000 7fd9cd1ef780 20 remove_watcher() i=4 2023-10-12T00:17:19.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.613+0000 7fd9cd1ef780 20 remove_watcher() i=5 2023-10-12T00:17:19.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.614+0000 7fd9cd1ef780 20 remove_watcher() i=6 2023-10-12T00:17:19.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.615+0000 7fd9cd1ef780 20 remove_watcher() i=7 2023-10-12T00:17:19.618 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2023-10-12T00:17:19.620 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T00:17:19.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:19.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:19.621 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T00:17:19.676 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:19.677 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:19.686 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.685+0000 7f26f0d24780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:19.686 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.685+0000 7f26f0d24780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:19.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.689+0000 7f26d2499700 20 reqs_thread_entry: start 2023-10-12T00:17:19.696 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:19.697 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "8669d6aa-a676-4c9b-955e-b4e2f447fb20", 2023-10-12T00:17:19.697 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default", 2023-10-12T00:17:19.697 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T00:17:19.697 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "default.rgw.control", 2023-10-12T00:17:19.698 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T00:17:19.698 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T00:17:19.698 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "default.rgw.log", 2023-10-12T00:17:19.698 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T00:17:19.698 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T00:17:19.699 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T00:17:19.699 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T00:17:19.699 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T00:17:19.699 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T00:17:19.699 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "", 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "" 2023-10-12T00:17:19.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:19.701 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:17:19.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:17:19.701 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:17:19.701 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:17:19.701 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T00:17:19.702 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:17:19.702 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:17:19.702 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T00:17:19.702 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2023-10-12T00:17:19.703 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:19.704 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T00:17:19.705 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:17:19.705 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:17:19.705 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.705 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.705 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:19.706 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2023-10-12T00:17:19.706 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T00:17:19.706 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:19.707 DEBUG:tasks.util.rgw: json result: {'id': '8669d6aa-a676-4c9b-955e-b4e2f447fb20', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T00:17:19.708 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-10-12T00:17:19.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-10-12T00:17:19.708 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-10-12T00:17:19.752 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:19.752 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:19.761 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.760+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 realm 2023-10-12T00:17:19.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-12T00:17:19.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:19.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.761+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.762+0000 7f09f9bfe780 20 rados_obj.operate() r=0 bl.length=890 2023-10-12T00:17:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.762+0000 7f09f9bfe780 20 searching for the correct realm 2023-10-12T00:17:19.770 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 RGWRados::pool_iterate: got zonegroup_info.6986cff7-c771-4b7d-af24-29b924610377 2023-10-12T00:17:19.770 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 RGWRados::pool_iterate: got zone_info.8669d6aa-a676-4c9b-955e-b4e2f447fb20 2023-10-12T00:17:19.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-12T00:17:19.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-12T00:17:19.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.769+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.772 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados_obj.operate() r=0 bl.length=46 2023-10-12T00:17:19.772 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.773 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados_obj.operate() r=0 bl.length=420 2023-10-12T00:17:19.773 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 zone default found 2023-10-12T00:17:19.773 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 4 Realm: () 2023-10-12T00:17:19.774 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 4 ZoneGroup: default (6986cff7-c771-4b7d-af24-29b924610377) 2023-10-12T00:17:19.774 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 4 Zone: default (8669d6aa-a676-4c9b-955e-b4e2f447fb20) 2023-10-12T00:17:19.774 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-12T00:17:19.774 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 zonegroup default 2023-10-12T00:17:19.774 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.775 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.775 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.775 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.775 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados->read ofs=0 len=0 2023-10-12T00:17:19.776 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-12T00:17:19.776 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.770+0000 7f09f9bfe780 20 started sync module instance, tier type = 2023-10-12T00:17:19.776 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.771+0000 7f09f9bfe780 20 started zone id=8669d6aa-a676-4c9b-955e-b4e2f447fb20 (name=default) with tier type = 2023-10-12T00:17:19.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=0 2023-10-12T00:17:19.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=1 2023-10-12T00:17:19.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=2 2023-10-12T00:17:19.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=3 2023-10-12T00:17:19.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=4 2023-10-12T00:17:19.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=5 2023-10-12T00:17:19.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.778+0000 7f09f9bfe780 20 add_watcher() i=6 2023-10-12T00:17:19.781 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.779+0000 7f09f9bfe780 20 add_watcher() i=7 2023-10-12T00:17:19.781 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.779+0000 7f09f9bfe780 2 all 8 watchers are set, enabling cache 2023-10-12T00:17:19.781 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.780+0000 7f09deb7a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:19.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.780+0000 7f09f9bfe780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:19.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.780+0000 7f09f9bfe780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:19.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.784+0000 7f09f9bfe780 5 note: GC not initialized 2023-10-12T00:17:19.785 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.784+0000 7f09da371700 20 reqs_thread_entry: start 2023-10-12T00:17:19.786 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.785+0000 7f09f9bfe780 20 init_complete bucket index max shards: 11 2023-10-12T00:17:19.786 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.785+0000 7f09f9bfe780 10 rgw notify: Started notification manager with: 1 workers 2023-10-12T00:17:19.786 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.785+0000 7f09d836d700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 12 00:17:50 2023 2023-10-12T00:17:19.786 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-12T00:17:19.786 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.785+0000 7f09d7b4b700 20 reqs_thread_entry: start 2023-10-12T00:17:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.792+0000 7f09deb7a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-12T00:17:19.794 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.794+0000 7f09f9bfe780 20 remove_watcher() i=0 2023-10-12T00:17:19.794 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.794+0000 7f09f9bfe780 2 removed watcher, disabling cache 2023-10-12T00:17:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.794+0000 7f09f9bfe780 20 remove_watcher() i=1 2023-10-12T00:17:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.795+0000 7f09f9bfe780 20 remove_watcher() i=2 2023-10-12T00:17:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.796+0000 7f09f9bfe780 20 remove_watcher() i=3 2023-10-12T00:17:19.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.797+0000 7f09f9bfe780 20 remove_watcher() i=4 2023-10-12T00:17:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.797+0000 7f09f9bfe780 20 remove_watcher() i=5 2023-10-12T00:17:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.798+0000 7f09f9bfe780 20 remove_watcher() i=6 2023-10-12T00:17:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.799+0000 7f09f9bfe780 20 remove_watcher() i=7 2023-10-12T00:17:19.802 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2023-10-12T00:17:19.804 DEBUG:tasks.util.rgw: json result: [] 2023-10-12T00:17:19.804 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:19.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-12T00:17:19.805 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-12T00:17:19.853 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:19.854 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:19.862 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.861+0000 7fcf2e2ac780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-12T00:17:19.862 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.861+0000 7fcf2e2ac780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-12T00:17:19.865 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:19.865+0000 7fcf0fa21700 20 reqs_thread_entry: start 2023-10-12T00:17:19.872 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "8669d6aa-a676-4c9b-955e-b4e2f447fb20", 2023-10-12T00:17:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default", 2023-10-12T00:17:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "default.rgw.meta:root", 2023-10-12T00:17:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "default.rgw.control", 2023-10-12T00:17:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-12T00:17:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-12T00:17:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "default.rgw.log", 2023-10-12T00:17:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-12T00:17:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-12T00:17:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-12T00:17:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-12T00:17:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-12T00:17:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "default.rgw.otp", 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "", 2023-10-12T00:17:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "" 2023-10-12T00:17:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:17:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:17:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:17:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:17:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-12T00:17:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:17:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:17:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-12T00:17:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2023-10-12T00:17:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-12T00:17:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:17:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:17:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2023-10-12T00:17:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-12T00:17:19.881 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:19.882 DEBUG:tasks.util.rgw: json result: {'id': '8669d6aa-a676-4c9b-955e-b4e2f447fb20', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-12T00:17:19.882 INFO:tasks.rgw:Starting rgw... 2023-10-12T00:17:19.883 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-10-12T00:17:19.883 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T00:17:19.883 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-12T00:17:19.884 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-12T00:17:19.886 INFO:tasks.rgw.c1.client.0:Started 2023-10-12T00:17:19.887 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-10-12T00:17:19.887 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T00:17:19.887 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T00:17:19.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T00:17:19.890 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T00:17:19.890 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-10-12T00:17:19.890 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T00:17:19.891 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-12T00:17:19.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-12T00:17:19.894 INFO:tasks.rgw.c2.client.0:Started 2023-10-12T00:17:19.894 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-10-12T00:17:19.894 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-12T00:17:19.894 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-12T00:17:19.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-12T00:17:19.897 INFO:tasks.rgw.c2.client.1:Started 2023-10-12T00:17:19.898 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8000/ 2023-10-12T00:17:19.898 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2023-10-12T00:17:19.960 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:19.960 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:17:19.962 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T00:17:20.963 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2023-10-12T00:17:20.989 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:20.989 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:20.990 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:20.991 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:17:20.991 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8001/ 2023-10-12T00:17:20.991 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001/ 2023-10-12T00:17:21.054 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:21.054 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:21.055 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:21.056 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:17:21.057 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi118.front.sepia.ceph.com:8000/ 2023-10-12T00:17:21.057 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000/ 2023-10-12T00:17:21.095 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:21.096 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:21.097 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:21.097 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2023-10-12T00:17:21.098 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi118.front.sepia.ceph.com:8001/ 2023-10-12T00:17:21.098 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001/ 2023-10-12T00:17:21.161 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:21.161 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:21.162 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:21.163 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2023-10-12T00:17:21.163 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-10-12T00:17:22.870 INFO:tasks.rgw_multisite:creating realm.. 2023-10-12T00:17:22.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:22.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:22.871 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:22.924 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:22.924 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:22.953 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:22.953 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:22.953 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "test-realm", 2023-10-12T00:17:22.953 INFO:teuthology.orchestra.run.smithi112.stdout: "current_period": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:17:22.953 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:17:22.954 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:22.955 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-10-12T00:17:22.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:22.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:22.956 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:23.005 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:23.006 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:23.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:23.018+0000 7fbbb79ca780 -1 old format 2023-10-12T00:17:23.032 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:23.032 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:17:23.032 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:17:23.032 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:17:23.032 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:17:23.033 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:23.033 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:17:23.033 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:23.033 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:17:23.033 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:17:23.034 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "", 2023-10-12T00:17:23.034 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [], 2023-10-12T00:17:23.034 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:17:23.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:23.034 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [] 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:17:23.035 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:23.036 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:17:23.036 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:17:23.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:23.036 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:17:23.036 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:23.037 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:23.037 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:23.038 INFO:tasks.rgw_multisite:creating master zone.. 2023-10-12T00:17:23.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:23.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:23.039 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --endpoints http://smithi112.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=a9c32acc-d391-49fc-954e-a55395784dbf 2023-10-12T00:17:23.166 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:23.166 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:23.166 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:17:23.166 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-12T00:17:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a1.rgw.control", 2023-10-12T00:17:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-12T00:17:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-12T00:17:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a1.rgw.log", 2023-10-12T00:17:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-12T00:17:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-12T00:17:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-12T00:17:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-12T00:17:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-12T00:17:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-12T00:17:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-12T00:17:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-12T00:17:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a1.rgw.otp", 2023-10-12T00:17:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-12T00:17:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a1.rgw.buckets.data" 2023-10-12T00:17:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-12T00:17:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-12T00:17:23.173 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:23.175 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-10-12T00:17:24.235 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'a1.rgw.buckets.data' created 2023-10-12T00:17:24.252 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-10-12T00:17:25.245 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-10-12T00:17:25.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:25.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:25.262 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:25.332 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:25.333 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:17:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-12T00:17:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a1.rgw.control", 2023-10-12T00:17:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-12T00:17:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-12T00:17:25.334 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a1.rgw.log", 2023-10-12T00:17:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-12T00:17:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-12T00:17:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-12T00:17:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-12T00:17:25.335 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a1.rgw.otp", 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:25.336 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:25.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-12T00:17:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:25.338 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a1.rgw.buckets.data", 2023-10-12T00:17:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:17:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:25.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-12T00:17:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:25.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-12T00:17:25.341 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:25.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:25.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:25.343 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:25.402+0000 7f60fd940780 0 period (26ed623f-4dfb-47f6-b4f4-62e562c80d92 does not have zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 configured 2023-10-12T00:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2023-10-12T00:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": false, 2023-10-12T00:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:17:32.340 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:32.340 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:32.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:32.340 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:32.341 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:17:32.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:32.341 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:32.341 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:17:32.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:32.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:32.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.343 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:17:32.343 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:32.343 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:17:32.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:17:32.343 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:32.344 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:32.344 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:32.344 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:32.344 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:32.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:32.345 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:32.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.346 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:17:32.346 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:32.346 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:32.346 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:32.347 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:32.347 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:32.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:17:32.347 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.348 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:17:32.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:32.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:32.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:32.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:32.350 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:32.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:32.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:32.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:32.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:17:32.351 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:17:32.351 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:32.352 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-12T00:17:32.353 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:17:32.353 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:17:32.352+0000 7ff694e0e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 119802) UID: 0 2023-10-12T00:17:32.354 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:17:32.352+0000 7ff6990777c0 -1 shutting down 2023-10-12T00:17:34.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-12T00:17:38.454 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-12T00:17:38.454 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-12T00:17:38.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-12T00:17:38.458 INFO:tasks.rgw.c1.client.0:Started 2023-10-12T00:17:38.458 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-10-12T00:17:38.483 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:38.483 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:17:38.485 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T00:17:39.486 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-10-12T00:17:39.508 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:39.509 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:39.510 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:39.510 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:17:39.511 INFO:tasks.rgw_multisite:creating admin user.. 2023-10-12T00:17:39.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:39.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:39.512 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:39.608 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:39.608 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:39.639 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "user_id": "realm-admin", 2023-10-12T00:17:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "display_name": "Realm Admin", 2023-10-12T00:17:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "email": "", 2023-10-12T00:17:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "suspended": 0, 2023-10-12T00:17:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "max_buckets": 1000, 2023-10-12T00:17:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "subusers": [], 2023-10-12T00:17:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "keys": [ 2023-10-12T00:17:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "realm-admin", 2023-10-12T00:17:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "swift_keys": [], 2023-10-12T00:17:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": [], 2023-10-12T00:17:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "op_mask": "read, write, delete", 2023-10-12T00:17:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "system": true, 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "", 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "default_storage_class": "", 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_tags": [], 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:17:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "temp_url_keys": [], 2023-10-12T00:17:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "rgw", 2023-10-12T00:17:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "mfa_ids": [] 2023-10-12T00:17:39.647 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:17:39.651 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T00:17:39.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:39.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:39.653 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --endpoints http://smithi112.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:39.707 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:39.708 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:39.721 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=a9c32acc-d391-49fc-954e-a55395784dbf 2023-10-12T00:17:39.735 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:17:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:17:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:17:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:17:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:17:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:17:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:17:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:17:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:17:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:17:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:17:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:17:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:17:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:17:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:17:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:17:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:17:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data" 2023-10-12T00:17:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:17:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:17:39.745 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:39.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-10-12T00:17:40.232 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'a2.rgw.buckets.data' created 2023-10-12T00:17:40.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-10-12T00:17:40.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-12T00:17:41.256 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-10-12T00:17:41.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:41.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:41.276 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:41.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:41.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:41.349 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:17:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:17:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:17:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:17:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:17:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:17:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:17:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:17:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:17:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:17:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:17:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:17:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:17:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:17:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:17:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:17:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:17:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:17:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:17:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:17:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:17:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:17:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:17:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:17:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:17:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:17:41.357 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:41.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:41.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:41.358 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:41.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:17:41.417+0000 7f5549ba4780 0 period (f8a42833-a8a4-4b21-a670-8605a7b80b21 does not have zone c93c7bc5-2183-452b-b2b7-6a812a26f746 configured 2023-10-12T00:17:48.302 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:17:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:17:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:17:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2, 2023-10-12T00:17:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:17:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:17:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:17:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:17:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:17:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:17:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:17:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:17:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:17:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:17:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:17:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:17:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:17:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:17:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:17:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:17:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:17:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:17:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:17:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:17:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:17:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:17:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:17:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:17:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:17:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:17:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:17:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:17:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:17:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:17:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:17:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:17:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:17:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:17:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:17:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:17:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:17:48.339 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:17:48.339 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:17:48.339 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:48.339 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:48.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:48.340 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.340 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:17:48.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.340 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:17:48.340 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:17:48.341 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:17:48.341 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:17:48.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:48.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:48.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.342 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:17:48.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:48.342 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:17:48.342 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:17:48.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:17:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:17:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:17:48.344 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:17:48.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.344 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:48.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:17:48.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:17:48.345 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:17:48.345 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:17:48.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:48.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:17:48.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.346 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:17:48.346 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:17:48.346 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:17:48.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:17:48.347 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:48.347 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:48.347 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:48.347 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:48.348 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:48.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.348 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:17:48.348 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:17:48.348 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:17:48.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:17:48.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:17:48.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:17:48.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.350 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:17:48.350 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:48.350 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:48.350 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:48.350 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:48.351 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:48.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:17:48.351 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:48.351 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:48.352 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:48.352 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:48.353 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:48.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.354 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:17:48.354 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:17:48.354 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:17:48.354 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:17:48.354 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:17:48.355 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:17:48.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:17:48.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:17:48.355 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:48.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:17:48.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:17:48.356 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:17:48.357 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T00:17:48.357 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:17:48.358 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:17:48.358+0000 7f223b378700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 119805) UID: 0 2023-10-12T00:17:48.358 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:17:48.358+0000 7f223f5e17c0 -1 shutting down 2023-10-12T00:17:51.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T00:17:54.459 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T00:17:54.459 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T00:17:54.460 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T00:17:54.463 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T00:17:54.463 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:17:54.488 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:54.489 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:54.489 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T00:17:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:17:55.491 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:17:55.514 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:17:55.514 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:17:55.515 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:17:55.516 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:17:55.516 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-12T00:17:55.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:55.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:55.517 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:55.578 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:55.579 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:56.621 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:56.621 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:56.621 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-realm", 2023-10-12T00:17:56.622 INFO:teuthology.orchestra.run.smithi118.stdout: "current_period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:17:56.622 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 2 2023-10-12T00:17:56.622 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:56.623 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-10-12T00:17:56.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:56.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:56.624 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:56.676 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:56.677 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:56.691 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:56.690+0000 7fdf149da780 -1 old format 2023-10-12T00:17:56.697 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:56.697 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "b", 2023-10-12T00:17:56.698 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b", 2023-10-12T00:17:56.698 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2023-10-12T00:17:56.698 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": false, 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2023-10-12T00:17:56.699 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "", 2023-10-12T00:17:56.700 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [], 2023-10-12T00:17:56.700 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2023-10-12T00:17:56.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:17:56.700 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2023-10-12T00:17:56.701 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2023-10-12T00:17:56.701 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [] 2023-10-12T00:17:56.701 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:56.701 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:56.701 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2023-10-12T00:17:56.702 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:56.702 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2023-10-12T00:17:56.702 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2023-10-12T00:17:56.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:56.702 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled_features": [ 2023-10-12T00:17:56.703 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:17:56.703 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:17:56.703 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:56.704 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T00:17:56.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:56.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:56.705 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --endpoints http://smithi118.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:56.802 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:56.802 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:56.816 INFO:teuthology.orchestra.run.smithi118.stderr:NOTICE: set zone's realm_id=a9c32acc-d391-49fc-954e-a55395784dbf 2023-10-12T00:17:56.830 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:56.830 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:17:56.830 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b1", 2023-10-12T00:17:56.831 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-12T00:17:56.831 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "b1.rgw.control", 2023-10-12T00:17:56.831 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-12T00:17:56.831 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-12T00:17:56.831 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "b1.rgw.log", 2023-10-12T00:17:56.832 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-12T00:17:56.832 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-12T00:17:56.832 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-12T00:17:56.832 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-12T00:17:56.832 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-12T00:17:56.833 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-12T00:17:56.833 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-12T00:17:56.833 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-12T00:17:56.833 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "b1.rgw.otp", 2023-10-12T00:17:56.833 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:17:56.834 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:56.834 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:56.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:56.834 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:17:56.835 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:17:56.836 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "b1.rgw.buckets.data" 2023-10-12T00:17:56.836 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:56.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:56.836 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-12T00:17:56.836 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:17:56.837 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:17:56.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:56.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:56.837 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:56.838 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:56.838 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-12T00:17:56.838 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:56.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-10-12T00:17:57.583 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'b1.rgw.buckets.data' created 2023-10-12T00:17:57.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-10-12T00:17:57.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T00:17:58.599 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-10-12T00:17:58.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:58.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:58.619 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:58.674 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:58.675 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:58.692 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:17:58.693 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:17:58.693 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b1", 2023-10-12T00:17:58.693 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-12T00:17:58.693 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "b1.rgw.control", 2023-10-12T00:17:58.694 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-12T00:17:58.694 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-12T00:17:58.694 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "b1.rgw.log", 2023-10-12T00:17:58.694 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-12T00:17:58.694 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-12T00:17:58.695 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-12T00:17:58.695 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-12T00:17:58.695 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-12T00:17:58.695 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-12T00:17:58.695 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-12T00:17:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-12T00:17:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "b1.rgw.otp", 2023-10-12T00:17:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:17:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:17:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:17:58.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:58.697 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:17:58.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:17:58.697 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:17:58.697 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:17:58.698 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-12T00:17:58.698 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:17:58.698 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:17:58.698 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "b1.rgw.buckets.data", 2023-10-12T00:17:58.698 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2023-10-12T00:17:58.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:58.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:17:58.699 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-12T00:17:58.699 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:17:58.699 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:17:58.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:58.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:17:58.700 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:17:58.700 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:17:58.700 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-12T00:17:58.701 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:17:58.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:58.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:17:58.703 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:17:58.751 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:17:58.751 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:17:58.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:17:58.762+0000 7f31471b6780 0 period (f8a42833-a8a4-4b21-a670-8605a7b80b21 does not have zone 759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5 configured 2023-10-12T00:18:05.638 INFO:teuthology.orchestra.run.smithi118.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:18:05.666 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:18:05.666 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:05.666 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 3, 2023-10-12T00:18:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:18:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": [], 2023-10-12T00:18:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: "period_map": { 2023-10-12T00:18:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: "zonegroups": [ 2023-10-12T00:18:05.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.668 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "a", 2023-10-12T00:18:05.668 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a", 2023-10-12T00:18:05.668 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2023-10-12T00:18:05.668 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": true, 2023-10-12T00:18:05.669 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:05.669 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:18:05.669 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.669 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2023-10-12T00:18:05.669 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2023-10-12T00:18:05.670 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:05.670 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2023-10-12T00:18:05.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.670 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:05.670 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a1", 2023-10-12T00:18:05.671 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:05.671 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:18:05.671 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.671 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:05.671 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:05.672 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:05.673 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:05.673 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:05.673 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:05.673 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.674 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:18:05.674 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a2", 2023-10-12T00:18:05.674 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:05.674 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:18:05.675 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.675 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:05.675 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:05.675 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:05.675 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:05.676 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:05.676 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:05.676 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:05.676 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:05.676 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.677 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2023-10-12T00:18:05.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.678 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2023-10-12T00:18:05.678 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2023-10-12T00:18:05.678 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2023-10-12T00:18:05.678 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2023-10-12T00:18:05.679 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.679 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.679 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2023-10-12T00:18:05.679 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled_features": [ 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:05.680 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.681 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "b", 2023-10-12T00:18:05.681 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b", 2023-10-12T00:18:05.681 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2023-10-12T00:18:05.682 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": false, 2023-10-12T00:18:05.682 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:05.682 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:18:05.682 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.682 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2023-10-12T00:18:05.683 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2023-10-12T00:18:05.683 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:05.683 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2023-10-12T00:18:05.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.683 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b1", 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:05.684 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": false, 2023-10-12T00:18:05.685 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:05.685 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:05.685 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:05.685 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:05.686 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:05.686 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:05.686 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:05.686 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:05.687 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:05.688 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.688 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.688 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2023-10-12T00:18:05.689 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.690 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.690 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.690 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2023-10-12T00:18:05.690 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled_features": [ 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:05.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.692 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:05.692 INFO:teuthology.orchestra.run.smithi118.stdout: "short_zone_ids": [ 2023-10-12T00:18:05.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.692 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:05.693 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3435037168 2023-10-12T00:18:05.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.693 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:05.693 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 2465750364 2023-10-12T00:18:05.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:05.694 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:18:05.694 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 1412291983 2023-10-12T00:18:05.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.695 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:05.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.695 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zonegroup": "a", 2023-10-12T00:18:05.695 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:05.695 INFO:teuthology.orchestra.run.smithi118.stdout: "period_config": { 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_quota": { 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2023-10-12T00:18:05.696 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2023-10-12T00:18:05.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.697 INFO:teuthology.orchestra.run.smithi118.stdout: "user_quota": { 2023-10-12T00:18:05.697 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2023-10-12T00:18:05.697 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2023-10-12T00:18:05.697 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2023-10-12T00:18:05.698 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2023-10-12T00:18:05.698 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2023-10-12T00:18:05.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.698 INFO:teuthology.orchestra.run.smithi118.stdout: "user_ratelimit": { 2023-10-12T00:18:05.698 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:05.699 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:05.699 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:05.699 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:05.699 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:05.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_ratelimit": { 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:05.700 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:05.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.701 INFO:teuthology.orchestra.run.smithi118.stdout: "anonymous_ratelimit": { 2023-10-12T00:18:05.701 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:05.701 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:05.701 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:05.702 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:05.702 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:05.702 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:05.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:05.702 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:05.703 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_name": "test-realm", 2023-10-12T00:18:05.703 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:18:05.703 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:18:05.704 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-12T00:18:05.704 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:18:05.705 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2023-10-12T00:18:05.704+0000 7f56b0423700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 119130) UID: 0 2023-10-12T00:18:05.705 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2023-10-12T00:18:05.704+0000 7f56b468c7c0 -1 shutting down 2023-10-12T00:18:11.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-12T00:18:11.806 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-12T00:18:11.806 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-12T00:18:11.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-12T00:18:11.810 INFO:tasks.rgw.c2.client.0:Started 2023-10-12T00:18:11.810 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000 2023-10-12T00:18:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:18:11.836 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:18:11.836 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:18:11.836 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi118.front.sepia.ceph.com port 8000: Connection refused 2023-10-12T00:18:12.837 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000 2023-10-12T00:18:12.861 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:18:12.862 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:18:12.862 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:18:12.864 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2023-10-12T00:18:12.864 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-12T00:18:12.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:12.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:12.866 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --url http://smithi112.front.sepia.ceph.com:8000 --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:18:12.962 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:12.962 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:12.995 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:18:12.995 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:12.995 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-realm", 2023-10-12T00:18:12.996 INFO:teuthology.orchestra.run.smithi118.stdout: "current_period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:12.996 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 2 2023-10-12T00:18:12.996 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:18:12.998 INFO:tasks.rgw_multisite:creating zone.. 2023-10-12T00:18:12.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:12.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:12.999 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --endpoints http://smithi118.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:18:13.052 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:13.053 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:13.066 INFO:teuthology.orchestra.run.smithi118.stderr:NOTICE: set zone's realm_id=a9c32acc-d391-49fc-954e-a55395784dbf 2023-10-12T00:18:13.080 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:18:13.080 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:18:13.080 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b2", 2023-10-12T00:18:13.081 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-12T00:18:13.081 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "b2.rgw.control", 2023-10-12T00:18:13.082 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-12T00:18:13.082 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-12T00:18:13.082 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "b2.rgw.log", 2023-10-12T00:18:13.083 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-12T00:18:13.083 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-12T00:18:13.083 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-12T00:18:13.084 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-12T00:18:13.084 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-12T00:18:13.084 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-12T00:18:13.085 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-12T00:18:13.085 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-12T00:18:13.086 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "b2.rgw.otp", 2023-10-12T00:18:13.086 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:18:13.086 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:18:13.086 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:18:13.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:13.087 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:18:13.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:13.088 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:18:13.088 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:18:13.088 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-12T00:18:13.088 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:18:13.089 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:18:13.089 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "b2.rgw.buckets.data" 2023-10-12T00:18:13.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:13.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:13.090 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-12T00:18:13.090 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:18:13.091 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:18:13.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:13.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:13.091 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:13.091 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:13.092 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-12T00:18:13.092 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:18:13.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-10-12T00:18:13.760 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'b2.rgw.buckets.data' created 2023-10-12T00:18:13.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-10-12T00:18:14.771 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-10-12T00:18:14.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:14.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:14.792 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:18:14.845 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:14.845 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:14.863 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:18:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:18:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b2", 2023-10-12T00:18:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-12T00:18:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "b2.rgw.control", 2023-10-12T00:18:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-12T00:18:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-12T00:18:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "b2.rgw.log", 2023-10-12T00:18:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-12T00:18:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-12T00:18:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-12T00:18:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-12T00:18:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-12T00:18:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-12T00:18:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-12T00:18:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-12T00:18:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "b2.rgw.otp", 2023-10-12T00:18:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2023-10-12T00:18:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:18:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:18:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2023-10-12T00:18:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2023-10-12T00:18:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:18:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-12T00:18:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2023-10-12T00:18:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2023-10-12T00:18:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "b2.rgw.buckets.data", 2023-10-12T00:18:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2023-10-12T00:18:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-12T00:18:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0, 2023-10-12T00:18:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "inline_data": true 2023-10-12T00:18:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-12T00:18:14.872 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:18:14.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:14.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:14.873 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-12T00:18:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:14.923 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:14.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:18:14.935+0000 7f5407782780 0 period (f8a42833-a8a4-4b21-a670-8605a7b80b21 does not have zone 3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6 configured 2023-10-12T00:18:16.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-12T00:18:21.823 INFO:teuthology.orchestra.run.smithi118.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:18:21.852 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:18:21.852 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:21.852 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 4, 2023-10-12T00:18:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:18:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": [], 2023-10-12T00:18:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: "period_map": { 2023-10-12T00:18:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: "zonegroups": [ 2023-10-12T00:18:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "a", 2023-10-12T00:18:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a", 2023-10-12T00:18:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2023-10-12T00:18:21.854 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": true, 2023-10-12T00:18:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:18:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2023-10-12T00:18:21.855 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a1", 2023-10-12T00:18:21.856 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.857 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:18:21.857 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.857 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:21.857 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:21.857 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:21.858 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:21.858 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:21.858 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:21.859 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:21.859 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:21.859 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:21.859 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:21.859 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.860 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:18:21.860 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "a2", 2023-10-12T00:18:21.861 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.861 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:18:21.861 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.861 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:21.861 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:21.862 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:21.863 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:21.863 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:21.863 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.863 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.864 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.864 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2023-10-12T00:18:21.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.864 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2023-10-12T00:18:21.865 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2023-10-12T00:18:21.865 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2023-10-12T00:18:21.865 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2023-10-12T00:18:21.865 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.866 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.866 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2023-10-12T00:18:21.866 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:21.866 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2023-10-12T00:18:21.867 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2023-10-12T00:18:21.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.867 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled_features": [ 2023-10-12T00:18:21.867 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.868 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "b", 2023-10-12T00:18:21.869 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b", 2023-10-12T00:18:21.869 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2023-10-12T00:18:21.869 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": false, 2023-10-12T00:18:21.869 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.869 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:18:21.870 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.870 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2023-10-12T00:18:21.870 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2023-10-12T00:18:21.870 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:21.870 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b2", 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:18:21.871 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.872 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:21.872 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:21.872 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:21.872 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:21.872 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:21.873 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:21.873 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:21.873 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:21.873 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:21.873 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:21.875 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "b1", 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.876 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": false, 2023-10-12T00:18:21.877 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": true, 2023-10-12T00:18:21.877 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:18:21.877 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": false, 2023-10-12T00:18:21.877 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2023-10-12T00:18:21.877 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": true, 2023-10-12T00:18:21.878 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2023-10-12T00:18:21.878 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "", 2023-10-12T00:18:21.878 INFO:teuthology.orchestra.run.smithi118.stdout: "supported_features": [ 2023-10-12T00:18:21.878 INFO:teuthology.orchestra.run.smithi118.stdout: "compress-encrypted", 2023-10-12T00:18:21.878 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.879 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.879 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.880 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2023-10-12T00:18:21.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.880 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2023-10-12T00:18:21.880 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2023-10-12T00:18:21.880 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2023-10-12T00:18:21.881 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2023-10-12T00:18:21.881 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.881 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.881 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.881 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled_features": [ 2023-10-12T00:18:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: "resharding" 2023-10-12T00:18:21.883 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.883 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.883 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2023-10-12T00:18:21.883 INFO:teuthology.orchestra.run.smithi118.stdout: "short_zone_ids": [ 2023-10-12T00:18:21.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.884 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:18:21.884 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3962893999 2023-10-12T00:18:21.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.884 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3435037168 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 2465750364 2023-10-12T00:18:21.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:18:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:18:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 1412291983 2023-10-12T00:18:21.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.889 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:18:21.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.889 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zonegroup": "a", 2023-10-12T00:18:21.889 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:18:21.889 INFO:teuthology.orchestra.run.smithi118.stdout: "period_config": { 2023-10-12T00:18:21.890 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_quota": { 2023-10-12T00:18:21.890 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2023-10-12T00:18:21.890 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2023-10-12T00:18:21.890 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2023-10-12T00:18:21.890 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2023-10-12T00:18:21.891 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2023-10-12T00:18:21.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.891 INFO:teuthology.orchestra.run.smithi118.stdout: "user_quota": { 2023-10-12T00:18:21.891 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2023-10-12T00:18:21.891 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2023-10-12T00:18:21.892 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2023-10-12T00:18:21.892 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2023-10-12T00:18:21.892 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2023-10-12T00:18:21.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.892 INFO:teuthology.orchestra.run.smithi118.stdout: "user_ratelimit": { 2023-10-12T00:18:21.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:21.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:21.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:21.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:21.894 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:21.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.894 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_ratelimit": { 2023-10-12T00:18:21.894 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:21.894 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:21.895 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:21.895 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:21.895 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:21.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.895 INFO:teuthology.orchestra.run.smithi118.stdout: "anonymous_ratelimit": { 2023-10-12T00:18:21.896 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_ops": 0, 2023-10-12T00:18:21.896 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_ops": 0, 2023-10-12T00:18:21.896 INFO:teuthology.orchestra.run.smithi118.stdout: "max_read_bytes": 0, 2023-10-12T00:18:21.896 INFO:teuthology.orchestra.run.smithi118.stdout: "max_write_bytes": 0, 2023-10-12T00:18:21.897 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false 2023-10-12T00:18:21.897 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:18:21.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:18:21.897 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:18:21.898 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_name": "test-realm", 2023-10-12T00:18:21.898 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:18:21.898 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:18:21.899 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-12T00:18:21.900 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:18:21.900 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:18:21.900+0000 7f64b9070700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 119133) UID: 0 2023-10-12T00:18:21.901 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:18:21.900+0000 7f64bd2d97c0 -1 shutting down 2023-10-12T00:18:22.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-12T00:18:28.002 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-12T00:18:28.002 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-12T00:18:28.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-12T00:18:28.005 INFO:tasks.rgw.c2.client.1:Started 2023-10-12T00:18:28.006 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001 2023-10-12T00:18:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:18:28.032 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:18:28.032 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:18:28.032 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi118.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T00:18:28.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2023-10-12T00:18:29.034 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001 2023-10-12T00:18:29.057 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:18:29.057 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:18:29.058 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:18:29.059 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2023-10-12T00:18:29.059 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-10-12T00:18:29.060 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-10-12T00:18:29.088 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi 2023-10-12T00:18:29.213 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-10-12T00:18:29.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:29.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:18:29.214 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:18:29.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:29.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:29.317 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: "user_id": "rgw-multisite-test-user", 2023-10-12T00:18:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: "display_name": "Multisite Test User", 2023-10-12T00:18:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: "email": "", 2023-10-12T00:18:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: "suspended": 0, 2023-10-12T00:18:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: "max_buckets": 1000, 2023-10-12T00:18:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: "subusers": [], 2023-10-12T00:18:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: "keys": [ 2023-10-12T00:18:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "rgw-multisite-test-user", 2023-10-12T00:18:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "L12R7HDAL0U7FKTOBSDN", 2023-10-12T00:18:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "tUeLRPF9wxBxB1dmeeKMMeljHodvIXvi4lIzHFcP" 2023-10-12T00:18:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:18:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: "swift_keys": [], 2023-10-12T00:18:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": [ 2023-10-12T00:18:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "roles", 2023-10-12T00:18:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: "perm": "*" 2023-10-12T00:18:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:18:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: "op_mask": "read, write, delete", 2023-10-12T00:18:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "", 2023-10-12T00:18:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: "default_storage_class": "", 2023-10-12T00:18:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_tags": [], 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:18:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:18:29.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.324 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:18:29.324 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:18:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:18:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:18:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:18:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:18:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.326 INFO:teuthology.orchestra.run.smithi112.stdout: "temp_url_keys": [], 2023-10-12T00:18:29.326 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "rgw", 2023-10-12T00:18:29.326 INFO:teuthology.orchestra.run.smithi112.stdout: "mfa_ids": [] 2023-10-12T00:18:29.326 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:29.326 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:18:29.329 INFO:rgw_multi.tests:meta checkpoint 2023-10-12T00:18:29.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:29.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:29.331 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:29.380 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:29.380 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:29.408 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:18:29.419 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:18:29.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:18:29.420 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:18:29.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.421 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:18:29.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.421 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:18:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:18:29.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.423 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:18:29.424 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:18:29.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:29.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:29.425 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:29.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:29.503+0000 7fb967e31780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:29.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:29.504+0000 7fb967e31780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:29.514 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:29.515 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:29.515 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:29.515 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:29.515 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:29.515 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:29.516 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:29.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.516 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:29.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.516 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:29.517 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:29.517 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:29.517 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.517 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:29.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.519 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:29.519 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:29.519 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:29.519 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.519 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:29.520 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:29.520 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:29.520 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:29.520 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:29.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:29.521 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.522 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:29.522 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:29.522 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:29.522 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:29.522 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:29.523 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:29.524 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:29.524 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:29.524 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:29.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:29.524 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.525 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:29.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:29.526 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:29.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:29.526 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:29Z" 2023-10-12T00:18:29.526 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:29.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:18:33.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-12T00:18:34.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:34.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:34.530 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:34.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:34.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:34.618+0000 7fb19a656780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:34.619+0000 7fb19a656780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:34.630 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:34.630 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:34.630 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:34.631 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:34.631 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:34.631 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:34.631 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:34.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.632 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:34.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:34.632 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:34.632 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:34.632 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:34.633 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:34.633 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:34.633 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:34.633 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:34.634 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:34.634 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:34.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:34.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:34.635 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:34.635 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:34.635 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:34.635 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:34.635 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:34.636 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:34.636 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:34.636 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:34.636 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:34.636 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:34.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:34.637 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:34.637 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:34.638 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:34.638 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:34.638 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:34.638 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:34.638 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:34.639 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:34.640 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:34.640 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:34.640 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:34.640 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:34.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:34.642 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:34.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.642 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:34.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:34.642 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:34.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:34.643 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:34Z" 2023-10-12T00:18:34.643 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:34.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:18:39.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:39.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:39.646 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:39.704 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:39.704 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:39.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:39.732+0000 7fc63c821780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:39.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:39.732+0000 7fc63c821780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:39.744 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:39Z" 2023-10-12T00:18:39.757 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:39.758 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:18:44.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:44.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:44.761 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:44.817 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:44.818 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:44.847+0000 7f380c0f8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:44.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:44.848+0000 7f380c0f8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:44.860 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:44Z" 2023-10-12T00:18:44.873 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:44.874 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:18:49.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:49.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:49.876 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:49.933 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:49.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:49.962+0000 7f9fac676780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:49.963+0000 7f9fac676780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:49.975 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:49.989 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:49.989 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:49.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:49.989 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:49.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:49.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:49.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.995 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:49.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:49Z" 2023-10-12T00:18:49.996 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:49.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:18:55.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:55.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:18:55.001 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:18:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:18:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:18:55.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:55.092+0000 7fbb17cde780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:55.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:18:55.093+0000 7fbb17cde780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:18:55.106 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:18:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:18:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:18:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:18:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:18:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:18:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:18:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:18:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:18:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:55.111 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:55.111 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:55.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:55.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:55.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:55.112 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:55.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:55.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:55.112 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:55.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:55.114 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:55.114 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:55.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:55.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:18:55.115 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:18:55.115 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:18:55.115 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:18:55.115 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:18:55.115 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:18:55.116 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:18:55.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:18:55.116 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:18:55.116 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:18:55.116 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:18:55.117 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:18:55.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:18:55.118 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:18:55Z" 2023-10-12T00:18:55.118 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:18:55.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:00.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:00.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:00.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:00.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:00.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:00.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:00.206+0000 7fd0dc7a3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:00.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:00.208+0000 7fd0dc7a3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:00.220 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:00Z" 2023-10-12T00:19:00.233 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:00.234 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:05.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:05.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:05.236 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:05.296 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:05.296 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:05.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:05.328+0000 7f2ff0e35780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:05.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:05.329+0000 7f2ff0e35780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:05.341 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:05.342 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:05.342 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:05.342 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:05.342 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:05.342 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:05.343 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:05.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.343 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:05.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:05.343 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:05.354 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:05Z" 2023-10-12T00:19:05.354 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:05.355 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:10.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:10.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:10.357 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:10.418 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:10.418 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:10.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:10.448+0000 7ff333437780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:10.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:10.450+0000 7ff333437780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:10.461 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:10.462 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:10.462 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:10.462 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:10.462 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:10.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:10.464 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:10.464 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:10.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:10.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:10.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:10.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:10.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:10.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:10.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:10.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:10.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:10.467 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:10.467 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:10.467 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:10.467 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:10.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:10.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:10.468 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:10.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:10.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:10.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:10Z" 2023-10-12T00:19:10.475 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:10.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:15.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:15.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:15.479 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:15.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:15.538 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:15.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:15.567+0000 7f6a8eee8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:15.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:15.568+0000 7f6a8eee8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:15.579 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:15.579 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:15.579 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:15.580 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:15.580 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:15.580 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:15.580 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:15.581 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:15.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:15.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:15.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:15.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:15.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:15.583 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:15.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:15.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:15.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:15.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:15.585 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:15.585 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:15.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:15.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:15.586 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:15.586 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:15.586 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:15.586 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:15.587 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:15.587 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:15.587 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:15.587 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:15.587 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:15.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:15.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:15.588 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:15.588 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:15.589 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:15.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:15.589 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:15.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:15.589 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:15.590 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:15.590 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:15.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:15.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:15.590 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:15.591 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:15Z" 2023-10-12T00:19:15.592 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:15.593 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:20.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:20.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:20.596 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:20.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:20.682+0000 7ff5eebbf780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:20.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:20.683+0000 7ff5eebbf780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:20.695 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:20.698 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:20.698 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:20.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:20.698 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:20.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:20.699 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:20.699 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:20.699 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:20.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:20.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:20.700 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:20.700 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:20.700 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:20.701 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:20.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:20.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:20.702 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:20.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:20.704 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:20.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:20.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:20.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:20.705 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:20.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:20.705 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:20.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:20.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:20.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:20.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:20.706 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:20.706 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:20.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:20.707 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:20.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:20.708 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:20Z" 2023-10-12T00:19:20.708 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:20.709 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:25.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:25.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:25.714 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:25.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:25.802+0000 7f179dd3a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:25.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:25.803+0000 7f179dd3a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:25.815 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:25.815 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:25.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:25.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:25.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:25.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:25.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:25.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:25.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:25.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:25.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:25.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:25.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:25.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:25.826 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:25.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:25.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:25.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:25.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:25.827 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:25.828 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:25.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:25.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:25.828 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:25.829 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:25Z" 2023-10-12T00:19:25.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:25.831 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:30.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:30.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:30.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:30.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:30.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:30.920+0000 7f3186529780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:30.921+0000 7f3186529780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:30.933 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:30.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:30.947 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:30Z" 2023-10-12T00:19:30.947 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:30.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:35.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:35.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:35.951 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:36.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:36.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:36.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:36.038+0000 7ff84e91a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:36.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:36.039+0000 7ff84e91a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:36.051 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:36.052 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:36.052 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:36.052 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:36.052 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:36.053 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:36.053 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:36.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.053 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:36.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:36.054 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:36.054 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:36.054 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:36.054 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:36.054 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:36.055 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:36.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:36.055 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:36.056 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:36.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:36.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:36.056 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:36.057 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:36.057 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:36.057 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:36.057 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:36.057 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:36.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:36.058 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:36.058 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:36.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:36.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:36.059 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:36.059 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:36.059 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:36.059 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:36.060 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:36.060 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:36.060 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:36.060 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:36.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:36.061 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:36.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:36.062 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:36.062 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:36.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:36.062 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:36.063 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:36.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:36.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:36.063 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:36.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.064 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:36.064 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:36.064 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:36.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:36.064 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:36Z" 2023-10-12T00:19:36.065 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:36.066 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:41.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:41.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:41.069 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:41.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:41.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:41.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:41.159+0000 7f28e88c4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:41.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:41.160+0000 7f28e88c4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:41.172 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:41.173 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:41.173 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:41.174 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:41.174 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:41.174 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:41.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:41.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.175 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:41.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:41.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:41.176 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:41.176 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:41.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:41.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:41.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:41.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:41.177 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:41.177 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:41.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:41.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:41.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:41.178 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:41.179 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:41.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:41.179 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:41.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:41.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:41.180 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:41.180 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:41.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:41.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:41.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:41.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:41.181 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:41.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:41.182 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:41.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:41.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:41.183 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:41.183 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:41.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:41.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:41.184 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:41.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:41.184 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:41.185 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:41.185 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:41.185 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:41.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:41.186 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:41.186 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:41.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:41.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:41.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:41.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.187 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:41.187 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:41.188 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:41.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:41.188 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:41Z" 2023-10-12T00:19:41.188 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:41.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:46.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:46.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:46.192 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:46.250 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:46.251 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:46.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:46.281+0000 7fc19b6df780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:46.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:46.282+0000 7fc19b6df780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:46.295 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:46.295 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:46.296 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:46.296 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:46.296 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:46.296 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:46.296 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:46.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.297 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:46.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:46.297 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:46.297 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:46.298 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:46.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:46.298 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:46.298 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:46.298 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:46.299 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:46.299 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:46.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:46.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:46.300 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:46.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:46.300 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:46.300 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:46.300 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:46.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:46.302 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:46.302 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:46.302 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:46.302 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:46.303 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:46.303 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:46.303 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:46.303 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:46.303 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:46.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:46.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:46.304 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:46.304 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:46.305 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:46.306 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:46.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:46.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:46.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:46.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:46Z" 2023-10-12T00:19:46.307 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:46.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:51.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:51.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:51.311 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:51.368 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:51.368 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:51.399+0000 7f257f319780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:51.400+0000 7f257f319780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:51.411 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:51.411 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:51.411 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:51.412 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:51.412 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:51.412 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:51.412 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:51.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.413 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:51.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:51.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:51.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:51.413 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:51.414 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:51.414 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:51.414 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:51.414 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:51.414 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:51.415 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:51.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:51.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:51.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:51.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:51.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:51.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:51.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:51.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:51.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:51.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:51.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:51.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:51.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:51.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:51.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:51.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:51Z" 2023-10-12T00:19:51.424 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:51.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:19:56.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:56.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:19:56.428 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:19:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:19:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:19:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:56.514+0000 7f3001f84780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:56.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:19:56.515+0000 7f3001f84780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:19:56.528 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:19:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:19:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:19:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:19:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:19:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:19:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:19:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:19:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:19:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:19:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:19:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:19:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:19:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:19:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:19:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:19:56.543 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:19:56.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:19:56.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:19:56.543 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:19:56.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:19:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:19:56.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:19:56.545 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:19:56Z" 2023-10-12T00:19:56.545 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:19:56.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:01.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:01.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:01.549 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:01.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:01.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:01.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:01.634+0000 7fc4d9eba780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:01.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:01.635+0000 7fc4d9eba780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:01.648 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:01.648 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:01.648 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:01.649 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:01.649 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:01.650 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:01.650 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:01.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.650 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:01.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:01.651 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:01.651 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:01.651 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:01.652 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:01.652 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:01.652 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:01.652 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:01.653 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:01.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:01.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:01.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:01.654 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:01.655 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:01.655 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:01.655 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:01.657 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:01.657 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:01.658 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:01.658 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:01.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:01.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:01.659 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:01.660 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:01.660 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:01.660 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:01.661 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:01.661 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:01.661 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:01.662 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:01.662 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:01.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:01.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:01.663 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:01.663 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:01.664 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:01.664 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:01.664 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:01.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:01.665 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:01.665 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:01.665 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:01.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:01.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:01.666 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:01.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.667 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:01.667 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:01.667 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:01.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:01.667 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:01Z" 2023-10-12T00:20:01.668 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:01.670 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:06.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:06.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:06.673 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:06.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:06.760+0000 7fa43d939780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:06.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:06.761+0000 7fa43d939780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:06.773 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:06.774 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:06.774 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:06.774 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:06.774 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:06.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:06.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:06.777 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:06.777 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:06.777 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:06.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:06.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:06.778 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:06.778 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:06.778 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:06.778 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:06.779 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:06.779 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:06.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:06.779 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:06.779 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:06.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:06.781 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:06.781 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:06.781 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:06.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:06.781 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:06.782 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:06.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:06.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:06.782 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:06.783 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:06.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:06.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:06.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:06.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:06.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:06.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.785 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:06.785 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:06.785 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:06.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:06.786 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:06Z" 2023-10-12T00:20:06.786 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:06.787 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:11.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:11.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:11.789 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:11.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:11.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:11.876+0000 7f623ad43780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:11.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:11.877+0000 7f623ad43780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:11.889 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:11Z" 2023-10-12T00:20:11.904 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:11.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:16.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:16.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:16.908 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:16.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:16.994+0000 7fabb62e2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:16.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:16.995+0000 7fabb62e2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:17.008 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:17.009 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:17.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:17.011 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:16Z" 2023-10-12T00:20:17.021 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:17.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:22.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:22.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:22.024 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:22.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:22.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:22.115+0000 7ff339206780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:22.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:22.117+0000 7ff339206780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:22.128 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:22.128 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:22.129 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:22.129 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:22.129 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:22.129 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:22.129 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:22.130 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:22.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:22.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:22.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:22.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:22.131 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:22.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:22.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:22.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:22.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:22.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:22.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:22.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:22.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:22.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:22.135 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:22.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:22.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:22.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:22.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:22.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:22Z" 2023-10-12T00:20:22.141 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:22.143 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697069909.315956_77.1 target= 2023-10-12T00:20:27.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.145 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:27.232+0000 7fe855b85780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:27.233+0000 7fe855b85780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.244 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:27.245 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:27.245 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:27.245 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:27.245 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:27.246 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.246 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.246 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:27.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.247 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:27.247 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.247 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.247 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.248 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:27.248 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.248 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.248 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.249 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.250 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.250 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.250 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.250 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.251 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:27.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.252 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.252 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.252 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.252 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.253 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.253 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.253 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:27.254 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.254 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.254 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.255 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.255 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:27.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:27.257 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:27.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.257 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.257 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:27.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:27.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-12T00:20:27.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.260 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.319 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.319 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.350 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.349+0000 7f636ee5f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.351 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.350+0000 7f636ee5f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.362 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:20:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2023-10-12T00:20:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2023-10-12T00:20:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2023-10-12T00:20:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2023-10-12T00:20:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2023-10-12T00:20:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2023-10-12T00:20:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2023-10-12T00:20:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2023-10-12T00:20:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2023-10-12T00:20:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:20:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2023-10-12T00:20:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2023-10-12T00:20:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2023-10-12T00:20:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.376 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:20:27.377 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-12T00:20:27.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-12T00:20:27.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.379 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.430 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.430 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.457 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.456+0000 7efe70425780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.457 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.457+0000 7efe70425780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.468 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:20:27.468 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2023-10-12T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2023-10-12T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2023-10-12T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2023-10-12T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2023-10-12T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2023-10-12T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2023-10-12T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.474 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.474 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.474 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.474 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2023-10-12T00:20:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.476 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.477 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2023-10-12T00:20:27.477 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.477 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.478 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.478 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.478 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.478 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.478 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.479 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2023-10-12T00:20:27.480 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2023-10-12T00:20:27.480 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2023-10-12T00:20:27.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.480 INFO:teuthology.orchestra.run.smithi118.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.480 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:20:27.481 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-12T00:20:27.482 INFO:rgw_multi.tests:meta checkpoint 2023-10-12T00:20:27.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.483 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.561 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:27.572 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:27.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.573 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.573 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:27.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.574 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:27.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.575 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:27.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.575 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.576 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:27.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.576 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:27.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:27.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.578 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.629 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.629 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:27.658+0000 7f6c10e9e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:27.659+0000 7f6c10e9e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.669 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:27.670 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:27.670 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:27.670 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:27.670 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:27.671 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.671 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.671 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:27.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.672 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:27.672 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.672 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.672 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.672 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.673 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:27.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.673 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.673 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:27.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.675 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.679 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:27.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:27.680 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:27.681 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.681 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:27.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:27.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:27.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.682 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:27.682 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:27.682 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:27.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:27.683 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.683 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:27.684 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:27.684 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-12T00:20:27.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.686 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.735 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.736 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.766 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.762+0000 7f882d7f5780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.766 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.763+0000 7f882d7f5780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.774 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:20:27.774 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2023-10-12T00:20:27.774 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2023-10-12T00:20:27.775 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2023-10-12T00:20:27.775 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2023-10-12T00:20:27.775 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.775 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.776 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2023-10-12T00:20:27.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.776 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2023-10-12T00:20:27.776 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.776 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.777 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.777 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.777 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2023-10-12T00:20:27.777 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.777 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.778 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.778 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2023-10-12T00:20:27.779 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.779 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.779 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.779 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.779 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.781 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2023-10-12T00:20:27.782 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.782 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.782 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.782 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.782 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.783 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.783 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.783 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.784 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.786 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.786 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:20:27.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.786 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2023-10-12T00:20:27.786 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2023-10-12T00:20:27.787 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2023-10-12T00:20:27.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.787 INFO:teuthology.orchestra.run.smithi118.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.787 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:20:27.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-12T00:20:27.789 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-12T00:20:27.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:27.790 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:27.839 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:27.839 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:27.867 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.866+0000 7fb7f8c0a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.867 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-12T00:20:27.867+0000 7fb7f8c0a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:27.878 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2023-10-12T00:20:27.878 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2023-10-12T00:20:27.879 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2023-10-12T00:20:27.879 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2023-10-12T00:20:27.879 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2023-10-12T00:20:27.879 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:27.879 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.880 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2023-10-12T00:20:27.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.880 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2023-10-12T00:20:27.880 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.881 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:27.882 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.882 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2023-10-12T00:20:27.883 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.883 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.883 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.883 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.883 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.885 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2023-10-12T00:20:27.885 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.885 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.885 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.885 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.886 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.886 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.886 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.886 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.886 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2023-10-12T00:20:27.887 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2023-10-12T00:20:27.888 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2023-10-12T00:20:27.888 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2023-10-12T00:20:27.888 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2023-10-12T00:20:27.888 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2023-10-12T00:20:27.888 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2023-10-12T00:20:27.889 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2023-10-12T00:20:27.890 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2023-10-12T00:20:27.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2023-10-12T00:20:27.890 INFO:teuthology.orchestra.run.smithi118.stdout: "current_time": "2023-10-12T00:20:27Z" 2023-10-12T00:20:27.890 INFO:teuthology.orchestra.run.smithi118.stdout:} 2023-10-12T00:20:27.891 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-12T00:20:27.892 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-10-12T00:20:28.151 DEBUG:boto:Using access key provided by client. 2023-10-12T00:20:28.300 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:20:28.302 DEBUG:boto:Using access key provided by client. 2023-10-12T00:20:28.320 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:20:28.321 DEBUG:boto:Using access key provided by client. 2023-10-12T00:20:28.322 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:20:28.323 DEBUG:boto:Using access key provided by client. 2023-10-12T00:20:28.325 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:20:28.325 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-1 2023-10-12T00:20:28.326 DEBUG:boto:path=/wafmoa-1/ 2023-10-12T00:20:28.326 DEBUG:boto:auth_path=/wafmoa-1/ 2023-10-12T00:20:28.327 DEBUG:boto:Method: PUT 2023-10-12T00:20:28.327 DEBUG:boto:Path: /wafmoa-1/ 2023-10-12T00:20:28.327 DEBUG:boto:Data: 2023-10-12T00:20:28.328 DEBUG:boto:Headers: {} 2023-10-12T00:20:28.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:28.329 DEBUG:boto:Port: 8000 2023-10-12T00:20:28.329 DEBUG:boto:Params: {} 2023-10-12T00:20:28.330 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:20:28.330 DEBUG:boto:Token: None 2023-10-12T00:20:28.331 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:28 GMT /wafmoa-1/ 2023-10-12T00:20:28.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J2AQ9PMsVTU9lEpkA0kaX08zEc0= 2023-10-12T00:20:28.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J2AQ9PMsVTU9lEpkA0kaX08zEc0=', 'Content-Length': '0'} 2023-10-12T00:20:31.367 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f3c5270059fd81f-0065273bcc-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:31.368 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-2 2023-10-12T00:20:31.369 DEBUG:boto:path=/wafmoa-2/ 2023-10-12T00:20:31.369 DEBUG:boto:auth_path=/wafmoa-2/ 2023-10-12T00:20:31.370 DEBUG:boto:Method: PUT 2023-10-12T00:20:31.370 DEBUG:boto:Path: /wafmoa-2/ 2023-10-12T00:20:31.371 DEBUG:boto:Data: 2023-10-12T00:20:31.371 DEBUG:boto:Headers: {} 2023-10-12T00:20:31.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:31.373 DEBUG:boto:Port: 8001 2023-10-12T00:20:31.373 DEBUG:boto:Params: {} 2023-10-12T00:20:31.374 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:20:31.374 DEBUG:boto:Token: None 2023-10-12T00:20:31.375 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:31 GMT /wafmoa-2/ 2023-10-12T00:20:31.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a6xEyASbw1Q/sOdDaOFNXb+R09o= 2023-10-12T00:20:31.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a6xEyASbw1Q/sOdDaOFNXb+R09o=', 'Content-Length': '0'} 2023-10-12T00:20:34.387 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e20b7f643986b8e7-0065273bcf-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:34.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:34.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:34.389 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:34.440 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:34.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:34.471 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:34.483 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:34.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:34.484 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:34.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.486 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:34.486 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:34.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.487 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:34.487 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:34.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.488 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070031.393989_85.1", 2023-10-12T00:20:34.488 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:31.393989Z" 2023-10-12T00:20:34.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.489 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:34.490 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070031.393989_85.1'} 2023-10-12T00:20:34.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:34.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:34.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:34.492 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:34.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:34.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:34.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:34.568+0000 7fa10b03f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:34.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:34.569+0000 7fa10b03f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:34.582 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:34.582 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:34.583 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:34.583 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:34.583 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:34.583 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:34.584 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:34.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.584 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:34.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.585 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:34.585 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:34.586 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:34.586 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:34.587 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:34.587 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:34.587 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:34.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:34.588 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:34.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.589 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:34.589 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:34.590 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:34.590 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:34.590 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:34.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:34.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:34.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:34.592 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:34.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:34.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:34.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:34.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:34.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:34.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:34.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:34.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:34.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:34.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070031.350406_78.1", 2023-10-12T00:20:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:31.350406Z", 2023-10-12T00:20:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:34.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:34.606 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:34Z" 2023-10-12T00:20:34.606 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:34.607 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070031.350406_78.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:31.350406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:34Z" } 2023-10-12T00:20:34.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070031.350406_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:31.350406Z', 'realm_epoch': 2}}] 2023-10-12T00:20:34.609 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070031.393989_85.1'} 2023-10-12T00:20:34.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070031.350406_78.1'} 2023-10-12T00:20:34.611 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697070031.393989_85.1 target=1_1697070031.350406_78.1 2023-10-12T00:20:39.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.615 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:39.676 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:39.677 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:39.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:39.704+0000 7fb62d24b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:39.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:39.705+0000 7fb62d24b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:39.718 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070031.393989_85.1", 2023-10-12T00:20:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:31.393989Z", 2023-10-12T00:20:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:39Z" 2023-10-12T00:20:39.734 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:39.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070031.393989_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:31.393989Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:39Z" } 2023-10-12T00:20:39.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070031.393989_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:31.393989Z', 'realm_epoch': 2}}] 2023-10-12T00:20:39.736 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070031.393989_85.1'} 2023-10-12T00:20:39.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070031.393989_85.1'} 2023-10-12T00:20:39.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:39.737 DEBUG:boto:path=/wafmoa-1/ 2023-10-12T00:20:39.738 DEBUG:boto:auth_path=/wafmoa-1/ 2023-10-12T00:20:39.738 DEBUG:boto:Method: HEAD 2023-10-12T00:20:39.739 DEBUG:boto:Path: /wafmoa-1/ 2023-10-12T00:20:39.739 DEBUG:boto:Data: 2023-10-12T00:20:39.740 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:39.740 DEBUG:boto:Port: 8000 2023-10-12T00:20:39.741 DEBUG:boto:Params: {} 2023-10-12T00:20:39.741 DEBUG:boto:Token: None 2023-10-12T00:20:39.742 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-1/ 2023-10-12T00:20:39.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKrnKmwWRPsLAEjyU1Ob7xEeD5U= 2023-10-12T00:20:39.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKrnKmwWRPsLAEjyU1Ob7xEeD5U=', 'Content-Length': '0'} 2023-10-12T00:20:39.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a246f1dbb647ef97-0065273bd7-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.746 DEBUG:boto:path=/wafmoa-2/ 2023-10-12T00:20:39.747 DEBUG:boto:auth_path=/wafmoa-2/ 2023-10-12T00:20:39.747 DEBUG:boto:Method: HEAD 2023-10-12T00:20:39.747 DEBUG:boto:Path: /wafmoa-2/ 2023-10-12T00:20:39.748 DEBUG:boto:Data: 2023-10-12T00:20:39.748 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:39.749 DEBUG:boto:Port: 8000 2023-10-12T00:20:39.750 DEBUG:boto:Params: {} 2023-10-12T00:20:39.750 DEBUG:boto:Token: None 2023-10-12T00:20:39.751 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-2/ 2023-10-12T00:20:39.751 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QPrEJPf/3oR8FwyxbyDckAuk+cQ= 2023-10-12T00:20:39.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QPrEJPf/3oR8FwyxbyDckAuk+cQ=', 'Content-Length': '0'} 2023-10-12T00:20:39.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de1c8c144ff26fbe-0065273bd7-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.757 DEBUG:boto:path=/wafmoa-1/ 2023-10-12T00:20:39.757 DEBUG:boto:auth_path=/wafmoa-1/ 2023-10-12T00:20:39.758 DEBUG:boto:Method: HEAD 2023-10-12T00:20:39.759 DEBUG:boto:Path: /wafmoa-1/ 2023-10-12T00:20:39.759 DEBUG:boto:Data: 2023-10-12T00:20:39.760 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:39.760 DEBUG:boto:Port: 8001 2023-10-12T00:20:39.761 DEBUG:boto:Params: {} 2023-10-12T00:20:39.761 DEBUG:boto:Token: None 2023-10-12T00:20:39.762 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-1/ 2023-10-12T00:20:39.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKrnKmwWRPsLAEjyU1Ob7xEeD5U= 2023-10-12T00:20:39.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKrnKmwWRPsLAEjyU1Ob7xEeD5U=', 'Content-Length': '0'} 2023-10-12T00:20:39.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7de35eafbd6346e-0065273bd7-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.766 DEBUG:boto:path=/wafmoa-2/ 2023-10-12T00:20:39.766 DEBUG:boto:auth_path=/wafmoa-2/ 2023-10-12T00:20:39.767 DEBUG:boto:Method: HEAD 2023-10-12T00:20:39.767 DEBUG:boto:Path: /wafmoa-2/ 2023-10-12T00:20:39.768 DEBUG:boto:Data: 2023-10-12T00:20:39.768 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:39.769 DEBUG:boto:Port: 8001 2023-10-12T00:20:39.769 DEBUG:boto:Params: {} 2023-10-12T00:20:39.770 DEBUG:boto:Token: None 2023-10-12T00:20:39.771 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-2/ 2023-10-12T00:20:39.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QPrEJPf/3oR8FwyxbyDckAuk+cQ= 2023-10-12T00:20:39.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QPrEJPf/3oR8FwyxbyDckAuk+cQ=', 'Content-Length': '0'} 2023-10-12T00:20:39.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d1fd68af5fde72c-0065273bd7-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-10-12T00:20:39.801 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-3 2023-10-12T00:20:39.802 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:39.803 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:39.804 DEBUG:boto:Method: PUT 2023-10-12T00:20:39.804 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:39.805 DEBUG:boto:Data: 2023-10-12T00:20:39.806 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:39.807 DEBUG:boto:Port: 8000 2023-10-12T00:20:39.808 DEBUG:boto:Params: {} 2023-10-12T00:20:39.808 DEBUG:boto:Token: None 2023-10-12T00:20:39.809 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-3/ 2023-10-12T00:20:39.810 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:660kXTJnmbkqZrPhpNtcdEwFCOI= 2023-10-12T00:20:39.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:660kXTJnmbkqZrPhpNtcdEwFCOI=', 'Content-Length': '0'} 2023-10-12T00:20:39.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc3e70ee5dda8550-0065273bd7-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.825 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-4 2023-10-12T00:20:39.826 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:39.826 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:39.826 DEBUG:boto:Method: PUT 2023-10-12T00:20:39.827 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:39.827 DEBUG:boto:Data: 2023-10-12T00:20:39.828 DEBUG:boto:Headers: {} 2023-10-12T00:20:39.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:39.828 DEBUG:boto:Port: 8001 2023-10-12T00:20:39.829 DEBUG:boto:Params: {} 2023-10-12T00:20:39.829 DEBUG:boto:Token: None 2023-10-12T00:20:39.830 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:39 GMT /wafmoa-4/ 2023-10-12T00:20:39.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MIE02eVrmsEKrO8hLV2LJZn2fA4= 2023-10-12T00:20:39.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MIE02eVrmsEKrO8hLV2LJZn2fA4=', 'Content-Length': '0'} 2023-10-12T00:20:39.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d60b37ce9cd0393b-0065273bd7-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:39.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.858 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:39.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:39.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:39.937 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:39.948 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:39.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.949 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:39.950 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:39.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.951 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:39.951 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:39.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:39.953 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:39.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:39.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:39.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070039.845356_93.1", 2023-10-12T00:20:39.954 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:39.845356Z" 2023-10-12T00:20:39.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:39.957 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:39.959 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:39.960 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:39.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:39.961 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:40.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:40.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:40.047+0000 7f55e3b2c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:40.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:40.048+0000 7f55e3b2c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:40.059 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070039.817543_86.1", 2023-10-12T00:20:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:39.817543Z", 2023-10-12T00:20:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:40Z" 2023-10-12T00:20:40.075 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:40.076 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070039.817543_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:39.817543Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:40Z" } 2023-10-12T00:20:40.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070039.817543_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:39.817543Z', 'realm_epoch': 2}}] 2023-10-12T00:20:40.077 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:40.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.817543_86.1'} 2023-10-12T00:20:40.078 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697070039.845356_93.1 target=1_1697070039.817543_86.1 2023-10-12T00:20:45.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.082 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:45.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.172+0000 7f8b2b8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.173+0000 7f8b2b8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.186 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:45.186 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:45.187 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:45.187 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:45.188 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:45.188 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:45.188 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.189 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:45.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.190 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:45.190 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.191 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.191 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:45.192 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.192 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:45.192 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.193 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:45.193 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:45.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.196 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.197 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.197 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.198 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.198 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.199 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.200 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:45.201 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.201 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.201 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.202 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.202 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.203 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.203 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.203 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.206 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:45.206 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.206 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070039.845356_93.1", 2023-10-12T00:20:45.207 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.207 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.208 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:39.845356Z", 2023-10-12T00:20:45.208 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.209 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:45Z" 2023-10-12T00:20:45.211 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:45.212 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070039.845356_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:39.845356Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:45Z" } 2023-10-12T00:20:45.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070039.845356_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:39.845356Z', 'realm_epoch': 2}}] 2023-10-12T00:20:45.213 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:45.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:45.214 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:45.215 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.215 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.216 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.216 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.216 DEBUG:boto:Data: 2023-10-12T00:20:45.217 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.218 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.218 DEBUG:boto:Params: {} 2023-10-12T00:20:45.219 DEBUG:boto:Token: None 2023-10-12T00:20:45.219 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea0b477d6cdba4fb-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.223 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.223 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.224 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.224 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.225 DEBUG:boto:Data: 2023-10-12T00:20:45.225 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.226 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.226 DEBUG:boto:Params: {} 2023-10-12T00:20:45.227 DEBUG:boto:Token: None 2023-10-12T00:20:45.227 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bccd27000e45567f-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.231 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.231 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.232 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.232 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.233 DEBUG:boto:Data: 2023-10-12T00:20:45.233 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.234 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.234 DEBUG:boto:Params: {} 2023-10-12T00:20:45.235 DEBUG:boto:Token: None 2023-10-12T00:20:45.237 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007590f497a911fb20-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.242 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.243 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.244 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.244 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.245 DEBUG:boto:Data: 2023-10-12T00:20:45.246 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.247 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.248 DEBUG:boto:Params: {} 2023-10-12T00:20:45.249 DEBUG:boto:Token: None 2023-10-12T00:20:45.249 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067c75ca28409a12a-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.254 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.254 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.255 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.255 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.256 DEBUG:boto:Data: 2023-10-12T00:20:45.256 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.257 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.258 DEBUG:boto:Params: {} 2023-10-12T00:20:45.259 DEBUG:boto:Token: None 2023-10-12T00:20:45.260 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QccetWXUMEDZo/T134snFaU1cWA= 2023-10-12T00:20:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QccetWXUMEDZo/T134snFaU1cWA=', 'Content-Length': '0'} 2023-10-12T00:20:45.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c8baa4db8868a12-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.266 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.266 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.267 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.268 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.268 DEBUG:boto:Data: 2023-10-12T00:20:45.269 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.270 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.271 DEBUG:boto:Params: {} 2023-10-12T00:20:45.272 DEBUG:boto:Token: None 2023-10-12T00:20:45.273 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1fFruyuTBY7wo8hyMuGJQEdN2Ow= 2023-10-12T00:20:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1fFruyuTBY7wo8hyMuGJQEdN2Ow=', 'Content-Length': '0'} 2023-10-12T00:20:45.277 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002b93a9b0a8850e83-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.277 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.278 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.279 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.279 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.280 DEBUG:boto:Data: 2023-10-12T00:20:45.280 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.281 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.281 DEBUG:boto:Params: {} 2023-10-12T00:20:45.282 DEBUG:boto:Token: None 2023-10-12T00:20:45.282 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QccetWXUMEDZo/T134snFaU1cWA= 2023-10-12T00:20:45.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QccetWXUMEDZo/T134snFaU1cWA=', 'Content-Length': '0'} 2023-10-12T00:20:45.292 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1682656f3bc0b40-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.292 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.293 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.293 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.294 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.294 DEBUG:boto:Data: 2023-10-12T00:20:45.294 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.295 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.296 DEBUG:boto:Params: {} 2023-10-12T00:20:45.296 DEBUG:boto:Token: None 2023-10-12T00:20:45.297 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1fFruyuTBY7wo8hyMuGJQEdN2Ow= 2023-10-12T00:20:45.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1fFruyuTBY7wo8hyMuGJQEdN2Ow=', 'Content-Length': '0'} 2023-10-12T00:20:45.344 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004de93ae5de900eff-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.345 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.345 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.345 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.346 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.346 DEBUG:boto:Data: 2023-10-12T00:20:45.347 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.348 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.348 DEBUG:boto:Params: {} 2023-10-12T00:20:45.348 DEBUG:boto:Token: None 2023-10-12T00:20:45.349 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005883134528c183fa-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.353 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.353 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.354 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.354 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.355 DEBUG:boto:Data: 2023-10-12T00:20:45.355 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.356 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.356 DEBUG:boto:Params: {} 2023-10-12T00:20:45.357 DEBUG:boto:Token: None 2023-10-12T00:20:45.358 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdc3ef159dff34cb-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.361 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.362 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.362 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.363 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.363 DEBUG:boto:Data: 2023-10-12T00:20:45.363 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.364 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.365 DEBUG:boto:Params: {} 2023-10-12T00:20:45.365 DEBUG:boto:Token: None 2023-10-12T00:20:45.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dde20ca4cfd38ac7-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.369 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.369 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.370 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.370 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.371 DEBUG:boto:Data: 2023-10-12T00:20:45.371 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.372 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.372 DEBUG:boto:Params: {} 2023-10-12T00:20:45.373 DEBUG:boto:Token: None 2023-10-12T00:20:45.373 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030032df80171e63b-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.378 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:45.426 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:45.456 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:45.467 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070039.845356_93.1", 2023-10-12T00:20:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:39.845356Z" 2023-10-12T00:20:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.471 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:45.472 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:45.473 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:45.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.474 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:45.524 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:45.525 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:45.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.550+0000 7fd50ce70780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.551+0000 7fd50ce70780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.562 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070039.845356_93.1", 2023-10-12T00:20:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:39.845356Z", 2023-10-12T00:20:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:45Z" 2023-10-12T00:20:45.577 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:45.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070039.845356_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:39.845356Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:45Z" } 2023-10-12T00:20:45.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070039.845356_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:39.845356Z', 'realm_epoch': 2}}] 2023-10-12T00:20:45.579 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:45.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070039.845356_93.1'} 2023-10-12T00:20:45.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:45.580 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.581 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.581 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.582 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.582 DEBUG:boto:Data: 2023-10-12T00:20:45.583 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.584 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.584 DEBUG:boto:Params: {} 2023-10-12T00:20:45.585 DEBUG:boto:Token: None 2023-10-12T00:20:45.585 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e52314ed2fdd10ad-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.589 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.589 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.590 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.590 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.591 DEBUG:boto:Data: 2023-10-12T00:20:45.591 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.592 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.592 DEBUG:boto:Params: {} 2023-10-12T00:20:45.593 DEBUG:boto:Token: None 2023-10-12T00:20:45.593 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a703ff25d2baa9d7-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.597 DEBUG:boto:path=/wafmoa-3/ 2023-10-12T00:20:45.598 DEBUG:boto:auth_path=/wafmoa-3/ 2023-10-12T00:20:45.598 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.598 DEBUG:boto:Path: /wafmoa-3/ 2023-10-12T00:20:45.599 DEBUG:boto:Data: 2023-10-12T00:20:45.599 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.600 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.600 DEBUG:boto:Params: {} 2023-10-12T00:20:45.601 DEBUG:boto:Token: None 2023-10-12T00:20:45.601 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-3/ 2023-10-12T00:20:45.602 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY= 2023-10-12T00:20:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQNIZDxubbutJsPBpm7Rmn/VVfY=', 'Content-Length': '0'} 2023-10-12T00:20:45.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008027e460f567b463-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.605 DEBUG:boto:path=/wafmoa-4/ 2023-10-12T00:20:45.605 DEBUG:boto:auth_path=/wafmoa-4/ 2023-10-12T00:20:45.606 DEBUG:boto:Method: HEAD 2023-10-12T00:20:45.607 DEBUG:boto:Path: /wafmoa-4/ 2023-10-12T00:20:45.607 DEBUG:boto:Data: 2023-10-12T00:20:45.607 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.608 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.609 DEBUG:boto:Params: {} 2023-10-12T00:20:45.609 DEBUG:boto:Token: None 2023-10-12T00:20:45.609 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-4/ 2023-10-12T00:20:45.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0= 2023-10-12T00:20:45.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2hu20espRwkLF2VaoQFi7//oOc0=', 'Content-Length': '0'} 2023-10-12T00:20:45.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f413645e7d84e9-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-10-12T00:20:45.614 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-5 2023-10-12T00:20:45.614 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:45.614 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:45.615 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.615 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:45.616 DEBUG:boto:Data: 2023-10-12T00:20:45.616 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:45.617 DEBUG:boto:Port: 8000 2023-10-12T00:20:45.617 DEBUG:boto:Params: {} 2023-10-12T00:20:45.618 DEBUG:boto:Token: None 2023-10-12T00:20:45.618 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-5/ 2023-10-12T00:20:45.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YZFFmeVt2EDGIYsYUaHOFNsB/xE= 2023-10-12T00:20:45.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YZFFmeVt2EDGIYsYUaHOFNsB/xE=', 'Content-Length': '0'} 2023-10-12T00:20:45.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7e1defd1c69ac1e-0065273bdd-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.635 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-6 2023-10-12T00:20:45.635 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:45.636 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:45.636 DEBUG:boto:Method: PUT 2023-10-12T00:20:45.636 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:45.637 DEBUG:boto:Data: 2023-10-12T00:20:45.637 DEBUG:boto:Headers: {} 2023-10-12T00:20:45.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:45.638 DEBUG:boto:Port: 8001 2023-10-12T00:20:45.639 DEBUG:boto:Params: {} 2023-10-12T00:20:45.639 DEBUG:boto:Token: None 2023-10-12T00:20:45.640 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:45 GMT /wafmoa-6/ 2023-10-12T00:20:45.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Eb6BJU87CANZeWDEy9jBUDxGfV4= 2023-10-12T00:20:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Eb6BJU87CANZeWDEy9jBUDxGfV4=', 'Content-Length': '0'} 2023-10-12T00:20:45.668 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000555e5b69760ec728-0065273bdd-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:45.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.670 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:45.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:45.719 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:45.747 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:45.758 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:45.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.758 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:45.759 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:45.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.759 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.760 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:45.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.760 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.760 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:45.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.761 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070045.656950_101.1", 2023-10-12T00:20:45.761 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:45.656950Z" 2023-10-12T00:20:45.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.762 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:45.763 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070045.656950_101.1'} 2023-10-12T00:20:45.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:45.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:45.765 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:45.813 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:45.813 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.842+0000 7fa2c1c82780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:45.843+0000 7fa2c1c82780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:45.854 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070045.626682_94.1", 2023-10-12T00:20:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:45.626682Z", 2023-10-12T00:20:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:45Z" 2023-10-12T00:20:45.878 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:45.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070045.626682_94.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:45.626682Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:45Z" } 2023-10-12T00:20:45.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070045.626682_94.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:45.626682Z', 'realm_epoch': 2}}] 2023-10-12T00:20:45.880 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070045.656950_101.1'} 2023-10-12T00:20:45.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070045.626682_94.1'} 2023-10-12T00:20:45.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697070045.656950_101.1 target=1_1697070045.626682_94.1 2023-10-12T00:20:50.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:50.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:50.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:50.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:50.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:50.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:50.970+0000 7f202cafd780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:50.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:50.972+0000 7f202cafd780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:50.986 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:50.987 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:50.989 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:50.990 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:50.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:50.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:50.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:50.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:50.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:50.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:50.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:50.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070045.656950_101.1", 2023-10-12T00:20:50.999 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:45.656950Z", 2023-10-12T00:20:51.000 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:51.002 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.003 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:50Z" 2023-10-12T00:20:51.003 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:51.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070045.656950_101.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:45.656950Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:50Z" } 2023-10-12T00:20:51.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070045.656950_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:45.656950Z', 'realm_epoch': 2}}] 2023-10-12T00:20:51.005 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070045.656950_101.1'} 2023-10-12T00:20:51.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070045.656950_101.1'} 2023-10-12T00:20:51.006 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:51.007 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:51.007 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:51.008 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.008 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:51.009 DEBUG:boto:Data: 2023-10-12T00:20:51.009 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.010 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.010 DEBUG:boto:Params: {} 2023-10-12T00:20:51.011 DEBUG:boto:Token: None 2023-10-12T00:20:51.012 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-5/ 2023-10-12T00:20:51.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo= 2023-10-12T00:20:51.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo=', 'Content-Length': '0'} 2023-10-12T00:20:51.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9ac3eb8cfe69eb6-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.017 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:51.017 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:51.018 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.018 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:51.019 DEBUG:boto:Data: 2023-10-12T00:20:51.019 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.020 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.020 DEBUG:boto:Params: {} 2023-10-12T00:20:51.021 DEBUG:boto:Token: None 2023-10-12T00:20:51.021 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-6/ 2023-10-12T00:20:51.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU= 2023-10-12T00:20:51.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU=', 'Content-Length': '0'} 2023-10-12T00:20:51.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000608cdbae7d233592-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.025 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:51.025 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:51.026 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.026 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:51.027 DEBUG:boto:Data: 2023-10-12T00:20:51.030 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.031 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.031 DEBUG:boto:Params: {} 2023-10-12T00:20:51.032 DEBUG:boto:Token: None 2023-10-12T00:20:51.032 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-5/ 2023-10-12T00:20:51.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo= 2023-10-12T00:20:51.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo=', 'Content-Length': '0'} 2023-10-12T00:20:51.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000829891a9d2a7b2e-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.036 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:51.037 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:51.037 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.037 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:51.038 DEBUG:boto:Data: 2023-10-12T00:20:51.038 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.039 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.039 DEBUG:boto:Params: {} 2023-10-12T00:20:51.040 DEBUG:boto:Token: None 2023-10-12T00:20:51.040 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-6/ 2023-10-12T00:20:51.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU= 2023-10-12T00:20:51.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU=', 'Content-Length': '0'} 2023-10-12T00:20:51.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d80e4122bf772b39-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.044 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:51.044 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:51.045 DEBUG:boto:Method: DELETE 2023-10-12T00:20:51.045 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:51.046 DEBUG:boto:Data: 2023-10-12T00:20:51.046 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.047 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.047 DEBUG:boto:Params: {} 2023-10-12T00:20:51.048 DEBUG:boto:Token: None 2023-10-12T00:20:51.048 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-5/ 2023-10-12T00:20:51.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V9WIOgVRFLY2VdI1QFn3fqX2IMQ= 2023-10-12T00:20:51.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V9WIOgVRFLY2VdI1QFn3fqX2IMQ=', 'Content-Length': '0'} 2023-10-12T00:20:51.076 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041727a90ffd0eaa4-0065273be3-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.076 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:51.077 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:51.077 DEBUG:boto:Method: DELETE 2023-10-12T00:20:51.078 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:51.078 DEBUG:boto:Data: 2023-10-12T00:20:51.078 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.079 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.079 DEBUG:boto:Params: {} 2023-10-12T00:20:51.080 DEBUG:boto:Token: None 2023-10-12T00:20:51.080 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-6/ 2023-10-12T00:20:51.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xzJ13GBn93gOlLM/q9Lo+BHuWhs= 2023-10-12T00:20:51.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xzJ13GBn93gOlLM/q9Lo+BHuWhs=', 'Content-Length': '0'} 2023-10-12T00:20:51.140 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077d6dcbb17114238-0065273be3-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.142 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:51.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:51.191 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:51.219 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:51.235 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:51.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.235 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:51.236 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:51.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.236 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.236 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:51.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.237 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.237 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:51.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070051.116801_109.1", 2023-10-12T00:20:51.238 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:51.116801Z" 2023-10-12T00:20:51.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.239 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:51.240 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.116801_109.1'} 2023-10-12T00:20:51.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:51.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.241 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:51.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:51.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:51.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:51.322+0000 7fdc3e71d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:51.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:51.323+0000 7fdc3e71d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:51.335 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:51.335 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:51.335 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:51.336 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:51.336 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:51.336 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:51.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.337 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:51.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:51.337 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.338 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:51.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:51.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.339 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:51.339 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.340 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:51.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.341 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.341 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.342 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:51.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.343 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:51.345 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.346 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:51.346 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.346 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.347 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070051.116801_109.1", 2023-10-12T00:20:51.347 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.347 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.347 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.347 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:51.116801Z", 2023-10-12T00:20:51.348 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:51.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.349 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:51.349 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:51.349 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:51.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.350 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:51Z" 2023-10-12T00:20:51.350 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:51.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070051.116801_109.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:51.116801Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:51Z" } 2023-10-12T00:20:51.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070051.116801_109.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:51.116801Z', 'realm_epoch': 2}}] 2023-10-12T00:20:51.352 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.116801_109.1'} 2023-10-12T00:20:51.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.116801_109.1'} 2023-10-12T00:20:51.353 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:51.353 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:51.354 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:51.354 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.354 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:51.355 DEBUG:boto:Data: 2023-10-12T00:20:51.355 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.356 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.356 DEBUG:boto:Params: {} 2023-10-12T00:20:51.357 DEBUG:boto:Token: None 2023-10-12T00:20:51.357 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-5/ 2023-10-12T00:20:51.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo= 2023-10-12T00:20:51.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo=', 'Content-Length': '0'} 2023-10-12T00:20:51.362 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000a5771465c3de8cc0-0065273be3-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.362 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:51.363 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:51.363 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.363 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:51.364 DEBUG:boto:Data: 2023-10-12T00:20:51.364 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.365 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.365 DEBUG:boto:Params: {} 2023-10-12T00:20:51.366 DEBUG:boto:Token: None 2023-10-12T00:20:51.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-6/ 2023-10-12T00:20:51.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU= 2023-10-12T00:20:51.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU=', 'Content-Length': '0'} 2023-10-12T00:20:51.368 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000f64ffaafa06a1404-0065273be3-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.369 DEBUG:boto:path=/wafmoa-5/ 2023-10-12T00:20:51.369 DEBUG:boto:auth_path=/wafmoa-5/ 2023-10-12T00:20:51.370 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.370 DEBUG:boto:Path: /wafmoa-5/ 2023-10-12T00:20:51.370 DEBUG:boto:Data: 2023-10-12T00:20:51.371 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.372 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.372 DEBUG:boto:Params: {} 2023-10-12T00:20:51.373 DEBUG:boto:Token: None 2023-10-12T00:20:51.373 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-5/ 2023-10-12T00:20:51.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo= 2023-10-12T00:20:51.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC24M+4HuCYhNl18DqcguzxnPSo=', 'Content-Length': '0'} 2023-10-12T00:20:51.376 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000002aede34ea4158f85-0065273be3-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.376 DEBUG:boto:path=/wafmoa-6/ 2023-10-12T00:20:51.377 DEBUG:boto:auth_path=/wafmoa-6/ 2023-10-12T00:20:51.377 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.377 DEBUG:boto:Path: /wafmoa-6/ 2023-10-12T00:20:51.378 DEBUG:boto:Data: 2023-10-12T00:20:51.378 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.379 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.379 DEBUG:boto:Params: {} 2023-10-12T00:20:51.380 DEBUG:boto:Token: None 2023-10-12T00:20:51.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-6/ 2023-10-12T00:20:51.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU= 2023-10-12T00:20:51.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i9vB/3GGstHs7wMUPFl64PnaKMU=', 'Content-Length': '0'} 2023-10-12T00:20:51.382 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000ed81376b46b98bae-0065273be3-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-10-12T00:20:51.384 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-7 2023-10-12T00:20:51.384 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:20:51.384 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:20:51.385 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.385 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:20:51.386 DEBUG:boto:Data: 2023-10-12T00:20:51.386 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.387 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.387 DEBUG:boto:Params: {} 2023-10-12T00:20:51.388 DEBUG:boto:Token: None 2023-10-12T00:20:51.388 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/ 2023-10-12T00:20:51.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wKuHHnF3UlfzZpFodZRKcsMNHrg= 2023-10-12T00:20:51.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wKuHHnF3UlfzZpFodZRKcsMNHrg=', 'Content-Length': '0'} 2023-10-12T00:20:51.405 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2a8677dd1a72f76-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.406 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-8 2023-10-12T00:20:51.406 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:20:51.406 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:20:51.407 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.407 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:20:51.408 DEBUG:boto:Data: 2023-10-12T00:20:51.408 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.409 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.409 DEBUG:boto:Params: {} 2023-10-12T00:20:51.410 DEBUG:boto:Token: None 2023-10-12T00:20:51.410 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/ 2023-10-12T00:20:51.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CvyGAUfDuG19v1IKagP7mblR2oM= 2023-10-12T00:20:51.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CvyGAUfDuG19v1IKagP7mblR2oM=', 'Content-Length': '0'} 2023-10-12T00:20:51.439 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005df3ccf5ea5c8813-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.440 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:20:51.441 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:20:51.441 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.441 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:20:51.442 DEBUG:boto:Data: 2023-10-12T00:20:51.442 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.444 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.444 DEBUG:boto:Params: {} 2023-10-12T00:20:51.445 DEBUG:boto:Token: None 2023-10-12T00:20:51.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/ 2023-10-12T00:20:51.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk= 2023-10-12T00:20:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk=', 'Content-Length': '0'} 2023-10-12T00:20:51.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1e21e5a40e82b60-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.452 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:20:51.489 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:20:51.489 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.490 DEBUG:boto:Path: /wafmoa-7/myobj 2023-10-12T00:20:51.490 DEBUG:boto:Data: 2023-10-12T00:20:51.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.491 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.492 DEBUG:boto:Params: {} 2023-10-12T00:20:51.492 DEBUG:boto:Token: None 2023-10-12T00:20:51.493 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/myobj 2023-10-12T00:20:51.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k1DHulaQZKBERulpBBBeK4mwU7U= 2023-10-12T00:20:51.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k1DHulaQZKBERulpBBBeK4mwU7U='} 2023-10-12T00:20:51.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000599326cfb1670a9a-0065273be3-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.503 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:20:51.504 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:20:51.504 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.504 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:20:51.505 DEBUG:boto:Data: 2023-10-12T00:20:51.505 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.506 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.506 DEBUG:boto:Params: {} 2023-10-12T00:20:51.507 DEBUG:boto:Token: None 2023-10-12T00:20:51.508 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/ 2023-10-12T00:20:51.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk= 2023-10-12T00:20:51.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk=', 'Content-Length': '0'} 2023-10-12T00:20:51.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013d8e9803c3096a7-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.512 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:20:51.512 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:20:51.513 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.513 DEBUG:boto:Path: /wafmoa-7/_myobj 2023-10-12T00:20:51.513 DEBUG:boto:Data: 2023-10-12T00:20:51.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.515 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.515 DEBUG:boto:Params: {} 2023-10-12T00:20:51.515 DEBUG:boto:Token: None 2023-10-12T00:20:51.516 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/_myobj 2023-10-12T00:20:51.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lgDmsc4E1d02u3GoS5+57E0cTL0= 2023-10-12T00:20:51.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lgDmsc4E1d02u3GoS5+57E0cTL0='} 2023-10-12T00:20:51.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3f58b100b2617fe-0065273be3-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.526 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:20:51.527 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:20:51.527 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.528 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:20:51.528 DEBUG:boto:Data: 2023-10-12T00:20:51.528 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.529 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.529 DEBUG:boto:Params: {} 2023-10-12T00:20:51.530 DEBUG:boto:Token: None 2023-10-12T00:20:51.530 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/ 2023-10-12T00:20:51.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk= 2023-10-12T00:20:51.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk=', 'Content-Length': '0'} 2023-10-12T00:20:51.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005830035e0f9b7439-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.535 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:20:51.535 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:20:51.535 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.536 DEBUG:boto:Path: /wafmoa-7/%3A 2023-10-12T00:20:51.536 DEBUG:boto:Data: 2023-10-12T00:20:51.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.539 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.539 DEBUG:boto:Params: {} 2023-10-12T00:20:51.540 DEBUG:boto:Token: None 2023-10-12T00:20:51.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/%3A 2023-10-12T00:20:51.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MQmRi7gvUm7TSeJ/blQ7NhYJgGU= 2023-10-12T00:20:51.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MQmRi7gvUm7TSeJ/blQ7NhYJgGU='} 2023-10-12T00:20:51.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000969f0eb3ee82c212-0065273be3-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.587 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:20:51.588 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:20:51.588 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.589 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:20:51.589 DEBUG:boto:Data: 2023-10-12T00:20:51.589 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.590 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.590 DEBUG:boto:Params: {} 2023-10-12T00:20:51.591 DEBUG:boto:Token: None 2023-10-12T00:20:51.591 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/ 2023-10-12T00:20:51.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk= 2023-10-12T00:20:51.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:06CdzFVOXY0Cp95kl4odPgQa8Qk=', 'Content-Length': '0'} 2023-10-12T00:20:51.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4378a9783fc5c56-0065273be3-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.595 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:20:51.595 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:20:51.596 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.596 DEBUG:boto:Path: /wafmoa-7/%26 2023-10-12T00:20:51.596 DEBUG:boto:Data: 2023-10-12T00:20:51.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:20:51.598 DEBUG:boto:Port: 8000 2023-10-12T00:20:51.598 DEBUG:boto:Params: {} 2023-10-12T00:20:51.599 DEBUG:boto:Token: None 2023-10-12T00:20:51.599 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-7/%26 2023-10-12T00:20:51.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/W3dKTXsBZsFpzznYFNSzfXUEmI= 2023-10-12T00:20:51.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/W3dKTXsBZsFpzznYFNSzfXUEmI='} 2023-10-12T00:20:51.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8dddf21995402b5-0065273be3-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.606 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:20:51.606 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:20:51.606 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.607 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:20:51.607 DEBUG:boto:Data: 2023-10-12T00:20:51.608 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.608 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.609 DEBUG:boto:Params: {} 2023-10-12T00:20:51.609 DEBUG:boto:Token: None 2023-10-12T00:20:51.610 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/ 2023-10-12T00:20:51.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8= 2023-10-12T00:20:51.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8=', 'Content-Length': '0'} 2023-10-12T00:20:51.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e86738f4000475a-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.614 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:20:51.615 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:20:51.615 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.615 DEBUG:boto:Path: /wafmoa-8/myobj 2023-10-12T00:20:51.616 DEBUG:boto:Data: 2023-10-12T00:20:51.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.617 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.617 DEBUG:boto:Params: {} 2023-10-12T00:20:51.618 DEBUG:boto:Token: None 2023-10-12T00:20:51.619 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/myobj 2023-10-12T00:20:51.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0ngPngFOZVDoS/3IvvRdkXaCFjk= 2023-10-12T00:20:51.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0ngPngFOZVDoS/3IvvRdkXaCFjk='} 2023-10-12T00:20:51.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e81bbeb1ca9db406-0065273be3-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.631 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:20:51.632 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:20:51.632 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.632 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:20:51.633 DEBUG:boto:Data: 2023-10-12T00:20:51.633 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.634 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.634 DEBUG:boto:Params: {} 2023-10-12T00:20:51.635 DEBUG:boto:Token: None 2023-10-12T00:20:51.635 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/ 2023-10-12T00:20:51.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8= 2023-10-12T00:20:51.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8=', 'Content-Length': '0'} 2023-10-12T00:20:51.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000581f20276674e2e7-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.639 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:20:51.639 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:20:51.640 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.640 DEBUG:boto:Path: /wafmoa-8/_myobj 2023-10-12T00:20:51.641 DEBUG:boto:Data: 2023-10-12T00:20:51.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.642 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.642 DEBUG:boto:Params: {} 2023-10-12T00:20:51.645 DEBUG:boto:Token: None 2023-10-12T00:20:51.646 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/_myobj 2023-10-12T00:20:51.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bh+/OcNtyasr1lestts5DAaJpt0= 2023-10-12T00:20:51.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bh+/OcNtyasr1lestts5DAaJpt0='} 2023-10-12T00:20:51.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006756e70bd75b5e10-0065273be3-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.691 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:20:51.692 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:20:51.692 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.692 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:20:51.693 DEBUG:boto:Data: 2023-10-12T00:20:51.693 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.694 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.694 DEBUG:boto:Params: {} 2023-10-12T00:20:51.695 DEBUG:boto:Token: None 2023-10-12T00:20:51.695 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/ 2023-10-12T00:20:51.696 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8= 2023-10-12T00:20:51.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8=', 'Content-Length': '0'} 2023-10-12T00:20:51.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc1d8a79fe694369-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.699 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:20:51.700 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:20:51.700 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.701 DEBUG:boto:Path: /wafmoa-8/%3A 2023-10-12T00:20:51.701 DEBUG:boto:Data: 2023-10-12T00:20:51.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.702 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.703 DEBUG:boto:Params: {} 2023-10-12T00:20:51.703 DEBUG:boto:Token: None 2023-10-12T00:20:51.704 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/%3A 2023-10-12T00:20:51.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:If18Zv9G6LkkPnei30zlMGpmesI= 2023-10-12T00:20:51.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:If18Zv9G6LkkPnei30zlMGpmesI='} 2023-10-12T00:20:51.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000690aa8e0d1816661-0065273be3-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.718 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:20:51.719 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:20:51.719 DEBUG:boto:Method: HEAD 2023-10-12T00:20:51.720 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:20:51.720 DEBUG:boto:Data: 2023-10-12T00:20:51.720 DEBUG:boto:Headers: {} 2023-10-12T00:20:51.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.721 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.722 DEBUG:boto:Params: {} 2023-10-12T00:20:51.722 DEBUG:boto:Token: None 2023-10-12T00:20:51.723 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/ 2023-10-12T00:20:51.723 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8= 2023-10-12T00:20:51.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XoRXxwdk9KOvGI2kVsUqyVMLXr8=', 'Content-Length': '0'} 2023-10-12T00:20:51.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042e027db89322324-0065273be3-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.726 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:20:51.726 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:20:51.727 DEBUG:boto:Method: PUT 2023-10-12T00:20:51.727 DEBUG:boto:Path: /wafmoa-8/%26 2023-10-12T00:20:51.728 DEBUG:boto:Data: 2023-10-12T00:20:51.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:20:51.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:20:51.729 DEBUG:boto:Port: 8001 2023-10-12T00:20:51.729 DEBUG:boto:Params: {} 2023-10-12T00:20:51.730 DEBUG:boto:Token: None 2023-10-12T00:20:51.730 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:20:51 GMT /wafmoa-8/%26 2023-10-12T00:20:51.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n0jZH3HHlstyXMKwh+rN9QHn0mA= 2023-10-12T00:20:51.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:20:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n0jZH3HHlstyXMKwh+rN9QHn0mA='} 2023-10-12T00:20:51.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002eae10eb10f9d81b-0065273be3-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:20:51.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.780 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:51.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:51.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:51.861 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:20:51.871 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:20:51.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.872 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:51.872 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:20:51.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.873 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.873 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:51.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.874 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:20:51.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.875 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070051.423750_117.1", 2023-10-12T00:20:51.875 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:20:51.423750Z" 2023-10-12T00:20:51.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.876 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:20:51.877 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.423750_117.1'} 2023-10-12T00:20:51.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:20:51.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:20:51.879 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:20:51.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:51.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:51.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:51.955+0000 7f9fb3ff2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:51.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:51.956+0000 7f9fb3ff2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:20:51.967 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:20:51.968 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:20:51.968 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:20:51.968 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:20:51.968 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:20:51.969 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:20:51.969 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.969 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:20:51.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:20:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:20:51.970 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:20:51.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.971 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:20:51.971 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.972 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:20:51.973 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.973 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.973 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.974 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.974 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:51.974 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:20:51.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.975 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.975 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:20:51.976 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.976 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.976 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:20:51.976 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:20:51.977 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:20:51.977 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:20:51.978 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:20:51.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070051.423750_117.1", 2023-10-12T00:20:51.979 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:20:51.979 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:20:51.979 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:20:51.979 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:20:51.423750Z", 2023-10-12T00:20:51.980 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:20:51.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:20:51.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:20:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:20:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:20:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:20:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:20:51.982 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:20:51Z" 2023-10-12T00:20:51.982 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:20:51.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070051.423750_117.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:20:51.423750Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:20:51Z" } 2023-10-12T00:20:51.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070051.423750_117.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:20:51.423750Z', 'realm_epoch': 2}}] 2023-10-12T00:20:51.984 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.423750_117.1'} 2023-10-12T00:20:51.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '', 3: '1_1697070051.423750_117.1'} 2023-10-12T00:20:51.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:20:51.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:20:51.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:20:51.986 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:20:52.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:20:52.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:20:52.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:52.074+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:20:57.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:20:57.076+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:02.076+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:07.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:07.077+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:12.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:12.076+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:17.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:17.077+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:22.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:22.078+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:27.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:27.079+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:32.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:32.079+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:37.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:37.081+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:42.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:42.081+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:47.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:47.082+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:52.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:52.081+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:21:57.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:21:57.082+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:02.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:02.082+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:07.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:07.083+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:12.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:12.084+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:17.085+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:22.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:22.086+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:27.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:27.087+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:32.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:32.086+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:37.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:37.087+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:42.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:42.087+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:47.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:47.088+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:52.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:52.090+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:22:57.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:22:57.090+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:23:02.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:02.091+0000 7f8148a8a780 1 waiting to reach incremental sync.. 2023-10-12T00:23:07.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:07.091+0000 7f8148a8a780 1 bucket sync caught up with source: 2023-10-12T00:23:07.093 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.3.6, , 00000000001.2.6, , , , 00000000001.6.6, 00000000001.6.6, , ] 2023-10-12T00:23:07.093 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.3.6, , 00000000001.2.6, , , , 00000000001.6.6, 00000000001.6.6, , ] 2023-10-12T00:23:07.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:07.091+0000 7f8148a8a780 0 bucket checkpoint complete 2023-10-12T00:23:07.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-7 zones={a2, a1} 2023-10-12T00:23:07.109 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.109 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.110 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.110 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:23:07.111 DEBUG:boto:Data: 2023-10-12T00:23:07.111 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.112 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.112 DEBUG:boto:Params: {} 2023-10-12T00:23:07.113 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:23:07.113 DEBUG:boto:Token: None 2023-10-12T00:23:07.114 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.115 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dyW1gWCddAjLTO5nTZD62DOC//Y= 2023-10-12T00:23:07.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dyW1gWCddAjLTO5nTZD62DOC//Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007084f8fc11daa880-0065273c6b-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.120 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.120 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.120 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.121 DEBUG:boto:Path: /wafmoa-7/ 2023-10-12T00:23:07.121 DEBUG:boto:Data: 2023-10-12T00:23:07.122 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.123 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.123 DEBUG:boto:Params: {} 2023-10-12T00:23:07.123 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:23:07.124 DEBUG:boto:Token: None 2023-10-12T00:23:07.124 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dyW1gWCddAjLTO5nTZD62DOC//Y= 2023-10-12T00:23:07.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dyW1gWCddAjLTO5nTZD62DOC//Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084c95d35ece9e75b-0065273c6b-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.135 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:23:07.136 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.167 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.168 DEBUG:boto:path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.169 DEBUG:boto:auth_path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.169 DEBUG:boto:Method: GET 2023-10-12T00:23:07.170 DEBUG:boto:Path: /wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.171 DEBUG:boto:Data: 2023-10-12T00:23:07.172 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.173 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.174 DEBUG:boto:Params: {} 2023-10-12T00:23:07.175 DEBUG:boto:Token: None 2023-10-12T00:23:07.176 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/?versions 2023-10-12T00:23:07.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI= 2023-10-12T00:23:07.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI=', 'Content-Length': '0'} 2023-10-12T00:23:07.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009918aaf5e09fb5b0-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.181 DEBUG:boto:b'wafmoa-7999false&nulltrue2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.250 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T00:23:07.250 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T00:23:07.251 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T00:23:07.251 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:07.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:23:07.252 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.252 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.253 DEBUG:boto:path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.253 DEBUG:boto:auth_path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.254 DEBUG:boto:Method: GET 2023-10-12T00:23:07.254 DEBUG:boto:Path: /wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.257 DEBUG:boto:Data: 2023-10-12T00:23:07.258 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.259 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.259 DEBUG:boto:Params: {} 2023-10-12T00:23:07.260 DEBUG:boto:Token: None 2023-10-12T00:23:07.260 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/?versions 2023-10-12T00:23:07.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI= 2023-10-12T00:23:07.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI=', 'Content-Length': '0'} 2023-10-12T00:23:07.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076a5d7f6a9759458-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.265 DEBUG:boto:b'wafmoa-7999false&nulltrue2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.266 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T00:23:07.266 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T00:23:07.266 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T00:23:07.267 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:07.267 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.268 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.268 DEBUG:boto:path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.268 DEBUG:boto:auth_path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.269 DEBUG:boto:Method: GET 2023-10-12T00:23:07.269 DEBUG:boto:Path: /wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.270 DEBUG:boto:Data: 2023-10-12T00:23:07.270 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.271 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.271 DEBUG:boto:Params: {} 2023-10-12T00:23:07.272 DEBUG:boto:Token: None 2023-10-12T00:23:07.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/?versions 2023-10-12T00:23:07.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI= 2023-10-12T00:23:07.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI=', 'Content-Length': '0'} 2023-10-12T00:23:07.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f34f5151ed0d20-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.276 DEBUG:boto:b'wafmoa-7999false&nulltrue2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.277 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.278 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.278 DEBUG:boto:path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.278 DEBUG:boto:auth_path=/wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.279 DEBUG:boto:Method: GET 2023-10-12T00:23:07.279 DEBUG:boto:Path: /wafmoa-7/?versions&max-keys=999 2023-10-12T00:23:07.280 DEBUG:boto:Data: 2023-10-12T00:23:07.280 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.281 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.281 DEBUG:boto:Params: {} 2023-10-12T00:23:07.282 DEBUG:boto:Token: None 2023-10-12T00:23:07.282 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/?versions 2023-10-12T00:23:07.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI= 2023-10-12T00:23:07.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tm8LhIo63hXLzvXdjLDNe60RhI=', 'Content-Length': '0'} 2023-10-12T00:23:07.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e76d5157d63d094f-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.287 DEBUG:boto:b'wafmoa-7999false&nulltrue2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:07.288 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.288 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.289 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.289 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.290 DEBUG:boto:Method: GET 2023-10-12T00:23:07.290 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.291 DEBUG:boto:Data: 2023-10-12T00:23:07.291 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.292 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.292 DEBUG:boto:Params: {} 2023-10-12T00:23:07.293 DEBUG:boto:Token: None 2023-10-12T00:23:07.293 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.294 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4= 2023-10-12T00:23:07.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4=', 'Content-Length': '0'} 2023-10-12T00:23:07.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0144d0be94e7d55-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.297 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.297 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.298 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.298 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.299 DEBUG:boto:Method: GET 2023-10-12T00:23:07.299 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.300 DEBUG:boto:Data: 2023-10-12T00:23:07.300 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.301 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.301 DEBUG:boto:Params: {} 2023-10-12T00:23:07.302 DEBUG:boto:Token: None 2023-10-12T00:23:07.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4= 2023-10-12T00:23:07.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4=', 'Content-Length': '0'} 2023-10-12T00:23:07.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071586094fcf9e693-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.306 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.306 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.307 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.307 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.308 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.308 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.308 DEBUG:boto:Data: 2023-10-12T00:23:07.309 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.310 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.310 DEBUG:boto:Params: {} 2023-10-12T00:23:07.310 DEBUG:boto:Token: None 2023-10-12T00:23:07.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQ3ESKLwQVTqsJSRnn3K0qwvHCo= 2023-10-12T00:23:07.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQ3ESKLwQVTqsJSRnn3K0qwvHCo=', 'Content-Length': '0'} 2023-10-12T00:23:07.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8373d5c04609abd-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.315 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.315 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.315 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.316 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.316 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.317 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.317 DEBUG:boto:Data: 2023-10-12T00:23:07.317 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.318 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.319 DEBUG:boto:Params: {} 2023-10-12T00:23:07.319 DEBUG:boto:Token: None 2023-10-12T00:23:07.320 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.320 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SQ3ESKLwQVTqsJSRnn3K0qwvHCo= 2023-10-12T00:23:07.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SQ3ESKLwQVTqsJSRnn3K0qwvHCo=', 'Content-Length': '0'} 2023-10-12T00:23:07.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b37245a858b02176-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:07.323 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.324 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.324 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.325 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.325 DEBUG:boto:Method: GET 2023-10-12T00:23:07.325 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.326 DEBUG:boto:Data: 2023-10-12T00:23:07.326 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.327 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.328 DEBUG:boto:Params: {} 2023-10-12T00:23:07.328 DEBUG:boto:Token: None 2023-10-12T00:23:07.329 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4= 2023-10-12T00:23:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4=', 'Content-Length': '0'} 2023-10-12T00:23:07.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002629980950d3380a-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.332 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.333 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.333 DEBUG:boto:path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.334 DEBUG:boto:auth_path=/wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.334 DEBUG:boto:Method: GET 2023-10-12T00:23:07.334 DEBUG:boto:Path: /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.335 DEBUG:boto:Data: 2023-10-12T00:23:07.335 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.336 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.337 DEBUG:boto:Params: {} 2023-10-12T00:23:07.337 DEBUG:boto:Token: None 2023-10-12T00:23:07.338 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26?versionId=null 2023-10-12T00:23:07.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4= 2023-10-12T00:23:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rKSF06m81rGovAyQcTy7S0eYUn4=', 'Content-Length': '0'} 2023-10-12T00:23:07.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e6592e5e3aa735-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.342 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.342 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.343 DEBUG:boto:path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.343 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.344 DEBUG:boto:Method: GET 2023-10-12T00:23:07.344 DEBUG:boto:Path: /wafmoa-7/?prefix=%26 2023-10-12T00:23:07.344 DEBUG:boto:Data: 2023-10-12T00:23:07.345 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.346 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.346 DEBUG:boto:Params: {} 2023-10-12T00:23:07.347 DEBUG:boto:Token: None 2023-10-12T00:23:07.348 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000456ae906c626fd2d-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.352 DEBUG:boto:b'wafmoa-7&1000false&2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.352 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.353 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.353 DEBUG:boto:path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.354 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.354 DEBUG:boto:Method: GET 2023-10-12T00:23:07.354 DEBUG:boto:Path: /wafmoa-7/?prefix=%26 2023-10-12T00:23:07.355 DEBUG:boto:Data: 2023-10-12T00:23:07.355 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.356 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.357 DEBUG:boto:Params: {} 2023-10-12T00:23:07.357 DEBUG:boto:Token: None 2023-10-12T00:23:07.358 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac4faf1a15a5a973-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.362 DEBUG:boto:b'wafmoa-7&1000false&2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.362 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.363 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.363 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.363 DEBUG:boto:Path: /wafmoa-7/%26 2023-10-12T00:23:07.364 DEBUG:boto:Data: 2023-10-12T00:23:07.364 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.365 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.365 DEBUG:boto:Params: {} 2023-10-12T00:23:07.366 DEBUG:boto:Token: None 2023-10-12T00:23:07.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26 2023-10-12T00:23:07.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AAs12V00z5kIQTt5VykzpkOdogA= 2023-10-12T00:23:07.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AAs12V00z5kIQTt5VykzpkOdogA=', 'Content-Length': '0'} 2023-10-12T00:23:07.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067532bf76d8c0e45-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.370 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.370 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.370 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.371 DEBUG:boto:Path: /wafmoa-7/%26 2023-10-12T00:23:07.371 DEBUG:boto:Data: 2023-10-12T00:23:07.372 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.372 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.373 DEBUG:boto:Params: {} 2023-10-12T00:23:07.373 DEBUG:boto:Token: None 2023-10-12T00:23:07.374 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26 2023-10-12T00:23:07.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AAs12V00z5kIQTt5VykzpkOdogA= 2023-10-12T00:23:07.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AAs12V00z5kIQTt5VykzpkOdogA=', 'Content-Length': '0'} 2023-10-12T00:23:07.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000040d5a0580ed1be5-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:07.378 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.378 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.379 DEBUG:boto:Method: GET 2023-10-12T00:23:07.379 DEBUG:boto:Path: /wafmoa-7/%26 2023-10-12T00:23:07.380 DEBUG:boto:Data: 2023-10-12T00:23:07.380 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.381 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.382 DEBUG:boto:Params: {} 2023-10-12T00:23:07.382 DEBUG:boto:Token: None 2023-10-12T00:23:07.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26 2023-10-12T00:23:07.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OXL9szq1XUoSLagqFhmIvSgp9Wk= 2023-10-12T00:23:07.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXL9szq1XUoSLagqFhmIvSgp9Wk=', 'Content-Length': '0'} 2023-10-12T00:23:07.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d8bf52e88ed0114-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.386 DEBUG:boto:path=/wafmoa-7/%26 2023-10-12T00:23:07.387 DEBUG:boto:auth_path=/wafmoa-7/%26 2023-10-12T00:23:07.387 DEBUG:boto:Method: GET 2023-10-12T00:23:07.388 DEBUG:boto:Path: /wafmoa-7/%26 2023-10-12T00:23:07.388 DEBUG:boto:Data: 2023-10-12T00:23:07.388 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.389 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.390 DEBUG:boto:Params: {} 2023-10-12T00:23:07.390 DEBUG:boto:Token: None 2023-10-12T00:23:07.391 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%26 2023-10-12T00:23:07.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OXL9szq1XUoSLagqFhmIvSgp9Wk= 2023-10-12T00:23:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXL9szq1XUoSLagqFhmIvSgp9Wk=', 'Content-Length': '0'} 2023-10-12T00:23:07.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000192883b4011b897c-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.394 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.395 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.395 DEBUG:boto:path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.396 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.396 DEBUG:boto:Method: GET 2023-10-12T00:23:07.397 DEBUG:boto:Path: /wafmoa-7/?prefix=%26 2023-10-12T00:23:07.397 DEBUG:boto:Data: 2023-10-12T00:23:07.397 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.398 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.399 DEBUG:boto:Params: {} 2023-10-12T00:23:07.399 DEBUG:boto:Token: None 2023-10-12T00:23:07.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054999b068ebc80c0-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.404 DEBUG:boto:b'wafmoa-7&1000false&2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.405 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.405 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.406 DEBUG:boto:path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.406 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%26 2023-10-12T00:23:07.406 DEBUG:boto:Method: GET 2023-10-12T00:23:07.407 DEBUG:boto:Path: /wafmoa-7/?prefix=%26 2023-10-12T00:23:07.407 DEBUG:boto:Data: 2023-10-12T00:23:07.408 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.409 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.409 DEBUG:boto:Params: {} 2023-10-12T00:23:07.409 DEBUG:boto:Token: None 2023-10-12T00:23:07.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc2fdbc56372cfe-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.414 DEBUG:boto:b'wafmoa-7&1000false&2023-10-12T00:20:51.603Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:07.415 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.416 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.416 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.416 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.417 DEBUG:boto:Method: GET 2023-10-12T00:23:07.417 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.418 DEBUG:boto:Data: 2023-10-12T00:23:07.418 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.419 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.420 DEBUG:boto:Params: {} 2023-10-12T00:23:07.420 DEBUG:boto:Token: None 2023-10-12T00:23:07.421 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y= 2023-10-12T00:23:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034ec5137b9dfdbc5-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.424 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.425 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.425 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.426 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.426 DEBUG:boto:Method: GET 2023-10-12T00:23:07.426 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.427 DEBUG:boto:Data: 2023-10-12T00:23:07.427 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.428 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.429 DEBUG:boto:Params: {} 2023-10-12T00:23:07.429 DEBUG:boto:Token: None 2023-10-12T00:23:07.430 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.430 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y= 2023-10-12T00:23:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbe2bd8b568bfabc-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.433 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.434 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.434 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.435 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.435 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.436 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.436 DEBUG:boto:Data: 2023-10-12T00:23:07.436 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.437 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.438 DEBUG:boto:Params: {} 2023-10-12T00:23:07.438 DEBUG:boto:Token: None 2023-10-12T00:23:07.439 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oIS29JYOsKXAgV7bQZdTBsgEO2A= 2023-10-12T00:23:07.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oIS29JYOsKXAgV7bQZdTBsgEO2A=', 'Content-Length': '0'} 2023-10-12T00:23:07.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5ed8d80c6170427-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.442 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.443 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.443 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.443 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.444 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.444 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.445 DEBUG:boto:Data: 2023-10-12T00:23:07.445 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.446 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.446 DEBUG:boto:Params: {} 2023-10-12T00:23:07.447 DEBUG:boto:Token: None 2023-10-12T00:23:07.447 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oIS29JYOsKXAgV7bQZdTBsgEO2A= 2023-10-12T00:23:07.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oIS29JYOsKXAgV7bQZdTBsgEO2A=', 'Content-Length': '0'} 2023-10-12T00:23:07.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073bd92b576e107ed-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:07.451 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.452 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.452 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.452 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.453 DEBUG:boto:Method: GET 2023-10-12T00:23:07.453 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.454 DEBUG:boto:Data: 2023-10-12T00:23:07.454 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.455 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.456 DEBUG:boto:Params: {} 2023-10-12T00:23:07.456 DEBUG:boto:Token: None 2023-10-12T00:23:07.457 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y= 2023-10-12T00:23:07.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dabca70fc408eb7b-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.462 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.462 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.463 DEBUG:boto:path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.463 DEBUG:boto:auth_path=/wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.464 DEBUG:boto:Method: GET 2023-10-12T00:23:07.464 DEBUG:boto:Path: /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.464 DEBUG:boto:Data: 2023-10-12T00:23:07.465 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.466 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.466 DEBUG:boto:Params: {} 2023-10-12T00:23:07.467 DEBUG:boto:Token: None 2023-10-12T00:23:07.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A?versionId=null 2023-10-12T00:23:07.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y= 2023-10-12T00:23:07.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezKLR6bBI8XI4GSRyEBUiXFm74Y=', 'Content-Length': '0'} 2023-10-12T00:23:07.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d5469f78ca19df5-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.471 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.471 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.472 DEBUG:boto:path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.472 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.473 DEBUG:boto:Method: GET 2023-10-12T00:23:07.473 DEBUG:boto:Path: /wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.474 DEBUG:boto:Data: 2023-10-12T00:23:07.474 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.475 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.475 DEBUG:boto:Params: {} 2023-10-12T00:23:07.476 DEBUG:boto:Token: None 2023-10-12T00:23:07.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030520c17f3e87bcc-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.480 DEBUG:boto:b'wafmoa-7:1000false:2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.481 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.481 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.482 DEBUG:boto:path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.482 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.483 DEBUG:boto:Method: GET 2023-10-12T00:23:07.483 DEBUG:boto:Path: /wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.483 DEBUG:boto:Data: 2023-10-12T00:23:07.484 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.485 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.485 DEBUG:boto:Params: {} 2023-10-12T00:23:07.486 DEBUG:boto:Token: None 2023-10-12T00:23:07.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.487 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a77d58778fe389-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.490 DEBUG:boto:b'wafmoa-7:1000false:2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.491 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.492 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.492 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.493 DEBUG:boto:Path: /wafmoa-7/%3A 2023-10-12T00:23:07.493 DEBUG:boto:Data: 2023-10-12T00:23:07.493 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.494 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.495 DEBUG:boto:Params: {} 2023-10-12T00:23:07.495 DEBUG:boto:Token: None 2023-10-12T00:23:07.496 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A 2023-10-12T00:23:07.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:slQoz4U4KPSxml21/EkEXzyYizc= 2023-10-12T00:23:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:slQoz4U4KPSxml21/EkEXzyYizc=', 'Content-Length': '0'} 2023-10-12T00:23:07.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029f1821635618aad-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.499 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.500 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.500 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.501 DEBUG:boto:Path: /wafmoa-7/%3A 2023-10-12T00:23:07.501 DEBUG:boto:Data: 2023-10-12T00:23:07.502 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.502 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.503 DEBUG:boto:Params: {} 2023-10-12T00:23:07.503 DEBUG:boto:Token: None 2023-10-12T00:23:07.504 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A 2023-10-12T00:23:07.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:slQoz4U4KPSxml21/EkEXzyYizc= 2023-10-12T00:23:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:slQoz4U4KPSxml21/EkEXzyYizc=', 'Content-Length': '0'} 2023-10-12T00:23:07.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093dbfcfcd00efb57-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:07.508 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.508 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.509 DEBUG:boto:Method: GET 2023-10-12T00:23:07.509 DEBUG:boto:Path: /wafmoa-7/%3A 2023-10-12T00:23:07.510 DEBUG:boto:Data: 2023-10-12T00:23:07.510 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.511 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.511 DEBUG:boto:Params: {} 2023-10-12T00:23:07.512 DEBUG:boto:Token: None 2023-10-12T00:23:07.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A 2023-10-12T00:23:07.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QPdTV/VqwgNT9iA+ydLTxPWz9mc= 2023-10-12T00:23:07.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QPdTV/VqwgNT9iA+ydLTxPWz9mc=', 'Content-Length': '0'} 2023-10-12T00:23:07.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049d007dfca2af769-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.562 DEBUG:boto:path=/wafmoa-7/%3A 2023-10-12T00:23:07.563 DEBUG:boto:auth_path=/wafmoa-7/%3A 2023-10-12T00:23:07.563 DEBUG:boto:Method: GET 2023-10-12T00:23:07.564 DEBUG:boto:Path: /wafmoa-7/%3A 2023-10-12T00:23:07.564 DEBUG:boto:Data: 2023-10-12T00:23:07.565 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.566 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.566 DEBUG:boto:Params: {} 2023-10-12T00:23:07.567 DEBUG:boto:Token: None 2023-10-12T00:23:07.567 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/%3A 2023-10-12T00:23:07.568 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QPdTV/VqwgNT9iA+ydLTxPWz9mc= 2023-10-12T00:23:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QPdTV/VqwgNT9iA+ydLTxPWz9mc=', 'Content-Length': '0'} 2023-10-12T00:23:07.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ed76cef27046d04-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.571 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.572 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.572 DEBUG:boto:path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.572 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.573 DEBUG:boto:Method: GET 2023-10-12T00:23:07.573 DEBUG:boto:Path: /wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.574 DEBUG:boto:Data: 2023-10-12T00:23:07.574 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.575 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.576 DEBUG:boto:Params: {} 2023-10-12T00:23:07.576 DEBUG:boto:Token: None 2023-10-12T00:23:07.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d84d25f200c1e72-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.582 DEBUG:boto:b'wafmoa-7:1000false:2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.583 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.583 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.584 DEBUG:boto:path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.584 DEBUG:boto:auth_path=/wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.584 DEBUG:boto:Method: GET 2023-10-12T00:23:07.585 DEBUG:boto:Path: /wafmoa-7/?prefix=%3A 2023-10-12T00:23:07.585 DEBUG:boto:Data: 2023-10-12T00:23:07.586 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.587 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.587 DEBUG:boto:Params: {} 2023-10-12T00:23:07.587 DEBUG:boto:Token: None 2023-10-12T00:23:07.588 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.588 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091acc9ef074a0d0d-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.638 DEBUG:boto:b'wafmoa-7:1000false:2023-10-12T00:20:51.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:07.640 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.640 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.641 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.641 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.641 DEBUG:boto:Method: GET 2023-10-12T00:23:07.642 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.642 DEBUG:boto:Data: 2023-10-12T00:23:07.643 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.644 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.644 DEBUG:boto:Params: {} 2023-10-12T00:23:07.645 DEBUG:boto:Token: None 2023-10-12T00:23:07.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k= 2023-10-12T00:23:07.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k=', 'Content-Length': '0'} 2023-10-12T00:23:07.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93021ba092faf17-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.649 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.649 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.650 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.650 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.651 DEBUG:boto:Method: GET 2023-10-12T00:23:07.651 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.652 DEBUG:boto:Data: 2023-10-12T00:23:07.652 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.653 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.653 DEBUG:boto:Params: {} 2023-10-12T00:23:07.654 DEBUG:boto:Token: None 2023-10-12T00:23:07.654 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k= 2023-10-12T00:23:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k=', 'Content-Length': '0'} 2023-10-12T00:23:07.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b24f8a723efaf8a-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.658 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.658 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.659 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.659 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.659 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.660 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.660 DEBUG:boto:Data: 2023-10-12T00:23:07.661 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.661 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.662 DEBUG:boto:Params: {} 2023-10-12T00:23:07.662 DEBUG:boto:Token: None 2023-10-12T00:23:07.663 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zQWp8DfnC1NG3ouIVkLwkygf8I0= 2023-10-12T00:23:07.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zQWp8DfnC1NG3ouIVkLwkygf8I0=', 'Content-Length': '0'} 2023-10-12T00:23:07.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000dd3c7ef5e87a99-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.666 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.667 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.667 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.668 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.668 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.668 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.669 DEBUG:boto:Data: 2023-10-12T00:23:07.669 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.670 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.670 DEBUG:boto:Params: {} 2023-10-12T00:23:07.671 DEBUG:boto:Token: None 2023-10-12T00:23:07.671 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zQWp8DfnC1NG3ouIVkLwkygf8I0= 2023-10-12T00:23:07.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zQWp8DfnC1NG3ouIVkLwkygf8I0=', 'Content-Length': '0'} 2023-10-12T00:23:07.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092ac9e873b078ba9-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:07.675 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.676 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.676 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.676 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.677 DEBUG:boto:Method: GET 2023-10-12T00:23:07.677 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.678 DEBUG:boto:Data: 2023-10-12T00:23:07.678 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.679 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.679 DEBUG:boto:Params: {} 2023-10-12T00:23:07.680 DEBUG:boto:Token: None 2023-10-12T00:23:07.680 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.681 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k= 2023-10-12T00:23:07.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k=', 'Content-Length': '0'} 2023-10-12T00:23:07.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e260225aa177827-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.684 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.685 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.685 DEBUG:boto:path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.685 DEBUG:boto:auth_path=/wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.686 DEBUG:boto:Method: GET 2023-10-12T00:23:07.686 DEBUG:boto:Path: /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.687 DEBUG:boto:Data: 2023-10-12T00:23:07.687 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.688 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.688 DEBUG:boto:Params: {} 2023-10-12T00:23:07.689 DEBUG:boto:Token: None 2023-10-12T00:23:07.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj?versionId=null 2023-10-12T00:23:07.690 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k= 2023-10-12T00:23:07.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9Ryq4sJRs18YmL+uUfGTXTUa5k=', 'Content-Length': '0'} 2023-10-12T00:23:07.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e2d2e4737c012e6-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.693 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.694 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.694 DEBUG:boto:path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.695 DEBUG:boto:auth_path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.695 DEBUG:boto:Method: GET 2023-10-12T00:23:07.695 DEBUG:boto:Path: /wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.696 DEBUG:boto:Data: 2023-10-12T00:23:07.696 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.697 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.697 DEBUG:boto:Params: {} 2023-10-12T00:23:07.698 DEBUG:boto:Token: None 2023-10-12T00:23:07.698 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f73d9f9f80829258-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.703 DEBUG:boto:b'wafmoa-7_myobj1000false_myobj2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.703 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.704 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.704 DEBUG:boto:path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.705 DEBUG:boto:auth_path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.705 DEBUG:boto:Method: GET 2023-10-12T00:23:07.705 DEBUG:boto:Path: /wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.706 DEBUG:boto:Data: 2023-10-12T00:23:07.706 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.707 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.708 DEBUG:boto:Params: {} 2023-10-12T00:23:07.708 DEBUG:boto:Token: None 2023-10-12T00:23:07.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002f77b6d7fb4017-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.713 DEBUG:boto:b'wafmoa-7_myobj1000false_myobj2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.714 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.714 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.715 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.715 DEBUG:boto:Path: /wafmoa-7/_myobj 2023-10-12T00:23:07.715 DEBUG:boto:Data: 2023-10-12T00:23:07.716 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.717 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.717 DEBUG:boto:Params: {} 2023-10-12T00:23:07.717 DEBUG:boto:Token: None 2023-10-12T00:23:07.718 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj 2023-10-12T00:23:07.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z0BJzxyhpEGlQPrgGUT2GHEqJYE= 2023-10-12T00:23:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z0BJzxyhpEGlQPrgGUT2GHEqJYE=', 'Content-Length': '0'} 2023-10-12T00:23:07.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066a39cefe49b9d40-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.721 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.722 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.722 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.723 DEBUG:boto:Path: /wafmoa-7/_myobj 2023-10-12T00:23:07.723 DEBUG:boto:Data: 2023-10-12T00:23:07.724 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.724 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.725 DEBUG:boto:Params: {} 2023-10-12T00:23:07.725 DEBUG:boto:Token: None 2023-10-12T00:23:07.726 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj 2023-10-12T00:23:07.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z0BJzxyhpEGlQPrgGUT2GHEqJYE= 2023-10-12T00:23:07.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z0BJzxyhpEGlQPrgGUT2GHEqJYE=', 'Content-Length': '0'} 2023-10-12T00:23:07.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087b2627f01f9cc71-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:07.730 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.730 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.731 DEBUG:boto:Method: GET 2023-10-12T00:23:07.731 DEBUG:boto:Path: /wafmoa-7/_myobj 2023-10-12T00:23:07.732 DEBUG:boto:Data: 2023-10-12T00:23:07.732 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.733 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.734 DEBUG:boto:Params: {} 2023-10-12T00:23:07.734 DEBUG:boto:Token: None 2023-10-12T00:23:07.735 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj 2023-10-12T00:23:07.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vz6RHSlMFnspgPmuwjUQAxr9e0A= 2023-10-12T00:23:07.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vz6RHSlMFnspgPmuwjUQAxr9e0A=', 'Content-Length': '0'} 2023-10-12T00:23:07.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007eb66e10fb104e21-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.739 DEBUG:boto:path=/wafmoa-7/_myobj 2023-10-12T00:23:07.739 DEBUG:boto:auth_path=/wafmoa-7/_myobj 2023-10-12T00:23:07.740 DEBUG:boto:Method: GET 2023-10-12T00:23:07.740 DEBUG:boto:Path: /wafmoa-7/_myobj 2023-10-12T00:23:07.740 DEBUG:boto:Data: 2023-10-12T00:23:07.741 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.742 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.742 DEBUG:boto:Params: {} 2023-10-12T00:23:07.743 DEBUG:boto:Token: None 2023-10-12T00:23:07.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/_myobj 2023-10-12T00:23:07.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vz6RHSlMFnspgPmuwjUQAxr9e0A= 2023-10-12T00:23:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vz6RHSlMFnspgPmuwjUQAxr9e0A=', 'Content-Length': '0'} 2023-10-12T00:23:07.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca9132df5b047360-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.747 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.747 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.748 DEBUG:boto:path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.748 DEBUG:boto:auth_path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.749 DEBUG:boto:Method: GET 2023-10-12T00:23:07.749 DEBUG:boto:Path: /wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.750 DEBUG:boto:Data: 2023-10-12T00:23:07.750 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.751 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.751 DEBUG:boto:Params: {} 2023-10-12T00:23:07.752 DEBUG:boto:Token: None 2023-10-12T00:23:07.752 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fbbd590ca95511d-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.756 DEBUG:boto:b'wafmoa-7_myobj1000false_myobj2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.757 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.757 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.758 DEBUG:boto:path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.758 DEBUG:boto:auth_path=/wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.759 DEBUG:boto:Method: GET 2023-10-12T00:23:07.759 DEBUG:boto:Path: /wafmoa-7/?prefix=_myobj 2023-10-12T00:23:07.759 DEBUG:boto:Data: 2023-10-12T00:23:07.760 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.761 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.761 DEBUG:boto:Params: {} 2023-10-12T00:23:07.762 DEBUG:boto:Token: None 2023-10-12T00:23:07.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccbf7d552c801c13-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.806 DEBUG:boto:b'wafmoa-7_myobj1000false_myobj2023-10-12T00:20:51.520Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:07.808 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.809 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.809 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.810 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.810 DEBUG:boto:Method: GET 2023-10-12T00:23:07.811 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.811 DEBUG:boto:Data: 2023-10-12T00:23:07.811 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.812 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.813 DEBUG:boto:Params: {} 2023-10-12T00:23:07.813 DEBUG:boto:Token: None 2023-10-12T00:23:07.814 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.814 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc= 2023-10-12T00:23:07.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc=', 'Content-Length': '0'} 2023-10-12T00:23:07.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a199cb7a69bdecc-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.818 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.818 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.819 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.819 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.820 DEBUG:boto:Method: GET 2023-10-12T00:23:07.820 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.821 DEBUG:boto:Data: 2023-10-12T00:23:07.821 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.822 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.822 DEBUG:boto:Params: {} 2023-10-12T00:23:07.823 DEBUG:boto:Token: None 2023-10-12T00:23:07.823 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc= 2023-10-12T00:23:07.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc=', 'Content-Length': '0'} 2023-10-12T00:23:07.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065662dda9901c0e9-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.827 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.827 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.828 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.828 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.829 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.829 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.829 DEBUG:boto:Data: 2023-10-12T00:23:07.830 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.831 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.831 DEBUG:boto:Params: {} 2023-10-12T00:23:07.832 DEBUG:boto:Token: None 2023-10-12T00:23:07.833 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:08herU0/Hj7l+Wz7W19J9TYMII4= 2023-10-12T00:23:07.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:08herU0/Hj7l+Wz7W19J9TYMII4=', 'Content-Length': '0'} 2023-10-12T00:23:07.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b77c09a782e830c-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.838 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.839 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.839 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.839 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.840 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.840 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.841 DEBUG:boto:Data: 2023-10-12T00:23:07.841 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.842 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.842 DEBUG:boto:Params: {} 2023-10-12T00:23:07.843 DEBUG:boto:Token: None 2023-10-12T00:23:07.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:08herU0/Hj7l+Wz7W19J9TYMII4= 2023-10-12T00:23:07.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:08herU0/Hj7l+Wz7W19J9TYMII4=', 'Content-Length': '0'} 2023-10-12T00:23:07.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000473d8875149ea07d-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:07.847 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.848 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.848 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.849 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.849 DEBUG:boto:Method: GET 2023-10-12T00:23:07.849 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.850 DEBUG:boto:Data: 2023-10-12T00:23:07.850 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.851 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.852 DEBUG:boto:Params: {} 2023-10-12T00:23:07.852 DEBUG:boto:Token: None 2023-10-12T00:23:07.853 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc= 2023-10-12T00:23:07.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc=', 'Content-Length': '0'} 2023-10-12T00:23:07.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023333cd164ef2ed0-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.856 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.857 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.857 DEBUG:boto:path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.858 DEBUG:boto:auth_path=/wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.858 DEBUG:boto:Method: GET 2023-10-12T00:23:07.859 DEBUG:boto:Path: /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.859 DEBUG:boto:Data: 2023-10-12T00:23:07.859 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.860 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.861 DEBUG:boto:Params: {} 2023-10-12T00:23:07.861 DEBUG:boto:Token: None 2023-10-12T00:23:07.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj?versionId=null 2023-10-12T00:23:07.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc= 2023-10-12T00:23:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGSK6so6DhRMRa934CNl59EQuXc=', 'Content-Length': '0'} 2023-10-12T00:23:07.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000676d4bf0d375b759-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.906 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.907 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.907 DEBUG:boto:path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.908 DEBUG:boto:auth_path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.908 DEBUG:boto:Method: GET 2023-10-12T00:23:07.909 DEBUG:boto:Path: /wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.909 DEBUG:boto:Data: 2023-10-12T00:23:07.910 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.910 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.911 DEBUG:boto:Params: {} 2023-10-12T00:23:07.911 DEBUG:boto:Token: None 2023-10-12T00:23:07.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c8dbfdf6a555ff-0065273c6b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.916 DEBUG:boto:b'wafmoa-7myobj1000falsemyobj2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.917 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:07.917 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:07.918 DEBUG:boto:path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.918 DEBUG:boto:auth_path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.919 DEBUG:boto:Method: GET 2023-10-12T00:23:07.919 DEBUG:boto:Path: /wafmoa-7/?prefix=myobj 2023-10-12T00:23:07.919 DEBUG:boto:Data: 2023-10-12T00:23:07.920 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.921 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.921 DEBUG:boto:Params: {} 2023-10-12T00:23:07.922 DEBUG:boto:Token: None 2023-10-12T00:23:07.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/ 2023-10-12T00:23:07.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA= 2023-10-12T00:23:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kNGR/sAHrGhOV+m00DWLP+3QbeA=', 'Content-Length': '0'} 2023-10-12T00:23:07.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae70e192304a14f-0065273c6b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.926 DEBUG:boto:b'wafmoa-7myobj1000falsemyobj2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:07.927 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.928 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.928 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.928 DEBUG:boto:Path: /wafmoa-7/myobj 2023-10-12T00:23:07.929 DEBUG:boto:Data: 2023-10-12T00:23:07.929 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.930 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.931 DEBUG:boto:Params: {} 2023-10-12T00:23:07.931 DEBUG:boto:Token: None 2023-10-12T00:23:07.932 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj 2023-10-12T00:23:07.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nKcapTyCMbtFWiKp3jZ9XILo+7k= 2023-10-12T00:23:07.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nKcapTyCMbtFWiKp3jZ9XILo+7k=', 'Content-Length': '0'} 2023-10-12T00:23:07.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da474e3615a5adfe-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.935 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.936 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.936 DEBUG:boto:Method: HEAD 2023-10-12T00:23:07.936 DEBUG:boto:Path: /wafmoa-7/myobj 2023-10-12T00:23:07.937 DEBUG:boto:Data: 2023-10-12T00:23:07.937 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:07.938 DEBUG:boto:Port: 8000 2023-10-12T00:23:07.939 DEBUG:boto:Params: {} 2023-10-12T00:23:07.939 DEBUG:boto:Token: None 2023-10-12T00:23:07.940 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj 2023-10-12T00:23:07.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nKcapTyCMbtFWiKp3jZ9XILo+7k= 2023-10-12T00:23:07.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nKcapTyCMbtFWiKp3jZ9XILo+7k=', 'Content-Length': '0'} 2023-10-12T00:23:07.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009222a302f770094e-0065273c6b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:07.944 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.944 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.945 DEBUG:boto:Method: GET 2023-10-12T00:23:07.945 DEBUG:boto:Path: /wafmoa-7/myobj 2023-10-12T00:23:07.946 DEBUG:boto:Data: 2023-10-12T00:23:07.946 DEBUG:boto:Headers: {} 2023-10-12T00:23:07.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:07.947 DEBUG:boto:Port: 8001 2023-10-12T00:23:07.947 DEBUG:boto:Params: {} 2023-10-12T00:23:07.948 DEBUG:boto:Token: None 2023-10-12T00:23:07.948 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:07 GMT /wafmoa-7/myobj 2023-10-12T00:23:07.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cmgj9bN3I6iTxB7jwaKa8ApAVrg= 2023-10-12T00:23:07.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cmgj9bN3I6iTxB7jwaKa8ApAVrg=', 'Content-Length': '0'} 2023-10-12T00:23:07.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ef58233749954f9-0065273c6b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:07.998 DEBUG:boto:path=/wafmoa-7/myobj 2023-10-12T00:23:07.999 DEBUG:boto:auth_path=/wafmoa-7/myobj 2023-10-12T00:23:07.999 DEBUG:boto:Method: GET 2023-10-12T00:23:08.000 DEBUG:boto:Path: /wafmoa-7/myobj 2023-10-12T00:23:08.000 DEBUG:boto:Data: 2023-10-12T00:23:08.001 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.002 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.002 DEBUG:boto:Params: {} 2023-10-12T00:23:08.003 DEBUG:boto:Token: None 2023-10-12T00:23:08.003 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-7/myobj 2023-10-12T00:23:08.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n1G9nHLMkT4WwT5HYhFGukuIF60= 2023-10-12T00:23:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n1G9nHLMkT4WwT5HYhFGukuIF60=', 'Content-Length': '0'} 2023-10-12T00:23:08.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4f87815d5486d1e-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.007 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:08.007 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:08.008 DEBUG:boto:path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.008 DEBUG:boto:auth_path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.008 DEBUG:boto:Method: GET 2023-10-12T00:23:08.009 DEBUG:boto:Path: /wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.009 DEBUG:boto:Data: 2023-10-12T00:23:08.010 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.011 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.011 DEBUG:boto:Params: {} 2023-10-12T00:23:08.011 DEBUG:boto:Token: None 2023-10-12T00:23:08.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-7/ 2023-10-12T00:23:08.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EtEnBLtAAra90gwOVuiHf08rdUc= 2023-10-12T00:23:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EtEnBLtAAra90gwOVuiHf08rdUc=', 'Content-Length': '0'} 2023-10-12T00:23:08.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f13273e48d6de1-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.016 DEBUG:boto:b'wafmoa-7myobj1000falsemyobj2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.017 DEBUG:boto:path=/wafmoa-7/ 2023-10-12T00:23:08.017 DEBUG:boto:auth_path=/wafmoa-7/ 2023-10-12T00:23:08.018 DEBUG:boto:path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.018 DEBUG:boto:auth_path=/wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.019 DEBUG:boto:Method: GET 2023-10-12T00:23:08.019 DEBUG:boto:Path: /wafmoa-7/?prefix=myobj 2023-10-12T00:23:08.020 DEBUG:boto:Data: 2023-10-12T00:23:08.020 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.021 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.021 DEBUG:boto:Params: {} 2023-10-12T00:23:08.022 DEBUG:boto:Token: None 2023-10-12T00:23:08.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-7/ 2023-10-12T00:23:08.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EtEnBLtAAra90gwOVuiHf08rdUc= 2023-10-12T00:23:08.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EtEnBLtAAra90gwOVuiHf08rdUc=', 'Content-Length': '0'} 2023-10-12T00:23:08.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006da623bbb652060-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.027 DEBUG:boto:b'wafmoa-7myobj1000falsemyobj2023-10-12T00:20:51.497Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.027 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-7 zones={a2, a1} 2023-10-12T00:23:08.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:08.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:08.029 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:23:08.092 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:08.092 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:08.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:08.131+0000 7f6960019780 1 bucket sync caught up with source: 2023-10-12T00:23:08.132 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.8.6, , 00000000001.10.6, , , , 00000000001.7.6, 00000000001.10.6, , ] 2023-10-12T00:23:08.132 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.8.6, , 00000000001.10.6, , , , 00000000001.7.6, 00000000001.10.6, , ] 2023-10-12T00:23:08.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:08.131+0000 7f6960019780 0 bucket checkpoint complete 2023-10-12T00:23:08.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-8 zones={a1, a2} 2023-10-12T00:23:08.145 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.145 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.146 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.146 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:23:08.147 DEBUG:boto:Data: 2023-10-12T00:23:08.147 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.148 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.148 DEBUG:boto:Params: {} 2023-10-12T00:23:08.149 DEBUG:boto:Token: None 2023-10-12T00:23:08.149 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cAN1njTnM7sogFzPAYcoWCQsk1s= 2023-10-12T00:23:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cAN1njTnM7sogFzPAYcoWCQsk1s=', 'Content-Length': '0'} 2023-10-12T00:23:08.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f78faade13face3d-0065273c6c-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.153 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.153 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.154 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.154 DEBUG:boto:Path: /wafmoa-8/ 2023-10-12T00:23:08.155 DEBUG:boto:Data: 2023-10-12T00:23:08.155 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.156 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.156 DEBUG:boto:Params: {} 2023-10-12T00:23:08.157 DEBUG:boto:Token: None 2023-10-12T00:23:08.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cAN1njTnM7sogFzPAYcoWCQsk1s= 2023-10-12T00:23:08.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cAN1njTnM7sogFzPAYcoWCQsk1s=', 'Content-Length': '0'} 2023-10-12T00:23:08.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bb21ef34753fb5b-0065273c6c-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.161 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:23:08.162 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.162 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.162 DEBUG:boto:path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.163 DEBUG:boto:auth_path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.163 DEBUG:boto:Method: GET 2023-10-12T00:23:08.164 DEBUG:boto:Path: /wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.164 DEBUG:boto:Data: 2023-10-12T00:23:08.165 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.166 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.166 DEBUG:boto:Params: {} 2023-10-12T00:23:08.166 DEBUG:boto:Token: None 2023-10-12T00:23:08.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/?versions 2023-10-12T00:23:08.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio= 2023-10-12T00:23:08.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio=', 'Content-Length': '0'} 2023-10-12T00:23:08.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c132e6a6e4e55eb-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.171 DEBUG:boto:b'wafmoa-8999false&nulltrue2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.172 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T00:23:08.172 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T00:23:08.173 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T00:23:08.173 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:08.173 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:23:08.174 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.174 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.175 DEBUG:boto:path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.175 DEBUG:boto:auth_path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.175 DEBUG:boto:Method: GET 2023-10-12T00:23:08.176 DEBUG:boto:Path: /wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.176 DEBUG:boto:Data: 2023-10-12T00:23:08.177 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.178 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.178 DEBUG:boto:Params: {} 2023-10-12T00:23:08.178 DEBUG:boto:Token: None 2023-10-12T00:23:08.179 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/?versions 2023-10-12T00:23:08.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio= 2023-10-12T00:23:08.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio=', 'Content-Length': '0'} 2023-10-12T00:23:08.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a83ac482b38e4ef8-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.183 DEBUG:boto:b'wafmoa-8999false&nulltrue2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.184 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-12T00:23:08.184 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-12T00:23:08.184 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-12T00:23:08.185 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:08.185 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.186 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.186 DEBUG:boto:path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.187 DEBUG:boto:auth_path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.187 DEBUG:boto:Method: GET 2023-10-12T00:23:08.187 DEBUG:boto:Path: /wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.188 DEBUG:boto:Data: 2023-10-12T00:23:08.188 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.189 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.190 DEBUG:boto:Params: {} 2023-10-12T00:23:08.190 DEBUG:boto:Token: None 2023-10-12T00:23:08.191 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/?versions 2023-10-12T00:23:08.191 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio= 2023-10-12T00:23:08.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio=', 'Content-Length': '0'} 2023-10-12T00:23:08.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30a4b18e569f301-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.194 DEBUG:boto:b'wafmoa-8999false&nulltrue2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.195 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.196 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.196 DEBUG:boto:path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.196 DEBUG:boto:auth_path=/wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.197 DEBUG:boto:Method: GET 2023-10-12T00:23:08.197 DEBUG:boto:Path: /wafmoa-8/?versions&max-keys=999 2023-10-12T00:23:08.198 DEBUG:boto:Data: 2023-10-12T00:23:08.198 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.199 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.199 DEBUG:boto:Params: {} 2023-10-12T00:23:08.200 DEBUG:boto:Token: None 2023-10-12T00:23:08.200 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/?versions 2023-10-12T00:23:08.201 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio= 2023-10-12T00:23:08.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6IRGtUT16Oy7Vx+aThH98sVrio=', 'Content-Length': '0'} 2023-10-12T00:23:08.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74b5b90cba7183d-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.204 DEBUG:boto:b'wafmoa-8999false&nulltrue2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:08.205 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.206 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.206 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.207 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.207 DEBUG:boto:Method: GET 2023-10-12T00:23:08.208 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.208 DEBUG:boto:Data: 2023-10-12T00:23:08.208 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.209 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.210 DEBUG:boto:Params: {} 2023-10-12T00:23:08.210 DEBUG:boto:Token: None 2023-10-12T00:23:08.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.211 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0= 2023-10-12T00:23:08.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0=', 'Content-Length': '0'} 2023-10-12T00:23:08.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063252d9d2f183452-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.215 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.215 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.215 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.216 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.216 DEBUG:boto:Method: GET 2023-10-12T00:23:08.217 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.217 DEBUG:boto:Data: 2023-10-12T00:23:08.218 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.219 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.219 DEBUG:boto:Params: {} 2023-10-12T00:23:08.219 DEBUG:boto:Token: None 2023-10-12T00:23:08.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0= 2023-10-12T00:23:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0=', 'Content-Length': '0'} 2023-10-12T00:23:08.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057904e88a7e82f04-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.224 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.224 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.225 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.225 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.225 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.226 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.226 DEBUG:boto:Data: 2023-10-12T00:23:08.227 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.228 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.228 DEBUG:boto:Params: {} 2023-10-12T00:23:08.228 DEBUG:boto:Token: None 2023-10-12T00:23:08.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kN1BKugLH0vIuMdkYUhxaO1pFIE= 2023-10-12T00:23:08.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kN1BKugLH0vIuMdkYUhxaO1pFIE=', 'Content-Length': '0'} 2023-10-12T00:23:08.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000765017dd2f5f002-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.232 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.233 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.233 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.233 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.234 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.234 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.235 DEBUG:boto:Data: 2023-10-12T00:23:08.235 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.236 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.236 DEBUG:boto:Params: {} 2023-10-12T00:23:08.237 DEBUG:boto:Token: None 2023-10-12T00:23:08.237 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kN1BKugLH0vIuMdkYUhxaO1pFIE= 2023-10-12T00:23:08.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kN1BKugLH0vIuMdkYUhxaO1pFIE=', 'Content-Length': '0'} 2023-10-12T00:23:08.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098667ef4809a380a-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:08.241 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.241 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.242 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.242 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.243 DEBUG:boto:Method: GET 2023-10-12T00:23:08.243 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.244 DEBUG:boto:Data: 2023-10-12T00:23:08.244 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.245 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.245 DEBUG:boto:Params: {} 2023-10-12T00:23:08.246 DEBUG:boto:Token: None 2023-10-12T00:23:08.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0= 2023-10-12T00:23:08.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0=', 'Content-Length': '0'} 2023-10-12T00:23:08.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cc170a02c8dc393-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.250 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.250 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.251 DEBUG:boto:path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.251 DEBUG:boto:auth_path=/wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.252 DEBUG:boto:Method: GET 2023-10-12T00:23:08.252 DEBUG:boto:Path: /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.253 DEBUG:boto:Data: 2023-10-12T00:23:08.253 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.254 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.254 DEBUG:boto:Params: {} 2023-10-12T00:23:08.255 DEBUG:boto:Token: None 2023-10-12T00:23:08.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26?versionId=null 2023-10-12T00:23:08.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0= 2023-10-12T00:23:08.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rq5rpqqXnT+Gl9TEPl/8Pht1hA0=', 'Content-Length': '0'} 2023-10-12T00:23:08.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027326e3822fb2fb0-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.302 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.303 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.303 DEBUG:boto:path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.304 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.304 DEBUG:boto:Method: GET 2023-10-12T00:23:08.305 DEBUG:boto:Path: /wafmoa-8/?prefix=%26 2023-10-12T00:23:08.305 DEBUG:boto:Data: 2023-10-12T00:23:08.306 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.307 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.307 DEBUG:boto:Params: {} 2023-10-12T00:23:08.308 DEBUG:boto:Token: None 2023-10-12T00:23:08.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023326d24fa6e2318-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.312 DEBUG:boto:b'wafmoa-8&1000false&2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.313 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.313 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.313 DEBUG:boto:path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.314 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.314 DEBUG:boto:Method: GET 2023-10-12T00:23:08.315 DEBUG:boto:Path: /wafmoa-8/?prefix=%26 2023-10-12T00:23:08.315 DEBUG:boto:Data: 2023-10-12T00:23:08.316 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.317 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.317 DEBUG:boto:Params: {} 2023-10-12T00:23:08.318 DEBUG:boto:Token: None 2023-10-12T00:23:08.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e8ca0eeab3e2af-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.322 DEBUG:boto:b'wafmoa-8&1000false&2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.323 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.323 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.324 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.324 DEBUG:boto:Path: /wafmoa-8/%26 2023-10-12T00:23:08.324 DEBUG:boto:Data: 2023-10-12T00:23:08.325 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.326 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.326 DEBUG:boto:Params: {} 2023-10-12T00:23:08.326 DEBUG:boto:Token: None 2023-10-12T00:23:08.327 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26 2023-10-12T00:23:08.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X1jEHlk6WvfnTTkVisQKsIoLZAM= 2023-10-12T00:23:08.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X1jEHlk6WvfnTTkVisQKsIoLZAM=', 'Content-Length': '0'} 2023-10-12T00:23:08.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009919f2608889d9fa-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.330 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.331 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.331 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.332 DEBUG:boto:Path: /wafmoa-8/%26 2023-10-12T00:23:08.332 DEBUG:boto:Data: 2023-10-12T00:23:08.332 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.333 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.334 DEBUG:boto:Params: {} 2023-10-12T00:23:08.334 DEBUG:boto:Token: None 2023-10-12T00:23:08.335 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26 2023-10-12T00:23:08.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X1jEHlk6WvfnTTkVisQKsIoLZAM= 2023-10-12T00:23:08.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X1jEHlk6WvfnTTkVisQKsIoLZAM=', 'Content-Length': '0'} 2023-10-12T00:23:08.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc559009d97422f9-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-12T00:23:08.338 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.339 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.339 DEBUG:boto:Method: GET 2023-10-12T00:23:08.340 DEBUG:boto:Path: /wafmoa-8/%26 2023-10-12T00:23:08.340 DEBUG:boto:Data: 2023-10-12T00:23:08.341 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.342 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.342 DEBUG:boto:Params: {} 2023-10-12T00:23:08.342 DEBUG:boto:Token: None 2023-10-12T00:23:08.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26 2023-10-12T00:23:08.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+nbjVrPoJff/dPdt2P5aDaGyVmY= 2023-10-12T00:23:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+nbjVrPoJff/dPdt2P5aDaGyVmY=', 'Content-Length': '0'} 2023-10-12T00:23:08.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000272e37b079dc1601-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.386 DEBUG:boto:path=/wafmoa-8/%26 2023-10-12T00:23:08.387 DEBUG:boto:auth_path=/wafmoa-8/%26 2023-10-12T00:23:08.387 DEBUG:boto:Method: GET 2023-10-12T00:23:08.388 DEBUG:boto:Path: /wafmoa-8/%26 2023-10-12T00:23:08.388 DEBUG:boto:Data: 2023-10-12T00:23:08.389 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.390 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.390 DEBUG:boto:Params: {} 2023-10-12T00:23:08.391 DEBUG:boto:Token: None 2023-10-12T00:23:08.391 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%26 2023-10-12T00:23:08.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+nbjVrPoJff/dPdt2P5aDaGyVmY= 2023-10-12T00:23:08.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+nbjVrPoJff/dPdt2P5aDaGyVmY=', 'Content-Length': '0'} 2023-10-12T00:23:08.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039d0007a265f9182-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.395 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.395 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.396 DEBUG:boto:path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.396 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.396 DEBUG:boto:Method: GET 2023-10-12T00:23:08.397 DEBUG:boto:Path: /wafmoa-8/?prefix=%26 2023-10-12T00:23:08.397 DEBUG:boto:Data: 2023-10-12T00:23:08.398 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.399 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.399 DEBUG:boto:Params: {} 2023-10-12T00:23:08.399 DEBUG:boto:Token: None 2023-10-12T00:23:08.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0150a44ea84c1c-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.403 DEBUG:boto:b'wafmoa-8&1000false&2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.404 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.405 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.405 DEBUG:boto:path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.406 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%26 2023-10-12T00:23:08.406 DEBUG:boto:Method: GET 2023-10-12T00:23:08.406 DEBUG:boto:Path: /wafmoa-8/?prefix=%26 2023-10-12T00:23:08.407 DEBUG:boto:Data: 2023-10-12T00:23:08.407 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.408 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.409 DEBUG:boto:Params: {} 2023-10-12T00:23:08.409 DEBUG:boto:Token: None 2023-10-12T00:23:08.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f99e7ed842d4dde-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.413 DEBUG:boto:b'wafmoa-8&1000false&2023-10-12T00:20:51.734Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:08.415 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.415 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.416 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.416 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.416 DEBUG:boto:Method: GET 2023-10-12T00:23:08.417 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.417 DEBUG:boto:Data: 2023-10-12T00:23:08.418 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.419 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.419 DEBUG:boto:Params: {} 2023-10-12T00:23:08.419 DEBUG:boto:Token: None 2023-10-12T00:23:08.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE= 2023-10-12T00:23:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE=', 'Content-Length': '0'} 2023-10-12T00:23:08.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b196b846f5e05cdf-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.423 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.424 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.424 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.425 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.425 DEBUG:boto:Method: GET 2023-10-12T00:23:08.426 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.426 DEBUG:boto:Data: 2023-10-12T00:23:08.427 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.427 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.428 DEBUG:boto:Params: {} 2023-10-12T00:23:08.428 DEBUG:boto:Token: None 2023-10-12T00:23:08.429 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE= 2023-10-12T00:23:08.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE=', 'Content-Length': '0'} 2023-10-12T00:23:08.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000054c95f0b8ddb01a-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.432 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.433 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.433 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.433 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.434 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.434 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.435 DEBUG:boto:Data: 2023-10-12T00:23:08.435 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.436 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.437 DEBUG:boto:Params: {} 2023-10-12T00:23:08.437 DEBUG:boto:Token: None 2023-10-12T00:23:08.438 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oSrBuidvEvUKRGzH/nwIi1NTSac= 2023-10-12T00:23:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oSrBuidvEvUKRGzH/nwIi1NTSac=', 'Content-Length': '0'} 2023-10-12T00:23:08.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3d0aa002151d938-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.441 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.442 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.442 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.443 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.443 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.443 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.444 DEBUG:boto:Data: 2023-10-12T00:23:08.444 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.445 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.445 DEBUG:boto:Params: {} 2023-10-12T00:23:08.446 DEBUG:boto:Token: None 2023-10-12T00:23:08.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oSrBuidvEvUKRGzH/nwIi1NTSac= 2023-10-12T00:23:08.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oSrBuidvEvUKRGzH/nwIi1NTSac=', 'Content-Length': '0'} 2023-10-12T00:23:08.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000953a6267833dcd07-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:08.450 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.451 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.451 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.451 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.452 DEBUG:boto:Method: GET 2023-10-12T00:23:08.452 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.453 DEBUG:boto:Data: 2023-10-12T00:23:08.453 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.454 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.455 DEBUG:boto:Params: {} 2023-10-12T00:23:08.455 DEBUG:boto:Token: None 2023-10-12T00:23:08.456 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE= 2023-10-12T00:23:08.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE=', 'Content-Length': '0'} 2023-10-12T00:23:08.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000140a68091e3f03ce-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.506 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.507 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.507 DEBUG:boto:path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.508 DEBUG:boto:auth_path=/wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.508 DEBUG:boto:Method: GET 2023-10-12T00:23:08.509 DEBUG:boto:Path: /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.509 DEBUG:boto:Data: 2023-10-12T00:23:08.509 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.510 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.511 DEBUG:boto:Params: {} 2023-10-12T00:23:08.511 DEBUG:boto:Token: None 2023-10-12T00:23:08.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A?versionId=null 2023-10-12T00:23:08.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE= 2023-10-12T00:23:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VgDI8yj+PnvvhKUFSTbSjLnEOlE=', 'Content-Length': '0'} 2023-10-12T00:23:08.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000563f91aaf5ebfb7e-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.516 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.516 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.516 DEBUG:boto:path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.517 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.517 DEBUG:boto:Method: GET 2023-10-12T00:23:08.518 DEBUG:boto:Path: /wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.518 DEBUG:boto:Data: 2023-10-12T00:23:08.519 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.519 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.520 DEBUG:boto:Params: {} 2023-10-12T00:23:08.520 DEBUG:boto:Token: None 2023-10-12T00:23:08.521 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f06f8b0a947eea5-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.524 DEBUG:boto:b'wafmoa-8:1000false:2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.525 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.526 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.526 DEBUG:boto:path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.526 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.527 DEBUG:boto:Method: GET 2023-10-12T00:23:08.527 DEBUG:boto:Path: /wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.528 DEBUG:boto:Data: 2023-10-12T00:23:08.528 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.529 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.529 DEBUG:boto:Params: {} 2023-10-12T00:23:08.530 DEBUG:boto:Token: None 2023-10-12T00:23:08.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ffacd2b359fe42-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.534 DEBUG:boto:b'wafmoa-8:1000false:2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.535 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.535 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.536 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.536 DEBUG:boto:Path: /wafmoa-8/%3A 2023-10-12T00:23:08.537 DEBUG:boto:Data: 2023-10-12T00:23:08.537 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.538 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.538 DEBUG:boto:Params: {} 2023-10-12T00:23:08.539 DEBUG:boto:Token: None 2023-10-12T00:23:08.539 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A 2023-10-12T00:23:08.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hcITq+u7+Ju8Y4yiwRldAm9Vc7k= 2023-10-12T00:23:08.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hcITq+u7+Ju8Y4yiwRldAm9Vc7k=', 'Content-Length': '0'} 2023-10-12T00:23:08.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e11bcebe4ce12c5-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.542 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.543 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.543 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.544 DEBUG:boto:Path: /wafmoa-8/%3A 2023-10-12T00:23:08.544 DEBUG:boto:Data: 2023-10-12T00:23:08.545 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.546 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.546 DEBUG:boto:Params: {} 2023-10-12T00:23:08.547 DEBUG:boto:Token: None 2023-10-12T00:23:08.547 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A 2023-10-12T00:23:08.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hcITq+u7+Ju8Y4yiwRldAm9Vc7k= 2023-10-12T00:23:08.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hcITq+u7+Ju8Y4yiwRldAm9Vc7k=', 'Content-Length': '0'} 2023-10-12T00:23:08.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000907fd117acf3d70d-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-12T00:23:08.551 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.551 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.551 DEBUG:boto:Method: GET 2023-10-12T00:23:08.552 DEBUG:boto:Path: /wafmoa-8/%3A 2023-10-12T00:23:08.552 DEBUG:boto:Data: 2023-10-12T00:23:08.553 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.554 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.554 DEBUG:boto:Params: {} 2023-10-12T00:23:08.555 DEBUG:boto:Token: None 2023-10-12T00:23:08.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A 2023-10-12T00:23:08.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ST3T5miWy2s76uc8hh8gkgsdyY4= 2023-10-12T00:23:08.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ST3T5miWy2s76uc8hh8gkgsdyY4=', 'Content-Length': '0'} 2023-10-12T00:23:08.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc30e178298548ae-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.559 DEBUG:boto:path=/wafmoa-8/%3A 2023-10-12T00:23:08.561 DEBUG:boto:auth_path=/wafmoa-8/%3A 2023-10-12T00:23:08.561 DEBUG:boto:Method: GET 2023-10-12T00:23:08.562 DEBUG:boto:Path: /wafmoa-8/%3A 2023-10-12T00:23:08.562 DEBUG:boto:Data: 2023-10-12T00:23:08.562 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.563 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.564 DEBUG:boto:Params: {} 2023-10-12T00:23:08.565 DEBUG:boto:Token: None 2023-10-12T00:23:08.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/%3A 2023-10-12T00:23:08.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ST3T5miWy2s76uc8hh8gkgsdyY4= 2023-10-12T00:23:08.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ST3T5miWy2s76uc8hh8gkgsdyY4=', 'Content-Length': '0'} 2023-10-12T00:23:08.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be5ea661f6830ec9-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.569 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.570 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.570 DEBUG:boto:path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.571 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.571 DEBUG:boto:Method: GET 2023-10-12T00:23:08.571 DEBUG:boto:Path: /wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.572 DEBUG:boto:Data: 2023-10-12T00:23:08.572 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.573 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.573 DEBUG:boto:Params: {} 2023-10-12T00:23:08.574 DEBUG:boto:Token: None 2023-10-12T00:23:08.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eda83e69caa6b9b-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.578 DEBUG:boto:b'wafmoa-8:1000false:2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.579 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.579 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.580 DEBUG:boto:path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.580 DEBUG:boto:auth_path=/wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.581 DEBUG:boto:Method: GET 2023-10-12T00:23:08.581 DEBUG:boto:Path: /wafmoa-8/?prefix=%3A 2023-10-12T00:23:08.581 DEBUG:boto:Data: 2023-10-12T00:23:08.582 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.583 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.583 DEBUG:boto:Params: {} 2023-10-12T00:23:08.584 DEBUG:boto:Token: None 2023-10-12T00:23:08.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000036d06fbc816fc5c-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.588 DEBUG:boto:b'wafmoa-8:1000false:2023-10-12T00:20:51.707Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:08.590 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.590 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.591 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.591 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.591 DEBUG:boto:Method: GET 2023-10-12T00:23:08.592 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.592 DEBUG:boto:Data: 2023-10-12T00:23:08.593 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.594 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.594 DEBUG:boto:Params: {} 2023-10-12T00:23:08.595 DEBUG:boto:Token: None 2023-10-12T00:23:08.595 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio= 2023-10-12T00:23:08.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio=', 'Content-Length': '0'} 2023-10-12T00:23:08.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ebe183e695cc7d0-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.599 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.600 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.601 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.601 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.601 DEBUG:boto:Method: GET 2023-10-12T00:23:08.602 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.602 DEBUG:boto:Data: 2023-10-12T00:23:08.603 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.604 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.604 DEBUG:boto:Params: {} 2023-10-12T00:23:08.604 DEBUG:boto:Token: None 2023-10-12T00:23:08.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.606 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio= 2023-10-12T00:23:08.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio=', 'Content-Length': '0'} 2023-10-12T00:23:08.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd2e6e74b1f5402f-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.609 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.609 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.610 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.610 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.610 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.611 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.611 DEBUG:boto:Data: 2023-10-12T00:23:08.612 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.612 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.613 DEBUG:boto:Params: {} 2023-10-12T00:23:08.613 DEBUG:boto:Token: None 2023-10-12T00:23:08.614 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.614 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:or/NTVqhHc3HAVYBSr+Kac/JdlQ= 2023-10-12T00:23:08.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:or/NTVqhHc3HAVYBSr+Kac/JdlQ=', 'Content-Length': '0'} 2023-10-12T00:23:08.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000451c6c3485977229-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.617 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.617 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.618 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.618 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.619 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.619 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.620 DEBUG:boto:Data: 2023-10-12T00:23:08.620 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.621 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.621 DEBUG:boto:Params: {} 2023-10-12T00:23:08.622 DEBUG:boto:Token: None 2023-10-12T00:23:08.622 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:or/NTVqhHc3HAVYBSr+Kac/JdlQ= 2023-10-12T00:23:08.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:or/NTVqhHc3HAVYBSr+Kac/JdlQ=', 'Content-Length': '0'} 2023-10-12T00:23:08.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0fc4bb643af08fc-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:08.626 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.627 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.627 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.628 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.628 DEBUG:boto:Method: GET 2023-10-12T00:23:08.629 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.629 DEBUG:boto:Data: 2023-10-12T00:23:08.629 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.630 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.631 DEBUG:boto:Params: {} 2023-10-12T00:23:08.631 DEBUG:boto:Token: None 2023-10-12T00:23:08.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio= 2023-10-12T00:23:08.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio=', 'Content-Length': '0'} 2023-10-12T00:23:08.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b264576ce9c0d0ae-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.635 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.636 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.636 DEBUG:boto:path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.637 DEBUG:boto:auth_path=/wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.637 DEBUG:boto:Method: GET 2023-10-12T00:23:08.637 DEBUG:boto:Path: /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.638 DEBUG:boto:Data: 2023-10-12T00:23:08.638 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.639 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.640 DEBUG:boto:Params: {} 2023-10-12T00:23:08.640 DEBUG:boto:Token: None 2023-10-12T00:23:08.641 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj?versionId=null 2023-10-12T00:23:08.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio= 2023-10-12T00:23:08.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+9vQsR+6rBjg6DYpoqOUwFzwio=', 'Content-Length': '0'} 2023-10-12T00:23:08.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021a5195135ae6a38-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.644 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.644 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.645 DEBUG:boto:path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.645 DEBUG:boto:auth_path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.646 DEBUG:boto:Method: GET 2023-10-12T00:23:08.646 DEBUG:boto:Path: /wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.647 DEBUG:boto:Data: 2023-10-12T00:23:08.647 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.648 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.648 DEBUG:boto:Params: {} 2023-10-12T00:23:08.649 DEBUG:boto:Token: None 2023-10-12T00:23:08.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d3e45ed5206aa6-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.694 DEBUG:boto:b'wafmoa-8_myobj1000false_myobj2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.695 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.696 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.696 DEBUG:boto:path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.697 DEBUG:boto:auth_path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.697 DEBUG:boto:Method: GET 2023-10-12T00:23:08.697 DEBUG:boto:Path: /wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.698 DEBUG:boto:Data: 2023-10-12T00:23:08.698 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.699 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.700 DEBUG:boto:Params: {} 2023-10-12T00:23:08.700 DEBUG:boto:Token: None 2023-10-12T00:23:08.701 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099bf6957e43ee4af-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.705 DEBUG:boto:b'wafmoa-8_myobj1000false_myobj2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.705 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.706 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.706 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.707 DEBUG:boto:Path: /wafmoa-8/_myobj 2023-10-12T00:23:08.707 DEBUG:boto:Data: 2023-10-12T00:23:08.707 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.708 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.709 DEBUG:boto:Params: {} 2023-10-12T00:23:08.709 DEBUG:boto:Token: None 2023-10-12T00:23:08.710 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj 2023-10-12T00:23:08.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JBpnqSZgcK2+3w9gmuHLErLhGXA= 2023-10-12T00:23:08.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JBpnqSZgcK2+3w9gmuHLErLhGXA=', 'Content-Length': '0'} 2023-10-12T00:23:08.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012fe8122b9a69240-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.713 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.713 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.714 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.714 DEBUG:boto:Path: /wafmoa-8/_myobj 2023-10-12T00:23:08.715 DEBUG:boto:Data: 2023-10-12T00:23:08.715 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.716 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.716 DEBUG:boto:Params: {} 2023-10-12T00:23:08.717 DEBUG:boto:Token: None 2023-10-12T00:23:08.717 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj 2023-10-12T00:23:08.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JBpnqSZgcK2+3w9gmuHLErLhGXA= 2023-10-12T00:23:08.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JBpnqSZgcK2+3w9gmuHLErLhGXA=', 'Content-Length': '0'} 2023-10-12T00:23:08.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c57ae355f7de13e9-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-12T00:23:08.721 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.721 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.722 DEBUG:boto:Method: GET 2023-10-12T00:23:08.722 DEBUG:boto:Path: /wafmoa-8/_myobj 2023-10-12T00:23:08.723 DEBUG:boto:Data: 2023-10-12T00:23:08.723 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.724 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.724 DEBUG:boto:Params: {} 2023-10-12T00:23:08.725 DEBUG:boto:Token: None 2023-10-12T00:23:08.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj 2023-10-12T00:23:08.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MWC138k2DxxtUbaApKTPqt0aNQs= 2023-10-12T00:23:08.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MWC138k2DxxtUbaApKTPqt0aNQs=', 'Content-Length': '0'} 2023-10-12T00:23:08.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008920f097fea5ac23-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.729 DEBUG:boto:path=/wafmoa-8/_myobj 2023-10-12T00:23:08.729 DEBUG:boto:auth_path=/wafmoa-8/_myobj 2023-10-12T00:23:08.730 DEBUG:boto:Method: GET 2023-10-12T00:23:08.730 DEBUG:boto:Path: /wafmoa-8/_myobj 2023-10-12T00:23:08.731 DEBUG:boto:Data: 2023-10-12T00:23:08.731 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.732 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.732 DEBUG:boto:Params: {} 2023-10-12T00:23:08.733 DEBUG:boto:Token: None 2023-10-12T00:23:08.733 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/_myobj 2023-10-12T00:23:08.734 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MWC138k2DxxtUbaApKTPqt0aNQs= 2023-10-12T00:23:08.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MWC138k2DxxtUbaApKTPqt0aNQs=', 'Content-Length': '0'} 2023-10-12T00:23:08.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d9642bfd198e1ae-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.737 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.737 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.738 DEBUG:boto:path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.738 DEBUG:boto:auth_path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.739 DEBUG:boto:Method: GET 2023-10-12T00:23:08.739 DEBUG:boto:Path: /wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.739 DEBUG:boto:Data: 2023-10-12T00:23:08.740 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.741 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.741 DEBUG:boto:Params: {} 2023-10-12T00:23:08.741 DEBUG:boto:Token: None 2023-10-12T00:23:08.742 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c74b093b0178817-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.746 DEBUG:boto:b'wafmoa-8_myobj1000false_myobj2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.746 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.747 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.747 DEBUG:boto:path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.748 DEBUG:boto:auth_path=/wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.748 DEBUG:boto:Method: GET 2023-10-12T00:23:08.749 DEBUG:boto:Path: /wafmoa-8/?prefix=_myobj 2023-10-12T00:23:08.749 DEBUG:boto:Data: 2023-10-12T00:23:08.749 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.750 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.751 DEBUG:boto:Params: {} 2023-10-12T00:23:08.751 DEBUG:boto:Token: None 2023-10-12T00:23:08.752 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004065f212485877da-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.756 DEBUG:boto:b'wafmoa-8_myobj1000false_myobj2023-10-12T00:20:51.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:08.757 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.757 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.758 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.758 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.759 DEBUG:boto:Method: GET 2023-10-12T00:23:08.759 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.759 DEBUG:boto:Data: 2023-10-12T00:23:08.760 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.761 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.761 DEBUG:boto:Params: {} 2023-10-12T00:23:08.762 DEBUG:boto:Token: None 2023-10-12T00:23:08.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I= 2023-10-12T00:23:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I=', 'Content-Length': '0'} 2023-10-12T00:23:08.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004752e5dc09985bf-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.766 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.766 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.767 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.767 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.767 DEBUG:boto:Method: GET 2023-10-12T00:23:08.768 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.768 DEBUG:boto:Data: 2023-10-12T00:23:08.769 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.770 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.770 DEBUG:boto:Params: {} 2023-10-12T00:23:08.771 DEBUG:boto:Token: None 2023-10-12T00:23:08.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I= 2023-10-12T00:23:08.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I=', 'Content-Length': '0'} 2023-10-12T00:23:08.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a9f538bf9d0907d-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.814 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.815 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.815 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.816 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.816 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.817 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.817 DEBUG:boto:Data: 2023-10-12T00:23:08.817 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.818 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.819 DEBUG:boto:Params: {} 2023-10-12T00:23:08.819 DEBUG:boto:Token: None 2023-10-12T00:23:08.820 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.820 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VYph1tFspmofmoSYVaY6BLmocdM= 2023-10-12T00:23:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VYph1tFspmofmoSYVaY6BLmocdM=', 'Content-Length': '0'} 2023-10-12T00:23:08.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e441124b495b1b-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.823 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.823 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.824 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.824 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.824 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.825 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.825 DEBUG:boto:Data: 2023-10-12T00:23:08.826 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.826 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.827 DEBUG:boto:Params: {} 2023-10-12T00:23:08.827 DEBUG:boto:Token: None 2023-10-12T00:23:08.828 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.828 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VYph1tFspmofmoSYVaY6BLmocdM= 2023-10-12T00:23:08.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VYph1tFspmofmoSYVaY6BLmocdM=', 'Content-Length': '0'} 2023-10-12T00:23:08.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2ad261a408f57a0-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:08.832 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.832 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.833 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.833 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.834 DEBUG:boto:Method: GET 2023-10-12T00:23:08.834 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.834 DEBUG:boto:Data: 2023-10-12T00:23:08.835 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.836 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.836 DEBUG:boto:Params: {} 2023-10-12T00:23:08.837 DEBUG:boto:Token: None 2023-10-12T00:23:08.837 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I= 2023-10-12T00:23:08.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I=', 'Content-Length': '0'} 2023-10-12T00:23:08.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a525be040d2cd0b-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.841 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.841 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.842 DEBUG:boto:path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.842 DEBUG:boto:auth_path=/wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.842 DEBUG:boto:Method: GET 2023-10-12T00:23:08.843 DEBUG:boto:Path: /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.843 DEBUG:boto:Data: 2023-10-12T00:23:08.844 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.845 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.845 DEBUG:boto:Params: {} 2023-10-12T00:23:08.845 DEBUG:boto:Token: None 2023-10-12T00:23:08.846 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj?versionId=null 2023-10-12T00:23:08.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I= 2023-10-12T00:23:08.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G7nPJ39+625ZpHPGjcMhR451m3I=', 'Content-Length': '0'} 2023-10-12T00:23:08.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b86d1ea4a63c150f-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.849 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.850 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.850 DEBUG:boto:path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.851 DEBUG:boto:auth_path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.851 DEBUG:boto:Method: GET 2023-10-12T00:23:08.852 DEBUG:boto:Path: /wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.852 DEBUG:boto:Data: 2023-10-12T00:23:08.853 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.853 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.854 DEBUG:boto:Params: {} 2023-10-12T00:23:08.854 DEBUG:boto:Token: None 2023-10-12T00:23:08.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.855 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccb4c06faad46dc-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.859 DEBUG:boto:b'wafmoa-8myobj1000falsemyobj2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.859 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.860 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.860 DEBUG:boto:path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.861 DEBUG:boto:auth_path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.861 DEBUG:boto:Method: GET 2023-10-12T00:23:08.862 DEBUG:boto:Path: /wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.862 DEBUG:boto:Data: 2023-10-12T00:23:08.862 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.863 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.864 DEBUG:boto:Params: {} 2023-10-12T00:23:08.864 DEBUG:boto:Token: None 2023-10-12T00:23:08.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014ebc1bc3049cd19-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.868 DEBUG:boto:b'wafmoa-8myobj1000falsemyobj2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.869 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.870 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.870 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.871 DEBUG:boto:Path: /wafmoa-8/myobj 2023-10-12T00:23:08.871 DEBUG:boto:Data: 2023-10-12T00:23:08.871 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.872 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.873 DEBUG:boto:Params: {} 2023-10-12T00:23:08.873 DEBUG:boto:Token: None 2023-10-12T00:23:08.874 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj 2023-10-12T00:23:08.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Aw39Nb37Fy2K19uD991mNlhuUe8= 2023-10-12T00:23:08.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Aw39Nb37Fy2K19uD991mNlhuUe8=', 'Content-Length': '0'} 2023-10-12T00:23:08.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003df967ad0a268385-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.877 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.877 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.878 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.878 DEBUG:boto:Path: /wafmoa-8/myobj 2023-10-12T00:23:08.879 DEBUG:boto:Data: 2023-10-12T00:23:08.879 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.880 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.880 DEBUG:boto:Params: {} 2023-10-12T00:23:08.881 DEBUG:boto:Token: None 2023-10-12T00:23:08.881 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj 2023-10-12T00:23:08.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Aw39Nb37Fy2K19uD991mNlhuUe8= 2023-10-12T00:23:08.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Aw39Nb37Fy2K19uD991mNlhuUe8=', 'Content-Length': '0'} 2023-10-12T00:23:08.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f3cc1065e678eb-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:08.885 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.886 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.886 DEBUG:boto:Method: GET 2023-10-12T00:23:08.886 DEBUG:boto:Path: /wafmoa-8/myobj 2023-10-12T00:23:08.887 DEBUG:boto:Data: 2023-10-12T00:23:08.887 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.888 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.889 DEBUG:boto:Params: {} 2023-10-12T00:23:08.889 DEBUG:boto:Token: None 2023-10-12T00:23:08.889 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj 2023-10-12T00:23:08.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GJMlIvZuMyxco2B6/QfGJrRlL1Y= 2023-10-12T00:23:08.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GJMlIvZuMyxco2B6/QfGJrRlL1Y=', 'Content-Length': '0'} 2023-10-12T00:23:08.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042c70c55a8da9c4e-0065273c6c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.894 DEBUG:boto:path=/wafmoa-8/myobj 2023-10-12T00:23:08.894 DEBUG:boto:auth_path=/wafmoa-8/myobj 2023-10-12T00:23:08.895 DEBUG:boto:Method: GET 2023-10-12T00:23:08.895 DEBUG:boto:Path: /wafmoa-8/myobj 2023-10-12T00:23:08.896 DEBUG:boto:Data: 2023-10-12T00:23:08.896 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.897 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.897 DEBUG:boto:Params: {} 2023-10-12T00:23:08.898 DEBUG:boto:Token: None 2023-10-12T00:23:08.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/myobj 2023-10-12T00:23:08.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GJMlIvZuMyxco2B6/QfGJrRlL1Y= 2023-10-12T00:23:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GJMlIvZuMyxco2B6/QfGJrRlL1Y=', 'Content-Length': '0'} 2023-10-12T00:23:08.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:20:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fefc23ffbb523faf-0065273c6c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.902 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.903 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.903 DEBUG:boto:path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.903 DEBUG:boto:auth_path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.904 DEBUG:boto:Method: GET 2023-10-12T00:23:08.904 DEBUG:boto:Path: /wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.905 DEBUG:boto:Data: 2023-10-12T00:23:08.905 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.906 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.906 DEBUG:boto:Params: {} 2023-10-12T00:23:08.907 DEBUG:boto:Token: None 2023-10-12T00:23:08.907 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092afb691d139de35-0065273c6c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.911 DEBUG:boto:b'wafmoa-8myobj1000falsemyobj2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.912 DEBUG:boto:path=/wafmoa-8/ 2023-10-12T00:23:08.912 DEBUG:boto:auth_path=/wafmoa-8/ 2023-10-12T00:23:08.913 DEBUG:boto:path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.913 DEBUG:boto:auth_path=/wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.914 DEBUG:boto:Method: GET 2023-10-12T00:23:08.914 DEBUG:boto:Path: /wafmoa-8/?prefix=myobj 2023-10-12T00:23:08.914 DEBUG:boto:Data: 2023-10-12T00:23:08.915 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.916 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.916 DEBUG:boto:Params: {} 2023-10-12T00:23:08.917 DEBUG:boto:Token: None 2023-10-12T00:23:08.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-8/ 2023-10-12T00:23:08.918 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo= 2023-10-12T00:23:08.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8dLtV8RF49QIIZ/XNj1FBnVMyo=', 'Content-Length': '0'} 2023-10-12T00:23:08.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff83509b0e38b7ad-0065273c6c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.921 DEBUG:boto:b'wafmoa-8myobj1000falsemyobj2023-10-12T00:20:51.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:08.921 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-8 zones={a1, a2} 2023-10-12T00:23:08.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-10-12T00:23:08.923 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-9 2023-10-12T00:23:08.924 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:08.924 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:08.924 DEBUG:boto:Method: PUT 2023-10-12T00:23:08.925 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:08.925 DEBUG:boto:Data: 2023-10-12T00:23:08.926 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.926 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.927 DEBUG:boto:Params: {} 2023-10-12T00:23:08.927 DEBUG:boto:Token: None 2023-10-12T00:23:08.928 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-9/ 2023-10-12T00:23:08.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kCMIJwhZ6nDCK3cjuq0x9ni73aM= 2023-10-12T00:23:08.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kCMIJwhZ6nDCK3cjuq0x9ni73aM=', 'Content-Length': '0'} 2023-10-12T00:23:08.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004231075038eb579d-0065273c6c-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.941 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-10 2023-10-12T00:23:08.942 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:08.942 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:08.943 DEBUG:boto:Method: PUT 2023-10-12T00:23:08.943 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:23:08.944 DEBUG:boto:Data: 2023-10-12T00:23:08.944 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:08.945 DEBUG:boto:Port: 8001 2023-10-12T00:23:08.945 DEBUG:boto:Params: {} 2023-10-12T00:23:08.946 DEBUG:boto:Token: None 2023-10-12T00:23:08.946 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-10/ 2023-10-12T00:23:08.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1RW9Js8/Y0iDuKkLqd7yOWByauM= 2023-10-12T00:23:08.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1RW9Js8/Y0iDuKkLqd7yOWByauM=', 'Content-Length': '0'} 2023-10-12T00:23:08.970 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3942c1d3b6a9e6b-0065273c6c-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.971 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:08.971 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:08.972 DEBUG:boto:Method: HEAD 2023-10-12T00:23:08.972 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:08.973 DEBUG:boto:Data: 2023-10-12T00:23:08.973 DEBUG:boto:Headers: {} 2023-10-12T00:23:08.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.974 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.974 DEBUG:boto:Params: {} 2023-10-12T00:23:08.975 DEBUG:boto:Token: None 2023-10-12T00:23:08.975 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-9/ 2023-10-12T00:23:08.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VnKAkdVKzICkJdoJIMuRrfjWGYA= 2023-10-12T00:23:08.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VnKAkdVKzICkJdoJIMuRrfjWGYA=', 'Content-Length': '0'} 2023-10-12T00:23:08.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ff15f21bf76556e-0065273c6c-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:08.979 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:08.980 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:08.980 DEBUG:boto:Method: PUT 2023-10-12T00:23:08.981 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:08.981 DEBUG:boto:Data: 2023-10-12T00:23:08.981 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:23:08.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:08.982 DEBUG:boto:Port: 8000 2023-10-12T00:23:08.983 DEBUG:boto:Params: {} 2023-10-12T00:23:08.983 DEBUG:boto:Token: None 2023-10-12T00:23:08.984 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:23:08 GMT /wafmoa-9/myobj 2023-10-12T00:23:08.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w2sG1L1EGa8EYMtELi9phXzhXtg= 2023-10-12T00:23:08.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:23:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w2sG1L1EGa8EYMtELi9phXzhXtg='} 2023-10-12T00:23:09.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003411461263da718d-0065273c6c-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:09.027 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:09.028 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:09.028 DEBUG:boto:Method: HEAD 2023-10-12T00:23:09.028 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:23:09.029 DEBUG:boto:Data: 2023-10-12T00:23:09.029 DEBUG:boto:Headers: {} 2023-10-12T00:23:09.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:09.030 DEBUG:boto:Port: 8001 2023-10-12T00:23:09.031 DEBUG:boto:Params: {} 2023-10-12T00:23:09.031 DEBUG:boto:Token: None 2023-10-12T00:23:09.032 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:09 GMT /wafmoa-10/ 2023-10-12T00:23:09.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oI7KNenb1h0fdUdzPAujmZ0DNgA= 2023-10-12T00:23:09.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oI7KNenb1h0fdUdzPAujmZ0DNgA=', 'Content-Length': '0'} 2023-10-12T00:23:09.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbcb391946f95bb0-0065273c6d-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:09.035 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:09.036 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:09.036 DEBUG:boto:Method: PUT 2023-10-12T00:23:09.037 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:09.037 DEBUG:boto:Data: 2023-10-12T00:23:09.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:23:09.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:09.038 DEBUG:boto:Port: 8001 2023-10-12T00:23:09.039 DEBUG:boto:Params: {} 2023-10-12T00:23:09.039 DEBUG:boto:Token: None 2023-10-12T00:23:09.040 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:23:09 GMT /wafmoa-10/myobj 2023-10-12T00:23:09.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QxZdF3QcGmcKHGMek7q+1bNvZ6k= 2023-10-12T00:23:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:23:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QxZdF3QcGmcKHGMek7q+1bNvZ6k='} 2023-10-12T00:23:09.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4432e8cfc4061d9-0065273c6d-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:09.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:23:09.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:23:09.084 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:23:09.136 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:09.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:09.169 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:23:09.181 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:23:09.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.181 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:23:09.182 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:23:09.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:23:09.183 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:23:09.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.183 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.960271_92.1", 2023-10-12T00:23:09.183 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.960271Z" 2023-10-12T00:23:09.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.184 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:23:09.184 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:23:09.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.185 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:23:09.186 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070188.960271_92.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:23:09.187 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:23:09.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:23:09.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:23:09.188 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:23:09.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:09.238 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:09.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:09.267+0000 7fb62037f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:23:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:09.268+0000 7fb62037f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:23:09.282 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:23:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:23:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:23:09.282 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:23:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:23:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:23:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:23:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.283 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:23:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:23:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:23:09.284 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:23:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:23:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:23:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:23:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:23:09.285 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:23:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:23:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.286 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:23:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:23:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:23:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:23:09.287 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:23:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:23:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:23:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:23:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:23:09.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:23:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:23:09.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:23:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.960271_92.1", 2023-10-12T00:23:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:23:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:23:09.290 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:23:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.960271Z", 2023-10-12T00:23:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:23:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:23:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:23:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:23:09.292 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:23:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:23:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:23:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:23:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:23:09.293 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:23:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:23:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:23:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:23:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:23:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:23:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:23:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:23:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:23:09Z" 2023-10-12T00:23:09.297 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:23:09.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070188.960271_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.960271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:23:09Z" } 2023-10-12T00:23:09.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070188.960271_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.960271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:23:09.299 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070188.960271_92.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:23:09.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070188.960271_92.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:23:09.300 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:23:09.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:09.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:09.302 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:23:09.358 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:09.358 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:09.393+0000 7fc8090cd780 1 waiting to reach incremental sync.. 2023-10-12T00:23:14.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:14.393+0000 7fc8090cd780 1 waiting to reach incremental sync.. 2023-10-12T00:23:19.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:19.394+0000 7fc8090cd780 1 waiting to reach incremental sync.. 2023-10-12T00:23:24.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:24.396+0000 7fc8090cd780 1 bucket sync caught up with source: 2023-10-12T00:23:24.397 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:24.397 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:24.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:24.396+0000 7fc8090cd780 0 bucket checkpoint complete 2023-10-12T00:23:24.413 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-9 zones={a2, a1} 2023-10-12T00:23:24.413 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.414 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.414 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.415 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:24.415 DEBUG:boto:Data: 2023-10-12T00:23:24.416 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.417 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.417 DEBUG:boto:Params: {} 2023-10-12T00:23:24.418 DEBUG:boto:Token: None 2023-10-12T00:23:24.418 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKjZYkVJVcWuPha58NN0uI5RND8= 2023-10-12T00:23:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKjZYkVJVcWuPha58NN0uI5RND8=', 'Content-Length': '0'} 2023-10-12T00:23:24.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000501989c8b8e4c636-0065273c7c-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.423 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.423 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.424 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.425 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:24.426 DEBUG:boto:Data: 2023-10-12T00:23:24.427 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.429 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.429 DEBUG:boto:Params: {} 2023-10-12T00:23:24.430 DEBUG:boto:Token: None 2023-10-12T00:23:24.431 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKjZYkVJVcWuPha58NN0uI5RND8= 2023-10-12T00:23:24.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKjZYkVJVcWuPha58NN0uI5RND8=', 'Content-Length': '0'} 2023-10-12T00:23:24.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a3dda697ca6e4a0-0065273c7c-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.436 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:23:24.436 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.437 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.437 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.437 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.438 DEBUG:boto:Method: GET 2023-10-12T00:23:24.438 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.439 DEBUG:boto:Data: 2023-10-12T00:23:24.439 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.440 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.440 DEBUG:boto:Params: {} 2023-10-12T00:23:24.441 DEBUG:boto:Token: None 2023-10-12T00:23:24.441 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/?versions 2023-10-12T00:23:24.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8= 2023-10-12T00:23:24.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8=', 'Content-Length': '0'} 2023-10-12T00:23:24.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016427733f2366983-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.445 DEBUG:boto:b'wafmoa-9999falsemyobjnulltrue2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.446 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:24.447 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:23:24.447 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.447 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.448 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.448 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.449 DEBUG:boto:Method: GET 2023-10-12T00:23:24.449 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.450 DEBUG:boto:Data: 2023-10-12T00:23:24.450 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.451 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.451 DEBUG:boto:Params: {} 2023-10-12T00:23:24.452 DEBUG:boto:Token: None 2023-10-12T00:23:24.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/?versions 2023-10-12T00:23:24.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8= 2023-10-12T00:23:24.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8=', 'Content-Length': '0'} 2023-10-12T00:23:24.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003982b35e819234c-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.456 DEBUG:boto:b'wafmoa-9999falsemyobjnulltrue2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.457 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:24.457 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.458 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.458 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.458 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.459 DEBUG:boto:Method: GET 2023-10-12T00:23:24.459 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.460 DEBUG:boto:Data: 2023-10-12T00:23:24.460 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.461 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.461 DEBUG:boto:Params: {} 2023-10-12T00:23:24.462 DEBUG:boto:Token: None 2023-10-12T00:23:24.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/?versions 2023-10-12T00:23:24.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8= 2023-10-12T00:23:24.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8=', 'Content-Length': '0'} 2023-10-12T00:23:24.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0f5ec937f5a5fa-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.469 DEBUG:boto:b'wafmoa-9999falsemyobjnulltrue2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.470 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.470 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.470 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.471 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.471 DEBUG:boto:Method: GET 2023-10-12T00:23:24.472 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:24.472 DEBUG:boto:Data: 2023-10-12T00:23:24.473 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.473 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.474 DEBUG:boto:Params: {} 2023-10-12T00:23:24.474 DEBUG:boto:Token: None 2023-10-12T00:23:24.475 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/?versions 2023-10-12T00:23:24.475 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8= 2023-10-12T00:23:24.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGCNs0FbMIYCMkL7q4dyQOQ4hq8=', 'Content-Length': '0'} 2023-10-12T00:23:24.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060553371f1f59e5c-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.479 DEBUG:boto:b'wafmoa-9999falsemyobjnulltrue2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.480 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.480 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.481 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.481 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.482 DEBUG:boto:Method: GET 2023-10-12T00:23:24.482 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.483 DEBUG:boto:Data: 2023-10-12T00:23:24.483 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.484 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.484 DEBUG:boto:Params: {} 2023-10-12T00:23:24.485 DEBUG:boto:Token: None 2023-10-12T00:23:24.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek= 2023-10-12T00:23:24.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek=', 'Content-Length': '0'} 2023-10-12T00:23:24.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f2a49af05ee2aca-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.489 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.490 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.490 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.491 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.491 DEBUG:boto:Method: GET 2023-10-12T00:23:24.491 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.492 DEBUG:boto:Data: 2023-10-12T00:23:24.492 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.493 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.493 DEBUG:boto:Params: {} 2023-10-12T00:23:24.494 DEBUG:boto:Token: None 2023-10-12T00:23:24.494 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek= 2023-10-12T00:23:24.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek=', 'Content-Length': '0'} 2023-10-12T00:23:24.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000127517c84104ce0a-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.538 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.539 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.539 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.540 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.540 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.540 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.541 DEBUG:boto:Data: 2023-10-12T00:23:24.541 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.542 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.543 DEBUG:boto:Params: {} 2023-10-12T00:23:24.543 DEBUG:boto:Token: None 2023-10-12T00:23:24.543 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X+SkTIRAbecWCwZhvlsNsKcqzeM= 2023-10-12T00:23:24.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X+SkTIRAbecWCwZhvlsNsKcqzeM=', 'Content-Length': '0'} 2023-10-12T00:23:24.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000501744d40ef3ba07-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.548 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.548 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.548 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.549 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.549 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.550 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.550 DEBUG:boto:Data: 2023-10-12T00:23:24.550 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.551 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.552 DEBUG:boto:Params: {} 2023-10-12T00:23:24.552 DEBUG:boto:Token: None 2023-10-12T00:23:24.553 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X+SkTIRAbecWCwZhvlsNsKcqzeM= 2023-10-12T00:23:24.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X+SkTIRAbecWCwZhvlsNsKcqzeM=', 'Content-Length': '0'} 2023-10-12T00:23:24.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009238e58561e80a9e-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.557 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.557 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.558 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.558 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.559 DEBUG:boto:Method: GET 2023-10-12T00:23:24.559 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.559 DEBUG:boto:Data: 2023-10-12T00:23:24.560 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.561 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.561 DEBUG:boto:Params: {} 2023-10-12T00:23:24.562 DEBUG:boto:Token: None 2023-10-12T00:23:24.562 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.563 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek= 2023-10-12T00:23:24.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek=', 'Content-Length': '0'} 2023-10-12T00:23:24.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecc3dd712af29aa8-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.566 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.566 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.567 DEBUG:boto:path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.567 DEBUG:boto:auth_path=/wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.568 DEBUG:boto:Method: GET 2023-10-12T00:23:24.568 DEBUG:boto:Path: /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.568 DEBUG:boto:Data: 2023-10-12T00:23:24.569 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.570 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.570 DEBUG:boto:Params: {} 2023-10-12T00:23:24.571 DEBUG:boto:Token: None 2023-10-12T00:23:24.571 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj?versionId=null 2023-10-12T00:23:24.572 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek= 2023-10-12T00:23:24.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jgG3c1i/kaLkfbStuL+WuQr0kek=', 'Content-Length': '0'} 2023-10-12T00:23:24.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000474597ea36dbf757-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.575 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.575 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.576 DEBUG:boto:path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.576 DEBUG:boto:auth_path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.577 DEBUG:boto:Method: GET 2023-10-12T00:23:24.577 DEBUG:boto:Path: /wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.577 DEBUG:boto:Data: 2023-10-12T00:23:24.578 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.579 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.579 DEBUG:boto:Params: {} 2023-10-12T00:23:24.580 DEBUG:boto:Token: None 2023-10-12T00:23:24.580 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE= 2023-10-12T00:23:24.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE=', 'Content-Length': '0'} 2023-10-12T00:23:24.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a35b4cc0f189ad-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.585 DEBUG:boto:b'wafmoa-9myobj1000falsemyobj2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.585 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.586 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.586 DEBUG:boto:path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.587 DEBUG:boto:auth_path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.587 DEBUG:boto:Method: GET 2023-10-12T00:23:24.588 DEBUG:boto:Path: /wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.588 DEBUG:boto:Data: 2023-10-12T00:23:24.588 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.589 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.590 DEBUG:boto:Params: {} 2023-10-12T00:23:24.590 DEBUG:boto:Token: None 2023-10-12T00:23:24.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE= 2023-10-12T00:23:24.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE=', 'Content-Length': '0'} 2023-10-12T00:23:24.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70050cb0f720494-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.595 DEBUG:boto:b'wafmoa-9myobj1000falsemyobj2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.595 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.596 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.596 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.597 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:24.597 DEBUG:boto:Data: 2023-10-12T00:23:24.598 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.598 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.599 DEBUG:boto:Params: {} 2023-10-12T00:23:24.599 DEBUG:boto:Token: None 2023-10-12T00:23:24.600 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj 2023-10-12T00:23:24.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfgoQACLN/p8OTXSw90oIF7aXjE= 2023-10-12T00:23:24.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfgoQACLN/p8OTXSw90oIF7aXjE=', 'Content-Length': '0'} 2023-10-12T00:23:24.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c29fcfc20985ac33-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.603 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.604 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.604 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.605 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:24.605 DEBUG:boto:Data: 2023-10-12T00:23:24.605 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.606 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.607 DEBUG:boto:Params: {} 2023-10-12T00:23:24.607 DEBUG:boto:Token: None 2023-10-12T00:23:24.608 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj 2023-10-12T00:23:24.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfgoQACLN/p8OTXSw90oIF7aXjE= 2023-10-12T00:23:24.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfgoQACLN/p8OTXSw90oIF7aXjE=', 'Content-Length': '0'} 2023-10-12T00:23:24.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000034c57d30502a106-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.611 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.612 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.612 DEBUG:boto:Method: GET 2023-10-12T00:23:24.613 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:24.613 DEBUG:boto:Data: 2023-10-12T00:23:24.614 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.614 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.615 DEBUG:boto:Params: {} 2023-10-12T00:23:24.615 DEBUG:boto:Token: None 2023-10-12T00:23:24.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj 2023-10-12T00:23:24.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DezYl6V5pI+TEQx+6EYRXtPmsaY= 2023-10-12T00:23:24.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DezYl6V5pI+TEQx+6EYRXtPmsaY=', 'Content-Length': '0'} 2023-10-12T00:23:24.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc72d228babe8957-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.662 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:24.663 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:24.663 DEBUG:boto:Method: GET 2023-10-12T00:23:24.664 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:24.664 DEBUG:boto:Data: 2023-10-12T00:23:24.665 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.665 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.666 DEBUG:boto:Params: {} 2023-10-12T00:23:24.666 DEBUG:boto:Token: None 2023-10-12T00:23:24.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/myobj 2023-10-12T00:23:24.667 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DezYl6V5pI+TEQx+6EYRXtPmsaY= 2023-10-12T00:23:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DezYl6V5pI+TEQx+6EYRXtPmsaY=', 'Content-Length': '0'} 2023-10-12T00:23:24.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc5e1db73c6c9e03-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.671 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.671 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.672 DEBUG:boto:path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.672 DEBUG:boto:auth_path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.673 DEBUG:boto:Method: GET 2023-10-12T00:23:24.673 DEBUG:boto:Path: /wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.674 DEBUG:boto:Data: 2023-10-12T00:23:24.674 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.675 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.675 DEBUG:boto:Params: {} 2023-10-12T00:23:24.676 DEBUG:boto:Token: None 2023-10-12T00:23:24.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE= 2023-10-12T00:23:24.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE=', 'Content-Length': '0'} 2023-10-12T00:23:24.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad8006beaaa1fa6-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.680 DEBUG:boto:b'wafmoa-9myobj1000falsemyobj2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.681 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:24.682 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:24.682 DEBUG:boto:path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.683 DEBUG:boto:auth_path=/wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.683 DEBUG:boto:Method: GET 2023-10-12T00:23:24.683 DEBUG:boto:Path: /wafmoa-9/?prefix=myobj 2023-10-12T00:23:24.684 DEBUG:boto:Data: 2023-10-12T00:23:24.684 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.685 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.686 DEBUG:boto:Params: {} 2023-10-12T00:23:24.686 DEBUG:boto:Token: None 2023-10-12T00:23:24.687 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-9/ 2023-10-12T00:23:24.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE= 2023-10-12T00:23:24.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RL2Gv5F4vxgl0Z/FDYZFwFLX7WE=', 'Content-Length': '0'} 2023-10-12T00:23:24.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0fd47c8c52e4bc-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.691 DEBUG:boto:b'wafmoa-9myobj1000falsemyobj2023-10-12T00:23:08.987Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.692 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-9 zones={a2, a1} 2023-10-12T00:23:24.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:24.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:24.693 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:23:24.757 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:24.758 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:24.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:24.794+0000 7f7d493cd780 1 bucket sync caught up with source: 2023-10-12T00:23:24.795 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:23:24.795 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:23:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:24.794+0000 7f7d493cd780 0 bucket checkpoint complete 2023-10-12T00:23:24.808 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-10 zones={a1, a2} 2023-10-12T00:23:24.808 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.809 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.809 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.809 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:23:24.810 DEBUG:boto:Data: 2023-10-12T00:23:24.810 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.811 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.812 DEBUG:boto:Params: {} 2023-10-12T00:23:24.812 DEBUG:boto:Token: None 2023-10-12T00:23:24.813 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/ 2023-10-12T00:23:24.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hl7O6rUs1qhXR9uppdYgh906p8U= 2023-10-12T00:23:24.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hl7O6rUs1qhXR9uppdYgh906p8U=', 'Content-Length': '0'} 2023-10-12T00:23:24.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0a04bff353ccac8-0065273c7c-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.816 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.817 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.817 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.818 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:23:24.818 DEBUG:boto:Data: 2023-10-12T00:23:24.819 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.819 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.820 DEBUG:boto:Params: {} 2023-10-12T00:23:24.820 DEBUG:boto:Token: None 2023-10-12T00:23:24.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/ 2023-10-12T00:23:24.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hl7O6rUs1qhXR9uppdYgh906p8U= 2023-10-12T00:23:24.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hl7O6rUs1qhXR9uppdYgh906p8U=', 'Content-Length': '0'} 2023-10-12T00:23:24.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a0729f1508a27a9-0065273c7c-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.825 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:23:24.825 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.826 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.826 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.827 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.827 DEBUG:boto:Method: GET 2023-10-12T00:23:24.827 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.828 DEBUG:boto:Data: 2023-10-12T00:23:24.828 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.829 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.829 DEBUG:boto:Params: {} 2023-10-12T00:23:24.830 DEBUG:boto:Token: None 2023-10-12T00:23:24.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/?versions 2023-10-12T00:23:24.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0= 2023-10-12T00:23:24.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0=', 'Content-Length': '0'} 2023-10-12T00:23:24.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b23494ac6b4a0358-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.834 DEBUG:boto:b'wafmoa-10999falsemyobjnulltrue2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.835 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:24.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:23:24.836 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.836 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.837 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.837 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.838 DEBUG:boto:Method: GET 2023-10-12T00:23:24.838 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.839 DEBUG:boto:Data: 2023-10-12T00:23:24.839 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.840 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.840 DEBUG:boto:Params: {} 2023-10-12T00:23:24.841 DEBUG:boto:Token: None 2023-10-12T00:23:24.841 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/?versions 2023-10-12T00:23:24.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0= 2023-10-12T00:23:24.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0=', 'Content-Length': '0'} 2023-10-12T00:23:24.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f5cf05e3fe4f3e-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.845 DEBUG:boto:b'wafmoa-10999falsemyobjnulltrue2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.846 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-12T00:23:24.846 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.847 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.847 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.847 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.848 DEBUG:boto:Method: GET 2023-10-12T00:23:24.848 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.849 DEBUG:boto:Data: 2023-10-12T00:23:24.849 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.850 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.850 DEBUG:boto:Params: {} 2023-10-12T00:23:24.851 DEBUG:boto:Token: None 2023-10-12T00:23:24.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/?versions 2023-10-12T00:23:24.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0= 2023-10-12T00:23:24.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0=', 'Content-Length': '0'} 2023-10-12T00:23:24.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f19e3e29d3239a-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.855 DEBUG:boto:b'wafmoa-10999falsemyobjnulltrue2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.856 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.857 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.857 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.857 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.858 DEBUG:boto:Method: GET 2023-10-12T00:23:24.858 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:23:24.859 DEBUG:boto:Data: 2023-10-12T00:23:24.859 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.860 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.861 DEBUG:boto:Params: {} 2023-10-12T00:23:24.861 DEBUG:boto:Token: None 2023-10-12T00:23:24.861 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/?versions 2023-10-12T00:23:24.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0= 2023-10-12T00:23:24.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IfT8oXjIOWFVTRMsfOjg010qV/0=', 'Content-Length': '0'} 2023-10-12T00:23:24.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ef7dfb10b2941f-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.865 DEBUG:boto:b'wafmoa-10999falsemyobjnulltrue2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.867 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.867 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.867 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.868 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.868 DEBUG:boto:Method: GET 2023-10-12T00:23:24.869 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.869 DEBUG:boto:Data: 2023-10-12T00:23:24.870 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.871 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.871 DEBUG:boto:Params: {} 2023-10-12T00:23:24.872 DEBUG:boto:Token: None 2023-10-12T00:23:24.872 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk= 2023-10-12T00:23:24.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk=', 'Content-Length': '0'} 2023-10-12T00:23:24.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f5a1f168df5d166-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.876 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.876 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.877 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.877 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.878 DEBUG:boto:Method: GET 2023-10-12T00:23:24.878 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.878 DEBUG:boto:Data: 2023-10-12T00:23:24.879 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.880 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.880 DEBUG:boto:Params: {} 2023-10-12T00:23:24.881 DEBUG:boto:Token: None 2023-10-12T00:23:24.881 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk= 2023-10-12T00:23:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk=', 'Content-Length': '0'} 2023-10-12T00:23:24.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001491348e3fde7370-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.885 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.885 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.886 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.886 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.887 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.887 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.887 DEBUG:boto:Data: 2023-10-12T00:23:24.888 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.889 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.889 DEBUG:boto:Params: {} 2023-10-12T00:23:24.890 DEBUG:boto:Token: None 2023-10-12T00:23:24.890 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I3TdcgSfLn7pVXqEO5TLQeE8mN4= 2023-10-12T00:23:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I3TdcgSfLn7pVXqEO5TLQeE8mN4=', 'Content-Length': '0'} 2023-10-12T00:23:24.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091af630bdc642672-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.893 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.894 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.894 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.895 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.895 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.896 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.896 DEBUG:boto:Data: 2023-10-12T00:23:24.896 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.897 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.898 DEBUG:boto:Params: {} 2023-10-12T00:23:24.898 DEBUG:boto:Token: None 2023-10-12T00:23:24.899 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I3TdcgSfLn7pVXqEO5TLQeE8mN4= 2023-10-12T00:23:24.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I3TdcgSfLn7pVXqEO5TLQeE8mN4=', 'Content-Length': '0'} 2023-10-12T00:23:24.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2cf056fbfa2f90a-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.902 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.903 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.903 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.904 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.904 DEBUG:boto:Method: GET 2023-10-12T00:23:24.905 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.905 DEBUG:boto:Data: 2023-10-12T00:23:24.905 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.906 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.907 DEBUG:boto:Params: {} 2023-10-12T00:23:24.907 DEBUG:boto:Token: None 2023-10-12T00:23:24.908 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk= 2023-10-12T00:23:24.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk=', 'Content-Length': '0'} 2023-10-12T00:23:24.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003de9a5f9f579ab11-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.954 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.955 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.955 DEBUG:boto:path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.956 DEBUG:boto:auth_path=/wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.956 DEBUG:boto:Method: GET 2023-10-12T00:23:24.957 DEBUG:boto:Path: /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.957 DEBUG:boto:Data: 2023-10-12T00:23:24.958 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.959 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.959 DEBUG:boto:Params: {} 2023-10-12T00:23:24.959 DEBUG:boto:Token: None 2023-10-12T00:23:24.960 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj?versionId=null 2023-10-12T00:23:24.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk= 2023-10-12T00:23:24.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uTL4j8L82GGj51hJ/dinBrNEgEk=', 'Content-Length': '0'} 2023-10-12T00:23:24.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ecd3fc2cef58833-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.964 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.964 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.964 DEBUG:boto:path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.965 DEBUG:boto:auth_path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.965 DEBUG:boto:Method: GET 2023-10-12T00:23:24.966 DEBUG:boto:Path: /wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.966 DEBUG:boto:Data: 2023-10-12T00:23:24.967 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.967 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.968 DEBUG:boto:Params: {} 2023-10-12T00:23:24.968 DEBUG:boto:Token: None 2023-10-12T00:23:24.969 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/ 2023-10-12T00:23:24.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UmYhRhILKrbxKte+HhasvsPBLWs= 2023-10-12T00:23:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UmYhRhILKrbxKte+HhasvsPBLWs=', 'Content-Length': '0'} 2023-10-12T00:23:24.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325ebfa20b7401ca-0065273c7c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.973 DEBUG:boto:b'wafmoa-10myobj1000falsemyobj2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.973 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:24.974 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:24.974 DEBUG:boto:path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.975 DEBUG:boto:auth_path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.975 DEBUG:boto:Method: GET 2023-10-12T00:23:24.975 DEBUG:boto:Path: /wafmoa-10/?prefix=myobj 2023-10-12T00:23:24.976 DEBUG:boto:Data: 2023-10-12T00:23:24.976 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.977 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.978 DEBUG:boto:Params: {} 2023-10-12T00:23:24.978 DEBUG:boto:Token: None 2023-10-12T00:23:24.979 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/ 2023-10-12T00:23:24.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UmYhRhILKrbxKte+HhasvsPBLWs= 2023-10-12T00:23:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UmYhRhILKrbxKte+HhasvsPBLWs=', 'Content-Length': '0'} 2023-10-12T00:23:24.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bddb13b87c6decc3-0065273c7c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.983 DEBUG:boto:b'wafmoa-10myobj1000falsemyobj2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:24.983 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.984 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.984 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.985 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:24.985 DEBUG:boto:Data: 2023-10-12T00:23:24.986 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:24.986 DEBUG:boto:Port: 8000 2023-10-12T00:23:24.987 DEBUG:boto:Params: {} 2023-10-12T00:23:24.987 DEBUG:boto:Token: None 2023-10-12T00:23:24.988 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj 2023-10-12T00:23:24.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SLcJNP69xWoM0i6esm9W1CttrtA= 2023-10-12T00:23:24.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SLcJNP69xWoM0i6esm9W1CttrtA=', 'Content-Length': '0'} 2023-10-12T00:23:24.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093c66871beceea13-0065273c7c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.991 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.991 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.992 DEBUG:boto:Method: HEAD 2023-10-12T00:23:24.992 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:24.993 DEBUG:boto:Data: 2023-10-12T00:23:24.993 DEBUG:boto:Headers: {} 2023-10-12T00:23:24.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:24.994 DEBUG:boto:Port: 8001 2023-10-12T00:23:24.994 DEBUG:boto:Params: {} 2023-10-12T00:23:24.995 DEBUG:boto:Token: None 2023-10-12T00:23:24.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:24 GMT /wafmoa-10/myobj 2023-10-12T00:23:24.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SLcJNP69xWoM0i6esm9W1CttrtA= 2023-10-12T00:23:24.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SLcJNP69xWoM0i6esm9W1CttrtA=', 'Content-Length': '0'} 2023-10-12T00:23:24.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006345f2e67d85d833-0065273c7c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:24.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-12T00:23:24.999 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:24.999 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:24.999 DEBUG:boto:Method: GET 2023-10-12T00:23:25.000 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:25.000 DEBUG:boto:Data: 2023-10-12T00:23:25.001 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:25.001 DEBUG:boto:Port: 8000 2023-10-12T00:23:25.002 DEBUG:boto:Params: {} 2023-10-12T00:23:25.002 DEBUG:boto:Token: None 2023-10-12T00:23:25.003 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-10/myobj 2023-10-12T00:23:25.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFd5D6QaHXwuaZTRqcY5JVLw2MI= 2023-10-12T00:23:25.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFd5D6QaHXwuaZTRqcY5JVLw2MI=', 'Content-Length': '0'} 2023-10-12T00:23:25.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee3083abad6a9887-0065273c7d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.006 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:25.007 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:25.007 DEBUG:boto:Method: GET 2023-10-12T00:23:25.007 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:25.008 DEBUG:boto:Data: 2023-10-12T00:23:25.008 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:25.009 DEBUG:boto:Port: 8001 2023-10-12T00:23:25.009 DEBUG:boto:Params: {} 2023-10-12T00:23:25.010 DEBUG:boto:Token: None 2023-10-12T00:23:25.010 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-10/myobj 2023-10-12T00:23:25.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFd5D6QaHXwuaZTRqcY5JVLw2MI= 2023-10-12T00:23:25.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFd5D6QaHXwuaZTRqcY5JVLw2MI=', 'Content-Length': '0'} 2023-10-12T00:23:25.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c24be049fdb80b2d-0065273c7d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.054 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:25.055 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:25.055 DEBUG:boto:path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.056 DEBUG:boto:auth_path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.056 DEBUG:boto:Method: GET 2023-10-12T00:23:25.057 DEBUG:boto:Path: /wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.057 DEBUG:boto:Data: 2023-10-12T00:23:25.057 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:25.058 DEBUG:boto:Port: 8000 2023-10-12T00:23:25.059 DEBUG:boto:Params: {} 2023-10-12T00:23:25.059 DEBUG:boto:Token: None 2023-10-12T00:23:25.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-10/ 2023-10-12T00:23:25.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BvQDEoF01/JER/oKYY3iNsrzD6I= 2023-10-12T00:23:25.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BvQDEoF01/JER/oKYY3iNsrzD6I=', 'Content-Length': '0'} 2023-10-12T00:23:25.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4841e75a9e3cc8-0065273c7d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.064 DEBUG:boto:b'wafmoa-10myobj1000falsemyobj2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:25.065 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:25.065 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:25.065 DEBUG:boto:path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.066 DEBUG:boto:auth_path=/wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.066 DEBUG:boto:Method: GET 2023-10-12T00:23:25.067 DEBUG:boto:Path: /wafmoa-10/?prefix=myobj 2023-10-12T00:23:25.067 DEBUG:boto:Data: 2023-10-12T00:23:25.067 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:25.068 DEBUG:boto:Port: 8001 2023-10-12T00:23:25.069 DEBUG:boto:Params: {} 2023-10-12T00:23:25.069 DEBUG:boto:Token: None 2023-10-12T00:23:25.070 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-10/ 2023-10-12T00:23:25.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BvQDEoF01/JER/oKYY3iNsrzD6I= 2023-10-12T00:23:25.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BvQDEoF01/JER/oKYY3iNsrzD6I=', 'Content-Length': '0'} 2023-10-12T00:23:25.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4a14d74a146bdd3-0065273c7d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.074 DEBUG:boto:b'wafmoa-10myobj1000falsemyobj2023-10-12T00:23:09.043Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:23:25.074 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-10 zones={a1, a2} 2023-10-12T00:23:25.075 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:25.075 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:25.076 DEBUG:boto:Method: HEAD 2023-10-12T00:23:25.076 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:25.076 DEBUG:boto:Data: 2023-10-12T00:23:25.078 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:25.079 DEBUG:boto:Port: 8000 2023-10-12T00:23:25.080 DEBUG:boto:Params: {} 2023-10-12T00:23:25.080 DEBUG:boto:Token: None 2023-10-12T00:23:25.080 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-9/ 2023-10-12T00:23:25.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YYqWxslgB1sw7t2Asxu5M7ZMAcE= 2023-10-12T00:23:25.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YYqWxslgB1sw7t2Asxu5M7ZMAcE=', 'Content-Length': '0'} 2023-10-12T00:23:25.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000472f34003b42579c-0065273c7d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.084 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:25.084 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:25.085 DEBUG:boto:Method: HEAD 2023-10-12T00:23:25.085 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:25.085 DEBUG:boto:Data: 2023-10-12T00:23:25.086 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:25.087 DEBUG:boto:Port: 8000 2023-10-12T00:23:25.087 DEBUG:boto:Params: {} 2023-10-12T00:23:25.087 DEBUG:boto:Token: None 2023-10-12T00:23:25.088 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-9/myobj 2023-10-12T00:23:25.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DwiTOKJcxawBqfgEkwDnMftfb+k= 2023-10-12T00:23:25.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DwiTOKJcxawBqfgEkwDnMftfb+k=', 'Content-Length': '0'} 2023-10-12T00:23:25.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f996c5264c7b82c7-0065273c7d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.091 DEBUG:boto:path=/wafmoa-9/myobj 2023-10-12T00:23:25.092 DEBUG:boto:auth_path=/wafmoa-9/myobj 2023-10-12T00:23:25.092 DEBUG:boto:Method: DELETE 2023-10-12T00:23:25.092 DEBUG:boto:Path: /wafmoa-9/myobj 2023-10-12T00:23:25.093 DEBUG:boto:Data: 2023-10-12T00:23:25.093 DEBUG:boto:Headers: {} 2023-10-12T00:23:25.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:25.094 DEBUG:boto:Port: 8000 2023-10-12T00:23:25.094 DEBUG:boto:Params: {} 2023-10-12T00:23:25.095 DEBUG:boto:Token: None 2023-10-12T00:23:25.095 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:23:25 GMT /wafmoa-9/myobj 2023-10-12T00:23:25.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K0NxnMPLB3hvjA/buFog8jU3+To= 2023-10-12T00:23:25.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K0NxnMPLB3hvjA/buFog8jU3+To=', 'Content-Length': '0'} 2023-10-12T00:23:25.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f35696aaa6249885-0065273c7d-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:23:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:25.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:25.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:25.101 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:23:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:25.160 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:25.193+0000 7f1f06e13780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:25.194 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:25.195 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:30.195+0000 7f1f06e13780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:30.196 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:30.196 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:35.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:35.197+0000 7f1f06e13780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:35.197 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:35.198 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:40.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:40.198+0000 7f1f06e13780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:40.199 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2023-10-12T00:23:40.199 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:45.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:45.200+0000 7f1f06e13780 1 bucket sync caught up with source: 2023-10-12T00:23:45.200 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:45.200 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-10-12T00:23:45.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:45.200+0000 7f1f06e13780 0 bucket checkpoint complete 2023-10-12T00:23:45.215 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-9 zones={a2, a1} 2023-10-12T00:23:45.216 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.216 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.217 DEBUG:boto:Method: HEAD 2023-10-12T00:23:45.217 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:45.217 DEBUG:boto:Data: 2023-10-12T00:23:45.218 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.219 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.219 DEBUG:boto:Params: {} 2023-10-12T00:23:45.220 DEBUG:boto:Token: None 2023-10-12T00:23:45.220 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/ 2023-10-12T00:23:45.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzJZg2O5MUNS83g9L0C30rP0A4A= 2023-10-12T00:23:45.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzJZg2O5MUNS83g9L0C30rP0A4A=', 'Content-Length': '0'} 2023-10-12T00:23:45.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000409aa124cd298ae9-0065273c91-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.225 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.225 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.225 DEBUG:boto:Method: HEAD 2023-10-12T00:23:45.226 DEBUG:boto:Path: /wafmoa-9/ 2023-10-12T00:23:45.226 DEBUG:boto:Data: 2023-10-12T00:23:45.227 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:45.227 DEBUG:boto:Port: 8000 2023-10-12T00:23:45.228 DEBUG:boto:Params: {} 2023-10-12T00:23:45.228 DEBUG:boto:Token: None 2023-10-12T00:23:45.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/ 2023-10-12T00:23:45.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzJZg2O5MUNS83g9L0C30rP0A4A= 2023-10-12T00:23:45.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzJZg2O5MUNS83g9L0C30rP0A4A=', 'Content-Length': '0'} 2023-10-12T00:23:45.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000215b17a3416de3a0-0065273c91-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.233 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:23:45.233 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.234 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.234 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.234 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.235 DEBUG:boto:Method: GET 2023-10-12T00:23:45.235 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.236 DEBUG:boto:Data: 2023-10-12T00:23:45.236 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.237 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.237 DEBUG:boto:Params: {} 2023-10-12T00:23:45.238 DEBUG:boto:Token: None 2023-10-12T00:23:45.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/?versions 2023-10-12T00:23:45.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc= 2023-10-12T00:23:45.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc=', 'Content-Length': '0'} 2023-10-12T00:23:45.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e04b89df7d894e3-0065273c91-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.242 DEBUG:boto:b'wafmoa-9999false' 2023-10-12T00:23:45.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:23:45.244 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.244 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.244 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.245 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.245 DEBUG:boto:Method: GET 2023-10-12T00:23:45.246 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.246 DEBUG:boto:Data: 2023-10-12T00:23:45.247 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:45.248 DEBUG:boto:Port: 8000 2023-10-12T00:23:45.248 DEBUG:boto:Params: {} 2023-10-12T00:23:45.248 DEBUG:boto:Token: None 2023-10-12T00:23:45.249 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/?versions 2023-10-12T00:23:45.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc= 2023-10-12T00:23:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc=', 'Content-Length': '0'} 2023-10-12T00:23:45.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863ed697f5fc252e-0065273c91-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.253 DEBUG:boto:b'wafmoa-9999false' 2023-10-12T00:23:45.254 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.254 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.254 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.255 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.255 DEBUG:boto:Method: GET 2023-10-12T00:23:45.256 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.256 DEBUG:boto:Data: 2023-10-12T00:23:45.257 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.257 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.258 DEBUG:boto:Params: {} 2023-10-12T00:23:45.258 DEBUG:boto:Token: None 2023-10-12T00:23:45.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/?versions 2023-10-12T00:23:45.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc= 2023-10-12T00:23:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc=', 'Content-Length': '0'} 2023-10-12T00:23:45.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff9dc675a02d9871-0065273c91-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.263 DEBUG:boto:b'wafmoa-9999false' 2023-10-12T00:23:45.263 DEBUG:boto:path=/wafmoa-9/ 2023-10-12T00:23:45.264 DEBUG:boto:auth_path=/wafmoa-9/ 2023-10-12T00:23:45.264 DEBUG:boto:path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.264 DEBUG:boto:auth_path=/wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.265 DEBUG:boto:Method: GET 2023-10-12T00:23:45.265 DEBUG:boto:Path: /wafmoa-9/?versions&max-keys=999 2023-10-12T00:23:45.266 DEBUG:boto:Data: 2023-10-12T00:23:45.266 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:23:45.267 DEBUG:boto:Port: 8000 2023-10-12T00:23:45.267 DEBUG:boto:Params: {} 2023-10-12T00:23:45.268 DEBUG:boto:Token: None 2023-10-12T00:23:45.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-9/?versions 2023-10-12T00:23:45.269 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc= 2023-10-12T00:23:45.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hONP53kbxs7FJzxGVkr92t28mLc=', 'Content-Length': '0'} 2023-10-12T00:23:45.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc8dd9de3780f67a-0065273c91-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.273 DEBUG:boto:b'wafmoa-9999false' 2023-10-12T00:23:45.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-9 zones={a2, a1} 2023-10-12T00:23:45.275 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:23:45.275 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:23:45.275 DEBUG:boto:Method: HEAD 2023-10-12T00:23:45.276 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:23:45.276 DEBUG:boto:Data: 2023-10-12T00:23:45.277 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.277 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.278 DEBUG:boto:Params: {} 2023-10-12T00:23:45.278 DEBUG:boto:Token: None 2023-10-12T00:23:45.279 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-10/ 2023-10-12T00:23:45.279 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BIPH4o3CkEMSWZg4Z8gkmNsRpJM= 2023-10-12T00:23:45.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BIPH4o3CkEMSWZg4Z8gkmNsRpJM=', 'Content-Length': '0'} 2023-10-12T00:23:45.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336e9e627906e806-0065273c91-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.283 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:45.283 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:45.284 DEBUG:boto:Method: HEAD 2023-10-12T00:23:45.284 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:45.284 DEBUG:boto:Data: 2023-10-12T00:23:45.285 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.285 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.286 DEBUG:boto:Params: {} 2023-10-12T00:23:45.286 DEBUG:boto:Token: None 2023-10-12T00:23:45.287 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-10/myobj 2023-10-12T00:23:45.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZseRfAJv4b4dg2SEnSZglSKHQmE= 2023-10-12T00:23:45.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZseRfAJv4b4dg2SEnSZglSKHQmE=', 'Content-Length': '0'} 2023-10-12T00:23:45.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:23:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a4423a835f24745-0065273c91-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.290 DEBUG:boto:path=/wafmoa-10/myobj 2023-10-12T00:23:45.291 DEBUG:boto:auth_path=/wafmoa-10/myobj 2023-10-12T00:23:45.291 DEBUG:boto:Method: DELETE 2023-10-12T00:23:45.292 DEBUG:boto:Path: /wafmoa-10/myobj 2023-10-12T00:23:45.292 DEBUG:boto:Data: 2023-10-12T00:23:45.292 DEBUG:boto:Headers: {} 2023-10-12T00:23:45.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:23:45.293 DEBUG:boto:Port: 8001 2023-10-12T00:23:45.294 DEBUG:boto:Params: {} 2023-10-12T00:23:45.294 DEBUG:boto:Token: None 2023-10-12T00:23:45.295 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:23:45 GMT /wafmoa-10/myobj 2023-10-12T00:23:45.295 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vqB3H6LOf6y/Tc8bjqiQhPgMGc0= 2023-10-12T00:23:45.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:23:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vqB3H6LOf6y/Tc8bjqiQhPgMGc0=', 'Content-Length': '0'} 2023-10-12T00:23:45.301 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003530241a8264b903-0065273c91-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:23:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:23:45.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:45.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:23:45.303 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:23:45.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:23:45.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:23:45.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:45.401+0000 7f854ad1d780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:45.402 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:23:45.402 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:23:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:50.402+0000 7f854ad1d780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:50.403 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:23:50.404 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:23:55.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:23:55.403+0000 7f854ad1d780 1 waiting for incremental sync to catch up: 2023-10-12T00:23:55.405 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:23:55.405 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:24:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:00.404+0000 7f854ad1d780 1 waiting for incremental sync to catch up: 2023-10-12T00:24:00.406 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-12T00:24:00.406 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:24:05.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:05.406+0000 7f854ad1d780 1 bucket sync caught up with source: 2023-10-12T00:24:05.407 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:24:05.407 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.16.6, , , , , , , , , ] 2023-10-12T00:24:05.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:05.406+0000 7f854ad1d780 0 bucket checkpoint complete 2023-10-12T00:24:05.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-10 zones={a1, a2} 2023-10-12T00:24:05.423 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.424 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.425 DEBUG:boto:Method: HEAD 2023-10-12T00:24:05.425 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:24:05.426 DEBUG:boto:Data: 2023-10-12T00:24:05.427 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:05.428 DEBUG:boto:Port: 8000 2023-10-12T00:24:05.428 DEBUG:boto:Params: {} 2023-10-12T00:24:05.429 DEBUG:boto:Token: None 2023-10-12T00:24:05.429 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/ 2023-10-12T00:24:05.430 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbcSWstLJM8cePVNDfGIb+zPXrw= 2023-10-12T00:24:05.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbcSWstLJM8cePVNDfGIb+zPXrw=', 'Content-Length': '0'} 2023-10-12T00:24:05.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b35fcfc5e10a3a3-0065273ca5-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.434 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.434 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.435 DEBUG:boto:Method: HEAD 2023-10-12T00:24:05.435 DEBUG:boto:Path: /wafmoa-10/ 2023-10-12T00:24:05.435 DEBUG:boto:Data: 2023-10-12T00:24:05.436 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:05.437 DEBUG:boto:Port: 8001 2023-10-12T00:24:05.437 DEBUG:boto:Params: {} 2023-10-12T00:24:05.437 DEBUG:boto:Token: None 2023-10-12T00:24:05.438 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/ 2023-10-12T00:24:05.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbcSWstLJM8cePVNDfGIb+zPXrw= 2023-10-12T00:24:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbcSWstLJM8cePVNDfGIb+zPXrw=', 'Content-Length': '0'} 2023-10-12T00:24:05.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5c0770ae68f7fbd-0065273ca5-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:24:05.443 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.445 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.446 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.446 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.447 DEBUG:boto:Method: GET 2023-10-12T00:24:05.447 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.448 DEBUG:boto:Data: 2023-10-12T00:24:05.448 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:05.449 DEBUG:boto:Port: 8000 2023-10-12T00:24:05.449 DEBUG:boto:Params: {} 2023-10-12T00:24:05.450 DEBUG:boto:Token: None 2023-10-12T00:24:05.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/?versions 2023-10-12T00:24:05.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw= 2023-10-12T00:24:05.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw=', 'Content-Length': '0'} 2023-10-12T00:24:05.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea28fe40aa0ed38-0065273ca5-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.455 DEBUG:boto:b'wafmoa-10999false' 2023-10-12T00:24:05.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:24:05.456 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.457 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.457 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.457 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.458 DEBUG:boto:Method: GET 2023-10-12T00:24:05.458 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.459 DEBUG:boto:Data: 2023-10-12T00:24:05.459 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:05.460 DEBUG:boto:Port: 8001 2023-10-12T00:24:05.460 DEBUG:boto:Params: {} 2023-10-12T00:24:05.461 DEBUG:boto:Token: None 2023-10-12T00:24:05.461 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/?versions 2023-10-12T00:24:05.462 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw= 2023-10-12T00:24:05.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw=', 'Content-Length': '0'} 2023-10-12T00:24:05.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583a3bf9cd87a55c-0065273ca5-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.466 DEBUG:boto:b'wafmoa-10999false' 2023-10-12T00:24:05.467 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.467 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.468 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.468 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.468 DEBUG:boto:Method: GET 2023-10-12T00:24:05.469 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.469 DEBUG:boto:Data: 2023-10-12T00:24:05.470 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:05.470 DEBUG:boto:Port: 8000 2023-10-12T00:24:05.471 DEBUG:boto:Params: {} 2023-10-12T00:24:05.471 DEBUG:boto:Token: None 2023-10-12T00:24:05.472 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/?versions 2023-10-12T00:24:05.472 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw= 2023-10-12T00:24:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw=', 'Content-Length': '0'} 2023-10-12T00:24:05.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af6c89c14003b96-0065273ca5-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.477 DEBUG:boto:b'wafmoa-10999false' 2023-10-12T00:24:05.478 DEBUG:boto:path=/wafmoa-10/ 2023-10-12T00:24:05.479 DEBUG:boto:auth_path=/wafmoa-10/ 2023-10-12T00:24:05.479 DEBUG:boto:path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.480 DEBUG:boto:auth_path=/wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.481 DEBUG:boto:Method: GET 2023-10-12T00:24:05.481 DEBUG:boto:Path: /wafmoa-10/?versions&max-keys=999 2023-10-12T00:24:05.482 DEBUG:boto:Data: 2023-10-12T00:24:05.483 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:05.484 DEBUG:boto:Port: 8001 2023-10-12T00:24:05.484 DEBUG:boto:Params: {} 2023-10-12T00:24:05.485 DEBUG:boto:Token: None 2023-10-12T00:24:05.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-10/?versions 2023-10-12T00:24:05.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw= 2023-10-12T00:24:05.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCWaweWNEqW5xq2eDnBD6i0haqw=', 'Content-Length': '0'} 2023-10-12T00:24:05.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003423a875d183cce3-0065273ca5-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.489 DEBUG:boto:b'wafmoa-10999false' 2023-10-12T00:24:05.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-10 zones={a1, a2} 2023-10-12T00:24:05.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-10-12T00:24:05.492 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-11 2023-10-12T00:24:05.492 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:24:05.492 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:24:05.493 DEBUG:boto:Method: PUT 2023-10-12T00:24:05.493 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:24:05.494 DEBUG:boto:Data: 2023-10-12T00:24:05.494 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:05.495 DEBUG:boto:Port: 8000 2023-10-12T00:24:05.496 DEBUG:boto:Params: {} 2023-10-12T00:24:05.496 DEBUG:boto:Token: None 2023-10-12T00:24:05.497 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-11/ 2023-10-12T00:24:05.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AVClXNnMi9c4qhhcmhLQ5XmsGus= 2023-10-12T00:24:05.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AVClXNnMi9c4qhhcmhLQ5XmsGus=', 'Content-Length': '0'} 2023-10-12T00:24:05.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000089d440ea74951c34-0065273ca5-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.518 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-12 2023-10-12T00:24:05.518 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:24:05.519 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:24:05.519 DEBUG:boto:Method: PUT 2023-10-12T00:24:05.519 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:24:05.520 DEBUG:boto:Data: 2023-10-12T00:24:05.520 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:05.521 DEBUG:boto:Port: 8001 2023-10-12T00:24:05.522 DEBUG:boto:Params: {} 2023-10-12T00:24:05.522 DEBUG:boto:Token: None 2023-10-12T00:24:05.523 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-12/ 2023-10-12T00:24:05.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cr9j7tSOyIpcYtBbqhSABHQ/fd0= 2023-10-12T00:24:05.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cr9j7tSOyIpcYtBbqhSABHQ/fd0=', 'Content-Length': '0'} 2023-10-12T00:24:05.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c7dc70959372d2f-0065273ca5-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.549 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:24:05.549 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:24:05.549 DEBUG:boto:path=/wafmoa-11/?versioning 2023-10-12T00:24:05.550 DEBUG:boto:auth_path=/wafmoa-11/?versioning 2023-10-12T00:24:05.550 DEBUG:boto:Method: PUT 2023-10-12T00:24:05.551 DEBUG:boto:Path: /wafmoa-11/?versioning 2023-10-12T00:24:05.551 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:24:05.551 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:05.552 DEBUG:boto:Port: 8000 2023-10-12T00:24:05.552 DEBUG:boto:Params: {} 2023-10-12T00:24:05.553 DEBUG:boto:Token: None 2023-10-12T00:24:05.553 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-11/?versioning 2023-10-12T00:24:05.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qGUq1XuecuUIFpWwdpJukY61Kxo= 2023-10-12T00:24:05.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qGUq1XuecuUIFpWwdpJukY61Kxo=', 'Content-Length': '228'} 2023-10-12T00:24:05.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000980559062e1c76db-0065273ca5-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.561 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:24:05.561 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:24:05.561 DEBUG:boto:path=/wafmoa-12/?versioning 2023-10-12T00:24:05.562 DEBUG:boto:auth_path=/wafmoa-12/?versioning 2023-10-12T00:24:05.562 DEBUG:boto:Method: PUT 2023-10-12T00:24:05.563 DEBUG:boto:Path: /wafmoa-12/?versioning 2023-10-12T00:24:05.563 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:24:05.563 DEBUG:boto:Headers: {} 2023-10-12T00:24:05.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:05.564 DEBUG:boto:Port: 8001 2023-10-12T00:24:05.565 DEBUG:boto:Params: {} 2023-10-12T00:24:05.565 DEBUG:boto:Token: None 2023-10-12T00:24:05.566 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:24:05 GMT /wafmoa-12/?versioning 2023-10-12T00:24:05.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9CSx9HPTvfG2Ji9Rz0ls6wGwTwA= 2023-10-12T00:24:05.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9CSx9HPTvfG2Ji9Rz0ls6wGwTwA=', 'Content-Length': '228'} 2023-10-12T00:24:05.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004129492f76637e04-0065273ca5-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:05.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:05.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:05.581 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:24:05.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:05.632 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:05.660 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:24:05.672 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:24:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:24:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:24:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:24:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:24:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070245.577107_105.1", 2023-10-12T00:24:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:24:05.577107Z" 2023-10-12T00:24:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:24:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:24:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.675 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:24:05.678 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070245.577107_105.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:24:05.678 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:24:05.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:05.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:05.679 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:24:05.733 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:05.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:05.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:05.762+0000 7f5099773780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:24:05.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:05.763+0000 7f5099773780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:24:05.776 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:24:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:24:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:24:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:24:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:24:05.777 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:24:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:24:05.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:24:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:05.779 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:24:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:24:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:24:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:05.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:24:05.782 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:24:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:05.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:24:05.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.785 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:24:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070245.509072_94.1", 2023-10-12T00:24:05.786 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:24:05.509072Z", 2023-10-12T00:24:05.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:24:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:05.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:24:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:24:05.790 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:24:05.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:24:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:24:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:24:05.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:05.793 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:24:05Z" 2023-10-12T00:24:05.793 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:24:05.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070245.509072_94.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:24:05.509072Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:24:05Z" } 2023-10-12T00:24:05.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070245.509072_94.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:24:05.509072Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:24:05.795 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070245.577107_105.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:24:05.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070245.509072_94.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:24:05.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070245.577107_105.1 target=1_1697070245.509072_94.1 2023-10-12T00:24:10.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:10.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:24:10.799 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:24:10.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:10.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:10.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:10.890+0000 7f2078a49780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:24:10.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:10.891+0000 7f2078a49780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:24:10.903 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:24:10.903 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:24:10.903 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:24:10.904 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:24:10.904 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:24:10.904 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:24:10.904 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:10.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.905 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:24:10.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:10.905 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:24:10.906 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:10.906 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:10.906 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:24:10.906 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:10.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:24:10.907 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:10.907 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:24:10.907 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:10.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:10.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:10.908 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:24:10.908 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:10.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:10.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:24:10.909 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:10.909 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:10.909 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:10.909 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:24:10.910 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:10.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:10.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:10.910 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:24:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070245.577107_105.1", 2023-10-12T00:24:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:10.911 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:24:05.577107Z", 2023-10-12T00:24:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:24:10.913 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:24:10.913 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:24:10.913 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:24:10.914 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:24:10.914 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:24:10.914 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:24:10.914 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:24:10.914 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:24:10.915 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:24:10.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:10.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:24:10.916 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:24:10.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.916 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:24:10.916 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:24:10.916 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:24:10.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:24:10.917 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:24:10Z" 2023-10-12T00:24:10.917 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:24:10.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070245.577107_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:24:05.577107Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:24:10Z" } 2023-10-12T00:24:10.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070245.577107_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:24:05.577107Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:24:10.919 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070245.577107_105.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:24:10.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070245.577107_105.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:24:10.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:24:10.920 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:24:10.921 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:24:10.921 DEBUG:boto:Method: HEAD 2023-10-12T00:24:10.922 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:24:10.922 DEBUG:boto:Data: 2023-10-12T00:24:10.922 DEBUG:boto:Headers: {} 2023-10-12T00:24:10.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:10.923 DEBUG:boto:Port: 8000 2023-10-12T00:24:10.924 DEBUG:boto:Params: {} 2023-10-12T00:24:10.924 DEBUG:boto:Token: None 2023-10-12T00:24:10.925 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:10 GMT /wafmoa-11/ 2023-10-12T00:24:10.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FyAvcx1ihmtShaoviCBZI79Ts9c= 2023-10-12T00:24:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FyAvcx1ihmtShaoviCBZI79Ts9c=', 'Content-Length': '0'} 2023-10-12T00:24:10.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a07c2691ebba25a-0065273caa-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:10.929 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:24:10.929 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:24:10.930 DEBUG:boto:Method: PUT 2023-10-12T00:24:10.930 DEBUG:boto:Path: /wafmoa-11/dummy 2023-10-12T00:24:10.931 DEBUG:boto:Data: 2023-10-12T00:24:10.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:24:10.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:10.932 DEBUG:boto:Port: 8000 2023-10-12T00:24:10.932 DEBUG:boto:Params: {} 2023-10-12T00:24:10.933 DEBUG:boto:Token: None 2023-10-12T00:24:10.933 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:24:10 GMT /wafmoa-11/dummy 2023-10-12T00:24:10.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SWk2ENdgGeagfXVJuBiWtAoykU= 2023-10-12T00:24:10.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SWk2ENdgGeagfXVJuBiWtAoykU='} 2023-10-12T00:24:10.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-amz-request-id', 'tx00000bb57f43229653b74-0065273caa-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:10.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:10.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:10.946 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:24:11.003 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:11.004 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:11.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:11.036+0000 7f693983a780 1 waiting to reach incremental sync.. 2023-10-12T00:24:16.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:16.037+0000 7f693983a780 1 waiting to reach incremental sync.. 2023-10-12T00:24:21.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:21.036+0000 7f693983a780 1 waiting to reach incremental sync.. 2023-10-12T00:24:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:26.037+0000 7f693983a780 1 bucket sync caught up with source: 2023-10-12T00:24:26.038 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:24:26.038 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:24:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:26.037+0000 7f693983a780 0 bucket checkpoint complete 2023-10-12T00:24:26.054 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:24:26.054 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:24:26.055 DEBUG:boto:Method: HEAD 2023-10-12T00:24:26.055 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:24:26.056 DEBUG:boto:Data: 2023-10-12T00:24:26.056 DEBUG:boto:Headers: {} 2023-10-12T00:24:26.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:26.057 DEBUG:boto:Port: 8001 2023-10-12T00:24:26.058 DEBUG:boto:Params: {} 2023-10-12T00:24:26.058 DEBUG:boto:Token: None 2023-10-12T00:24:26.059 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:26 GMT /wafmoa-12/ 2023-10-12T00:24:26.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:67oYELPCzYC/ssM+M/dpNqlzhBE= 2023-10-12T00:24:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:67oYELPCzYC/ssM+M/dpNqlzhBE=', 'Content-Length': '0'} 2023-10-12T00:24:26.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d360f6ca38545f58-0065273cba-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:26.063 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:24:26.064 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:24:26.064 DEBUG:boto:Method: PUT 2023-10-12T00:24:26.065 DEBUG:boto:Path: /wafmoa-12/dummy 2023-10-12T00:24:26.065 DEBUG:boto:Data: 2023-10-12T00:24:26.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:24:26.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:26.066 DEBUG:boto:Port: 8001 2023-10-12T00:24:26.067 DEBUG:boto:Params: {} 2023-10-12T00:24:26.067 DEBUG:boto:Token: None 2023-10-12T00:24:26.068 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:24:26 GMT /wafmoa-12/dummy 2023-10-12T00:24:26.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hvm+HDHHcFwTko0UJqt3wWh8In4= 2023-10-12T00:24:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hvm+HDHHcFwTko0UJqt3wWh8In4='} 2023-10-12T00:24:26.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-amz-request-id', 'tx000000f19ec69a178a422-0065273cba-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:26.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:26.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:26.082 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:24:26.143 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:26.144 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:26.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:26.178+0000 7f6ed01a5780 1 waiting to reach incremental sync.. 2023-10-12T00:24:31.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:31.179+0000 7f6ed01a5780 1 waiting to reach incremental sync.. 2023-10-12T00:24:36.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:36.180+0000 7f6ed01a5780 1 waiting to reach incremental sync.. 2023-10-12T00:24:41.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:41.180+0000 7f6ed01a5780 1 waiting to reach incremental sync.. 2023-10-12T00:24:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:46.182+0000 7f6ed01a5780 1 bucket sync caught up with source: 2023-10-12T00:24:46.183 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.17.11, , , , , ] 2023-10-12T00:24:46.183 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.17.11, , , , , ] 2023-10-12T00:24:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:46.182+0000 7f6ed01a5780 0 bucket checkpoint complete 2023-10-12T00:24:46.201 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:24:46.202 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:24:46.202 DEBUG:boto:Method: HEAD 2023-10-12T00:24:46.202 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:24:46.203 DEBUG:boto:Data: 2023-10-12T00:24:46.203 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.204 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.204 DEBUG:boto:Params: {} 2023-10-12T00:24:46.205 DEBUG:boto:Token: None 2023-10-12T00:24:46.205 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/ 2023-10-12T00:24:46.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:am6t1S0hqHgjQ/2TSFlt1aZP44g= 2023-10-12T00:24:46.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:am6t1S0hqHgjQ/2TSFlt1aZP44g=', 'Content-Length': '0'} 2023-10-12T00:24:46.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f66188b121445bf-0065273cce-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.210 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.211 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.211 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.212 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:24:46.212 DEBUG:boto:Data: 2023-10-12T00:24:46.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.213 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.214 DEBUG:boto:Params: {} 2023-10-12T00:24:46.214 DEBUG:boto:Token: None 2023-10-12T00:24:46.215 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a1 2023-10-12T00:24:46.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W+P873Gw+XzZYo+oxcqEfXSg92Q= 2023-10-12T00:24:46.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W+P873Gw+XzZYo+oxcqEfXSg92Q='} 2023-10-12T00:24:46.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-amz-request-id', 'tx00000397610a786a74527-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.227 DEBUG:rgw_multi.tests:version1 id=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:24:46.228 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.228 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.229 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.229 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:24:46.229 DEBUG:boto:Data: 2023-10-12T00:24:46.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.231 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.231 DEBUG:boto:Params: {} 2023-10-12T00:24:46.232 DEBUG:boto:Token: None 2023-10-12T00:24:46.232 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a1 2023-10-12T00:24:46.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmZsaE9UzycGRIpiY7F+jsFYW2k= 2023-10-12T00:24:46.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmZsaE9UzycGRIpiY7F+jsFYW2k='} 2023-10-12T00:24:46.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7'), ('x-amz-request-id', 'tx000008b45013c9196ab62-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.279 DEBUG:rgw_multi.tests:version2 id=fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7 2023-10-12T00:24:46.280 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.280 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.280 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7 2023-10-12T00:24:46.281 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7 2023-10-12T00:24:46.281 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.282 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7 2023-10-12T00:24:46.282 DEBUG:boto:Data: 2023-10-12T00:24:46.282 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.283 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.284 DEBUG:boto:Params: {} 2023-10-12T00:24:46.284 DEBUG:boto:Token: None 2023-10-12T00:24:46.285 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a1?versionId=fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7 2023-10-12T00:24:46.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UtJosmxAxsEod8pau3mqMeMTSRg= 2023-10-12T00:24:46.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UtJosmxAxsEod8pau3mqMeMTSRg=', 'Content-Length': '0'} 2023-10-12T00:24:46.300 DEBUG:boto:Response headers: [('x-amz-version-id', 'fYAJvYsrm8wNuiczDIHxmeVIKtKOIg7'), ('x-amz-request-id', 'tx000005e183c8099f8e767-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.301 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.301 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.302 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.302 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:24:46.303 DEBUG:boto:Data: 2023-10-12T00:24:46.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.304 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.304 DEBUG:boto:Params: {} 2023-10-12T00:24:46.305 DEBUG:boto:Token: None 2023-10-12T00:24:46.305 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a1 2023-10-12T00:24:46.306 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AEKQxXTorxeRgwqaW9+frNt9UYA= 2023-10-12T00:24:46.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AEKQxXTorxeRgwqaW9+frNt9UYA='} 2023-10-12T00:24:46.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DdWiHjOaor3HFGjrPWkJyATy67CmsLR'), ('x-amz-request-id', 'tx0000023779544f1f3daf9-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.318 DEBUG:rgw_multi.tests:version3 id=DdWiHjOaor3HFGjrPWkJyATy67CmsLR 2023-10-12T00:24:46.319 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.319 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:24:46.320 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=DdWiHjOaor3HFGjrPWkJyATy67CmsLR 2023-10-12T00:24:46.320 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=DdWiHjOaor3HFGjrPWkJyATy67CmsLR 2023-10-12T00:24:46.320 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.321 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=DdWiHjOaor3HFGjrPWkJyATy67CmsLR 2023-10-12T00:24:46.321 DEBUG:boto:Data: 2023-10-12T00:24:46.322 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.323 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.323 DEBUG:boto:Params: {} 2023-10-12T00:24:46.323 DEBUG:boto:Token: None 2023-10-12T00:24:46.324 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a1?versionId=DdWiHjOaor3HFGjrPWkJyATy67CmsLR 2023-10-12T00:24:46.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hmpG2dNrHAyxQgwjo58Ps1LHh9g= 2023-10-12T00:24:46.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hmpG2dNrHAyxQgwjo58Ps1LHh9g=', 'Content-Length': '0'} 2023-10-12T00:24:46.335 DEBUG:boto:Response headers: [('x-amz-version-id', 'DdWiHjOaor3HFGjrPWkJyATy67CmsLR'), ('x-amz-request-id', 'tx000001cc5637e52f8dd3a-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.335 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:24:46.336 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:24:46.336 DEBUG:boto:Method: HEAD 2023-10-12T00:24:46.336 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:24:46.337 DEBUG:boto:Data: 2023-10-12T00:24:46.337 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.338 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.338 DEBUG:boto:Params: {} 2023-10-12T00:24:46.339 DEBUG:boto:Token: None 2023-10-12T00:24:46.339 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/ 2023-10-12T00:24:46.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:am6t1S0hqHgjQ/2TSFlt1aZP44g= 2023-10-12T00:24:46.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:am6t1S0hqHgjQ/2TSFlt1aZP44g=', 'Content-Length': '0'} 2023-10-12T00:24:46.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad5e3a4f55164880-0065273cce-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.343 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.344 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.344 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.345 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:24:46.345 DEBUG:boto:Data: 2023-10-12T00:24:46.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.346 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.347 DEBUG:boto:Params: {} 2023-10-12T00:24:46.347 DEBUG:boto:Token: None 2023-10-12T00:24:46.348 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a2 2023-10-12T00:24:46.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2i0RX3tPW298KnIe4SS3nUAAbWU= 2023-10-12T00:24:46.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2i0RX3tPW298KnIe4SS3nUAAbWU='} 2023-10-12T00:24:46.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-amz-request-id', 'tx0000071306701fcb588ec-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.366 DEBUG:rgw_multi.tests:version1 id=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:24:46.366 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.367 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.367 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.368 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:24:46.368 DEBUG:boto:Data: 2023-10-12T00:24:46.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.369 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.370 DEBUG:boto:Params: {} 2023-10-12T00:24:46.370 DEBUG:boto:Token: None 2023-10-12T00:24:46.371 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a2 2023-10-12T00:24:46.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nG+lGAxpZRvQoljwCXRsgr0l9cI= 2023-10-12T00:24:46.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nG+lGAxpZRvQoljwCXRsgr0l9cI='} 2023-10-12T00:24:46.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LtbJxUcLLYZrlnypwhqdticXiKo-cR.'), ('x-amz-request-id', 'tx00000fa546c7f8ff06e43-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.419 DEBUG:rgw_multi.tests:version2 id=LtbJxUcLLYZrlnypwhqdticXiKo-cR. 2023-10-12T00:24:46.420 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.420 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.420 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=LtbJxUcLLYZrlnypwhqdticXiKo-cR. 2023-10-12T00:24:46.421 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=LtbJxUcLLYZrlnypwhqdticXiKo-cR. 2023-10-12T00:24:46.421 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.422 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=LtbJxUcLLYZrlnypwhqdticXiKo-cR. 2023-10-12T00:24:46.422 DEBUG:boto:Data: 2023-10-12T00:24:46.423 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.423 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.424 DEBUG:boto:Params: {} 2023-10-12T00:24:46.424 DEBUG:boto:Token: None 2023-10-12T00:24:46.425 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a2?versionId=LtbJxUcLLYZrlnypwhqdticXiKo-cR. 2023-10-12T00:24:46.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w4NXdiSknQXKZ8Wlbo/gX/nNUuk= 2023-10-12T00:24:46.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w4NXdiSknQXKZ8Wlbo/gX/nNUuk=', 'Content-Length': '0'} 2023-10-12T00:24:46.435 DEBUG:boto:Response headers: [('x-amz-version-id', 'LtbJxUcLLYZrlnypwhqdticXiKo-cR.'), ('x-amz-request-id', 'tx0000014dd7458da8d83c7-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.436 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.436 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.437 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.437 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:24:46.437 DEBUG:boto:Data: 2023-10-12T00:24:46.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.439 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.439 DEBUG:boto:Params: {} 2023-10-12T00:24:46.440 DEBUG:boto:Token: None 2023-10-12T00:24:46.440 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a2 2023-10-12T00:24:46.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lVniBJgOkzT4m9wedXESuXp6Lzc= 2023-10-12T00:24:46.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lVniBJgOkzT4m9wedXESuXp6Lzc='} 2023-10-12T00:24:46.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'syil2ou.p3kNKoSk006XNtfV00Ppupj'), ('x-amz-request-id', 'tx00000d69894961c2721f7-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.453 DEBUG:rgw_multi.tests:version3 id=syil2ou.p3kNKoSk006XNtfV00Ppupj 2023-10-12T00:24:46.453 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.453 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:24:46.454 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=syil2ou.p3kNKoSk006XNtfV00Ppupj 2023-10-12T00:24:46.454 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=syil2ou.p3kNKoSk006XNtfV00Ppupj 2023-10-12T00:24:46.455 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.455 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=syil2ou.p3kNKoSk006XNtfV00Ppupj 2023-10-12T00:24:46.455 DEBUG:boto:Data: 2023-10-12T00:24:46.456 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.457 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.457 DEBUG:boto:Params: {} 2023-10-12T00:24:46.457 DEBUG:boto:Token: None 2023-10-12T00:24:46.458 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-11/obj-a2?versionId=syil2ou.p3kNKoSk006XNtfV00Ppupj 2023-10-12T00:24:46.458 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ptULMMItBXP7VrpFTx9lTudB6N0= 2023-10-12T00:24:46.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ptULMMItBXP7VrpFTx9lTudB6N0=', 'Content-Length': '0'} 2023-10-12T00:24:46.468 DEBUG:boto:Response headers: [('x-amz-version-id', 'syil2ou.p3kNKoSk006XNtfV00Ppupj'), ('x-amz-request-id', 'tx000008c3793ac78a450bb-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.469 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:24:46.470 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:24:46.470 DEBUG:boto:Method: HEAD 2023-10-12T00:24:46.471 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:24:46.471 DEBUG:boto:Data: 2023-10-12T00:24:46.472 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.472 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.473 DEBUG:boto:Params: {} 2023-10-12T00:24:46.473 DEBUG:boto:Token: None 2023-10-12T00:24:46.474 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/ 2023-10-12T00:24:46.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bjc6qgoUdWIimJnVavktbQi8Rtk= 2023-10-12T00:24:46.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bjc6qgoUdWIimJnVavktbQi8Rtk=', 'Content-Length': '0'} 2023-10-12T00:24:46.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f834edfad5180c49-0065273cce-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.478 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.479 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.479 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.479 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:24:46.480 DEBUG:boto:Data: 2023-10-12T00:24:46.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.481 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.482 DEBUG:boto:Params: {} 2023-10-12T00:24:46.482 DEBUG:boto:Token: None 2023-10-12T00:24:46.483 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a1 2023-10-12T00:24:46.483 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/p4Ot0przx2IOEUOEjBO/9janlY= 2023-10-12T00:24:46.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/p4Ot0przx2IOEUOEjBO/9janlY='} 2023-10-12T00:24:46.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-amz-request-id', 'tx00000d57fb3528fb078d4-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.494 DEBUG:rgw_multi.tests:version1 id=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:24:46.495 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.496 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.496 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.497 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:24:46.498 DEBUG:boto:Data: 2023-10-12T00:24:46.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.500 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.500 DEBUG:boto:Params: {} 2023-10-12T00:24:46.501 DEBUG:boto:Token: None 2023-10-12T00:24:46.502 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a1 2023-10-12T00:24:46.503 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gifo+3Fr+iPbCs54lz104KtQfaA= 2023-10-12T00:24:46.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gifo+3Fr+iPbCs54lz104KtQfaA='} 2023-10-12T00:24:46.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP'), ('x-amz-request-id', 'tx0000016b8d864282a5382-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.513 DEBUG:rgw_multi.tests:version2 id=ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP 2023-10-12T00:24:46.513 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.514 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.514 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP 2023-10-12T00:24:46.514 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP 2023-10-12T00:24:46.515 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.515 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP 2023-10-12T00:24:46.516 DEBUG:boto:Data: 2023-10-12T00:24:46.516 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.517 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.517 DEBUG:boto:Params: {} 2023-10-12T00:24:46.518 DEBUG:boto:Token: None 2023-10-12T00:24:46.518 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a1?versionId=ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP 2023-10-12T00:24:46.519 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFERC4Sk8DlDOB/OvlG7iBnkJKM= 2023-10-12T00:24:46.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFERC4Sk8DlDOB/OvlG7iBnkJKM=', 'Content-Length': '0'} 2023-10-12T00:24:46.530 DEBUG:boto:Response headers: [('x-amz-version-id', 'ljUCp2cjdLlNE7Akv.53ZVKk9aSh9PP'), ('x-amz-request-id', 'tx0000059172dbe1d1d168c-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.531 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.531 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.531 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.532 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:24:46.533 DEBUG:boto:Data: 2023-10-12T00:24:46.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.534 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.534 DEBUG:boto:Params: {} 2023-10-12T00:24:46.535 DEBUG:boto:Token: None 2023-10-12T00:24:46.536 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a1 2023-10-12T00:24:46.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:txgnKz8Io6OeDwNS6I58LYleexg= 2023-10-12T00:24:46.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:txgnKz8Io6OeDwNS6I58LYleexg='} 2023-10-12T00:24:46.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bN1yacfia10YIDTMiHfS9vNnh-4z3hD'), ('x-amz-request-id', 'tx00000ecfa554a9a058c3f-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.583 DEBUG:rgw_multi.tests:version3 id=bN1yacfia10YIDTMiHfS9vNnh-4z3hD 2023-10-12T00:24:46.584 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.584 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:24:46.584 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=bN1yacfia10YIDTMiHfS9vNnh-4z3hD 2023-10-12T00:24:46.585 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=bN1yacfia10YIDTMiHfS9vNnh-4z3hD 2023-10-12T00:24:46.585 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.586 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=bN1yacfia10YIDTMiHfS9vNnh-4z3hD 2023-10-12T00:24:46.586 DEBUG:boto:Data: 2023-10-12T00:24:46.587 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:24:46.587 DEBUG:boto:Port: 8000 2023-10-12T00:24:46.588 DEBUG:boto:Params: {} 2023-10-12T00:24:46.588 DEBUG:boto:Token: None 2023-10-12T00:24:46.589 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a1?versionId=bN1yacfia10YIDTMiHfS9vNnh-4z3hD 2023-10-12T00:24:46.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sbzvpj7NurD7FLowPZ5bAhXUV3w= 2023-10-12T00:24:46.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sbzvpj7NurD7FLowPZ5bAhXUV3w=', 'Content-Length': '0'} 2023-10-12T00:24:46.598 DEBUG:boto:Response headers: [('x-amz-version-id', 'bN1yacfia10YIDTMiHfS9vNnh-4z3hD'), ('x-amz-request-id', 'tx0000053d1064f30a79c67-0065273cce-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.599 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:24:46.600 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:24:46.600 DEBUG:boto:Method: HEAD 2023-10-12T00:24:46.601 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:24:46.601 DEBUG:boto:Data: 2023-10-12T00:24:46.601 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.602 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.603 DEBUG:boto:Params: {} 2023-10-12T00:24:46.603 DEBUG:boto:Token: None 2023-10-12T00:24:46.603 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/ 2023-10-12T00:24:46.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bjc6qgoUdWIimJnVavktbQi8Rtk= 2023-10-12T00:24:46.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bjc6qgoUdWIimJnVavktbQi8Rtk=', 'Content-Length': '0'} 2023-10-12T00:24:46.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d34d9617c841a4a-0065273cce-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.607 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.607 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.608 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.608 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:24:46.609 DEBUG:boto:Data: 2023-10-12T00:24:46.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.610 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.610 DEBUG:boto:Params: {} 2023-10-12T00:24:46.611 DEBUG:boto:Token: None 2023-10-12T00:24:46.611 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a2 2023-10-12T00:24:46.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vFoPUv3jj+znBAIQyK0feTnqXzQ= 2023-10-12T00:24:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vFoPUv3jj+znBAIQyK0feTnqXzQ='} 2023-10-12T00:24:46.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-amz-request-id', 'tx000008fec0bcd6447f112-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.625 DEBUG:rgw_multi.tests:version1 id=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:24:46.625 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.626 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.626 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.627 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:24:46.627 DEBUG:boto:Data: 2023-10-12T00:24:46.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.629 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.629 DEBUG:boto:Params: {} 2023-10-12T00:24:46.630 DEBUG:boto:Token: None 2023-10-12T00:24:46.630 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a2 2023-10-12T00:24:46.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yDVbg8oUkWlbQrIIlUIyUngFyCY= 2023-10-12T00:24:46.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yDVbg8oUkWlbQrIIlUIyUngFyCY='} 2023-10-12T00:24:46.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'n07Ehok.ZQmlUZJaGjYck85B5eQrnxa'), ('x-amz-request-id', 'tx00000badfe6181779d975-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.642 DEBUG:rgw_multi.tests:version2 id=n07Ehok.ZQmlUZJaGjYck85B5eQrnxa 2023-10-12T00:24:46.642 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.643 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.643 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=n07Ehok.ZQmlUZJaGjYck85B5eQrnxa 2023-10-12T00:24:46.644 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=n07Ehok.ZQmlUZJaGjYck85B5eQrnxa 2023-10-12T00:24:46.644 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.645 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=n07Ehok.ZQmlUZJaGjYck85B5eQrnxa 2023-10-12T00:24:46.645 DEBUG:boto:Data: 2023-10-12T00:24:46.646 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.647 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.647 DEBUG:boto:Params: {} 2023-10-12T00:24:46.648 DEBUG:boto:Token: None 2023-10-12T00:24:46.648 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a2?versionId=n07Ehok.ZQmlUZJaGjYck85B5eQrnxa 2023-10-12T00:24:46.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jbmH7g6KY8z+0ktW9jzvtl8tEmI= 2023-10-12T00:24:46.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jbmH7g6KY8z+0ktW9jzvtl8tEmI=', 'Content-Length': '0'} 2023-10-12T00:24:46.658 DEBUG:boto:Response headers: [('x-amz-version-id', 'n07Ehok.ZQmlUZJaGjYck85B5eQrnxa'), ('x-amz-request-id', 'tx0000004ced7e63c456110-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.659 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.660 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.660 DEBUG:boto:Method: PUT 2023-10-12T00:24:46.660 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:24:46.661 DEBUG:boto:Data: 2023-10-12T00:24:46.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:24:46.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.662 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.662 DEBUG:boto:Params: {} 2023-10-12T00:24:46.663 DEBUG:boto:Token: None 2023-10-12T00:24:46.663 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a2 2023-10-12T00:24:46.664 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GU7z7i7FlqbTDQl3aUD/un4elEY= 2023-10-12T00:24:46.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GU7z7i7FlqbTDQl3aUD/un4elEY='} 2023-10-12T00:24:46.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cYRp3QCdthzGGe07xEoQaCxD5X7bOSS'), ('x-amz-request-id', 'tx000002c4fb5c3990e45c2-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.707 DEBUG:rgw_multi.tests:version3 id=cYRp3QCdthzGGe07xEoQaCxD5X7bOSS 2023-10-12T00:24:46.708 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.708 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:24:46.709 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=cYRp3QCdthzGGe07xEoQaCxD5X7bOSS 2023-10-12T00:24:46.709 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=cYRp3QCdthzGGe07xEoQaCxD5X7bOSS 2023-10-12T00:24:46.709 DEBUG:boto:Method: DELETE 2023-10-12T00:24:46.710 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=cYRp3QCdthzGGe07xEoQaCxD5X7bOSS 2023-10-12T00:24:46.710 DEBUG:boto:Data: 2023-10-12T00:24:46.711 DEBUG:boto:Headers: {} 2023-10-12T00:24:46.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:24:46.712 DEBUG:boto:Port: 8001 2023-10-12T00:24:46.712 DEBUG:boto:Params: {} 2023-10-12T00:24:46.713 DEBUG:boto:Token: None 2023-10-12T00:24:46.713 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:24:46 GMT /wafmoa-12/obj-a2?versionId=cYRp3QCdthzGGe07xEoQaCxD5X7bOSS 2023-10-12T00:24:46.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SOloUM1IOIH4XSvIfiAOw7OQFms= 2023-10-12T00:24:46.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:24:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SOloUM1IOIH4XSvIfiAOw7OQFms=', 'Content-Length': '0'} 2023-10-12T00:24:46.723 DEBUG:boto:Response headers: [('x-amz-version-id', 'cYRp3QCdthzGGe07xEoQaCxD5X7bOSS'), ('x-amz-request-id', 'tx0000014e05321d362c4e4-0065273cce-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:24:46.724 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-11 2023-10-12T00:24:46.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:46.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:24:46.725 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:24:46.782 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:24:46.783 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:24:46.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:46.815+0000 7f65a39b1780 1 waiting for incremental sync to catch up: 2023-10-12T00:24:46.818 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:24:46.818 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:24:51.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:51.818+0000 7f65a39b1780 1 waiting for incremental sync to catch up: 2023-10-12T00:24:51.818 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:24:51.818 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:24:56.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:24:56.819+0000 7f65a39b1780 1 waiting for incremental sync to catch up: 2023-10-12T00:24:56.819 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:24:56.819 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:25:01.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:01.820+0000 7f65a39b1780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:01.821 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-12T00:25:01.821 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:25:06.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:06.820+0000 7f65a39b1780 1 bucket sync caught up with source: 2023-10-12T00:25:06.822 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:25:06.822 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-10-12T00:25:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:06.820+0000 7f65a39b1780 0 bucket checkpoint complete 2023-10-12T00:25:06.837 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-11 2023-10-12T00:25:06.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:06.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:06.838 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:06.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:06.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:06.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:06.939+0000 7f43c65eb780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:06.940 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000010.40.6, , ] 2023-10-12T00:25:06.940 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:11.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:11.940+0000 7f43c65eb780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:11.941 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000010.40.6, , ] 2023-10-12T00:25:11.941 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:16.941+0000 7f43c65eb780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:16.943 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000010.40.6, , ] 2023-10-12T00:25:16.943 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:21.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:21.943+0000 7f43c65eb780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:21.944 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000010.40.6, , ] 2023-10-12T00:25:21.944 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:26.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:26.945+0000 7f43c65eb780 1 bucket sync caught up with source: 2023-10-12T00:25:26.946 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:26.946 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000010.40.6, 00000000002.32.11, ] 2023-10-12T00:25:26.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:26.945+0000 7f43c65eb780 0 bucket checkpoint complete 2023-10-12T00:25:26.960 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-11 zones={a2, a1} 2023-10-12T00:25:26.961 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:26.962 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:26.962 DEBUG:boto:Method: HEAD 2023-10-12T00:25:26.963 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:26.963 DEBUG:boto:Data: 2023-10-12T00:25:26.963 DEBUG:boto:Headers: {} 2023-10-12T00:25:26.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:26.964 DEBUG:boto:Port: 8001 2023-10-12T00:25:26.965 DEBUG:boto:Params: {} 2023-10-12T00:25:26.965 DEBUG:boto:Token: None 2023-10-12T00:25:26.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:26 GMT /wafmoa-11/ 2023-10-12T00:25:26.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eg0PEG44PRrUeS2eB0E/Lf8j+eQ= 2023-10-12T00:25:26.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eg0PEG44PRrUeS2eB0E/Lf8j+eQ=', 'Content-Length': '0'} 2023-10-12T00:25:26.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092b89f58e2eafcc4-0065273cf6-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:26.970 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:26.971 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:26.971 DEBUG:boto:Method: HEAD 2023-10-12T00:25:26.972 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:26.972 DEBUG:boto:Data: 2023-10-12T00:25:26.972 DEBUG:boto:Headers: {} 2023-10-12T00:25:26.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:26.973 DEBUG:boto:Port: 8000 2023-10-12T00:25:26.974 DEBUG:boto:Params: {} 2023-10-12T00:25:26.974 DEBUG:boto:Token: None 2023-10-12T00:25:26.975 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:26 GMT /wafmoa-11/ 2023-10-12T00:25:26.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eg0PEG44PRrUeS2eB0E/Lf8j+eQ= 2023-10-12T00:25:26.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eg0PEG44PRrUeS2eB0E/Lf8j+eQ=', 'Content-Length': '0'} 2023-10-12T00:25:26.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7981a6ec781ef21-0065273cf6-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:26.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:25:26.979 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:26.979 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:26.980 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.980 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.981 DEBUG:boto:Method: GET 2023-10-12T00:25:26.981 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.981 DEBUG:boto:Data: 2023-10-12T00:25:26.982 DEBUG:boto:Headers: {} 2023-10-12T00:25:26.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:26.983 DEBUG:boto:Port: 8001 2023-10-12T00:25:26.983 DEBUG:boto:Params: {} 2023-10-12T00:25:26.984 DEBUG:boto:Token: None 2023-10-12T00:25:26.984 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:26 GMT /wafmoa-11/?versions 2023-10-12T00:25:26.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0EwQp4S6J6c55ZkvFIrZfW/dS1g= 2023-10-12T00:25:26.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0EwQp4S6J6c55ZkvFIrZfW/dS1g=', 'Content-Length': '0'} 2023-10-12T00:25:26.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ffba9bba6acec4-0065273cf6-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:26.988 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:26.989 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:26.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:26.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:26.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:25:26.991 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:26.991 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:26.992 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.992 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.992 DEBUG:boto:Method: GET 2023-10-12T00:25:26.993 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:26.993 DEBUG:boto:Data: 2023-10-12T00:25:26.994 DEBUG:boto:Headers: {} 2023-10-12T00:25:26.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:26.994 DEBUG:boto:Port: 8000 2023-10-12T00:25:26.995 DEBUG:boto:Params: {} 2023-10-12T00:25:26.995 DEBUG:boto:Token: None 2023-10-12T00:25:26.996 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:26 GMT /wafmoa-11/?versions 2023-10-12T00:25:26.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0EwQp4S6J6c55ZkvFIrZfW/dS1g= 2023-10-12T00:25:26.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0EwQp4S6J6c55ZkvFIrZfW/dS1g=', 'Content-Length': '0'} 2023-10-12T00:25:27.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd83e315c556b5f1-0065273cf6-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.000 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:27.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:27.002 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.003 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.003 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.003 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.004 DEBUG:boto:Method: GET 2023-10-12T00:25:27.004 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.005 DEBUG:boto:Data: 2023-10-12T00:25:27.005 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.006 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.006 DEBUG:boto:Params: {} 2023-10-12T00:25:27.007 DEBUG:boto:Token: None 2023-10-12T00:25:27.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/?versions 2023-10-12T00:25:27.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xxp5TgI4ZUcX09NMPjYgjyQrmOM= 2023-10-12T00:25:27.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xxp5TgI4ZUcX09NMPjYgjyQrmOM=', 'Content-Length': '0'} 2023-10-12T00:25:27.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088724ac36d4dfe83-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.011 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.012 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.013 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.013 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.013 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.014 DEBUG:boto:Method: GET 2023-10-12T00:25:27.014 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:27.015 DEBUG:boto:Data: 2023-10-12T00:25:27.015 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.016 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.016 DEBUG:boto:Params: {} 2023-10-12T00:25:27.017 DEBUG:boto:Token: None 2023-10-12T00:25:27.017 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/?versions 2023-10-12T00:25:27.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xxp5TgI4ZUcX09NMPjYgjyQrmOM= 2023-10-12T00:25:27.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xxp5TgI4ZUcX09NMPjYgjyQrmOM=', 'Content-Length': '0'} 2023-10-12T00:25:27.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029bf7a758d043495-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.021 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:27.022 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.023 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.023 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.024 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.024 DEBUG:boto:Method: GET 2023-10-12T00:25:27.025 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.025 DEBUG:boto:Data: 2023-10-12T00:25:27.026 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.027 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.027 DEBUG:boto:Params: {} 2023-10-12T00:25:27.027 DEBUG:boto:Token: None 2023-10-12T00:25:27.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff4b532cd759ea02-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.032 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.032 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.033 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.033 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.034 DEBUG:boto:Method: GET 2023-10-12T00:25:27.034 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.034 DEBUG:boto:Data: 2023-10-12T00:25:27.035 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.036 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.036 DEBUG:boto:Params: {} 2023-10-12T00:25:27.037 DEBUG:boto:Token: None 2023-10-12T00:25:27.037 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d080c2c428e5ff03-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.041 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.041 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.042 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.042 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.042 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.043 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.043 DEBUG:boto:Data: 2023-10-12T00:25:27.044 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.045 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.045 DEBUG:boto:Params: {} 2023-10-12T00:25:27.045 DEBUG:boto:Token: None 2023-10-12T00:25:27.046 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VCrZSzgcuzcJAYKJtOeV01Q/MvM= 2023-10-12T00:25:27.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VCrZSzgcuzcJAYKJtOeV01Q/MvM=', 'Content-Length': '0'} 2023-10-12T00:25:27.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c23d931ca8177377-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.049 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.050 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.050 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.051 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.051 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.051 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.052 DEBUG:boto:Data: 2023-10-12T00:25:27.052 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.053 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.054 DEBUG:boto:Params: {} 2023-10-12T00:25:27.054 DEBUG:boto:Token: None 2023-10-12T00:25:27.055 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VCrZSzgcuzcJAYKJtOeV01Q/MvM= 2023-10-12T00:25:27.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VCrZSzgcuzcJAYKJtOeV01Q/MvM=', 'Content-Length': '0'} 2023-10-12T00:25:27.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c372e4eeca5f9168-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:27.059 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.059 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.059 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.060 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.060 DEBUG:boto:Method: GET 2023-10-12T00:25:27.061 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.061 DEBUG:boto:Data: 2023-10-12T00:25:27.062 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.063 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.063 DEBUG:boto:Params: {} 2023-10-12T00:25:27.064 DEBUG:boto:Token: None 2023-10-12T00:25:27.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7ce4bdc92778325-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.068 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.068 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.069 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.069 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.070 DEBUG:boto:Method: GET 2023-10-12T00:25:27.070 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.070 DEBUG:boto:Data: 2023-10-12T00:25:27.071 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.072 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.072 DEBUG:boto:Params: {} 2023-10-12T00:25:27.073 DEBUG:boto:Token: None 2023-10-12T00:25:27.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b05ece4bfefb4811-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.077 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.077 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.078 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.078 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.079 DEBUG:boto:Method: GET 2023-10-12T00:25:27.079 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.079 DEBUG:boto:Data: 2023-10-12T00:25:27.080 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.081 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.081 DEBUG:boto:Params: {} 2023-10-12T00:25:27.082 DEBUG:boto:Token: None 2023-10-12T00:25:27.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efcc6d8fed2c23bb-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.126 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.127 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.128 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.128 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.129 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.129 DEBUG:boto:Method: GET 2023-10-12T00:25:27.130 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.130 DEBUG:boto:Data: 2023-10-12T00:25:27.130 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.131 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.132 DEBUG:boto:Params: {} 2023-10-12T00:25:27.132 DEBUG:boto:Token: None 2023-10-12T00:25:27.133 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.133 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd0953188a3af162-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.137 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.138 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.138 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.139 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.139 DEBUG:boto:Path: /wafmoa-11/dummy 2023-10-12T00:25:27.139 DEBUG:boto:Data: 2023-10-12T00:25:27.140 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.141 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.141 DEBUG:boto:Params: {} 2023-10-12T00:25:27.142 DEBUG:boto:Token: None 2023-10-12T00:25:27.142 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy 2023-10-12T00:25:27.142 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tGEB7tJJlzKmfj93dUkIk7CAgjQ= 2023-10-12T00:25:27.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tGEB7tJJlzKmfj93dUkIk7CAgjQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085f7e814956de862-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.146 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.146 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.147 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.147 DEBUG:boto:Path: /wafmoa-11/dummy 2023-10-12T00:25:27.148 DEBUG:boto:Data: 2023-10-12T00:25:27.148 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.149 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.150 DEBUG:boto:Params: {} 2023-10-12T00:25:27.150 DEBUG:boto:Token: None 2023-10-12T00:25:27.151 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy 2023-10-12T00:25:27.151 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tGEB7tJJlzKmfj93dUkIk7CAgjQ= 2023-10-12T00:25:27.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tGEB7tJJlzKmfj93dUkIk7CAgjQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000167c51cf9dcd2d31-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:27.155 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.155 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.156 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.156 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.157 DEBUG:boto:Method: GET 2023-10-12T00:25:27.157 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.158 DEBUG:boto:Data: 2023-10-12T00:25:27.158 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.159 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.159 DEBUG:boto:Params: {} 2023-10-12T00:25:27.160 DEBUG:boto:Token: None 2023-10-12T00:25:27.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003009df94704452ff-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.164 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:27.165 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:27.165 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.166 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.166 DEBUG:boto:Method: GET 2023-10-12T00:25:27.166 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.167 DEBUG:boto:Data: 2023-10-12T00:25:27.167 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.168 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.169 DEBUG:boto:Params: {} 2023-10-12T00:25:27.169 DEBUG:boto:Token: None 2023-10-12T00:25:27.170 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:27.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg= 2023-10-12T00:25:27.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TsJpPoyg0QlOqmPGUpyLAbVujjg=', 'Content-Length': '0'} 2023-10-12T00:25:27.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027b1e4c8817f6778-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.173 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.174 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.174 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.174 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.175 DEBUG:boto:Method: GET 2023-10-12T00:25:27.175 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.176 DEBUG:boto:Data: 2023-10-12T00:25:27.176 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.177 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.177 DEBUG:boto:Params: {} 2023-10-12T00:25:27.178 DEBUG:boto:Token: None 2023-10-12T00:25:27.178 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd173668a9909690-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.182 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.183 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.184 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.184 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.184 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.185 DEBUG:boto:Method: GET 2023-10-12T00:25:27.185 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:27.186 DEBUG:boto:Data: 2023-10-12T00:25:27.186 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.187 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.187 DEBUG:boto:Params: {} 2023-10-12T00:25:27.188 DEBUG:boto:Token: None 2023-10-12T00:25:27.188 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b9e2ad10ba20373-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.192 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:27.193 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.194 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.194 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.195 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.195 DEBUG:boto:Method: GET 2023-10-12T00:25:27.196 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.196 DEBUG:boto:Data: 2023-10-12T00:25:27.196 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.197 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.198 DEBUG:boto:Params: {} 2023-10-12T00:25:27.198 DEBUG:boto:Token: None 2023-10-12T00:25:27.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.199 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037d41b46ad45f416-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.203 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.203 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.203 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.204 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.204 DEBUG:boto:Method: GET 2023-10-12T00:25:27.205 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.205 DEBUG:boto:Data: 2023-10-12T00:25:27.206 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.206 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.207 DEBUG:boto:Params: {} 2023-10-12T00:25:27.207 DEBUG:boto:Token: None 2023-10-12T00:25:27.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbe926cde044139b-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.211 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.212 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.212 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.213 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.213 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.214 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.214 DEBUG:boto:Data: 2023-10-12T00:25:27.214 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.215 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.216 DEBUG:boto:Params: {} 2023-10-12T00:25:27.216 DEBUG:boto:Token: None 2023-10-12T00:25:27.216 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:alGMKFEa1g5jNvvcRW29NouA7RI= 2023-10-12T00:25:27.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:alGMKFEa1g5jNvvcRW29NouA7RI=', 'Content-Length': '0'} 2023-10-12T00:25:27.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbc288a9122daf67-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.220 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.221 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.221 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.221 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.222 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.222 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.223 DEBUG:boto:Data: 2023-10-12T00:25:27.223 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.224 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.224 DEBUG:boto:Params: {} 2023-10-12T00:25:27.225 DEBUG:boto:Token: None 2023-10-12T00:25:27.225 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:alGMKFEa1g5jNvvcRW29NouA7RI= 2023-10-12T00:25:27.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:alGMKFEa1g5jNvvcRW29NouA7RI=', 'Content-Length': '0'} 2023-10-12T00:25:27.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000285af79c370113c8-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:27.229 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.230 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.230 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.231 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.231 DEBUG:boto:Method: GET 2023-10-12T00:25:27.231 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.232 DEBUG:boto:Data: 2023-10-12T00:25:27.232 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.233 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.233 DEBUG:boto:Params: {} 2023-10-12T00:25:27.234 DEBUG:boto:Token: None 2023-10-12T00:25:27.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1c10ef74c7a0fa0-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.238 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.238 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.239 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.239 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.240 DEBUG:boto:Method: GET 2023-10-12T00:25:27.240 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.241 DEBUG:boto:Data: 2023-10-12T00:25:27.241 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.242 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.242 DEBUG:boto:Params: {} 2023-10-12T00:25:27.243 DEBUG:boto:Token: None 2023-10-12T00:25:27.243 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b995e1405799e9b2-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.286 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.287 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.287 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.288 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.288 DEBUG:boto:Method: GET 2023-10-12T00:25:27.288 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.289 DEBUG:boto:Data: 2023-10-12T00:25:27.289 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.290 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.290 DEBUG:boto:Params: {} 2023-10-12T00:25:27.291 DEBUG:boto:Token: None 2023-10-12T00:25:27.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004948f8ad865a3e1-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.295 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.296 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.297 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.297 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.298 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.298 DEBUG:boto:Method: GET 2023-10-12T00:25:27.299 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.299 DEBUG:boto:Data: 2023-10-12T00:25:27.299 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.300 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.301 DEBUG:boto:Params: {} 2023-10-12T00:25:27.301 DEBUG:boto:Token: None 2023-10-12T00:25:27.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688c906b268a3369-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.306 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.306 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.307 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.307 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.308 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:25:27.308 DEBUG:boto:Data: 2023-10-12T00:25:27.309 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.309 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.310 DEBUG:boto:Params: {} 2023-10-12T00:25:27.310 DEBUG:boto:Token: None 2023-10-12T00:25:27.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1 2023-10-12T00:25:27.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWG07iM0odw4cmymxu0Nait57A8= 2023-10-12T00:25:27.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWG07iM0odw4cmymxu0Nait57A8=', 'Content-Length': '0'} 2023-10-12T00:25:27.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aad8782c281f45a-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.315 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.315 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.317 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.318 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:25:27.318 DEBUG:boto:Data: 2023-10-12T00:25:27.318 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.319 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.320 DEBUG:boto:Params: {} 2023-10-12T00:25:27.320 DEBUG:boto:Token: None 2023-10-12T00:25:27.321 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1 2023-10-12T00:25:27.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWG07iM0odw4cmymxu0Nait57A8= 2023-10-12T00:25:27.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWG07iM0odw4cmymxu0Nait57A8=', 'Content-Length': '0'} 2023-10-12T00:25:27.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085467fbe99c1bcd0-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:27.325 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.326 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.326 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.327 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.327 DEBUG:boto:Method: GET 2023-10-12T00:25:27.328 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.328 DEBUG:boto:Data: 2023-10-12T00:25:27.328 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.329 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.330 DEBUG:boto:Params: {} 2023-10-12T00:25:27.330 DEBUG:boto:Token: None 2023-10-12T00:25:27.331 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002679eeef84844a7e-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.378 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.379 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:27.379 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.380 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.380 DEBUG:boto:Method: GET 2023-10-12T00:25:27.381 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.381 DEBUG:boto:Data: 2023-10-12T00:25:27.382 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.383 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.383 DEBUG:boto:Params: {} 2023-10-12T00:25:27.384 DEBUG:boto:Token: None 2023-10-12T00:25:27.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:27.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ= 2023-10-12T00:25:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CxwmhWKqklQjNVFBnp5ulhK1+JQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b47463d838bf6c5f-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.388 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.388 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.389 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.389 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.390 DEBUG:boto:Method: GET 2023-10-12T00:25:27.390 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.391 DEBUG:boto:Data: 2023-10-12T00:25:27.391 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.392 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.393 DEBUG:boto:Params: {} 2023-10-12T00:25:27.393 DEBUG:boto:Token: None 2023-10-12T00:25:27.394 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.394 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bddb1d39014d6d68-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.397 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.398 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.399 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.399 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.400 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.400 DEBUG:boto:Method: GET 2023-10-12T00:25:27.401 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:27.401 DEBUG:boto:Data: 2023-10-12T00:25:27.401 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.402 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.403 DEBUG:boto:Params: {} 2023-10-12T00:25:27.403 DEBUG:boto:Token: None 2023-10-12T00:25:27.404 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e53ce5646d7048-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.408 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:27.409 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.410 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.410 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.411 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.411 DEBUG:boto:Method: GET 2023-10-12T00:25:27.411 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.412 DEBUG:boto:Data: 2023-10-12T00:25:27.413 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.413 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.414 DEBUG:boto:Params: {} 2023-10-12T00:25:27.414 DEBUG:boto:Token: None 2023-10-12T00:25:27.415 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ad3c9f141c3f3d-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.419 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.419 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.420 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.420 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.421 DEBUG:boto:Method: GET 2023-10-12T00:25:27.421 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.421 DEBUG:boto:Data: 2023-10-12T00:25:27.422 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.423 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.423 DEBUG:boto:Params: {} 2023-10-12T00:25:27.424 DEBUG:boto:Token: None 2023-10-12T00:25:27.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a6a5c07cfd8781f-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.470 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.471 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.471 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.472 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.472 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.473 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.473 DEBUG:boto:Data: 2023-10-12T00:25:27.474 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.474 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.475 DEBUG:boto:Params: {} 2023-10-12T00:25:27.475 DEBUG:boto:Token: None 2023-10-12T00:25:27.476 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p1RgM233pedxMSEl1hWSFGZbRSQ= 2023-10-12T00:25:27.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p1RgM233pedxMSEl1hWSFGZbRSQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3fa7f2ab57e893b-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.480 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.480 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.481 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.481 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.481 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.482 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.482 DEBUG:boto:Data: 2023-10-12T00:25:27.483 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.483 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.484 DEBUG:boto:Params: {} 2023-10-12T00:25:27.484 DEBUG:boto:Token: None 2023-10-12T00:25:27.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p1RgM233pedxMSEl1hWSFGZbRSQ= 2023-10-12T00:25:27.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p1RgM233pedxMSEl1hWSFGZbRSQ=', 'Content-Length': '0'} 2023-10-12T00:25:27.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000972001b49022423b-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:27.489 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.489 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.489 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.490 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.490 DEBUG:boto:Method: GET 2023-10-12T00:25:27.491 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.491 DEBUG:boto:Data: 2023-10-12T00:25:27.492 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.493 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.493 DEBUG:boto:Params: {} 2023-10-12T00:25:27.494 DEBUG:boto:Token: None 2023-10-12T00:25:27.494 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eecfafae4ef2db09-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.498 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.498 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.499 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.499 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.500 DEBUG:boto:Method: GET 2023-10-12T00:25:27.501 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.501 DEBUG:boto:Data: 2023-10-12T00:25:27.502 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.503 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.503 DEBUG:boto:Params: {} 2023-10-12T00:25:27.504 DEBUG:boto:Token: None 2023-10-12T00:25:27.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbc0b3716189c115-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.508 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.509 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.509 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.509 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.510 DEBUG:boto:Method: GET 2023-10-12T00:25:27.510 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.511 DEBUG:boto:Data: 2023-10-12T00:25:27.511 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.512 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.513 DEBUG:boto:Params: {} 2023-10-12T00:25:27.513 DEBUG:boto:Token: None 2023-10-12T00:25:27.514 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd95d30e18528f69-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.518 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.519 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.519 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.520 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.522 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.522 DEBUG:boto:Method: GET 2023-10-12T00:25:27.523 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.523 DEBUG:boto:Data: 2023-10-12T00:25:27.524 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.525 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.525 DEBUG:boto:Params: {} 2023-10-12T00:25:27.526 DEBUG:boto:Token: None 2023-10-12T00:25:27.526 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d10c47ebb65db37d-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.531 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.532 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.532 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.533 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.533 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:25:27.534 DEBUG:boto:Data: 2023-10-12T00:25:27.534 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.535 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.535 DEBUG:boto:Params: {} 2023-10-12T00:25:27.536 DEBUG:boto:Token: None 2023-10-12T00:25:27.536 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2 2023-10-12T00:25:27.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LhNRjxatp5pOw7Tp24jZJCexMT0= 2023-10-12T00:25:27.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LhNRjxatp5pOw7Tp24jZJCexMT0=', 'Content-Length': '0'} 2023-10-12T00:25:27.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f4b2c8403841425-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.541 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.541 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.542 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.542 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:25:27.543 DEBUG:boto:Data: 2023-10-12T00:25:27.543 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.544 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.544 DEBUG:boto:Params: {} 2023-10-12T00:25:27.545 DEBUG:boto:Token: None 2023-10-12T00:25:27.545 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2 2023-10-12T00:25:27.546 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LhNRjxatp5pOw7Tp24jZJCexMT0= 2023-10-12T00:25:27.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LhNRjxatp5pOw7Tp24jZJCexMT0=', 'Content-Length': '0'} 2023-10-12T00:25:27.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b7a5b84ae34f849-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:27.550 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.550 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.550 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.551 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.551 DEBUG:boto:Method: GET 2023-10-12T00:25:27.552 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.552 DEBUG:boto:Data: 2023-10-12T00:25:27.553 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.554 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.554 DEBUG:boto:Params: {} 2023-10-12T00:25:27.555 DEBUG:boto:Token: None 2023-10-12T00:25:27.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc294fc627c6c323-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.559 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.559 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:27.560 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.560 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.561 DEBUG:boto:Method: GET 2023-10-12T00:25:27.561 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.562 DEBUG:boto:Data: 2023-10-12T00:25:27.562 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.563 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.563 DEBUG:boto:Params: {} 2023-10-12T00:25:27.564 DEBUG:boto:Token: None 2023-10-12T00:25:27.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:27.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc= 2023-10-12T00:25:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7QVZEU0cOK3NQ1H3uCe03Vg1OEc=', 'Content-Length': '0'} 2023-10-12T00:25:27.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064823d2c32f4e20e-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.569 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.570 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.570 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.571 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.571 DEBUG:boto:Method: GET 2023-10-12T00:25:27.571 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.572 DEBUG:boto:Data: 2023-10-12T00:25:27.572 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.573 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.574 DEBUG:boto:Params: {} 2023-10-12T00:25:27.574 DEBUG:boto:Token: None 2023-10-12T00:25:27.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cb68856348df18-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.579 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.580 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:27.580 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:27.581 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.581 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.582 DEBUG:boto:Method: GET 2023-10-12T00:25:27.582 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:27.583 DEBUG:boto:Data: 2023-10-12T00:25:27.583 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.584 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.585 DEBUG:boto:Params: {} 2023-10-12T00:25:27.585 DEBUG:boto:Token: None 2023-10-12T00:25:27.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-11/ 2023-10-12T00:25:27.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU= 2023-10-12T00:25:27.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jjmkuBEzDWjzuzQqwP0hY6WcHZU=', 'Content-Length': '0'} 2023-10-12T00:25:27.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740290bf58d19518-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.590 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.591 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-11 zones={a2, a1} 2023-10-12T00:25:27.591 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-12 2023-10-12T00:25:27.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:27.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:27.593 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:27.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:27.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:27.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:27.694+0000 7f4779689780 1 bucket sync caught up with source: 2023-10-12T00:25:27.694 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.4.11, , , 00000000002.8.11, 00000000010.30.6, ] 2023-10-12T00:25:27.694 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.4.11, , , 00000000002.8.11, 00000000010.30.6, ] 2023-10-12T00:25:27.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:27.694+0000 7f4779689780 0 bucket checkpoint complete 2023-10-12T00:25:27.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-12 2023-10-12T00:25:27.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:27.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:27.709 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:27.765 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:27.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:27.801+0000 7fa7650ef780 1 bucket sync caught up with source: 2023-10-12T00:25:27.801 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000010.49.6, 00000000002.42.11, ] 2023-10-12T00:25:27.801 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000010.49.6, 00000000002.42.11, ] 2023-10-12T00:25:27.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:27.801+0000 7fa7650ef780 0 bucket checkpoint complete 2023-10-12T00:25:27.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-12 zones={a2, a1} 2023-10-12T00:25:27.813 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.813 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.814 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.814 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:27.815 DEBUG:boto:Data: 2023-10-12T00:25:27.815 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.816 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.816 DEBUG:boto:Params: {} 2023-10-12T00:25:27.817 DEBUG:boto:Token: None 2023-10-12T00:25:27.817 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/ 2023-10-12T00:25:27.818 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0dt/kDkiYwhLWjV/HLst79eLxKY= 2023-10-12T00:25:27.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0dt/kDkiYwhLWjV/HLst79eLxKY=', 'Content-Length': '0'} 2023-10-12T00:25:27.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082d6ad322cb0f833-0065273cf7-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.821 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.822 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.822 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.823 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:27.823 DEBUG:boto:Data: 2023-10-12T00:25:27.824 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.824 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.825 DEBUG:boto:Params: {} 2023-10-12T00:25:27.825 DEBUG:boto:Token: None 2023-10-12T00:25:27.826 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/ 2023-10-12T00:25:27.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0dt/kDkiYwhLWjV/HLst79eLxKY= 2023-10-12T00:25:27.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0dt/kDkiYwhLWjV/HLst79eLxKY=', 'Content-Length': '0'} 2023-10-12T00:25:27.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048d2f98a82af52d4-0065273cf7-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.830 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:25:27.830 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.831 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.831 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.832 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.832 DEBUG:boto:Method: GET 2023-10-12T00:25:27.833 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.834 DEBUG:boto:Data: 2023-10-12T00:25:27.834 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.835 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.835 DEBUG:boto:Params: {} 2023-10-12T00:25:27.836 DEBUG:boto:Token: None 2023-10-12T00:25:27.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/?versions 2023-10-12T00:25:27.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g= 2023-10-12T00:25:27.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g=', 'Content-Length': '0'} 2023-10-12T00:25:27.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c0090c9625555b-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.840 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.840 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:27.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:27.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:27.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:25:27.842 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.842 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.843 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.843 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.844 DEBUG:boto:Method: GET 2023-10-12T00:25:27.844 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.845 DEBUG:boto:Data: 2023-10-12T00:25:27.845 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.846 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.846 DEBUG:boto:Params: {} 2023-10-12T00:25:27.847 DEBUG:boto:Token: None 2023-10-12T00:25:27.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/?versions 2023-10-12T00:25:27.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g= 2023-10-12T00:25:27.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g=', 'Content-Length': '0'} 2023-10-12T00:25:27.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe7ba68955fa576a-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.851 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.851 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:27.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:27.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:27.853 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.853 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.853 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.854 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.854 DEBUG:boto:Method: GET 2023-10-12T00:25:27.855 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.855 DEBUG:boto:Data: 2023-10-12T00:25:27.855 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.856 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.857 DEBUG:boto:Params: {} 2023-10-12T00:25:27.857 DEBUG:boto:Token: None 2023-10-12T00:25:27.858 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/?versions 2023-10-12T00:25:27.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g= 2023-10-12T00:25:27.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g=', 'Content-Length': '0'} 2023-10-12T00:25:27.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001721895e58fd6a31-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.861 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.862 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.862 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.863 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.863 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.864 DEBUG:boto:Method: GET 2023-10-12T00:25:27.864 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:27.865 DEBUG:boto:Data: 2023-10-12T00:25:27.865 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.866 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.866 DEBUG:boto:Params: {} 2023-10-12T00:25:27.867 DEBUG:boto:Token: None 2023-10-12T00:25:27.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/?versions 2023-10-12T00:25:27.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g= 2023-10-12T00:25:27.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9Y/ogcrgNHrnKFHWkBTLZpUJ3g=', 'Content-Length': '0'} 2023-10-12T00:25:27.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6080b5b5c2634eb-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.871 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:27.872 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.873 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.873 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.873 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.874 DEBUG:boto:Method: GET 2023-10-12T00:25:27.874 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.875 DEBUG:boto:Data: 2023-10-12T00:25:27.875 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.876 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.877 DEBUG:boto:Params: {} 2023-10-12T00:25:27.877 DEBUG:boto:Token: None 2023-10-12T00:25:27.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40= 2023-10-12T00:25:27.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40=', 'Content-Length': '0'} 2023-10-12T00:25:27.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cb2b3e4ddb7e6ef-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.881 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.881 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.882 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.882 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.883 DEBUG:boto:Method: GET 2023-10-12T00:25:27.883 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.884 DEBUG:boto:Data: 2023-10-12T00:25:27.884 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.885 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.885 DEBUG:boto:Params: {} 2023-10-12T00:25:27.886 DEBUG:boto:Token: None 2023-10-12T00:25:27.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40= 2023-10-12T00:25:27.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40=', 'Content-Length': '0'} 2023-10-12T00:25:27.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000efa5c0c997949eb2-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.890 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.890 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.891 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.891 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.891 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.892 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.892 DEBUG:boto:Data: 2023-10-12T00:25:27.893 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.893 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.894 DEBUG:boto:Params: {} 2023-10-12T00:25:27.894 DEBUG:boto:Token: None 2023-10-12T00:25:27.895 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1T4Z7TTttTxak0/3GStm3uaRGcY= 2023-10-12T00:25:27.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1T4Z7TTttTxak0/3GStm3uaRGcY=', 'Content-Length': '0'} 2023-10-12T00:25:27.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bab05656333d40d-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.898 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.899 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.899 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.900 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.900 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.900 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.901 DEBUG:boto:Data: 2023-10-12T00:25:27.901 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.902 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.902 DEBUG:boto:Params: {} 2023-10-12T00:25:27.903 DEBUG:boto:Token: None 2023-10-12T00:25:27.903 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1T4Z7TTttTxak0/3GStm3uaRGcY= 2023-10-12T00:25:27.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1T4Z7TTttTxak0/3GStm3uaRGcY=', 'Content-Length': '0'} 2023-10-12T00:25:27.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a29789140af58fbf-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:27.907 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.907 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.908 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.908 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.909 DEBUG:boto:Method: GET 2023-10-12T00:25:27.909 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.910 DEBUG:boto:Data: 2023-10-12T00:25:27.910 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.911 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.911 DEBUG:boto:Params: {} 2023-10-12T00:25:27.912 DEBUG:boto:Token: None 2023-10-12T00:25:27.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40= 2023-10-12T00:25:27.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40=', 'Content-Length': '0'} 2023-10-12T00:25:27.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000565688303cb58ead-0065273cf7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.916 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.916 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.917 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.917 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.917 DEBUG:boto:Method: GET 2023-10-12T00:25:27.918 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.918 DEBUG:boto:Data: 2023-10-12T00:25:27.919 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.920 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.920 DEBUG:boto:Params: {} 2023-10-12T00:25:27.921 DEBUG:boto:Token: None 2023-10-12T00:25:27.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:27.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40= 2023-10-12T00:25:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ldK6KEnz8lSxqOvFgZiBBVi3a40=', 'Content-Length': '0'} 2023-10-12T00:25:27.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000640695b748859abd-0065273cf7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.925 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.925 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.926 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.926 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.927 DEBUG:boto:Method: GET 2023-10-12T00:25:27.929 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.930 DEBUG:boto:Data: 2023-10-12T00:25:27.931 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.932 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.933 DEBUG:boto:Params: {} 2023-10-12T00:25:27.933 DEBUG:boto:Token: None 2023-10-12T00:25:27.934 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/ 2023-10-12T00:25:27.935 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P03XzPBAEndi5jnTutGuG1bEwRg= 2023-10-12T00:25:27.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P03XzPBAEndi5jnTutGuG1bEwRg=', 'Content-Length': '0'} 2023-10-12T00:25:27.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f71e5041f1f4bc5-0065273cf7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.982 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.983 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:27.984 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:27.984 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.985 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.985 DEBUG:boto:Method: GET 2023-10-12T00:25:27.986 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:27.986 DEBUG:boto:Data: 2023-10-12T00:25:27.987 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:27.988 DEBUG:boto:Port: 8000 2023-10-12T00:25:27.988 DEBUG:boto:Params: {} 2023-10-12T00:25:27.989 DEBUG:boto:Token: None 2023-10-12T00:25:27.989 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/ 2023-10-12T00:25:27.990 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P03XzPBAEndi5jnTutGuG1bEwRg= 2023-10-12T00:25:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P03XzPBAEndi5jnTutGuG1bEwRg=', 'Content-Length': '0'} 2023-10-12T00:25:27.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b97bd0ca79a4c57b-0065273cf7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:27.994 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:27.994 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:27.995 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:27.995 DEBUG:boto:Method: HEAD 2023-10-12T00:25:27.996 DEBUG:boto:Path: /wafmoa-12/dummy 2023-10-12T00:25:27.996 DEBUG:boto:Data: 2023-10-12T00:25:27.996 DEBUG:boto:Headers: {} 2023-10-12T00:25:27.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:27.997 DEBUG:boto:Port: 8001 2023-10-12T00:25:27.998 DEBUG:boto:Params: {} 2023-10-12T00:25:27.998 DEBUG:boto:Token: None 2023-10-12T00:25:27.999 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:27 GMT /wafmoa-12/dummy 2023-10-12T00:25:27.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yL2e9aW5UxLKv0ofTkJlvqPG7aE= 2023-10-12T00:25:28.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yL2e9aW5UxLKv0ofTkJlvqPG7aE=', 'Content-Length': '0'} 2023-10-12T00:25:28.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000779322cbdb5835dd-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.003 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:28.003 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:28.003 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.004 DEBUG:boto:Path: /wafmoa-12/dummy 2023-10-12T00:25:28.004 DEBUG:boto:Data: 2023-10-12T00:25:28.005 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.005 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.006 DEBUG:boto:Params: {} 2023-10-12T00:25:28.006 DEBUG:boto:Token: None 2023-10-12T00:25:28.007 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/dummy 2023-10-12T00:25:28.007 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:egOPQ7UKduyyhJUSizHODFX7Srg= 2023-10-12T00:25:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:egOPQ7UKduyyhJUSizHODFX7Srg=', 'Content-Length': '0'} 2023-10-12T00:25:28.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7915c2af2327581-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:28.011 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:28.011 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:28.012 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.012 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.013 DEBUG:boto:Method: GET 2023-10-12T00:25:28.013 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.014 DEBUG:boto:Data: 2023-10-12T00:25:28.014 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.015 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.016 DEBUG:boto:Params: {} 2023-10-12T00:25:28.016 DEBUG:boto:Token: None 2023-10-12T00:25:28.017 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fhlIrFgSGGQ6T7XTO3Zv68HOf34= 2023-10-12T00:25:28.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fhlIrFgSGGQ6T7XTO3Zv68HOf34=', 'Content-Length': '0'} 2023-10-12T00:25:28.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e02219463cd64ea-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.021 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:28.021 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:28.021 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.022 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.022 DEBUG:boto:Method: GET 2023-10-12T00:25:28.023 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.023 DEBUG:boto:Data: 2023-10-12T00:25:28.024 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.025 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.025 DEBUG:boto:Params: {} 2023-10-12T00:25:28.025 DEBUG:boto:Token: None 2023-10-12T00:25:28.026 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:28.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fhlIrFgSGGQ6T7XTO3Zv68HOf34= 2023-10-12T00:25:28.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fhlIrFgSGGQ6T7XTO3Zv68HOf34=', 'Content-Length': '0'} 2023-10-12T00:25:28.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000323ce2e4adb53a60-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.029 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.030 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.030 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.031 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.031 DEBUG:boto:Method: GET 2023-10-12T00:25:28.032 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.032 DEBUG:boto:Data: 2023-10-12T00:25:28.032 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.033 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.034 DEBUG:boto:Params: {} 2023-10-12T00:25:28.034 DEBUG:boto:Token: None 2023-10-12T00:25:28.035 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.035 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d3330b0c2baad4-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.038 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.039 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.040 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.040 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.040 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.041 DEBUG:boto:Method: GET 2023-10-12T00:25:28.041 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:28.042 DEBUG:boto:Data: 2023-10-12T00:25:28.042 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.043 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.044 DEBUG:boto:Params: {} 2023-10-12T00:25:28.044 DEBUG:boto:Token: None 2023-10-12T00:25:28.045 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee70ab9a14652f3a-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.048 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:28.050 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.050 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.050 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.051 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.051 DEBUG:boto:Method: GET 2023-10-12T00:25:28.052 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.052 DEBUG:boto:Data: 2023-10-12T00:25:28.053 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.054 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.054 DEBUG:boto:Params: {} 2023-10-12T00:25:28.055 DEBUG:boto:Token: None 2023-10-12T00:25:28.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3313d807eb8543b-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.059 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.059 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.059 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.060 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.060 DEBUG:boto:Method: GET 2023-10-12T00:25:28.061 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.061 DEBUG:boto:Data: 2023-10-12T00:25:28.062 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.063 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.063 DEBUG:boto:Params: {} 2023-10-12T00:25:28.064 DEBUG:boto:Token: None 2023-10-12T00:25:28.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060c7141fd8d99b1d-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.068 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.068 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.069 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.069 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.069 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.070 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.070 DEBUG:boto:Data: 2023-10-12T00:25:28.071 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.072 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.072 DEBUG:boto:Params: {} 2023-10-12T00:25:28.072 DEBUG:boto:Token: None 2023-10-12T00:25:28.073 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.077 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d7kIKN3tE2UTWytf7T5aI7dqET0= 2023-10-12T00:25:28.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d7kIKN3tE2UTWytf7T5aI7dqET0=', 'Content-Length': '0'} 2023-10-12T00:25:28.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001559f85cb619b96f-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.080 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.080 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.080 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.081 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.081 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.082 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.082 DEBUG:boto:Data: 2023-10-12T00:25:28.083 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.083 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.084 DEBUG:boto:Params: {} 2023-10-12T00:25:28.084 DEBUG:boto:Token: None 2023-10-12T00:25:28.085 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d7kIKN3tE2UTWytf7T5aI7dqET0= 2023-10-12T00:25:28.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d7kIKN3tE2UTWytf7T5aI7dqET0=', 'Content-Length': '0'} 2023-10-12T00:25:28.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8df35a568fca148-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:28.088 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.089 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.089 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.090 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.090 DEBUG:boto:Method: GET 2023-10-12T00:25:28.090 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.091 DEBUG:boto:Data: 2023-10-12T00:25:28.091 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.092 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.093 DEBUG:boto:Params: {} 2023-10-12T00:25:28.093 DEBUG:boto:Token: None 2023-10-12T00:25:28.094 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000002c4ceb382abec1-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.097 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.097 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.098 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.099 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.099 DEBUG:boto:Method: GET 2023-10-12T00:25:28.100 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.100 DEBUG:boto:Data: 2023-10-12T00:25:28.100 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.101 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.102 DEBUG:boto:Params: {} 2023-10-12T00:25:28.102 DEBUG:boto:Token: None 2023-10-12T00:25:28.103 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.103 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b598de6bb80456ae-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.107 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.107 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.107 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.108 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.108 DEBUG:boto:Method: GET 2023-10-12T00:25:28.109 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.109 DEBUG:boto:Data: 2023-10-12T00:25:28.110 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.110 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.111 DEBUG:boto:Params: {} 2023-10-12T00:25:28.111 DEBUG:boto:Token: None 2023-10-12T00:25:28.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a19d439634688d-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.115 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.116 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.116 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.117 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.117 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.118 DEBUG:boto:Method: GET 2023-10-12T00:25:28.118 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.119 DEBUG:boto:Data: 2023-10-12T00:25:28.119 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.120 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.120 DEBUG:boto:Params: {} 2023-10-12T00:25:28.121 DEBUG:boto:Token: None 2023-10-12T00:25:28.121 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94540599df5c54f-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.125 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.126 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.126 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.127 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.127 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:25:28.128 DEBUG:boto:Data: 2023-10-12T00:25:28.128 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.129 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.129 DEBUG:boto:Params: {} 2023-10-12T00:25:28.130 DEBUG:boto:Token: None 2023-10-12T00:25:28.130 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1 2023-10-12T00:25:28.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2m+xEa73Q1MhMRW6IMIN3TPRfhY= 2023-10-12T00:25:28.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2m+xEa73Q1MhMRW6IMIN3TPRfhY=', 'Content-Length': '0'} 2023-10-12T00:25:28.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000854c81832e65cd23-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.134 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.134 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.135 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.135 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:25:28.135 DEBUG:boto:Data: 2023-10-12T00:25:28.136 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.137 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.137 DEBUG:boto:Params: {} 2023-10-12T00:25:28.137 DEBUG:boto:Token: None 2023-10-12T00:25:28.138 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1 2023-10-12T00:25:28.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2m+xEa73Q1MhMRW6IMIN3TPRfhY= 2023-10-12T00:25:28.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2m+xEa73Q1MhMRW6IMIN3TPRfhY=', 'Content-Length': '0'} 2023-10-12T00:25:28.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca30c680e7b0c42c-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:28.142 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.142 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.143 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.143 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.144 DEBUG:boto:Method: GET 2023-10-12T00:25:28.144 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.144 DEBUG:boto:Data: 2023-10-12T00:25:28.145 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.146 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.146 DEBUG:boto:Params: {} 2023-10-12T00:25:28.147 DEBUG:boto:Token: None 2023-10-12T00:25:28.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075b2d8bb0a9c1806-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.190 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.191 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.191 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.192 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.192 DEBUG:boto:Method: GET 2023-10-12T00:25:28.193 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.193 DEBUG:boto:Data: 2023-10-12T00:25:28.194 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.195 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.195 DEBUG:boto:Params: {} 2023-10-12T00:25:28.196 DEBUG:boto:Token: None 2023-10-12T00:25:28.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:28.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM= 2023-10-12T00:25:28.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xd7Mf2zwODhVR4Muw63FNvzR3vM=', 'Content-Length': '0'} 2023-10-12T00:25:28.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca47f3ef4aa3300f-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.200 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.200 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.200 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.201 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.201 DEBUG:boto:Method: GET 2023-10-12T00:25:28.202 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.202 DEBUG:boto:Data: 2023-10-12T00:25:28.203 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.204 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.204 DEBUG:boto:Params: {} 2023-10-12T00:25:28.204 DEBUG:boto:Token: None 2023-10-12T00:25:28.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.205 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ba24f3fcbae7e3-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.209 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.209 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.210 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.210 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.211 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.211 DEBUG:boto:Method: GET 2023-10-12T00:25:28.212 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:28.212 DEBUG:boto:Data: 2023-10-12T00:25:28.213 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.213 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.214 DEBUG:boto:Params: {} 2023-10-12T00:25:28.214 DEBUG:boto:Token: None 2023-10-12T00:25:28.215 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000886f97d80af7c3ed-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.219 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:28.220 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.220 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.221 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.221 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.222 DEBUG:boto:Method: GET 2023-10-12T00:25:28.222 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.223 DEBUG:boto:Data: 2023-10-12T00:25:28.223 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.224 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.224 DEBUG:boto:Params: {} 2023-10-12T00:25:28.225 DEBUG:boto:Token: None 2023-10-12T00:25:28.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f03415a696263e26-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.229 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.230 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.230 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.230 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.231 DEBUG:boto:Method: GET 2023-10-12T00:25:28.231 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.232 DEBUG:boto:Data: 2023-10-12T00:25:28.232 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.233 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.233 DEBUG:boto:Params: {} 2023-10-12T00:25:28.234 DEBUG:boto:Token: None 2023-10-12T00:25:28.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a5fa2ac0fe3666b-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.238 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.239 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.239 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.239 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.240 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.240 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.241 DEBUG:boto:Data: 2023-10-12T00:25:28.241 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.242 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.242 DEBUG:boto:Params: {} 2023-10-12T00:25:28.243 DEBUG:boto:Token: None 2023-10-12T00:25:28.243 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z8Ubbz+n49DyK4fEM+Truor6Np8= 2023-10-12T00:25:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z8Ubbz+n49DyK4fEM+Truor6Np8=', 'Content-Length': '0'} 2023-10-12T00:25:28.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012c9f664c7773d1e-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.246 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.247 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.247 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.248 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.248 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.249 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.249 DEBUG:boto:Data: 2023-10-12T00:25:28.249 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.250 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.251 DEBUG:boto:Params: {} 2023-10-12T00:25:28.251 DEBUG:boto:Token: None 2023-10-12T00:25:28.252 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z8Ubbz+n49DyK4fEM+Truor6Np8= 2023-10-12T00:25:28.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z8Ubbz+n49DyK4fEM+Truor6Np8=', 'Content-Length': '0'} 2023-10-12T00:25:28.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cadba18a8729f21a-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:28.255 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.256 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.256 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.257 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.257 DEBUG:boto:Method: GET 2023-10-12T00:25:28.257 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.258 DEBUG:boto:Data: 2023-10-12T00:25:28.258 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.259 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.260 DEBUG:boto:Params: {} 2023-10-12T00:25:28.260 DEBUG:boto:Token: None 2023-10-12T00:25:28.261 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065a75d360715441f-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.264 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.264 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.265 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.265 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.266 DEBUG:boto:Method: GET 2023-10-12T00:25:28.266 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.266 DEBUG:boto:Data: 2023-10-12T00:25:28.267 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.268 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.268 DEBUG:boto:Params: {} 2023-10-12T00:25:28.269 DEBUG:boto:Token: None 2023-10-12T00:25:28.269 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065dff4848cde2c8a-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.314 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.315 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.316 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.316 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.316 DEBUG:boto:Method: GET 2023-10-12T00:25:28.317 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.317 DEBUG:boto:Data: 2023-10-12T00:25:28.318 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.319 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.319 DEBUG:boto:Params: {} 2023-10-12T00:25:28.320 DEBUG:boto:Token: None 2023-10-12T00:25:28.320 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d068fbf4774a5870-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.324 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.325 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.325 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.325 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.326 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.326 DEBUG:boto:Method: GET 2023-10-12T00:25:28.327 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.327 DEBUG:boto:Data: 2023-10-12T00:25:28.328 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.329 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.329 DEBUG:boto:Params: {} 2023-10-12T00:25:28.330 DEBUG:boto:Token: None 2023-10-12T00:25:28.330 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c19fb2ccb2d0b5-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.334 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.335 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.335 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.336 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.336 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:25:28.337 DEBUG:boto:Data: 2023-10-12T00:25:28.337 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.338 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.338 DEBUG:boto:Params: {} 2023-10-12T00:25:28.339 DEBUG:boto:Token: None 2023-10-12T00:25:28.339 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2 2023-10-12T00:25:28.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tBatAvL2ggcPdt2q9ry8JxENP/c= 2023-10-12T00:25:28.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tBatAvL2ggcPdt2q9ry8JxENP/c=', 'Content-Length': '0'} 2023-10-12T00:25:28.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000338dbd01eeb160c1-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.345 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.346 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.346 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.346 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:25:28.347 DEBUG:boto:Data: 2023-10-12T00:25:28.347 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.348 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.349 DEBUG:boto:Params: {} 2023-10-12T00:25:28.349 DEBUG:boto:Token: None 2023-10-12T00:25:28.350 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2 2023-10-12T00:25:28.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tBatAvL2ggcPdt2q9ry8JxENP/c= 2023-10-12T00:25:28.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tBatAvL2ggcPdt2q9ry8JxENP/c=', 'Content-Length': '0'} 2023-10-12T00:25:28.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000774bd595ca3941f8-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:28.354 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.355 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.355 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.355 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.356 DEBUG:boto:Method: GET 2023-10-12T00:25:28.357 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.357 DEBUG:boto:Data: 2023-10-12T00:25:28.357 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.358 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.359 DEBUG:boto:Params: {} 2023-10-12T00:25:28.359 DEBUG:boto:Token: None 2023-10-12T00:25:28.360 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf0035dcd1a279fd-0065273cf8-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.406 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.407 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.408 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.408 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.409 DEBUG:boto:Method: GET 2023-10-12T00:25:28.409 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.410 DEBUG:boto:Data: 2023-10-12T00:25:28.410 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.411 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.411 DEBUG:boto:Params: {} 2023-10-12T00:25:28.412 DEBUG:boto:Token: None 2023-10-12T00:25:28.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:28.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I= 2023-10-12T00:25:28.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTjdTZ5cy4aO/AjsdWVR8Zehp1I=', 'Content-Length': '0'} 2023-10-12T00:25:28.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054b355e040ca11df-0065273cf8-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.416 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.416 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.417 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.417 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.418 DEBUG:boto:Method: GET 2023-10-12T00:25:28.418 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.419 DEBUG:boto:Data: 2023-10-12T00:25:28.419 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.420 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.420 DEBUG:boto:Params: {} 2023-10-12T00:25:28.421 DEBUG:boto:Token: None 2023-10-12T00:25:28.421 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a9119646df70df-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.426 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.426 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.427 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.427 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.428 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.428 DEBUG:boto:Method: GET 2023-10-12T00:25:28.429 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:28.429 DEBUG:boto:Data: 2023-10-12T00:25:28.429 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.430 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.431 DEBUG:boto:Params: {} 2023-10-12T00:25:28.431 DEBUG:boto:Token: None 2023-10-12T00:25:28.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s= 2023-10-12T00:25:28.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJzHoGAgWIkqO53rfLCLLRt0x+s=', 'Content-Length': '0'} 2023-10-12T00:25:28.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635bc1afaca47963-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.436 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.436 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-12 zones={a2, a1} 2023-10-12T00:25:28.437 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:28.437 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:28.438 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.438 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:28.438 DEBUG:boto:Data: 2023-10-12T00:25:28.439 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.440 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.440 DEBUG:boto:Params: {} 2023-10-12T00:25:28.440 DEBUG:boto:Token: None 2023-10-12T00:25:28.441 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-11/ 2023-10-12T00:25:28.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:whqgMNifjZd9TwC2lN58VI30GOY= 2023-10-12T00:25:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:whqgMNifjZd9TwC2lN58VI30GOY=', 'Content-Length': '0'} 2023-10-12T00:25:28.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f3b922f230808ec-0065273cf8-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.445 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:28.445 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:28.446 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.446 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.447 DEBUG:boto:Method: GET 2023-10-12T00:25:28.447 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.447 DEBUG:boto:Data: 2023-10-12T00:25:28.448 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.449 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.449 DEBUG:boto:Params: {} 2023-10-12T00:25:28.450 DEBUG:boto:Token: None 2023-10-12T00:25:28.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-11/?versions 2023-10-12T00:25:28.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uvrkinZKE0cakaO4M6d2TPAIL2k= 2023-10-12T00:25:28.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uvrkinZKE0cakaO4M6d2TPAIL2k=', 'Content-Length': '0'} 2023-10-12T00:25:28.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd3e7f744f51343-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.498 DEBUG:boto:b'wafmoa-11obj-a1999falseobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.499 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:28.500 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:28.500 DEBUG:boto:path=/wafmoa-11/obj-a1?acl 2023-10-12T00:25:28.500 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?acl 2023-10-12T00:25:28.501 DEBUG:boto:Method: PUT 2023-10-12T00:25:28.501 DEBUG:boto:Path: /wafmoa-11/obj-a1?acl 2023-10-12T00:25:28.502 DEBUG:boto:Data: 2023-10-12T00:25:28.502 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:25:28.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.503 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.503 DEBUG:boto:Params: {} 2023-10-12T00:25:28.504 DEBUG:boto:Token: None 2023-10-12T00:25:28.504 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:28 GMT x-amz-acl:public-read /wafmoa-11/obj-a1?acl 2023-10-12T00:25:28.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gAcPlkFnPCCbwysSBE/E7mrdVwg= 2023-10-12T00:25:28.505 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gAcPlkFnPCCbwysSBE/E7mrdVwg=', 'Content-Length': '0'} 2023-10-12T00:25:28.511 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e085fca577072f21-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.511 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:28.512 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:28.512 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.512 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:28.513 DEBUG:boto:Data: 2023-10-12T00:25:28.513 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.514 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.514 DEBUG:boto:Params: {} 2023-10-12T00:25:28.515 DEBUG:boto:Token: None 2023-10-12T00:25:28.515 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-11/ 2023-10-12T00:25:28.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:whqgMNifjZd9TwC2lN58VI30GOY= 2023-10-12T00:25:28.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:whqgMNifjZd9TwC2lN58VI30GOY=', 'Content-Length': '0'} 2023-10-12T00:25:28.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025165f3bab33a309-0065273cf8-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.519 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:28.519 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:28.520 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.520 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.520 DEBUG:boto:Method: GET 2023-10-12T00:25:28.521 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.521 DEBUG:boto:Data: 2023-10-12T00:25:28.522 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.522 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.523 DEBUG:boto:Params: {} 2023-10-12T00:25:28.523 DEBUG:boto:Token: None 2023-10-12T00:25:28.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-11/?versions 2023-10-12T00:25:28.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uvrkinZKE0cakaO4M6d2TPAIL2k= 2023-10-12T00:25:28.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uvrkinZKE0cakaO4M6d2TPAIL2k=', 'Content-Length': '0'} 2023-10-12T00:25:28.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d13fb81818f0c8-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.527 DEBUG:boto:b'wafmoa-11obj-a2999falseobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.528 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:28.528 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:28.529 DEBUG:boto:path=/wafmoa-11/obj-a2?acl 2023-10-12T00:25:28.529 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?acl 2023-10-12T00:25:28.530 DEBUG:boto:Method: PUT 2023-10-12T00:25:28.530 DEBUG:boto:Path: /wafmoa-11/obj-a2?acl 2023-10-12T00:25:28.530 DEBUG:boto:Data: 2023-10-12T00:25:28.531 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:25:28.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.532 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.532 DEBUG:boto:Params: {} 2023-10-12T00:25:28.533 DEBUG:boto:Token: None 2023-10-12T00:25:28.533 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:28 GMT x-amz-acl:public-read /wafmoa-11/obj-a2?acl 2023-10-12T00:25:28.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zhymUB4Y2Vfs5D6jhORtOuCnDkA= 2023-10-12T00:25:28.534 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zhymUB4Y2Vfs5D6jhORtOuCnDkA=', 'Content-Length': '0'} 2023-10-12T00:25:28.539 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c450ff1c3203312-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.540 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.540 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.541 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.541 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:28.541 DEBUG:boto:Data: 2023-10-12T00:25:28.542 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.543 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.543 DEBUG:boto:Params: {} 2023-10-12T00:25:28.543 DEBUG:boto:Token: None 2023-10-12T00:25:28.544 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7ASrMvnNoFtCP1mX+jRDjyCZnow= 2023-10-12T00:25:28.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7ASrMvnNoFtCP1mX+jRDjyCZnow=', 'Content-Length': '0'} 2023-10-12T00:25:28.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb7f59b85dbd2cf0-0065273cf8-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.549 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.550 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.551 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.551 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.551 DEBUG:boto:Method: GET 2023-10-12T00:25:28.552 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999&prefix=obj-a1 2023-10-12T00:25:28.552 DEBUG:boto:Data: 2023-10-12T00:25:28.553 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.553 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.554 DEBUG:boto:Params: {} 2023-10-12T00:25:28.555 DEBUG:boto:Token: None 2023-10-12T00:25:28.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/?versions 2023-10-12T00:25:28.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJaYalAEFrbjzaAdUp1Olp5pYSc= 2023-10-12T00:25:28.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJaYalAEFrbjzaAdUp1Olp5pYSc=', 'Content-Length': '0'} 2023-10-12T00:25:28.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010459dd653e4b5d8-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.606 DEBUG:boto:b'wafmoa-12obj-a1999falseobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.607 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.608 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:28.608 DEBUG:boto:path=/wafmoa-12/obj-a1?acl 2023-10-12T00:25:28.609 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?acl 2023-10-12T00:25:28.609 DEBUG:boto:Method: PUT 2023-10-12T00:25:28.609 DEBUG:boto:Path: /wafmoa-12/obj-a1?acl 2023-10-12T00:25:28.610 DEBUG:boto:Data: 2023-10-12T00:25:28.610 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:25:28.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:28.611 DEBUG:boto:Port: 8000 2023-10-12T00:25:28.611 DEBUG:boto:Params: {} 2023-10-12T00:25:28.612 DEBUG:boto:Token: None 2023-10-12T00:25:28.612 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:28 GMT x-amz-acl:public-read /wafmoa-12/obj-a1?acl 2023-10-12T00:25:28.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FkhhapwSHmEo5m9gMg5/bATPea0= 2023-10-12T00:25:28.613 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FkhhapwSHmEo5m9gMg5/bATPea0=', 'Content-Length': '0'} 2023-10-12T00:25:28.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a39bc469008b365c-0065273cf8-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.619 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.620 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.620 DEBUG:boto:Method: HEAD 2023-10-12T00:25:28.621 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:28.621 DEBUG:boto:Data: 2023-10-12T00:25:28.621 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.622 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.623 DEBUG:boto:Params: {} 2023-10-12T00:25:28.623 DEBUG:boto:Token: None 2023-10-12T00:25:28.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/ 2023-10-12T00:25:28.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7ASrMvnNoFtCP1mX+jRDjyCZnow= 2023-10-12T00:25:28.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7ASrMvnNoFtCP1mX+jRDjyCZnow=', 'Content-Length': '0'} 2023-10-12T00:25:28.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc0d1a99b0699ea9-0065273cf8-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.627 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:28.628 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:28.628 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.629 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.629 DEBUG:boto:Method: GET 2023-10-12T00:25:28.630 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999&prefix=obj-a2 2023-10-12T00:25:28.630 DEBUG:boto:Data: 2023-10-12T00:25:28.630 DEBUG:boto:Headers: {} 2023-10-12T00:25:28.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.631 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.632 DEBUG:boto:Params: {} 2023-10-12T00:25:28.632 DEBUG:boto:Token: None 2023-10-12T00:25:28.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:28 GMT /wafmoa-12/?versions 2023-10-12T00:25:28.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJaYalAEFrbjzaAdUp1Olp5pYSc= 2023-10-12T00:25:28.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJaYalAEFrbjzaAdUp1Olp5pYSc=', 'Content-Length': '0'} 2023-10-12T00:25:28.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29665bd358c77b7-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.636 DEBUG:boto:b'wafmoa-12obj-a2999falseobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:28.637 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.637 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:28.638 DEBUG:boto:path=/wafmoa-12/obj-a2?acl 2023-10-12T00:25:28.638 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?acl 2023-10-12T00:25:28.639 DEBUG:boto:Method: PUT 2023-10-12T00:25:28.639 DEBUG:boto:Path: /wafmoa-12/obj-a2?acl 2023-10-12T00:25:28.639 DEBUG:boto:Data: 2023-10-12T00:25:28.640 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:25:28.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:28.641 DEBUG:boto:Port: 8001 2023-10-12T00:25:28.641 DEBUG:boto:Params: {} 2023-10-12T00:25:28.642 DEBUG:boto:Token: None 2023-10-12T00:25:28.642 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:28 GMT x-amz-acl:public-read /wafmoa-12/obj-a2?acl 2023-10-12T00:25:28.642 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tBuR+zkwF8UzPtdcdk0PX0s6DNk= 2023-10-12T00:25:28.643 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tBuR+zkwF8UzPtdcdk0PX0s6DNk=', 'Content-Length': '0'} 2023-10-12T00:25:28.648 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028d6eb58488ebc9f-0065273cf8-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:28.648 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-11 2023-10-12T00:25:28.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:28.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:28.650 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:28.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:28.740+0000 7fd200cae780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:28.741 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.30.6, ] 2023-10-12T00:25:28.741 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.33.6, ] 2023-10-12T00:25:33.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:33.741+0000 7fd200cae780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:33.741 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.30.6, ] 2023-10-12T00:25:33.741 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.33.6, ] 2023-10-12T00:25:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:38.742+0000 7fd200cae780 1 waiting for incremental sync to catch up: 2023-10-12T00:25:38.742 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.30.6, ] 2023-10-12T00:25:38.743 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.33.6, ] 2023-10-12T00:25:43.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:43.742+0000 7fd200cae780 1 bucket sync caught up with source: 2023-10-12T00:25:43.743 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.33.6, ] 2023-10-12T00:25:43.744 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.33.6, ] 2023-10-12T00:25:43.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:43.742+0000 7fd200cae780 0 bucket checkpoint complete 2023-10-12T00:25:43.759 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-11 2023-10-12T00:25:43.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:43.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:43.760 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:43.823 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:43.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:43.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:43.861+0000 7fb59932c780 1 bucket sync caught up with source: 2023-10-12T00:25:43.862 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.38.11, , , 00000000011.43.6, 00000000002.32.11, ] 2023-10-12T00:25:43.863 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.38.11, , , 00000000011.43.6, 00000000002.32.11, ] 2023-10-12T00:25:43.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:43.861+0000 7fb59932c780 0 bucket checkpoint complete 2023-10-12T00:25:43.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-11 zones={a2, a1} 2023-10-12T00:25:43.875 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.875 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.876 DEBUG:boto:Method: HEAD 2023-10-12T00:25:43.876 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:43.877 DEBUG:boto:Data: 2023-10-12T00:25:43.877 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.878 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.879 DEBUG:boto:Params: {} 2023-10-12T00:25:43.879 DEBUG:boto:Token: None 2023-10-12T00:25:43.880 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/ 2023-10-12T00:25:43.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E7aqEAMx3PvOdFtzalbgAdcmkrc= 2023-10-12T00:25:43.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E7aqEAMx3PvOdFtzalbgAdcmkrc=', 'Content-Length': '0'} 2023-10-12T00:25:43.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c33218b00f208b95-0065273d07-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.884 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.884 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.884 DEBUG:boto:Method: HEAD 2023-10-12T00:25:43.885 DEBUG:boto:Path: /wafmoa-11/ 2023-10-12T00:25:43.885 DEBUG:boto:Data: 2023-10-12T00:25:43.886 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.887 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.887 DEBUG:boto:Params: {} 2023-10-12T00:25:43.887 DEBUG:boto:Token: None 2023-10-12T00:25:43.888 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/ 2023-10-12T00:25:43.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E7aqEAMx3PvOdFtzalbgAdcmkrc= 2023-10-12T00:25:43.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E7aqEAMx3PvOdFtzalbgAdcmkrc=', 'Content-Length': '0'} 2023-10-12T00:25:43.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000353dda468a93e5c9-0065273d07-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.892 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:25:43.892 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.893 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.893 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.893 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.894 DEBUG:boto:Method: GET 2023-10-12T00:25:43.894 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.895 DEBUG:boto:Data: 2023-10-12T00:25:43.895 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.896 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.897 DEBUG:boto:Params: {} 2023-10-12T00:25:43.897 DEBUG:boto:Token: None 2023-10-12T00:25:43.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/?versions 2023-10-12T00:25:43.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8= 2023-10-12T00:25:43.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8=', 'Content-Length': '0'} 2023-10-12T00:25:43.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef9a817753067c7-0065273d07-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.901 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:43.902 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:43.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:43.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:43.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:25:43.904 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.904 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.905 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.905 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.905 DEBUG:boto:Method: GET 2023-10-12T00:25:43.906 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.906 DEBUG:boto:Data: 2023-10-12T00:25:43.907 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.908 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.908 DEBUG:boto:Params: {} 2023-10-12T00:25:43.908 DEBUG:boto:Token: None 2023-10-12T00:25:43.909 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/?versions 2023-10-12T00:25:43.909 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8= 2023-10-12T00:25:43.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8=', 'Content-Length': '0'} 2023-10-12T00:25:43.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b9cd1f47aeccad-0065273d07-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.913 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:43.914 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:43.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:43.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:43.915 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.915 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.916 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.916 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.916 DEBUG:boto:Method: GET 2023-10-12T00:25:43.917 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.917 DEBUG:boto:Data: 2023-10-12T00:25:43.918 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.919 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.919 DEBUG:boto:Params: {} 2023-10-12T00:25:43.920 DEBUG:boto:Token: None 2023-10-12T00:25:43.920 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/?versions 2023-10-12T00:25:43.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8= 2023-10-12T00:25:43.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8=', 'Content-Length': '0'} 2023-10-12T00:25:43.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d26529b8998a6d-0065273d07-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.924 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:43.924 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.925 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.925 DEBUG:boto:path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.926 DEBUG:boto:auth_path=/wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.926 DEBUG:boto:Method: GET 2023-10-12T00:25:43.927 DEBUG:boto:Path: /wafmoa-11/?versions&max-keys=999 2023-10-12T00:25:43.927 DEBUG:boto:Data: 2023-10-12T00:25:43.927 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.928 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.929 DEBUG:boto:Params: {} 2023-10-12T00:25:43.929 DEBUG:boto:Token: None 2023-10-12T00:25:43.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/?versions 2023-10-12T00:25:43.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8= 2023-10-12T00:25:43.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ku9h/IBGRWFG0fg5ZSff642bNW8=', 'Content-Length': '0'} 2023-10-12T00:25:43.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5533be721b9e0aa-0065273d07-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.933 DEBUG:boto:b'wafmoa-11999falsedummylImvWALzPEm2zZKItVytOIMUSFZgn1ztrue2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1true2023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2azELJcGCiG2WSz4.mxte.dD48473uSFtrue2023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:43.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:43.935 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.935 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.936 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.936 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.936 DEBUG:boto:Method: GET 2023-10-12T00:25:43.937 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.937 DEBUG:boto:Data: 2023-10-12T00:25:43.938 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.939 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.939 DEBUG:boto:Params: {} 2023-10-12T00:25:43.939 DEBUG:boto:Token: None 2023-10-12T00:25:43.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c= 2023-10-12T00:25:43.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c=', 'Content-Length': '0'} 2023-10-12T00:25:43.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7ee5b6f03b2f487-0065273d07-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.943 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.944 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.944 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.945 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.945 DEBUG:boto:Method: GET 2023-10-12T00:25:43.946 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.946 DEBUG:boto:Data: 2023-10-12T00:25:43.946 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.947 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.948 DEBUG:boto:Params: {} 2023-10-12T00:25:43.948 DEBUG:boto:Token: None 2023-10-12T00:25:43.949 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c= 2023-10-12T00:25:43.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c=', 'Content-Length': '0'} 2023-10-12T00:25:43.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b0df8c3f8589f3e-0065273d07-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.952 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.953 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.953 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.953 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.954 DEBUG:boto:Method: HEAD 2023-10-12T00:25:43.954 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.955 DEBUG:boto:Data: 2023-10-12T00:25:43.955 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.956 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.956 DEBUG:boto:Params: {} 2023-10-12T00:25:43.957 DEBUG:boto:Token: None 2023-10-12T00:25:43.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iBB4gCyQ0yo6DUdDDEv8eFk5ABc= 2023-10-12T00:25:43.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iBB4gCyQ0yo6DUdDDEv8eFk5ABc=', 'Content-Length': '0'} 2023-10-12T00:25:43.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052b23fe11e37f575-0065273d07-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.960 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.961 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.961 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.962 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.962 DEBUG:boto:Method: HEAD 2023-10-12T00:25:43.963 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.963 DEBUG:boto:Data: 2023-10-12T00:25:43.963 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.964 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.965 DEBUG:boto:Params: {} 2023-10-12T00:25:43.965 DEBUG:boto:Token: None 2023-10-12T00:25:43.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iBB4gCyQ0yo6DUdDDEv8eFk5ABc= 2023-10-12T00:25:43.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iBB4gCyQ0yo6DUdDDEv8eFk5ABc=', 'Content-Length': '0'} 2023-10-12T00:25:43.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d13d0d7ab995653-0065273d07-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:43.970 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.970 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.971 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.971 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.972 DEBUG:boto:Method: GET 2023-10-12T00:25:43.972 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.972 DEBUG:boto:Data: 2023-10-12T00:25:43.973 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.974 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.974 DEBUG:boto:Params: {} 2023-10-12T00:25:43.975 DEBUG:boto:Token: None 2023-10-12T00:25:43.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c= 2023-10-12T00:25:43.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c=', 'Content-Length': '0'} 2023-10-12T00:25:43.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074886bc47e967413-0065273d07-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.979 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:43.979 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:43.979 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.980 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.980 DEBUG:boto:Method: GET 2023-10-12T00:25:43.981 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.981 DEBUG:boto:Data: 2023-10-12T00:25:43.982 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:43.982 DEBUG:boto:Port: 8000 2023-10-12T00:25:43.983 DEBUG:boto:Params: {} 2023-10-12T00:25:43.983 DEBUG:boto:Token: None 2023-10-12T00:25:43.984 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:43.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c= 2023-10-12T00:25:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N2sBLZDXY7IZ7w3T4E9bNaJll4c=', 'Content-Length': '0'} 2023-10-12T00:25:43.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b938e674a8634800-0065273d07-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:43.987 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:43.988 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:43.988 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:43.989 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:43.989 DEBUG:boto:Method: GET 2023-10-12T00:25:43.989 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:43.990 DEBUG:boto:Data: 2023-10-12T00:25:43.990 DEBUG:boto:Headers: {} 2023-10-12T00:25:43.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:43.991 DEBUG:boto:Port: 8001 2023-10-12T00:25:43.992 DEBUG:boto:Params: {} 2023-10-12T00:25:43.992 DEBUG:boto:Token: None 2023-10-12T00:25:43.993 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:43 GMT /wafmoa-11/ 2023-10-12T00:25:43.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pym7Z57EB+NySNYI07QL+HbTmFs= 2023-10-12T00:25:43.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pym7Z57EB+NySNYI07QL+HbTmFs=', 'Content-Length': '0'} 2023-10-12T00:25:43.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52bca4b273097e2-0065273d07-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.038 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.039 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.040 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.040 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.040 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.041 DEBUG:boto:Method: GET 2023-10-12T00:25:44.041 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.042 DEBUG:boto:Data: 2023-10-12T00:25:44.042 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.043 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.043 DEBUG:boto:Params: {} 2023-10-12T00:25:44.044 DEBUG:boto:Token: None 2023-10-12T00:25:44.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449206440b6be896-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.049 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.049 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:44.050 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:44.050 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.051 DEBUG:boto:Path: /wafmoa-11/dummy 2023-10-12T00:25:44.051 DEBUG:boto:Data: 2023-10-12T00:25:44.051 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.052 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.053 DEBUG:boto:Params: {} 2023-10-12T00:25:44.053 DEBUG:boto:Token: None 2023-10-12T00:25:44.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/dummy 2023-10-12T00:25:44.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nNCjTRFY/FWfKLkx3RRTqcnGZ6A= 2023-10-12T00:25:44.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nNCjTRFY/FWfKLkx3RRTqcnGZ6A=', 'Content-Length': '0'} 2023-10-12T00:25:44.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f77d32fc9b6fa94-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.057 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:44.058 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:44.058 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.058 DEBUG:boto:Path: /wafmoa-11/dummy 2023-10-12T00:25:44.059 DEBUG:boto:Data: 2023-10-12T00:25:44.059 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.060 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.061 DEBUG:boto:Params: {} 2023-10-12T00:25:44.061 DEBUG:boto:Token: None 2023-10-12T00:25:44.061 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/dummy 2023-10-12T00:25:44.062 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nNCjTRFY/FWfKLkx3RRTqcnGZ6A= 2023-10-12T00:25:44.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nNCjTRFY/FWfKLkx3RRTqcnGZ6A=', 'Content-Length': '0'} 2023-10-12T00:25:44.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009750430c4f852ddd-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:44.066 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:44.066 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:44.067 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.067 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.068 DEBUG:boto:Method: GET 2023-10-12T00:25:44.068 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.069 DEBUG:boto:Data: 2023-10-12T00:25:44.069 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.070 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.071 DEBUG:boto:Params: {} 2023-10-12T00:25:44.071 DEBUG:boto:Token: None 2023-10-12T00:25:44.071 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9uDXKl0VMUHuP/INfsE+GIkdCuw= 2023-10-12T00:25:44.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9uDXKl0VMUHuP/INfsE+GIkdCuw=', 'Content-Length': '0'} 2023-10-12T00:25:44.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000928c6a82b71a34bb-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.075 DEBUG:boto:path=/wafmoa-11/dummy 2023-10-12T00:25:44.075 DEBUG:boto:auth_path=/wafmoa-11/dummy 2023-10-12T00:25:44.076 DEBUG:boto:path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.076 DEBUG:boto:auth_path=/wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.076 DEBUG:boto:Method: GET 2023-10-12T00:25:44.077 DEBUG:boto:Path: /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.077 DEBUG:boto:Data: 2023-10-12T00:25:44.078 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.079 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.079 DEBUG:boto:Params: {} 2023-10-12T00:25:44.080 DEBUG:boto:Token: None 2023-10-12T00:25:44.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/dummy?versionId=lImvWALzPEm2zZKItVytOIMUSFZgn1z 2023-10-12T00:25:44.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9uDXKl0VMUHuP/INfsE+GIkdCuw= 2023-10-12T00:25:44.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9uDXKl0VMUHuP/INfsE+GIkdCuw=', 'Content-Length': '0'} 2023-10-12T00:25:44.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:10 GMT'), ('x-amz-version-id', 'lImvWALzPEm2zZKItVytOIMUSFZgn1z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b29f4d2326887cd-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.083 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.084 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.084 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.085 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.085 DEBUG:boto:Method: GET 2023-10-12T00:25:44.086 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.086 DEBUG:boto:Data: 2023-10-12T00:25:44.086 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.087 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.088 DEBUG:boto:Params: {} 2023-10-12T00:25:44.088 DEBUG:boto:Token: None 2023-10-12T00:25:44.089 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.089 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b16fc1a97a6a7ab0-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.093 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.093 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.094 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.094 DEBUG:boto:path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.095 DEBUG:boto:auth_path=/wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.095 DEBUG:boto:Method: GET 2023-10-12T00:25:44.096 DEBUG:boto:Path: /wafmoa-11/?prefix=dummy 2023-10-12T00:25:44.096 DEBUG:boto:Data: 2023-10-12T00:25:44.097 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.098 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.098 DEBUG:boto:Params: {} 2023-10-12T00:25:44.099 DEBUG:boto:Token: None 2023-10-12T00:25:44.099 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e760adab5ae9bcb-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.146 DEBUG:boto:b'wafmoa-11dummy1000falsedummy2023-10-12T00:24:10.937Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:44.148 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.148 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.149 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.149 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.149 DEBUG:boto:Method: GET 2023-10-12T00:25:44.150 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.150 DEBUG:boto:Data: 2023-10-12T00:25:44.151 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.151 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.152 DEBUG:boto:Params: {} 2023-10-12T00:25:44.152 DEBUG:boto:Token: None 2023-10-12T00:25:44.153 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2d4e9a1c51c8584-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.156 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.157 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.157 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.158 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.158 DEBUG:boto:Method: GET 2023-10-12T00:25:44.159 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.159 DEBUG:boto:Data: 2023-10-12T00:25:44.160 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.160 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.161 DEBUG:boto:Params: {} 2023-10-12T00:25:44.161 DEBUG:boto:Token: None 2023-10-12T00:25:44.162 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c80caaf14f261a59-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.165 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.166 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.166 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.166 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.167 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.167 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.168 DEBUG:boto:Data: 2023-10-12T00:25:44.168 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.169 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.169 DEBUG:boto:Params: {} 2023-10-12T00:25:44.170 DEBUG:boto:Token: None 2023-10-12T00:25:44.170 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.171 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ZPFFW2Q0kB8njOV8TaMrW0SD1Y= 2023-10-12T00:25:44.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ZPFFW2Q0kB8njOV8TaMrW0SD1Y=', 'Content-Length': '0'} 2023-10-12T00:25:44.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007da7d429341866d-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.173 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.174 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.174 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.175 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.175 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.176 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.176 DEBUG:boto:Data: 2023-10-12T00:25:44.176 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.177 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.178 DEBUG:boto:Params: {} 2023-10-12T00:25:44.178 DEBUG:boto:Token: None 2023-10-12T00:25:44.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ZPFFW2Q0kB8njOV8TaMrW0SD1Y= 2023-10-12T00:25:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ZPFFW2Q0kB8njOV8TaMrW0SD1Y=', 'Content-Length': '0'} 2023-10-12T00:25:44.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003188d83f5837b6f6-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:44.183 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.183 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.183 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.184 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.184 DEBUG:boto:Method: GET 2023-10-12T00:25:44.185 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.185 DEBUG:boto:Data: 2023-10-12T00:25:44.185 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.186 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.187 DEBUG:boto:Params: {} 2023-10-12T00:25:44.187 DEBUG:boto:Token: None 2023-10-12T00:25:44.188 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b71bd7a82d6c24c-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.191 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.191 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.192 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.192 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.193 DEBUG:boto:Method: GET 2023-10-12T00:25:44.193 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.194 DEBUG:boto:Data: 2023-10-12T00:25:44.194 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.195 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.195 DEBUG:boto:Params: {} 2023-10-12T00:25:44.196 DEBUG:boto:Token: None 2023-10-12T00:25:44.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031bd1d36269dc9ec-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.200 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.200 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.201 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.201 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.202 DEBUG:boto:Method: GET 2023-10-12T00:25:44.202 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.202 DEBUG:boto:Data: 2023-10-12T00:25:44.203 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.204 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.204 DEBUG:boto:Params: {} 2023-10-12T00:25:44.205 DEBUG:boto:Token: None 2023-10-12T00:25:44.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f7439d8d13dca0-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.250 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.251 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.251 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.252 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.252 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.253 DEBUG:boto:Method: GET 2023-10-12T00:25:44.253 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.254 DEBUG:boto:Data: 2023-10-12T00:25:44.254 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.255 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.256 DEBUG:boto:Params: {} 2023-10-12T00:25:44.256 DEBUG:boto:Token: None 2023-10-12T00:25:44.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c9c21958a51844c-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.261 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.261 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.262 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.262 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.263 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:25:44.263 DEBUG:boto:Data: 2023-10-12T00:25:44.264 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.265 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.265 DEBUG:boto:Params: {} 2023-10-12T00:25:44.265 DEBUG:boto:Token: None 2023-10-12T00:25:44.266 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1 2023-10-12T00:25:44.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4ISx01hhd1J8yW6tibCUXUOoHN8= 2023-10-12T00:25:44.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4ISx01hhd1J8yW6tibCUXUOoHN8=', 'Content-Length': '0'} 2023-10-12T00:25:44.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dea0c1ce9da867af-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.269 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.270 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.270 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.271 DEBUG:boto:Path: /wafmoa-11/obj-a1 2023-10-12T00:25:44.271 DEBUG:boto:Data: 2023-10-12T00:25:44.272 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.273 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.273 DEBUG:boto:Params: {} 2023-10-12T00:25:44.273 DEBUG:boto:Token: None 2023-10-12T00:25:44.274 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1 2023-10-12T00:25:44.274 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4ISx01hhd1J8yW6tibCUXUOoHN8= 2023-10-12T00:25:44.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4ISx01hhd1J8yW6tibCUXUOoHN8=', 'Content-Length': '0'} 2023-10-12T00:25:44.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000383cab3a4d292b64-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:44.278 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.278 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.279 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.279 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.280 DEBUG:boto:Method: GET 2023-10-12T00:25:44.280 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.281 DEBUG:boto:Data: 2023-10-12T00:25:44.281 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.282 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.283 DEBUG:boto:Params: {} 2023-10-12T00:25:44.283 DEBUG:boto:Token: None 2023-10-12T00:25:44.284 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e193ddbfdf5d2e4-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.287 DEBUG:boto:path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.288 DEBUG:boto:auth_path=/wafmoa-11/obj-a1 2023-10-12T00:25:44.288 DEBUG:boto:path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.289 DEBUG:boto:auth_path=/wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.290 DEBUG:boto:Method: GET 2023-10-12T00:25:44.290 DEBUG:boto:Path: /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.291 DEBUG:boto:Data: 2023-10-12T00:25:44.291 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.292 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.293 DEBUG:boto:Params: {} 2023-10-12T00:25:44.293 DEBUG:boto:Token: None 2023-10-12T00:25:44.294 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a1?versionId=iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1 2023-10-12T00:25:44.294 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78= 2023-10-12T00:25:44.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k9btfX5CULTim5VB4W+gz4bYW78=', 'Content-Length': '0'} 2023-10-12T00:25:44.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'iaMSUbSdoHu5UR9G3ffsB5BEKCI08H1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ff496eec5f24d4a-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.298 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.298 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.299 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.299 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.300 DEBUG:boto:Method: GET 2023-10-12T00:25:44.300 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.301 DEBUG:boto:Data: 2023-10-12T00:25:44.302 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.303 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.303 DEBUG:boto:Params: {} 2023-10-12T00:25:44.304 DEBUG:boto:Token: None 2023-10-12T00:25:44.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2807a5dd7c792e-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.308 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.309 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.310 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.311 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.311 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.312 DEBUG:boto:Method: GET 2023-10-12T00:25:44.312 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a1 2023-10-12T00:25:44.313 DEBUG:boto:Data: 2023-10-12T00:25:44.313 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.314 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.315 DEBUG:boto:Params: {} 2023-10-12T00:25:44.315 DEBUG:boto:Token: None 2023-10-12T00:25:44.316 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050067fbb753f3761-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.320 DEBUG:boto:b'wafmoa-11obj-a11000falseobj-a12023-10-12T00:24:46.218Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:44.321 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.321 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.322 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.322 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.323 DEBUG:boto:Method: GET 2023-10-12T00:25:44.323 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.324 DEBUG:boto:Data: 2023-10-12T00:25:44.324 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.325 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.326 DEBUG:boto:Params: {} 2023-10-12T00:25:44.326 DEBUG:boto:Token: None 2023-10-12T00:25:44.327 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000628791d7dad62fe-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.331 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.331 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.331 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.332 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.332 DEBUG:boto:Method: GET 2023-10-12T00:25:44.333 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.333 DEBUG:boto:Data: 2023-10-12T00:25:44.334 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.335 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.335 DEBUG:boto:Params: {} 2023-10-12T00:25:44.336 DEBUG:boto:Token: None 2023-10-12T00:25:44.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ec8693833c81e1f-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.340 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.340 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.341 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.341 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.342 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.342 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.343 DEBUG:boto:Data: 2023-10-12T00:25:44.343 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.344 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.345 DEBUG:boto:Params: {} 2023-10-12T00:25:44.345 DEBUG:boto:Token: None 2023-10-12T00:25:44.346 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:szoEIXB+/fBs13woiFZDv77QrZ8= 2023-10-12T00:25:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:szoEIXB+/fBs13woiFZDv77QrZ8=', 'Content-Length': '0'} 2023-10-12T00:25:44.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055d2040a46c7e929-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.350 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.350 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.351 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.351 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.352 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.352 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.353 DEBUG:boto:Data: 2023-10-12T00:25:44.353 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.355 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.355 DEBUG:boto:Params: {} 2023-10-12T00:25:44.356 DEBUG:boto:Token: None 2023-10-12T00:25:44.356 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:szoEIXB+/fBs13woiFZDv77QrZ8= 2023-10-12T00:25:44.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:szoEIXB+/fBs13woiFZDv77QrZ8=', 'Content-Length': '0'} 2023-10-12T00:25:44.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064647ce2f0f5f1d6-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:44.361 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.362 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.362 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.363 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.363 DEBUG:boto:Method: GET 2023-10-12T00:25:44.364 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.364 DEBUG:boto:Data: 2023-10-12T00:25:44.365 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.366 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.366 DEBUG:boto:Params: {} 2023-10-12T00:25:44.367 DEBUG:boto:Token: None 2023-10-12T00:25:44.367 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d353d16d7a68fb68-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.371 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.371 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.372 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.372 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.373 DEBUG:boto:Method: GET 2023-10-12T00:25:44.373 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.374 DEBUG:boto:Data: 2023-10-12T00:25:44.374 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.375 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.376 DEBUG:boto:Params: {} 2023-10-12T00:25:44.376 DEBUG:boto:Token: None 2023-10-12T00:25:44.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b51d72c06c476dc-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.380 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.381 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.381 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.382 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.382 DEBUG:boto:Method: GET 2023-10-12T00:25:44.383 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.383 DEBUG:boto:Data: 2023-10-12T00:25:44.384 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.384 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.385 DEBUG:boto:Params: {} 2023-10-12T00:25:44.385 DEBUG:boto:Token: None 2023-10-12T00:25:44.386 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009510505e6fd9fb42-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.430 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.431 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.431 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.432 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.432 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.433 DEBUG:boto:Method: GET 2023-10-12T00:25:44.433 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.434 DEBUG:boto:Data: 2023-10-12T00:25:44.434 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.435 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.436 DEBUG:boto:Params: {} 2023-10-12T00:25:44.436 DEBUG:boto:Token: None 2023-10-12T00:25:44.437 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb687eb3c50ddb84-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.441 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.441 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.442 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.442 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.443 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:25:44.443 DEBUG:boto:Data: 2023-10-12T00:25:44.443 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.444 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.444 DEBUG:boto:Params: {} 2023-10-12T00:25:44.445 DEBUG:boto:Token: None 2023-10-12T00:25:44.445 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2 2023-10-12T00:25:44.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ws2gbIxkSBu5K55LuGQ84HEgngI= 2023-10-12T00:25:44.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ws2gbIxkSBu5K55LuGQ84HEgngI=', 'Content-Length': '0'} 2023-10-12T00:25:44.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000176f6a4fc99c04d0-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.449 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.449 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.450 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.450 DEBUG:boto:Path: /wafmoa-11/obj-a2 2023-10-12T00:25:44.450 DEBUG:boto:Data: 2023-10-12T00:25:44.451 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.451 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.452 DEBUG:boto:Params: {} 2023-10-12T00:25:44.452 DEBUG:boto:Token: None 2023-10-12T00:25:44.453 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2 2023-10-12T00:25:44.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ws2gbIxkSBu5K55LuGQ84HEgngI= 2023-10-12T00:25:44.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ws2gbIxkSBu5K55LuGQ84HEgngI=', 'Content-Length': '0'} 2023-10-12T00:25:44.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bec4c39054f64a23-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:44.457 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.457 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.458 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.458 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.459 DEBUG:boto:Method: GET 2023-10-12T00:25:44.459 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.459 DEBUG:boto:Data: 2023-10-12T00:25:44.460 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.461 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.461 DEBUG:boto:Params: {} 2023-10-12T00:25:44.462 DEBUG:boto:Token: None 2023-10-12T00:25:44.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000183f1903cc0aa1ad-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.465 DEBUG:boto:path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.466 DEBUG:boto:auth_path=/wafmoa-11/obj-a2 2023-10-12T00:25:44.466 DEBUG:boto:path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.467 DEBUG:boto:auth_path=/wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.467 DEBUG:boto:Method: GET 2023-10-12T00:25:44.468 DEBUG:boto:Path: /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.468 DEBUG:boto:Data: 2023-10-12T00:25:44.469 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.470 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.470 DEBUG:boto:Params: {} 2023-10-12T00:25:44.470 DEBUG:boto:Token: None 2023-10-12T00:25:44.471 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/obj-a2?versionId=azELJcGCiG2WSz4.mxte.dD48473uSF 2023-10-12T00:25:44.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q= 2023-10-12T00:25:44.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ONFOeSnbow3OvqlsUW3j05m+T2Q=', 'Content-Length': '0'} 2023-10-12T00:25:44.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'azELJcGCiG2WSz4.mxte.dD48473uSF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca5af4eb09b8fe0-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.474 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.475 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.475 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.476 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.476 DEBUG:boto:Method: GET 2023-10-12T00:25:44.477 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.477 DEBUG:boto:Data: 2023-10-12T00:25:44.477 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.478 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.479 DEBUG:boto:Params: {} 2023-10-12T00:25:44.479 DEBUG:boto:Token: None 2023-10-12T00:25:44.480 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.480 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359ed948c91be466-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.483 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.484 DEBUG:boto:path=/wafmoa-11/ 2023-10-12T00:25:44.484 DEBUG:boto:auth_path=/wafmoa-11/ 2023-10-12T00:25:44.484 DEBUG:boto:path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.485 DEBUG:boto:auth_path=/wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.485 DEBUG:boto:Method: GET 2023-10-12T00:25:44.486 DEBUG:boto:Path: /wafmoa-11/?prefix=obj-a2 2023-10-12T00:25:44.486 DEBUG:boto:Data: 2023-10-12T00:25:44.487 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.487 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.488 DEBUG:boto:Params: {} 2023-10-12T00:25:44.488 DEBUG:boto:Token: None 2023-10-12T00:25:44.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-11/ 2023-10-12T00:25:44.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg= 2023-10-12T00:25:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JPuZ1M0XdwT6ZCS4jg+q+Sdphmg=', 'Content-Length': '0'} 2023-10-12T00:25:44.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8d555213875411-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.493 DEBUG:boto:b'wafmoa-11obj-a21000falseobj-a22023-10-12T00:24:46.351Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-11 zones={a2, a1} 2023-10-12T00:25:44.494 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-12 2023-10-12T00:25:44.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:44.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:44.495 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:44.552 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:44.552 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:44.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:44.584+0000 7f8ae6c92780 1 bucket sync caught up with source: 2023-10-12T00:25:44.585 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.4.11, , , 00000000002.8.11, 00000000011.33.6, ] 2023-10-12T00:25:44.585 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.4.11, , , 00000000002.8.11, 00000000011.33.6, ] 2023-10-12T00:25:44.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:44.584+0000 7f8ae6c92780 0 bucket checkpoint complete 2023-10-12T00:25:44.596 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-12 2023-10-12T00:25:44.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:44.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:44.598 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:44.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:44.655 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:44.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:44.687+0000 7fa881cbd780 1 bucket sync caught up with source: 2023-10-12T00:25:44.688 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000011.53.6, 00000000002.42.11, ] 2023-10-12T00:25:44.688 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000011.53.6, 00000000002.42.11, ] 2023-10-12T00:25:44.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:44.687+0000 7fa881cbd780 0 bucket checkpoint complete 2023-10-12T00:25:44.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-12 zones={a2, a1} 2023-10-12T00:25:44.702 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.702 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.703 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.703 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:44.704 DEBUG:boto:Data: 2023-10-12T00:25:44.704 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.705 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.705 DEBUG:boto:Params: {} 2023-10-12T00:25:44.706 DEBUG:boto:Token: None 2023-10-12T00:25:44.706 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6Iw2BwSnuhiT08ywLzz3FeTYi5k= 2023-10-12T00:25:44.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6Iw2BwSnuhiT08ywLzz3FeTYi5k=', 'Content-Length': '0'} 2023-10-12T00:25:44.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000addd338e73559238-0065273d08-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.710 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.711 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.711 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.711 DEBUG:boto:Path: /wafmoa-12/ 2023-10-12T00:25:44.712 DEBUG:boto:Data: 2023-10-12T00:25:44.712 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.713 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.714 DEBUG:boto:Params: {} 2023-10-12T00:25:44.714 DEBUG:boto:Token: None 2023-10-12T00:25:44.715 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6Iw2BwSnuhiT08ywLzz3FeTYi5k= 2023-10-12T00:25:44.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6Iw2BwSnuhiT08ywLzz3FeTYi5k=', 'Content-Length': '0'} 2023-10-12T00:25:44.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d8ece5f1eed69a-0065273d08-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.718 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:25:44.719 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.719 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.719 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.720 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.720 DEBUG:boto:Method: GET 2023-10-12T00:25:44.721 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.721 DEBUG:boto:Data: 2023-10-12T00:25:44.721 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.722 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.723 DEBUG:boto:Params: {} 2023-10-12T00:25:44.723 DEBUG:boto:Token: None 2023-10-12T00:25:44.724 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/?versions 2023-10-12T00:25:44.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc= 2023-10-12T00:25:44.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc=', 'Content-Length': '0'} 2023-10-12T00:25:44.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f931a5136fd683e1-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.728 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:44.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:44.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:25:44.730 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.731 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.731 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.732 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.732 DEBUG:boto:Method: GET 2023-10-12T00:25:44.732 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.733 DEBUG:boto:Data: 2023-10-12T00:25:44.733 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.734 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.734 DEBUG:boto:Params: {} 2023-10-12T00:25:44.735 DEBUG:boto:Token: None 2023-10-12T00:25:44.735 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/?versions 2023-10-12T00:25:44.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc= 2023-10-12T00:25:44.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc=', 'Content-Length': '0'} 2023-10-12T00:25:44.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094cabc47c0f5a8fa-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.739 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:25:44.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-12T00:25:44.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-12T00:25:44.741 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.741 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.742 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.742 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.743 DEBUG:boto:Method: GET 2023-10-12T00:25:44.743 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.744 DEBUG:boto:Data: 2023-10-12T00:25:44.744 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.745 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.745 DEBUG:boto:Params: {} 2023-10-12T00:25:44.746 DEBUG:boto:Token: None 2023-10-12T00:25:44.746 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/?versions 2023-10-12T00:25:44.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc= 2023-10-12T00:25:44.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc=', 'Content-Length': '0'} 2023-10-12T00:25:44.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7564b5b87aa896-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.751 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.752 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.752 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.753 DEBUG:boto:path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.753 DEBUG:boto:auth_path=/wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.754 DEBUG:boto:Method: GET 2023-10-12T00:25:44.754 DEBUG:boto:Path: /wafmoa-12/?versions&max-keys=999 2023-10-12T00:25:44.754 DEBUG:boto:Data: 2023-10-12T00:25:44.755 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.756 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.756 DEBUG:boto:Params: {} 2023-10-12T00:25:44.756 DEBUG:boto:Token: None 2023-10-12T00:25:44.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/?versions 2023-10-12T00:25:44.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc= 2023-10-12T00:25:44.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H8acZuRrshx4wqSY6hH9RKerWXc=', 'Content-Length': '0'} 2023-10-12T00:25:44.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000936299a30f2134d6-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.761 DEBUG:boto:b'wafmoa-12999falsedummy1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJtrue2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7true2023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2K16BzI0q1qGkqpfB4TyZSGkYxAF1pCTtrue2023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:44.762 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.762 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.763 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.763 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.764 DEBUG:boto:Method: GET 2023-10-12T00:25:44.764 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.764 DEBUG:boto:Data: 2023-10-12T00:25:44.765 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.766 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.766 DEBUG:boto:Params: {} 2023-10-12T00:25:44.767 DEBUG:boto:Token: None 2023-10-12T00:25:44.767 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000faf1ed6dd671bf56-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.771 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.771 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.771 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.772 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.772 DEBUG:boto:Method: GET 2023-10-12T00:25:44.773 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.773 DEBUG:boto:Data: 2023-10-12T00:25:44.774 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.774 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.775 DEBUG:boto:Params: {} 2023-10-12T00:25:44.775 DEBUG:boto:Token: None 2023-10-12T00:25:44.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ed5c154abb97762-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.779 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.780 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.780 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.780 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.781 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.781 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.782 DEBUG:boto:Data: 2023-10-12T00:25:44.782 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.783 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.783 DEBUG:boto:Params: {} 2023-10-12T00:25:44.784 DEBUG:boto:Token: None 2023-10-12T00:25:44.784 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ksPbMCvsFPJrrTJEVCOppqgBXU4= 2023-10-12T00:25:44.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ksPbMCvsFPJrrTJEVCOppqgBXU4=', 'Content-Length': '0'} 2023-10-12T00:25:44.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087c4d566781b79ef-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.787 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.788 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.788 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.789 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.789 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.789 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.790 DEBUG:boto:Data: 2023-10-12T00:25:44.790 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.791 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.791 DEBUG:boto:Params: {} 2023-10-12T00:25:44.792 DEBUG:boto:Token: None 2023-10-12T00:25:44.792 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ksPbMCvsFPJrrTJEVCOppqgBXU4= 2023-10-12T00:25:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ksPbMCvsFPJrrTJEVCOppqgBXU4=', 'Content-Length': '0'} 2023-10-12T00:25:44.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bf2516293b18def-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:44.796 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.796 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.797 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.797 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.798 DEBUG:boto:Method: GET 2023-10-12T00:25:44.798 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.799 DEBUG:boto:Data: 2023-10-12T00:25:44.799 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.800 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.800 DEBUG:boto:Params: {} 2023-10-12T00:25:44.801 DEBUG:boto:Token: None 2023-10-12T00:25:44.801 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.802 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2e09980a37cebe8-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.804 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.805 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.805 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.806 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.806 DEBUG:boto:Method: GET 2023-10-12T00:25:44.807 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.807 DEBUG:boto:Data: 2023-10-12T00:25:44.808 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.808 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.809 DEBUG:boto:Params: {} 2023-10-12T00:25:44.809 DEBUG:boto:Token: None 2023-10-12T00:25:44.810 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.810 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d579c2a303cf1793-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.813 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.813 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.814 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.814 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.815 DEBUG:boto:Method: GET 2023-10-12T00:25:44.815 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.816 DEBUG:boto:Data: 2023-10-12T00:25:44.816 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.817 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.817 DEBUG:boto:Params: {} 2023-10-12T00:25:44.818 DEBUG:boto:Token: None 2023-10-12T00:25:44.818 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI= 2023-10-12T00:25:44.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI=', 'Content-Length': '0'} 2023-10-12T00:25:44.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b948507b9024cb9-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.862 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.863 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.864 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.864 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.864 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.865 DEBUG:boto:Method: GET 2023-10-12T00:25:44.865 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.866 DEBUG:boto:Data: 2023-10-12T00:25:44.866 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.867 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.868 DEBUG:boto:Params: {} 2023-10-12T00:25:44.868 DEBUG:boto:Token: None 2023-10-12T00:25:44.869 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI= 2023-10-12T00:25:44.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI=', 'Content-Length': '0'} 2023-10-12T00:25:44.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeed481efc827d2c-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.873 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.873 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.874 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.874 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.875 DEBUG:boto:Path: /wafmoa-12/dummy 2023-10-12T00:25:44.875 DEBUG:boto:Data: 2023-10-12T00:25:44.876 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.877 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.877 DEBUG:boto:Params: {} 2023-10-12T00:25:44.878 DEBUG:boto:Token: None 2023-10-12T00:25:44.878 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy 2023-10-12T00:25:44.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ptSW3URVeQBpi9HzNtKjQ+ZN7LM= 2023-10-12T00:25:44.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ptSW3URVeQBpi9HzNtKjQ+ZN7LM=', 'Content-Length': '0'} 2023-10-12T00:25:44.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f1d3218cf139761-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.882 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.882 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.883 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.883 DEBUG:boto:Path: /wafmoa-12/dummy 2023-10-12T00:25:44.884 DEBUG:boto:Data: 2023-10-12T00:25:44.884 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.885 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.885 DEBUG:boto:Params: {} 2023-10-12T00:25:44.886 DEBUG:boto:Token: None 2023-10-12T00:25:44.886 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy 2023-10-12T00:25:44.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ptSW3URVeQBpi9HzNtKjQ+ZN7LM= 2023-10-12T00:25:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ptSW3URVeQBpi9HzNtKjQ+ZN7LM=', 'Content-Length': '0'} 2023-10-12T00:25:44.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0003bda5eb592ea-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:25:44.890 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.891 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.891 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.892 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.892 DEBUG:boto:Method: GET 2023-10-12T00:25:44.893 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.893 DEBUG:boto:Data: 2023-10-12T00:25:44.894 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.895 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.895 DEBUG:boto:Params: {} 2023-10-12T00:25:44.896 DEBUG:boto:Token: None 2023-10-12T00:25:44.896 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000064529a92c4fb54fe-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.900 DEBUG:boto:path=/wafmoa-12/dummy 2023-10-12T00:25:44.901 DEBUG:boto:auth_path=/wafmoa-12/dummy 2023-10-12T00:25:44.901 DEBUG:boto:path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.902 DEBUG:boto:auth_path=/wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.902 DEBUG:boto:Method: GET 2023-10-12T00:25:44.903 DEBUG:boto:Path: /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.903 DEBUG:boto:Data: 2023-10-12T00:25:44.903 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.904 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.905 DEBUG:boto:Params: {} 2023-10-12T00:25:44.905 DEBUG:boto:Token: None 2023-10-12T00:25:44.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/dummy?versionId=1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ 2023-10-12T00:25:44.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8= 2023-10-12T00:25:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OrBjyFB5wDlWnQ2v/Pkf62K4HB8=', 'Content-Length': '0'} 2023-10-12T00:25:44.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:26 GMT'), ('x-amz-version-id', '1GdjGHCbC7lBipYQ.T1Vjg061mk8lpJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4b65837ebdc9cae-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.909 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.910 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.910 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.911 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.911 DEBUG:boto:Method: GET 2023-10-12T00:25:44.912 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.912 DEBUG:boto:Data: 2023-10-12T00:25:44.913 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.915 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.915 DEBUG:boto:Params: {} 2023-10-12T00:25:44.916 DEBUG:boto:Token: None 2023-10-12T00:25:44.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI= 2023-10-12T00:25:44.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI=', 'Content-Length': '0'} 2023-10-12T00:25:44.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ed08c84c063e58-0065273d08-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.921 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.922 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:44.923 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:44.924 DEBUG:boto:path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.924 DEBUG:boto:auth_path=/wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.925 DEBUG:boto:Method: GET 2023-10-12T00:25:44.925 DEBUG:boto:Path: /wafmoa-12/?prefix=dummy 2023-10-12T00:25:44.926 DEBUG:boto:Data: 2023-10-12T00:25:44.927 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.927 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.928 DEBUG:boto:Params: {} 2023-10-12T00:25:44.928 DEBUG:boto:Token: None 2023-10-12T00:25:44.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/ 2023-10-12T00:25:44.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI= 2023-10-12T00:25:44.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR3YaRfsfdYk2Pqq5Lx0NPNqBSI=', 'Content-Length': '0'} 2023-10-12T00:25:44.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7825d2cc1800f2-0065273d08-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.934 DEBUG:boto:b'wafmoa-12dummy1000falsedummy2023-10-12T00:24:26.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:44.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:44.935 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.936 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.936 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.937 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.938 DEBUG:boto:Method: GET 2023-10-12T00:25:44.938 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.939 DEBUG:boto:Data: 2023-10-12T00:25:44.940 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.941 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.942 DEBUG:boto:Params: {} 2023-10-12T00:25:44.942 DEBUG:boto:Token: None 2023-10-12T00:25:44.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY= 2023-10-12T00:25:44.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY=', 'Content-Length': '0'} 2023-10-12T00:25:44.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf3de160923aec75-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.948 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.949 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.949 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.950 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.951 DEBUG:boto:Method: GET 2023-10-12T00:25:44.951 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.952 DEBUG:boto:Data: 2023-10-12T00:25:44.953 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.954 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.955 DEBUG:boto:Params: {} 2023-10-12T00:25:44.955 DEBUG:boto:Token: None 2023-10-12T00:25:44.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY= 2023-10-12T00:25:44.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY=', 'Content-Length': '0'} 2023-10-12T00:25:44.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e992b77f1852c96e-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.960 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.961 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.961 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.962 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.962 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.963 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.964 DEBUG:boto:Data: 2023-10-12T00:25:44.964 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.966 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.966 DEBUG:boto:Params: {} 2023-10-12T00:25:44.967 DEBUG:boto:Token: None 2023-10-12T00:25:44.968 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p2zrnbwB2zVHh+VhkrHyV0FMUnU= 2023-10-12T00:25:44.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p2zrnbwB2zVHh+VhkrHyV0FMUnU=', 'Content-Length': '0'} 2023-10-12T00:25:44.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004eebd3537d50282f-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.972 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.972 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.973 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.974 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.974 DEBUG:boto:Method: HEAD 2023-10-12T00:25:44.975 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.975 DEBUG:boto:Data: 2023-10-12T00:25:44.976 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:44.977 DEBUG:boto:Port: 8000 2023-10-12T00:25:44.978 DEBUG:boto:Params: {} 2023-10-12T00:25:44.978 DEBUG:boto:Token: None 2023-10-12T00:25:44.979 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p2zrnbwB2zVHh+VhkrHyV0FMUnU= 2023-10-12T00:25:44.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p2zrnbwB2zVHh+VhkrHyV0FMUnU=', 'Content-Length': '0'} 2023-10-12T00:25:44.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f1821f19683fb9-0065273d08-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:44.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:44.984 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.984 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:44.985 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.986 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.986 DEBUG:boto:Method: GET 2023-10-12T00:25:44.987 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.988 DEBUG:boto:Data: 2023-10-12T00:25:44.989 DEBUG:boto:Headers: {} 2023-10-12T00:25:44.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:44.990 DEBUG:boto:Port: 8001 2023-10-12T00:25:44.991 DEBUG:boto:Params: {} 2023-10-12T00:25:44.992 DEBUG:boto:Token: None 2023-10-12T00:25:44.992 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:44 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:44.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY= 2023-10-12T00:25:44.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ef9MmH+fMJQ6QRvR5L7u4aFM0QY=', 'Content-Length': '0'} 2023-10-12T00:25:44.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffc6b1191c13165e-0065273d08-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.038 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.039 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.039 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.039 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.040 DEBUG:boto:Method: GET 2023-10-12T00:25:45.040 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.041 DEBUG:boto:Data: 2023-10-12T00:25:45.041 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.042 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.042 DEBUG:boto:Params: {} 2023-10-12T00:25:45.043 DEBUG:boto:Token: None 2023-10-12T00:25:45.043 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q= 2023-10-12T00:25:45.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q=', 'Content-Length': '0'} 2023-10-12T00:25:45.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6217c8b886ccef4-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.047 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.047 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.047 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.048 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.048 DEBUG:boto:Method: GET 2023-10-12T00:25:45.049 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.049 DEBUG:boto:Data: 2023-10-12T00:25:45.049 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.050 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.051 DEBUG:boto:Params: {} 2023-10-12T00:25:45.051 DEBUG:boto:Token: None 2023-10-12T00:25:45.052 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ca2f93f0cbe6a8-0065273d09-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.056 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.057 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.057 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.058 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.058 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.059 DEBUG:boto:Method: GET 2023-10-12T00:25:45.059 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.059 DEBUG:boto:Data: 2023-10-12T00:25:45.060 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.061 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.061 DEBUG:boto:Params: {} 2023-10-12T00:25:45.062 DEBUG:boto:Token: None 2023-10-12T00:25:45.062 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a80d36bcd6d7018d-0065273d09-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.066 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.066 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.067 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.067 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.068 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:25:45.068 DEBUG:boto:Data: 2023-10-12T00:25:45.069 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.069 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.070 DEBUG:boto:Params: {} 2023-10-12T00:25:45.070 DEBUG:boto:Token: None 2023-10-12T00:25:45.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a1 2023-10-12T00:25:45.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGKPz7yUM5MyHoWXLgH0wItJum4= 2023-10-12T00:25:45.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGKPz7yUM5MyHoWXLgH0wItJum4=', 'Content-Length': '0'} 2023-10-12T00:25:45.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002870faf201130cbb-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.075 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.075 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.076 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.076 DEBUG:boto:Path: /wafmoa-12/obj-a1 2023-10-12T00:25:45.077 DEBUG:boto:Data: 2023-10-12T00:25:45.078 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.079 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.079 DEBUG:boto:Params: {} 2023-10-12T00:25:45.080 DEBUG:boto:Token: None 2023-10-12T00:25:45.080 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a1 2023-10-12T00:25:45.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGKPz7yUM5MyHoWXLgH0wItJum4= 2023-10-12T00:25:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGKPz7yUM5MyHoWXLgH0wItJum4=', 'Content-Length': '0'} 2023-10-12T00:25:45.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aec6dd1448a025e4-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-12T00:25:45.084 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.085 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.085 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.085 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.086 DEBUG:boto:Method: GET 2023-10-12T00:25:45.086 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.087 DEBUG:boto:Data: 2023-10-12T00:25:45.087 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.088 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.088 DEBUG:boto:Params: {} 2023-10-12T00:25:45.089 DEBUG:boto:Token: None 2023-10-12T00:25:45.089 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q= 2023-10-12T00:25:45.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q=', 'Content-Length': '0'} 2023-10-12T00:25:45.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2320f08a02d7c85-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.093 DEBUG:boto:path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.093 DEBUG:boto:auth_path=/wafmoa-12/obj-a1 2023-10-12T00:25:45.093 DEBUG:boto:path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.094 DEBUG:boto:auth_path=/wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.094 DEBUG:boto:Method: GET 2023-10-12T00:25:45.095 DEBUG:boto:Path: /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.095 DEBUG:boto:Data: 2023-10-12T00:25:45.096 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.096 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.097 DEBUG:boto:Params: {} 2023-10-12T00:25:45.097 DEBUG:boto:Token: None 2023-10-12T00:25:45.098 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a1?versionId=xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7 2023-10-12T00:25:45.098 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q= 2023-10-12T00:25:45.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BBHwo93jlRt7mqfdN+otrChtZ0Q=', 'Content-Length': '0'} 2023-10-12T00:25:45.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'xuE1xbN9f0hQQgd6rIeZVbTOvS8SQG7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012b5546bb29dbe69-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.101 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.102 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.102 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.103 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.103 DEBUG:boto:Method: GET 2023-10-12T00:25:45.104 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.104 DEBUG:boto:Data: 2023-10-12T00:25:45.104 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.105 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.106 DEBUG:boto:Params: {} 2023-10-12T00:25:45.106 DEBUG:boto:Token: None 2023-10-12T00:25:45.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001355c9d67d1570fc-0065273d09-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.111 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.111 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.112 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.112 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.113 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.113 DEBUG:boto:Method: GET 2023-10-12T00:25:45.113 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a1 2023-10-12T00:25:45.114 DEBUG:boto:Data: 2023-10-12T00:25:45.114 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.115 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.116 DEBUG:boto:Params: {} 2023-10-12T00:25:45.116 DEBUG:boto:Token: None 2023-10-12T00:25:45.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af544e705284ff0d-0065273d09-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.121 DEBUG:boto:b'wafmoa-12obj-a11000falseobj-a12023-10-12T00:24:46.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:45.122 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.122 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.122 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.123 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.123 DEBUG:boto:Method: GET 2023-10-12T00:25:45.124 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.124 DEBUG:boto:Data: 2023-10-12T00:25:45.124 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.125 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.126 DEBUG:boto:Params: {} 2023-10-12T00:25:45.126 DEBUG:boto:Token: None 2023-10-12T00:25:45.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb48a548cfae456-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.131 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.131 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.132 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.132 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.132 DEBUG:boto:Method: GET 2023-10-12T00:25:45.133 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.133 DEBUG:boto:Data: 2023-10-12T00:25:45.134 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.135 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.135 DEBUG:boto:Params: {} 2023-10-12T00:25:45.136 DEBUG:boto:Token: None 2023-10-12T00:25:45.136 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc883f6137d50bda-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.139 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.140 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.140 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.141 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.141 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.141 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.142 DEBUG:boto:Data: 2023-10-12T00:25:45.142 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.143 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.143 DEBUG:boto:Params: {} 2023-10-12T00:25:45.144 DEBUG:boto:Token: None 2023-10-12T00:25:45.144 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t5Xp6n6hYZVFdihPe5LNSkpG3r8= 2023-10-12T00:25:45.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t5Xp6n6hYZVFdihPe5LNSkpG3r8=', 'Content-Length': '0'} 2023-10-12T00:25:45.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c793a4f23e29cdf-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.148 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.148 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.149 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.149 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.150 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.150 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.151 DEBUG:boto:Data: 2023-10-12T00:25:45.151 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.152 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.152 DEBUG:boto:Params: {} 2023-10-12T00:25:45.152 DEBUG:boto:Token: None 2023-10-12T00:25:45.153 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t5Xp6n6hYZVFdihPe5LNSkpG3r8= 2023-10-12T00:25:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t5Xp6n6hYZVFdihPe5LNSkpG3r8=', 'Content-Length': '0'} 2023-10-12T00:25:45.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038c979e0039b984b-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:45.157 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.157 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.158 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.158 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.159 DEBUG:boto:Method: GET 2023-10-12T00:25:45.159 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.159 DEBUG:boto:Data: 2023-10-12T00:25:45.160 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.161 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.161 DEBUG:boto:Params: {} 2023-10-12T00:25:45.161 DEBUG:boto:Token: None 2023-10-12T00:25:45.162 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fd10ba92b657dc8-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.210 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.211 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.212 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.212 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.212 DEBUG:boto:Method: GET 2023-10-12T00:25:45.213 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.213 DEBUG:boto:Data: 2023-10-12T00:25:45.214 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.215 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.215 DEBUG:boto:Params: {} 2023-10-12T00:25:45.216 DEBUG:boto:Token: None 2023-10-12T00:25:45.216 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000110b28e8a3291fd6-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.219 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.220 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.220 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.221 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.221 DEBUG:boto:Method: GET 2023-10-12T00:25:45.222 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.222 DEBUG:boto:Data: 2023-10-12T00:25:45.223 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.223 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.224 DEBUG:boto:Params: {} 2023-10-12T00:25:45.224 DEBUG:boto:Token: None 2023-10-12T00:25:45.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6db8be5b410f1d5-0065273d09-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.228 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.229 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.229 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.230 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.230 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.231 DEBUG:boto:Method: GET 2023-10-12T00:25:45.231 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.232 DEBUG:boto:Data: 2023-10-12T00:25:45.232 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.233 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.233 DEBUG:boto:Params: {} 2023-10-12T00:25:45.234 DEBUG:boto:Token: None 2023-10-12T00:25:45.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9def71832faeda-0065273d09-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.238 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.239 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.239 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.239 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.240 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:25:45.240 DEBUG:boto:Data: 2023-10-12T00:25:45.241 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.241 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.242 DEBUG:boto:Params: {} 2023-10-12T00:25:45.242 DEBUG:boto:Token: None 2023-10-12T00:25:45.243 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2 2023-10-12T00:25:45.243 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFQNVAPyk8GBs/EGvEla6IeRdig= 2023-10-12T00:25:45.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFQNVAPyk8GBs/EGvEla6IeRdig=', 'Content-Length': '0'} 2023-10-12T00:25:45.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049985eaabe7deede-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.247 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.247 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.248 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.248 DEBUG:boto:Path: /wafmoa-12/obj-a2 2023-10-12T00:25:45.249 DEBUG:boto:Data: 2023-10-12T00:25:45.249 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.250 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.250 DEBUG:boto:Params: {} 2023-10-12T00:25:45.251 DEBUG:boto:Token: None 2023-10-12T00:25:45.251 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2 2023-10-12T00:25:45.251 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFQNVAPyk8GBs/EGvEla6IeRdig= 2023-10-12T00:25:45.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFQNVAPyk8GBs/EGvEla6IeRdig=', 'Content-Length': '0'} 2023-10-12T00:25:45.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4f130fe39ce3273-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-12T00:25:45.255 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.255 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.256 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.256 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.257 DEBUG:boto:Method: GET 2023-10-12T00:25:45.257 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.257 DEBUG:boto:Data: 2023-10-12T00:25:45.258 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.259 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.259 DEBUG:boto:Params: {} 2023-10-12T00:25:45.260 DEBUG:boto:Token: None 2023-10-12T00:25:45.260 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8bf4c3e2b5384cb-0065273d09-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.263 DEBUG:boto:path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.264 DEBUG:boto:auth_path=/wafmoa-12/obj-a2 2023-10-12T00:25:45.264 DEBUG:boto:path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.265 DEBUG:boto:auth_path=/wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.265 DEBUG:boto:Method: GET 2023-10-12T00:25:45.266 DEBUG:boto:Path: /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.266 DEBUG:boto:Data: 2023-10-12T00:25:45.267 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.267 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.268 DEBUG:boto:Params: {} 2023-10-12T00:25:45.268 DEBUG:boto:Token: None 2023-10-12T00:25:45.269 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/obj-a2?versionId=K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT 2023-10-12T00:25:45.269 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY= 2023-10-12T00:25:45.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zJCkHUK9MVrhpKSMc01gqgGQKwY=', 'Content-Length': '0'} 2023-10-12T00:25:45.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:24:46 GMT'), ('x-amz-version-id', 'K16BzI0q1qGkqpfB4TyZSGkYxAF1pCT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcb044357dde8ad2-0065273d09-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.272 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.273 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.273 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.273 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.274 DEBUG:boto:Method: GET 2023-10-12T00:25:45.274 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.275 DEBUG:boto:Data: 2023-10-12T00:25:45.275 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:25:45.276 DEBUG:boto:Port: 8001 2023-10-12T00:25:45.276 DEBUG:boto:Params: {} 2023-10-12T00:25:45.277 DEBUG:boto:Token: None 2023-10-12T00:25:45.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de6bc4137084fc1-0065273d09-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.322 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.323 DEBUG:boto:path=/wafmoa-12/ 2023-10-12T00:25:45.324 DEBUG:boto:auth_path=/wafmoa-12/ 2023-10-12T00:25:45.324 DEBUG:boto:path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.324 DEBUG:boto:auth_path=/wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.325 DEBUG:boto:Method: GET 2023-10-12T00:25:45.325 DEBUG:boto:Path: /wafmoa-12/?prefix=obj-a2 2023-10-12T00:25:45.326 DEBUG:boto:Data: 2023-10-12T00:25:45.326 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.327 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.327 DEBUG:boto:Params: {} 2023-10-12T00:25:45.328 DEBUG:boto:Token: None 2023-10-12T00:25:45.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-12/ 2023-10-12T00:25:45.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8= 2023-10-12T00:25:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g8StgaKZdYtmFZlJIVjXtTwS/C8=', 'Content-Length': '0'} 2023-10-12T00:25:45.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123f8b15ca8acaba-0065273d09-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.332 DEBUG:boto:b'wafmoa-12obj-a21000falseobj-a22023-10-12T00:24:46.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:25:45.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-12 zones={a2, a1} 2023-10-12T00:25:45.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-10-12T00:25:45.335 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:25:45.335 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:25:45.336 DEBUG:boto:Method: PUT 2023-10-12T00:25:45.336 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:25:45.337 DEBUG:boto:Data: 2023-10-12T00:25:45.337 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.338 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.338 DEBUG:boto:Params: {} 2023-10-12T00:25:45.339 DEBUG:boto:Token: None 2023-10-12T00:25:45.339 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-13/ 2023-10-12T00:25:45.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H6TDfPXMjsVqNgXiFpL+tkbxfCI= 2023-10-12T00:25:45.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H6TDfPXMjsVqNgXiFpL+tkbxfCI=', 'Content-Length': '0'} 2023-10-12T00:25:45.355 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b82c0e03c7eae342-0065273d09-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.356 DEBUG:rgw_multi.tests:created bucket=wafmoa-13 2023-10-12T00:25:45.356 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:25:45.357 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:25:45.357 DEBUG:boto:path=/wafmoa-13/?versioning 2023-10-12T00:25:45.358 DEBUG:boto:auth_path=/wafmoa-13/?versioning 2023-10-12T00:25:45.358 DEBUG:boto:Method: PUT 2023-10-12T00:25:45.358 DEBUG:boto:Path: /wafmoa-13/?versioning 2023-10-12T00:25:45.359 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:25:45.359 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.360 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.360 DEBUG:boto:Params: {} 2023-10-12T00:25:45.361 DEBUG:boto:Token: None 2023-10-12T00:25:45.361 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-13/?versioning 2023-10-12T00:25:45.362 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fLDpgKhEaJbpjT3XuNbq6YDpQQ8= 2023-10-12T00:25:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fLDpgKhEaJbpjT3XuNbq6YDpQQ8=', 'Content-Length': '228'} 2023-10-12T00:25:45.370 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce279734d9d53845-0065273d09-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:25:45.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:25:45.371 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:25:45.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:45.425 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:45.452 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:25:45.463 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:25:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:25:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:25:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:25:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:25:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070345.370186_111.1", 2023-10-12T00:25:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:25:45.370186Z" 2023-10-12T00:25:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:25:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:25:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.467 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:25:45.468 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070345.370186_111.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:25:45.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:25:45.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:25:45.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:25:45.470 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:25:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:45.520 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:45.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:45.546+0000 7ffaedf25780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:25:45.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:45.547+0000 7ffaedf25780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:25:45.559 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:25:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:25:45.559 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:25:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:25:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:25:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:25:45.560 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:25:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:25:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:25:45.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:25:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:25:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:25:45.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:25:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:25:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:25:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:25:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:25:45.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:25:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:25:45.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:25:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:25:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:25:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:25:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:25:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:25:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:25:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:25:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:25:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:25:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070345.370186_111.1", 2023-10-12T00:25:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:25:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:25:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:25:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:25:45.370186Z", 2023-10-12T00:25:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:25:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:25:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:25:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:25:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:25:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:25:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:25:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:25:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:25:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:25:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:25:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:25:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:25:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:25:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:25:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:25:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:25:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:25:45Z" 2023-10-12T00:25:45.574 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:25:45.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070345.370186_111.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:25:45.370186Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:25:45Z" } 2023-10-12T00:25:45.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070345.370186_111.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:25:45.370186Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:25:45.576 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070345.370186_111.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:25:45.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070345.370186_111.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:25:45.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:25:45.577 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:25:45.577 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:25:45.578 DEBUG:boto:Method: HEAD 2023-10-12T00:25:45.578 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:25:45.579 DEBUG:boto:Data: 2023-10-12T00:25:45.579 DEBUG:boto:Headers: {} 2023-10-12T00:25:45.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.580 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.580 DEBUG:boto:Params: {} 2023-10-12T00:25:45.581 DEBUG:boto:Token: None 2023-10-12T00:25:45.581 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-13/ 2023-10-12T00:25:45.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uaBkzQoWgnGZcAAe8uo46hFOrb0= 2023-10-12T00:25:45.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uaBkzQoWgnGZcAAe8uo46hFOrb0=', 'Content-Length': '0'} 2023-10-12T00:25:45.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000920d922ba9bae46d-0065273d09-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.584 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:25:45.585 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:25:45.585 DEBUG:boto:Method: PUT 2023-10-12T00:25:45.586 DEBUG:boto:Path: /wafmoa-13/dummy 2023-10-12T00:25:45.586 DEBUG:boto:Data: 2023-10-12T00:25:45.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:25:45.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:25:45.587 DEBUG:boto:Port: 8000 2023-10-12T00:25:45.588 DEBUG:boto:Params: {} 2023-10-12T00:25:45.588 DEBUG:boto:Token: None 2023-10-12T00:25:45.589 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:25:45 GMT /wafmoa-13/dummy 2023-10-12T00:25:45.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TNretqfSNxyHd5mntfzsctMuynE= 2023-10-12T00:25:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:25:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TNretqfSNxyHd5mntfzsctMuynE='} 2023-10-12T00:25:45.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-amz-request-id', 'tx0000078585d298138ee2e-0065273d09-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:25:45.602 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-13 2023-10-12T00:25:45.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:45.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:25:45.603 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:25:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:25:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:25:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:45.691+0000 7f697691b780 1 waiting to reach incremental sync.. 2023-10-12T00:25:50.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:50.692+0000 7f697691b780 1 waiting to reach incremental sync.. 2023-10-12T00:25:55.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:25:55.692+0000 7f697691b780 1 waiting to reach incremental sync.. 2023-10-12T00:26:00.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:00.693+0000 7f697691b780 1 waiting to reach incremental sync.. 2023-10-12T00:26:05.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:05.695+0000 7f697691b780 1 bucket sync caught up with source: 2023-10-12T00:26:05.696 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:05.696 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:05.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:05.695+0000 7f697691b780 0 bucket checkpoint complete 2023-10-12T00:26:05.710 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-13 2023-10-12T00:26:05.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:05.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:05.712 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:26:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:26:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:26:05.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:05.809+0000 7f84c6e68780 1 waiting to reach incremental sync.. 2023-10-12T00:26:10.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:10.809+0000 7f84c6e68780 1 waiting to reach incremental sync.. 2023-10-12T00:26:15.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:15.810+0000 7f84c6e68780 1 waiting to reach incremental sync.. 2023-10-12T00:26:20.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:20.809+0000 7f84c6e68780 1 waiting to reach incremental sync.. 2023-10-12T00:26:25.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:25.811+0000 7f84c6e68780 1 bucket sync caught up with source: 2023-10-12T00:26:25.812 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , , , ] 2023-10-12T00:26:25.812 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , , , ] 2023-10-12T00:26:25.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:25.811+0000 7f84c6e68780 0 bucket checkpoint complete 2023-10-12T00:26:25.828 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-13 zones={a2, a1} 2023-10-12T00:26:25.828 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.829 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.829 DEBUG:boto:Method: HEAD 2023-10-12T00:26:25.830 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:25.830 DEBUG:boto:Data: 2023-10-12T00:26:25.831 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.831 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.832 DEBUG:boto:Params: {} 2023-10-12T00:26:25.832 DEBUG:boto:Token: None 2023-10-12T00:26:25.833 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/ 2023-10-12T00:26:25.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rc5DaAkc5KWlYT62B4sSIOgaJFU= 2023-10-12T00:26:25.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rc5DaAkc5KWlYT62B4sSIOgaJFU=', 'Content-Length': '0'} 2023-10-12T00:26:25.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd049a2f3e3e536d-0065273d31-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.837 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.838 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.838 DEBUG:boto:Method: HEAD 2023-10-12T00:26:25.838 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:25.839 DEBUG:boto:Data: 2023-10-12T00:26:25.839 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.840 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.840 DEBUG:boto:Params: {} 2023-10-12T00:26:25.841 DEBUG:boto:Token: None 2023-10-12T00:26:25.842 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/ 2023-10-12T00:26:25.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rc5DaAkc5KWlYT62B4sSIOgaJFU= 2023-10-12T00:26:25.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rc5DaAkc5KWlYT62B4sSIOgaJFU=', 'Content-Length': '0'} 2023-10-12T00:26:25.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc4bb205a9fe5ca9-0065273d31-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.846 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:26:25.846 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.847 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.847 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.848 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.848 DEBUG:boto:Method: GET 2023-10-12T00:26:25.848 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.849 DEBUG:boto:Data: 2023-10-12T00:26:25.849 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.850 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.851 DEBUG:boto:Params: {} 2023-10-12T00:26:25.851 DEBUG:boto:Token: None 2023-10-12T00:26:25.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/?versions 2023-10-12T00:26:25.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U= 2023-10-12T00:26:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U=', 'Content-Length': '0'} 2023-10-12T00:26:25.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff16178d0a740f31-0065273d31-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.855 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:25.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:26:25.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:26:25.857 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.858 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.858 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.859 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.859 DEBUG:boto:Method: GET 2023-10-12T00:26:25.859 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.860 DEBUG:boto:Data: 2023-10-12T00:26:25.860 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.861 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.862 DEBUG:boto:Params: {} 2023-10-12T00:26:25.862 DEBUG:boto:Token: None 2023-10-12T00:26:25.863 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/?versions 2023-10-12T00:26:25.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U= 2023-10-12T00:26:25.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U=', 'Content-Length': '0'} 2023-10-12T00:26:25.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac094ad07bfb0f6c-0065273d31-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.867 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:25.867 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:26:25.868 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.868 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.869 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.869 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.870 DEBUG:boto:Method: GET 2023-10-12T00:26:25.870 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.871 DEBUG:boto:Data: 2023-10-12T00:26:25.871 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.872 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.872 DEBUG:boto:Params: {} 2023-10-12T00:26:25.873 DEBUG:boto:Token: None 2023-10-12T00:26:25.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/?versions 2023-10-12T00:26:25.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U= 2023-10-12T00:26:25.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U=', 'Content-Length': '0'} 2023-10-12T00:26:25.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043616ffb56791b43-0065273d31-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.878 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:25.878 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.879 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.879 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.880 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.880 DEBUG:boto:Method: GET 2023-10-12T00:26:25.881 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:26:25.881 DEBUG:boto:Data: 2023-10-12T00:26:25.881 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.882 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.883 DEBUG:boto:Params: {} 2023-10-12T00:26:25.883 DEBUG:boto:Token: None 2023-10-12T00:26:25.884 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/?versions 2023-10-12T00:26:25.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U= 2023-10-12T00:26:25.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WByppebDuVf77ZH5uzucvDani9U=', 'Content-Length': '0'} 2023-10-12T00:26:25.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e36e03cb0cf1f4-0065273d31-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.934 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:25.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:26:25.936 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.936 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.937 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.937 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.938 DEBUG:boto:Method: GET 2023-10-12T00:26:25.938 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.938 DEBUG:boto:Data: 2023-10-12T00:26:25.939 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.940 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.940 DEBUG:boto:Params: {} 2023-10-12T00:26:25.941 DEBUG:boto:Token: None 2023-10-12T00:26:25.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44= 2023-10-12T00:26:25.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44=', 'Content-Length': '0'} 2023-10-12T00:26:25.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049a7b52e9c327d16-0065273d31-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.945 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.946 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.946 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.947 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.947 DEBUG:boto:Method: GET 2023-10-12T00:26:25.948 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.948 DEBUG:boto:Data: 2023-10-12T00:26:25.948 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.949 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.950 DEBUG:boto:Params: {} 2023-10-12T00:26:25.950 DEBUG:boto:Token: None 2023-10-12T00:26:25.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44= 2023-10-12T00:26:25.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44=', 'Content-Length': '0'} 2023-10-12T00:26:25.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000426113dadd3ba2e-0065273d31-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.954 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.955 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.955 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.955 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.956 DEBUG:boto:Method: HEAD 2023-10-12T00:26:25.956 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.957 DEBUG:boto:Data: 2023-10-12T00:26:25.957 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.958 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.958 DEBUG:boto:Params: {} 2023-10-12T00:26:25.959 DEBUG:boto:Token: None 2023-10-12T00:26:25.959 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mTWIovM6d2CIdyUdD8Rl9Y9MadI= 2023-10-12T00:26:25.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mTWIovM6d2CIdyUdD8Rl9Y9MadI=', 'Content-Length': '0'} 2023-10-12T00:26:25.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b784cd5a0182619-0065273d31-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.963 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.963 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.964 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.964 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.964 DEBUG:boto:Method: HEAD 2023-10-12T00:26:25.965 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.965 DEBUG:boto:Data: 2023-10-12T00:26:25.966 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.967 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.967 DEBUG:boto:Params: {} 2023-10-12T00:26:25.967 DEBUG:boto:Token: None 2023-10-12T00:26:25.968 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mTWIovM6d2CIdyUdD8Rl9Y9MadI= 2023-10-12T00:26:25.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mTWIovM6d2CIdyUdD8Rl9Y9MadI=', 'Content-Length': '0'} 2023-10-12T00:26:25.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc05a5913d658e6-0065273d31-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:26:25.972 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.972 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.973 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.973 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.973 DEBUG:boto:Method: GET 2023-10-12T00:26:25.974 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.974 DEBUG:boto:Data: 2023-10-12T00:26:25.975 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.976 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.976 DEBUG:boto:Params: {} 2023-10-12T00:26:25.976 DEBUG:boto:Token: None 2023-10-12T00:26:25.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44= 2023-10-12T00:26:25.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44=', 'Content-Length': '0'} 2023-10-12T00:26:25.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073e981d57ef1875a-0065273d31-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.980 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:25.981 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:25.981 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.982 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.982 DEBUG:boto:Method: GET 2023-10-12T00:26:25.983 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.983 DEBUG:boto:Data: 2023-10-12T00:26:25.983 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:25.984 DEBUG:boto:Port: 8000 2023-10-12T00:26:25.985 DEBUG:boto:Params: {} 2023-10-12T00:26:25.985 DEBUG:boto:Token: None 2023-10-12T00:26:25.986 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:25.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44= 2023-10-12T00:26:25.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YQMyBVcz3EDNFQJPB+v0RsbCb44=', 'Content-Length': '0'} 2023-10-12T00:26:25.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000607c84111b948b20-0065273d31-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.989 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:25.990 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:25.990 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:25.991 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:25.991 DEBUG:boto:Method: GET 2023-10-12T00:26:25.992 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:26:25.992 DEBUG:boto:Data: 2023-10-12T00:26:25.993 DEBUG:boto:Headers: {} 2023-10-12T00:26:25.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:25.993 DEBUG:boto:Port: 8001 2023-10-12T00:26:25.994 DEBUG:boto:Params: {} 2023-10-12T00:26:25.994 DEBUG:boto:Token: None 2023-10-12T00:26:25.995 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:25 GMT /wafmoa-13/ 2023-10-12T00:26:25.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6BljtPXwSimRqmhwx6QCDML3AA= 2023-10-12T00:26:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6BljtPXwSimRqmhwx6QCDML3AA=', 'Content-Length': '0'} 2023-10-12T00:26:25.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006486a85bf7156df3-0065273d31-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:25.999 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:26.000 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.000 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.001 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.001 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.002 DEBUG:boto:Method: GET 2023-10-12T00:26:26.002 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.002 DEBUG:boto:Data: 2023-10-12T00:26:26.003 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.004 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.004 DEBUG:boto:Params: {} 2023-10-12T00:26:26.005 DEBUG:boto:Token: None 2023-10-12T00:26:26.005 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0= 2023-10-12T00:26:26.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0=', 'Content-Length': '0'} 2023-10-12T00:26:26.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e786273e92e0132f-0065273d32-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.009 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:26.010 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:26.010 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:26.011 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.011 DEBUG:boto:Path: /wafmoa-13/dummy 2023-10-12T00:26:26.012 DEBUG:boto:Data: 2023-10-12T00:26:26.012 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.013 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.013 DEBUG:boto:Params: {} 2023-10-12T00:26:26.014 DEBUG:boto:Token: None 2023-10-12T00:26:26.014 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/dummy 2023-10-12T00:26:26.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PXN7lQIGzoZ0IF2XKZSCUXwFkqg= 2023-10-12T00:26:26.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PXN7lQIGzoZ0IF2XKZSCUXwFkqg=', 'Content-Length': '0'} 2023-10-12T00:26:26.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000624da55c26118dab-0065273d32-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.018 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:26.019 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:26.019 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.020 DEBUG:boto:Path: /wafmoa-13/dummy 2023-10-12T00:26:26.020 DEBUG:boto:Data: 2023-10-12T00:26:26.020 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.021 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.022 DEBUG:boto:Params: {} 2023-10-12T00:26:26.022 DEBUG:boto:Token: None 2023-10-12T00:26:26.022 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/dummy 2023-10-12T00:26:26.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PXN7lQIGzoZ0IF2XKZSCUXwFkqg= 2023-10-12T00:26:26.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PXN7lQIGzoZ0IF2XKZSCUXwFkqg=', 'Content-Length': '0'} 2023-10-12T00:26:26.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000530fe195a3081182-0065273d32-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:26:26.027 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:26.027 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:26.027 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.028 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.028 DEBUG:boto:Method: GET 2023-10-12T00:26:26.029 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.029 DEBUG:boto:Data: 2023-10-12T00:26:26.030 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.031 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.031 DEBUG:boto:Params: {} 2023-10-12T00:26:26.032 DEBUG:boto:Token: None 2023-10-12T00:26:26.032 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euHv2oyfOYjVIanQOcrmUeqNFkM= 2023-10-12T00:26:26.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euHv2oyfOYjVIanQOcrmUeqNFkM=', 'Content-Length': '0'} 2023-10-12T00:26:26.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1ce853ff3e5bc9b-0065273d32-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.036 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:26:26.036 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:26:26.037 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.037 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.037 DEBUG:boto:Method: GET 2023-10-12T00:26:26.038 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.038 DEBUG:boto:Data: 2023-10-12T00:26:26.039 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.040 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.040 DEBUG:boto:Params: {} 2023-10-12T00:26:26.041 DEBUG:boto:Token: None 2023-10-12T00:26:26.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:26:26.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euHv2oyfOYjVIanQOcrmUeqNFkM= 2023-10-12T00:26:26.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euHv2oyfOYjVIanQOcrmUeqNFkM=', 'Content-Length': '0'} 2023-10-12T00:26:26.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b6aa4e93f38eed5-0065273d32-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.045 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.045 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.046 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.046 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.047 DEBUG:boto:Method: GET 2023-10-12T00:26:26.047 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.047 DEBUG:boto:Data: 2023-10-12T00:26:26.048 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.049 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.049 DEBUG:boto:Params: {} 2023-10-12T00:26:26.050 DEBUG:boto:Token: None 2023-10-12T00:26:26.050 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0= 2023-10-12T00:26:26.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0=', 'Content-Length': '0'} 2023-10-12T00:26:26.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000027dfd2eb0fa3f25-0065273d32-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.094 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:26.095 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.095 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.096 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.096 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.097 DEBUG:boto:Method: GET 2023-10-12T00:26:26.097 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:26:26.098 DEBUG:boto:Data: 2023-10-12T00:26:26.098 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.099 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.100 DEBUG:boto:Params: {} 2023-10-12T00:26:26.100 DEBUG:boto:Token: None 2023-10-12T00:26:26.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0= 2023-10-12T00:26:26.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Eg09rIizh6T2mgLf/1U7oR31tw0=', 'Content-Length': '0'} 2023-10-12T00:26:26.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df2b8eeb4cbd47fa-0065273d32-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.104 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:26:26.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-13 zones={a2, a1} 2023-10-12T00:26:26.105 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.106 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.106 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.107 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.107 DEBUG:boto:Data: 2023-10-12T00:26:26.107 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.108 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.109 DEBUG:boto:Params: {} 2023-10-12T00:26:26.109 DEBUG:boto:Token: None 2023-10-12T00:26:26.109 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000147df4d660b2e02c-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.113 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.114 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.114 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.115 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.115 DEBUG:boto:Data: 2023-10-12T00:26:26.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.116 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.117 DEBUG:boto:Params: {} 2023-10-12T00:26:26.117 DEBUG:boto:Token: None 2023-10-12T00:26:26.117 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.118 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-amz-request-id', 'tx0000022d80b022015db9d-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.163 DEBUG:rgw_multi.tests:zone=a1 version=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:26:26.163 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.164 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.164 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.165 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.165 DEBUG:boto:Data: 2023-10-12T00:26:26.165 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.166 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.167 DEBUG:boto:Params: {} 2023-10-12T00:26:26.167 DEBUG:boto:Token: None 2023-10-12T00:26:26.167 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbac368ddfb64355-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.172 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.172 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.173 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.173 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.173 DEBUG:boto:Data: 2023-10-12T00:26:26.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.174 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.175 DEBUG:boto:Params: {} 2023-10-12T00:26:26.175 DEBUG:boto:Token: None 2023-10-12T00:26:26.176 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.189 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-amz-request-id', 'tx000001f05c76becb84ab0-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.190 DEBUG:rgw_multi.tests:zone=a2 version=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:26:26.190 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.191 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.191 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.191 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.192 DEBUG:boto:Data: 2023-10-12T00:26:26.192 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.193 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.193 DEBUG:boto:Params: {} 2023-10-12T00:26:26.194 DEBUG:boto:Token: None 2023-10-12T00:26:26.194 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5c2107df88c99c2-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.198 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.198 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.199 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.199 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.200 DEBUG:boto:Data: 2023-10-12T00:26:26.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.201 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.201 DEBUG:boto:Params: {} 2023-10-12T00:26:26.202 DEBUG:boto:Token: None 2023-10-12T00:26:26.202 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-amz-request-id', 'tx00000d7ff80a9aeb59d30-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.217 DEBUG:rgw_multi.tests:zone=a1 version=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:26:26.218 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.218 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.218 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.219 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.219 DEBUG:boto:Data: 2023-10-12T00:26:26.220 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.220 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.221 DEBUG:boto:Params: {} 2023-10-12T00:26:26.221 DEBUG:boto:Token: None 2023-10-12T00:26:26.222 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095981b5613f74dd8-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.225 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.226 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.226 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.227 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.227 DEBUG:boto:Data: 2023-10-12T00:26:26.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.228 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.229 DEBUG:boto:Params: {} 2023-10-12T00:26:26.229 DEBUG:boto:Token: None 2023-10-12T00:26:26.230 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.230 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-amz-request-id', 'tx000000a9891a9a2a42403-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.241 DEBUG:rgw_multi.tests:zone=a2 version=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:26:26.241 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.241 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.242 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.242 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.243 DEBUG:boto:Data: 2023-10-12T00:26:26.243 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.244 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.244 DEBUG:boto:Params: {} 2023-10-12T00:26:26.245 DEBUG:boto:Token: None 2023-10-12T00:26:26.245 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d3e2ce384f32cf8-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.249 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.249 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.250 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.250 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.251 DEBUG:boto:Data: 2023-10-12T00:26:26.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.252 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.252 DEBUG:boto:Params: {} 2023-10-12T00:26:26.253 DEBUG:boto:Token: None 2023-10-12T00:26:26.253 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-amz-request-id', 'tx000004e156862069ba904-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.263 DEBUG:rgw_multi.tests:zone=a1 version=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:26:26.263 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.264 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.264 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.265 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.265 DEBUG:boto:Data: 2023-10-12T00:26:26.266 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.267 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.267 DEBUG:boto:Params: {} 2023-10-12T00:26:26.267 DEBUG:boto:Token: None 2023-10-12T00:26:26.268 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005abad807d6cc73-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.272 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.272 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.272 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.273 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.273 DEBUG:boto:Data: 2023-10-12T00:26:26.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.274 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.275 DEBUG:boto:Params: {} 2023-10-12T00:26:26.275 DEBUG:boto:Token: None 2023-10-12T00:26:26.276 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-amz-request-id', 'tx0000030bab3319faad616-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.319 DEBUG:rgw_multi.tests:zone=a2 version=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:26:26.319 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.320 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.320 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.321 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.321 DEBUG:boto:Data: 2023-10-12T00:26:26.321 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.322 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.322 DEBUG:boto:Params: {} 2023-10-12T00:26:26.323 DEBUG:boto:Token: None 2023-10-12T00:26:26.323 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f30d32f602e8688d-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.327 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.327 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.328 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.328 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.329 DEBUG:boto:Data: 2023-10-12T00:26:26.329 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.330 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.330 DEBUG:boto:Params: {} 2023-10-12T00:26:26.331 DEBUG:boto:Token: None 2023-10-12T00:26:26.331 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-amz-request-id', 'tx00000c6a93e9207f42877-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.375 DEBUG:rgw_multi.tests:zone=a1 version=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:26:26.375 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.376 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.376 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.377 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.377 DEBUG:boto:Data: 2023-10-12T00:26:26.377 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.378 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.379 DEBUG:boto:Params: {} 2023-10-12T00:26:26.379 DEBUG:boto:Token: None 2023-10-12T00:26:26.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e6adc39f996be35-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.383 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.384 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.384 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.385 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.385 DEBUG:boto:Data: 2023-10-12T00:26:26.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.387 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.387 DEBUG:boto:Params: {} 2023-10-12T00:26:26.387 DEBUG:boto:Token: None 2023-10-12T00:26:26.388 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-amz-request-id', 'tx00000377bcc3b634c37e8-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.398 DEBUG:rgw_multi.tests:zone=a2 version=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:26:26.398 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.399 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.399 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.399 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.400 DEBUG:boto:Data: 2023-10-12T00:26:26.400 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.401 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.401 DEBUG:boto:Params: {} 2023-10-12T00:26:26.402 DEBUG:boto:Token: None 2023-10-12T00:26:26.402 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033a4ce1b431c5eb8-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.406 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.406 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.407 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.407 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.408 DEBUG:boto:Data: 2023-10-12T00:26:26.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.409 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.409 DEBUG:boto:Params: {} 2023-10-12T00:26:26.409 DEBUG:boto:Token: None 2023-10-12T00:26:26.410 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-amz-request-id', 'tx000003fb77b0ecf0303fd-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.420 DEBUG:rgw_multi.tests:zone=a1 version=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:26:26.420 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.421 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.421 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.422 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.422 DEBUG:boto:Data: 2023-10-12T00:26:26.423 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.423 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.424 DEBUG:boto:Params: {} 2023-10-12T00:26:26.424 DEBUG:boto:Token: None 2023-10-12T00:26:26.424 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000de6ed69304100b-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.428 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.429 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.429 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.430 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.430 DEBUG:boto:Data: 2023-10-12T00:26:26.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.431 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.432 DEBUG:boto:Params: {} 2023-10-12T00:26:26.432 DEBUG:boto:Token: None 2023-10-12T00:26:26.433 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-amz-request-id', 'tx00000b2c1483f93da2566-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.443 DEBUG:rgw_multi.tests:zone=a2 version=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:26:26.443 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.444 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.444 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.444 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.445 DEBUG:boto:Data: 2023-10-12T00:26:26.445 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.446 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.447 DEBUG:boto:Params: {} 2023-10-12T00:26:26.447 DEBUG:boto:Token: None 2023-10-12T00:26:26.447 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc3650837310bbc-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.451 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.451 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.452 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.452 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.452 DEBUG:boto:Data: 2023-10-12T00:26:26.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.454 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.454 DEBUG:boto:Params: {} 2023-10-12T00:26:26.454 DEBUG:boto:Token: None 2023-10-12T00:26:26.455 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-amz-request-id', 'tx00000eb84d4d38d51835a-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.499 DEBUG:rgw_multi.tests:zone=a1 version=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:26:26.499 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.500 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.500 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.500 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.501 DEBUG:boto:Data: 2023-10-12T00:26:26.501 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.502 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.502 DEBUG:boto:Params: {} 2023-10-12T00:26:26.503 DEBUG:boto:Token: None 2023-10-12T00:26:26.503 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008262e292ca88c721-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.507 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.507 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.508 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.508 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.509 DEBUG:boto:Data: 2023-10-12T00:26:26.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.510 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.510 DEBUG:boto:Params: {} 2023-10-12T00:26:26.511 DEBUG:boto:Token: None 2023-10-12T00:26:26.511 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.511 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-amz-request-id', 'tx000001d976aca896ade09-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.555 DEBUG:rgw_multi.tests:zone=a2 version=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:26:26.555 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.556 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.556 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.556 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.557 DEBUG:boto:Data: 2023-10-12T00:26:26.557 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.558 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.558 DEBUG:boto:Params: {} 2023-10-12T00:26:26.559 DEBUG:boto:Token: None 2023-10-12T00:26:26.559 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.559 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c189ba836c4449df-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.563 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.563 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.563 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.564 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.564 DEBUG:boto:Data: 2023-10-12T00:26:26.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.566 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.566 DEBUG:boto:Params: {} 2023-10-12T00:26:26.566 DEBUG:boto:Token: None 2023-10-12T00:26:26.567 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-amz-request-id', 'tx000006b4b073c6865c349-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.578 DEBUG:rgw_multi.tests:zone=a1 version=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:26:26.578 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.578 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.579 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.579 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.580 DEBUG:boto:Data: 2023-10-12T00:26:26.580 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.581 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.581 DEBUG:boto:Params: {} 2023-10-12T00:26:26.581 DEBUG:boto:Token: None 2023-10-12T00:26:26.582 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.582 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c13f01eb1d9afd-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.586 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.586 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.587 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.587 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.587 DEBUG:boto:Data: 2023-10-12T00:26:26.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.589 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.589 DEBUG:boto:Params: {} 2023-10-12T00:26:26.589 DEBUG:boto:Token: None 2023-10-12T00:26:26.590 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.590 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-amz-request-id', 'tx000003b41d67fd1cc4114-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.600 DEBUG:rgw_multi.tests:zone=a2 version=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:26:26.600 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.601 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.601 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.601 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.602 DEBUG:boto:Data: 2023-10-12T00:26:26.602 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.603 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.603 DEBUG:boto:Params: {} 2023-10-12T00:26:26.604 DEBUG:boto:Token: None 2023-10-12T00:26:26.604 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004457fb6247131d06-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.608 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.608 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.609 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.609 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.609 DEBUG:boto:Data: 2023-10-12T00:26:26.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.610 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.611 DEBUG:boto:Params: {} 2023-10-12T00:26:26.611 DEBUG:boto:Token: None 2023-10-12T00:26:26.612 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-amz-request-id', 'tx000005491f7db240813a5-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.623 DEBUG:rgw_multi.tests:zone=a1 version=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:26:26.623 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.624 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.624 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.625 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.625 DEBUG:boto:Data: 2023-10-12T00:26:26.625 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.626 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.627 DEBUG:boto:Params: {} 2023-10-12T00:26:26.627 DEBUG:boto:Token: None 2023-10-12T00:26:26.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049d83073794a6e3a-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.631 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.631 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.632 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.632 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.632 DEBUG:boto:Data: 2023-10-12T00:26:26.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.633 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.634 DEBUG:boto:Params: {} 2023-10-12T00:26:26.634 DEBUG:boto:Token: None 2023-10-12T00:26:26.635 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-amz-request-id', 'tx00000e88f40c2078fbec2-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.679 DEBUG:rgw_multi.tests:zone=a2 version=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:26:26.679 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.680 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.680 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.680 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.681 DEBUG:boto:Data: 2023-10-12T00:26:26.681 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.682 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.682 DEBUG:boto:Params: {} 2023-10-12T00:26:26.683 DEBUG:boto:Token: None 2023-10-12T00:26:26.683 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000337d1a0ee67fed72-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.687 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.687 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.688 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.688 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.689 DEBUG:boto:Data: 2023-10-12T00:26:26.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.690 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.690 DEBUG:boto:Params: {} 2023-10-12T00:26:26.691 DEBUG:boto:Token: None 2023-10-12T00:26:26.691 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-amz-request-id', 'tx00000c8e9273e627db223-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.736 DEBUG:rgw_multi.tests:zone=a1 version=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:26:26.736 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.737 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.737 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.737 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.738 DEBUG:boto:Data: 2023-10-12T00:26:26.738 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.739 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.739 DEBUG:boto:Params: {} 2023-10-12T00:26:26.740 DEBUG:boto:Token: None 2023-10-12T00:26:26.740 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088bf7d56847fd650-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.744 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.744 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.745 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.745 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.746 DEBUG:boto:Data: 2023-10-12T00:26:26.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.747 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.747 DEBUG:boto:Params: {} 2023-10-12T00:26:26.748 DEBUG:boto:Token: None 2023-10-12T00:26:26.748 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-amz-request-id', 'tx000007ea1a15f504e4e3d-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.757 DEBUG:rgw_multi.tests:zone=a2 version=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:26:26.758 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.758 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.758 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.759 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.759 DEBUG:boto:Data: 2023-10-12T00:26:26.760 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.760 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.761 DEBUG:boto:Params: {} 2023-10-12T00:26:26.761 DEBUG:boto:Token: None 2023-10-12T00:26:26.761 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ce47234c07c8ecb-0065273d32-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.765 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.765 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.766 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.766 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.767 DEBUG:boto:Data: 2023-10-12T00:26:26.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:26:26.768 DEBUG:boto:Port: 8000 2023-10-12T00:26:26.768 DEBUG:boto:Params: {} 2023-10-12T00:26:26.769 DEBUG:boto:Token: None 2023-10-12T00:26:26.769 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-amz-request-id', 'tx000003176e64aa4376711-0065273d32-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.778 DEBUG:rgw_multi.tests:zone=a1 version=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:26:26.779 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:26:26.779 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:26:26.779 DEBUG:boto:Method: HEAD 2023-10-12T00:26:26.780 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:26:26.780 DEBUG:boto:Data: 2023-10-12T00:26:26.780 DEBUG:boto:Headers: {} 2023-10-12T00:26:26.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.781 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.782 DEBUG:boto:Params: {} 2023-10-12T00:26:26.782 DEBUG:boto:Token: None 2023-10-12T00:26:26.782 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/ 2023-10-12T00:26:26.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo= 2023-10-12T00:26:26.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYRi7k1Se7VomzlrCf052HVMfSo=', 'Content-Length': '0'} 2023-10-12T00:26:26.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de0b6778056421d-0065273d32-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.786 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:26:26.786 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:26:26.787 DEBUG:boto:Method: PUT 2023-10-12T00:26:26.787 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:26:26.788 DEBUG:boto:Data: 2023-10-12T00:26:26.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-12T00:26:26.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:26:26.789 DEBUG:boto:Port: 8001 2023-10-12T00:26:26.789 DEBUG:boto:Params: {} 2023-10-12T00:26:26.790 DEBUG:boto:Token: None 2023-10-12T00:26:26.790 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Thu, 12 Oct 2023 00:26:26 GMT /wafmoa-13/obj 2023-10-12T00:26:26.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs= 2023-10-12T00:26:26.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:26:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNyw1J/yCvRE/Uy1fLf7Bmtl7Qs='} 2023-10-12T00:26:26.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-amz-request-id', 'tx00000e065926d574dd2d0-0065273d32-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:26:26.799 DEBUG:rgw_multi.tests:zone=a2 version=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:26:26.800 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-13 2023-10-12T00:26:26.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:26.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:26.801 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:26:26.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:26:26.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:26:26.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:26.890+0000 7fb8986dc780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:26.891 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:26.891 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000020.46.12, , ] 2023-10-12T00:26:31.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:31.892+0000 7fb8986dc780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:31.893 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:31.893 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000020.46.12, , ] 2023-10-12T00:26:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:36.893+0000 7fb8986dc780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:36.894 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:36.894 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000020.46.12, , ] 2023-10-12T00:26:41.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:41.895+0000 7fb8986dc780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:41.895 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2023-10-12T00:26:41.895 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000020.46.12, , ] 2023-10-12T00:26:46.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:46.896+0000 7fb8986dc780 1 bucket sync caught up with source: 2023-10-12T00:26:46.896 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.12.10, , , 00000000028.70.6, , ] 2023-10-12T00:26:46.897 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000020.46.12, , ] 2023-10-12T00:26:46.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:46.896+0000 7fb8986dc780 0 bucket checkpoint complete 2023-10-12T00:26:46.911 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-13 2023-10-12T00:26:46.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:46.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:26:46.913 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:26:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:26:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:26:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:47.012+0000 7feb963d5780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:47.012 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , 00000000029.107.6, , ] 2023-10-12T00:26:47.012 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:26:52.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:52.013+0000 7feb963d5780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:52.013 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , 00000000029.107.6, , ] 2023-10-12T00:26:52.014 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:26:57.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:26:57.014+0000 7feb963d5780 1 waiting for incremental sync to catch up: 2023-10-12T00:26:57.016 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , 00000000029.107.6, , ] 2023-10-12T00:26:57.016 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:27:02.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:02.015+0000 7feb963d5780 1 waiting for incremental sync to catch up: 2023-10-12T00:27:02.016 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , 00000000029.107.6, , ] 2023-10-12T00:27:02.016 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:27:07.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:07.016+0000 7feb963d5780 1 bucket sync caught up with source: 2023-10-12T00:27:07.017 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:27:07.017 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.44.11, , , 00000000031.111.9, , ] 2023-10-12T00:27:07.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:07.016+0000 7feb963d5780 0 bucket checkpoint complete 2023-10-12T00:27:07.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-13 zones={a2, a1} 2023-10-12T00:27:07.034 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.034 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.035 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.035 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:27:07.035 DEBUG:boto:Data: 2023-10-12T00:27:07.036 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.037 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.037 DEBUG:boto:Params: {} 2023-10-12T00:27:07.038 DEBUG:boto:Token: None 2023-10-12T00:27:07.038 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/DH2rDzmd9BgMfyxi3LFzZH2y/U= 2023-10-12T00:27:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/DH2rDzmd9BgMfyxi3LFzZH2y/U=', 'Content-Length': '0'} 2023-10-12T00:27:07.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056327d3bc0af465b-0065273d5b-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.042 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.043 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.043 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.044 DEBUG:boto:Path: /wafmoa-13/ 2023-10-12T00:27:07.044 DEBUG:boto:Data: 2023-10-12T00:27:07.044 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.045 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.045 DEBUG:boto:Params: {} 2023-10-12T00:27:07.046 DEBUG:boto:Token: None 2023-10-12T00:27:07.046 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/DH2rDzmd9BgMfyxi3LFzZH2y/U= 2023-10-12T00:27:07.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/DH2rDzmd9BgMfyxi3LFzZH2y/U=', 'Content-Length': '0'} 2023-10-12T00:27:07.049 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0fe5abbd1f73115-0065273d5b-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.050 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:27:07.050 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.051 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.051 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.051 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.052 DEBUG:boto:Method: GET 2023-10-12T00:27:07.052 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.053 DEBUG:boto:Data: 2023-10-12T00:27:07.053 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.054 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.054 DEBUG:boto:Params: {} 2023-10-12T00:27:07.055 DEBUG:boto:Token: None 2023-10-12T00:27:07.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/?versions 2023-10-12T00:27:07.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY= 2023-10-12T00:27:07.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY=', 'Content-Length': '0'} 2023-10-12T00:27:07.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ece98751fbea0e7-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.059 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcCtrue2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9false2023-10-12T00:26:26.772Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKzSVGht51uCwIbejDUKpHTrvMCcWbAVfalse2023-10-12T00:26:26.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjih9Ogh.lZQJwApzn2a1cuUYUks7.GhSfalse2023-10-12T00:26:26.695Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYGWhHG0Ib6FCdA6at2MMDfIDcgO2wxhfalse2023-10-12T00:26:26.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqOdYv7tgNZreFYF8opEwj3jAqY-OBoCfalse2023-10-12T00:26:26.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.false2023-10-12T00:26:26.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqdRaE86GREIsJ3Eg774dLKCsX96HV3Gfalse2023-10-12T00:26:26.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqpfalse2023-10-12T00:26:26.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaREAq6CYlwscobe6hxBhqtoXwgZf5Iafalse2023-10-12T00:26:26.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8YF.vOWyk5Y0FN1lc5uqCulDrladR24false2023-10-12T00:26:26.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcW5Ye2CI-TLxfd9wO06BlCDsesBA4Pdfalse2023-10-12T00:26:26.413Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAwL75t2G8eZE-9BVIOBGa7w0CBFJ56Efalse2023-10-12T00:26:26.391Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUKcEH2brpi8Ybi0biABxFN1eyql3Efgfalse2023-10-12T00:26:26.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoXfalse2023-10-12T00:26:26.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbcLIwmkxUK6LImzWGZigFgqMDTZIkF3false2023-10-12T00:26:26.279Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaeNQEx4UDrOBog-IThXBKYcMJlbKvu3false2023-10-12T00:26:26.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl1NxRoVSzbfCB0g0y82beIxhCWs2wv3false2023-10-12T00:26:26.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA1GhQw.uNyVkATOZA23KGcLCah.1lK7false2023-10-12T00:26:26.179Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx3qNoJF7sAKv4z2L03tHhlOnXeMpWhqfalse2023-10-12T00:26:26.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.061 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:27:07.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.068 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:27:07.069 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.069 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.069 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.070 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.070 DEBUG:boto:Method: GET 2023-10-12T00:27:07.071 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.071 DEBUG:boto:Data: 2023-10-12T00:27:07.071 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.072 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.073 DEBUG:boto:Params: {} 2023-10-12T00:27:07.073 DEBUG:boto:Token: None 2023-10-12T00:27:07.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/?versions 2023-10-12T00:27:07.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY= 2023-10-12T00:27:07.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY=', 'Content-Length': '0'} 2023-10-12T00:27:07.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c01720835ea080-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.077 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcCtrue2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9false2023-10-12T00:26:26.772Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKzSVGht51uCwIbejDUKpHTrvMCcWbAVfalse2023-10-12T00:26:26.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjih9Ogh.lZQJwApzn2a1cuUYUks7.GhSfalse2023-10-12T00:26:26.695Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYGWhHG0Ib6FCdA6at2MMDfIDcgO2wxhfalse2023-10-12T00:26:26.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqOdYv7tgNZreFYF8opEwj3jAqY-OBoCfalse2023-10-12T00:26:26.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.false2023-10-12T00:26:26.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqdRaE86GREIsJ3Eg774dLKCsX96HV3Gfalse2023-10-12T00:26:26.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqpfalse2023-10-12T00:26:26.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaREAq6CYlwscobe6hxBhqtoXwgZf5Iafalse2023-10-12T00:26:26.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8YF.vOWyk5Y0FN1lc5uqCulDrladR24false2023-10-12T00:26:26.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcW5Ye2CI-TLxfd9wO06BlCDsesBA4Pdfalse2023-10-12T00:26:26.413Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAwL75t2G8eZE-9BVIOBGa7w0CBFJ56Efalse2023-10-12T00:26:26.391Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUKcEH2brpi8Ybi0biABxFN1eyql3Efgfalse2023-10-12T00:26:26.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoXfalse2023-10-12T00:26:26.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbcLIwmkxUK6LImzWGZigFgqMDTZIkF3false2023-10-12T00:26:26.279Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaeNQEx4UDrOBog-IThXBKYcMJlbKvu3false2023-10-12T00:26:26.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl1NxRoVSzbfCB0g0y82beIxhCWs2wv3false2023-10-12T00:26:26.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA1GhQw.uNyVkATOZA23KGcLCah.1lK7false2023-10-12T00:26:26.179Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx3qNoJF7sAKv4z2L03tHhlOnXeMpWhqfalse2023-10-12T00:26:26.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.079 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T00:27:07.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:07.086 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.086 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.087 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.087 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.087 DEBUG:boto:Method: GET 2023-10-12T00:27:07.088 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.088 DEBUG:boto:Data: 2023-10-12T00:27:07.089 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.089 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.090 DEBUG:boto:Params: {} 2023-10-12T00:27:07.090 DEBUG:boto:Token: None 2023-10-12T00:27:07.091 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/?versions 2023-10-12T00:27:07.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY= 2023-10-12T00:27:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY=', 'Content-Length': '0'} 2023-10-12T00:27:07.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c77e490f4c8e35-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.094 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcCtrue2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9false2023-10-12T00:26:26.772Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKzSVGht51uCwIbejDUKpHTrvMCcWbAVfalse2023-10-12T00:26:26.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjih9Ogh.lZQJwApzn2a1cuUYUks7.GhSfalse2023-10-12T00:26:26.695Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYGWhHG0Ib6FCdA6at2MMDfIDcgO2wxhfalse2023-10-12T00:26:26.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqOdYv7tgNZreFYF8opEwj3jAqY-OBoCfalse2023-10-12T00:26:26.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.false2023-10-12T00:26:26.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqdRaE86GREIsJ3Eg774dLKCsX96HV3Gfalse2023-10-12T00:26:26.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqpfalse2023-10-12T00:26:26.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaREAq6CYlwscobe6hxBhqtoXwgZf5Iafalse2023-10-12T00:26:26.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8YF.vOWyk5Y0FN1lc5uqCulDrladR24false2023-10-12T00:26:26.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcW5Ye2CI-TLxfd9wO06BlCDsesBA4Pdfalse2023-10-12T00:26:26.413Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAwL75t2G8eZE-9BVIOBGa7w0CBFJ56Efalse2023-10-12T00:26:26.391Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUKcEH2brpi8Ybi0biABxFN1eyql3Efgfalse2023-10-12T00:26:26.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoXfalse2023-10-12T00:26:26.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbcLIwmkxUK6LImzWGZigFgqMDTZIkF3false2023-10-12T00:26:26.279Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaeNQEx4UDrOBog-IThXBKYcMJlbKvu3false2023-10-12T00:26:26.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl1NxRoVSzbfCB0g0y82beIxhCWs2wv3false2023-10-12T00:26:26.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA1GhQw.uNyVkATOZA23KGcLCah.1lK7false2023-10-12T00:26:26.179Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx3qNoJF7sAKv4z2L03tHhlOnXeMpWhqfalse2023-10-12T00:26:26.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.096 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.096 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.097 DEBUG:boto:path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.097 DEBUG:boto:auth_path=/wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.097 DEBUG:boto:Method: GET 2023-10-12T00:27:07.098 DEBUG:boto:Path: /wafmoa-13/?versions&max-keys=999 2023-10-12T00:27:07.098 DEBUG:boto:Data: 2023-10-12T00:27:07.098 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.099 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.100 DEBUG:boto:Params: {} 2023-10-12T00:27:07.100 DEBUG:boto:Token: None 2023-10-12T00:27:07.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/?versions 2023-10-12T00:27:07.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY= 2023-10-12T00:27:07.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RrHcUXrjfWrdO9tcPJqF+2n48DY=', 'Content-Length': '0'} 2023-10-12T00:27:07.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb4fb0f76c9a9b5d-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.105 DEBUG:boto:b'wafmoa-13999falsedummyVELXn-JFmtILsDuZBAM5MxEQ-hyl6Ivtrue2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcCtrue2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9false2023-10-12T00:26:26.772Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKzSVGht51uCwIbejDUKpHTrvMCcWbAVfalse2023-10-12T00:26:26.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjih9Ogh.lZQJwApzn2a1cuUYUks7.GhSfalse2023-10-12T00:26:26.695Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYGWhHG0Ib6FCdA6at2MMDfIDcgO2wxhfalse2023-10-12T00:26:26.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqOdYv7tgNZreFYF8opEwj3jAqY-OBoCfalse2023-10-12T00:26:26.615Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjq262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.false2023-10-12T00:26:26.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqdRaE86GREIsJ3Eg774dLKCsX96HV3Gfalse2023-10-12T00:26:26.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqpfalse2023-10-12T00:26:26.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaREAq6CYlwscobe6hxBhqtoXwgZf5Iafalse2023-10-12T00:26:26.514Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8YF.vOWyk5Y0FN1lc5uqCulDrladR24false2023-10-12T00:26:26.436Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcW5Ye2CI-TLxfd9wO06BlCDsesBA4Pdfalse2023-10-12T00:26:26.413Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAwL75t2G8eZE-9BVIOBGa7w0CBFJ56Efalse2023-10-12T00:26:26.391Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUKcEH2brpi8Ybi0biABxFN1eyql3Efgfalse2023-10-12T00:26:26.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoXfalse2023-10-12T00:26:26.256Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbcLIwmkxUK6LImzWGZigFgqMDTZIkF3false2023-10-12T00:26:26.279Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaeNQEx4UDrOBog-IThXBKYcMJlbKvu3false2023-10-12T00:26:26.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl1NxRoVSzbfCB0g0y82beIxhCWs2wv3false2023-10-12T00:26:26.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjA1GhQw.uNyVkATOZA23KGcLCah.1lK7false2023-10-12T00:26:26.179Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjx3qNoJF7sAKv4z2L03tHhlOnXeMpWhqfalse2023-10-12T00:26:26.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:27:07.107 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.107 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.107 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.108 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.108 DEBUG:boto:Method: GET 2023-10-12T00:27:07.109 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.109 DEBUG:boto:Data: 2023-10-12T00:27:07.109 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.110 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.111 DEBUG:boto:Params: {} 2023-10-12T00:27:07.111 DEBUG:boto:Token: None 2023-10-12T00:27:07.111 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086c49a518fcea5d6-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.115 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.115 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.116 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.116 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.116 DEBUG:boto:Method: GET 2023-10-12T00:27:07.117 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.117 DEBUG:boto:Data: 2023-10-12T00:27:07.118 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.118 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.119 DEBUG:boto:Params: {} 2023-10-12T00:27:07.119 DEBUG:boto:Token: None 2023-10-12T00:27:07.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c15191001eca40c8-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.123 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.123 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.124 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.124 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.124 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.125 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.125 DEBUG:boto:Data: 2023-10-12T00:27:07.125 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.126 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.127 DEBUG:boto:Params: {} 2023-10-12T00:27:07.127 DEBUG:boto:Token: None 2023-10-12T00:27:07.127 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y/FWM72nyfrUCPtRPk8KjTFjMJg= 2023-10-12T00:27:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y/FWM72nyfrUCPtRPk8KjTFjMJg=', 'Content-Length': '0'} 2023-10-12T00:27:07.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8b76ca3c1a01e11-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.131 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.131 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.132 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.132 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.132 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.133 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.133 DEBUG:boto:Data: 2023-10-12T00:27:07.133 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.134 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.134 DEBUG:boto:Params: {} 2023-10-12T00:27:07.135 DEBUG:boto:Token: None 2023-10-12T00:27:07.135 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y/FWM72nyfrUCPtRPk8KjTFjMJg= 2023-10-12T00:27:07.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y/FWM72nyfrUCPtRPk8KjTFjMJg=', 'Content-Length': '0'} 2023-10-12T00:27:07.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003be17829afaafadc-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:27:07.139 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.139 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.140 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.140 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.140 DEBUG:boto:Method: GET 2023-10-12T00:27:07.141 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.141 DEBUG:boto:Data: 2023-10-12T00:27:07.142 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.142 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.143 DEBUG:boto:Params: {} 2023-10-12T00:27:07.143 DEBUG:boto:Token: None 2023-10-12T00:27:07.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000927f332e6ff7f8c0-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.147 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.147 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.148 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.148 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.148 DEBUG:boto:Method: GET 2023-10-12T00:27:07.149 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.149 DEBUG:boto:Data: 2023-10-12T00:27:07.150 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.150 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.151 DEBUG:boto:Params: {} 2023-10-12T00:27:07.151 DEBUG:boto:Token: None 2023-10-12T00:27:07.152 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a078b1e62613bda-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.155 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.156 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.156 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.157 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.157 DEBUG:boto:Method: GET 2023-10-12T00:27:07.157 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.158 DEBUG:boto:Data: 2023-10-12T00:27:07.158 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.159 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.159 DEBUG:boto:Params: {} 2023-10-12T00:27:07.160 DEBUG:boto:Token: None 2023-10-12T00:27:07.160 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd56fa41bd37f313-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.206 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.207 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.207 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.208 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.208 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.209 DEBUG:boto:Method: GET 2023-10-12T00:27:07.209 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.209 DEBUG:boto:Data: 2023-10-12T00:27:07.210 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.211 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.211 DEBUG:boto:Params: {} 2023-10-12T00:27:07.212 DEBUG:boto:Token: None 2023-10-12T00:27:07.212 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009abbf9b2eb1ff5f-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.216 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.217 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.217 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.218 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.218 DEBUG:boto:Path: /wafmoa-13/dummy 2023-10-12T00:27:07.218 DEBUG:boto:Data: 2023-10-12T00:27:07.219 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.220 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.220 DEBUG:boto:Params: {} 2023-10-12T00:27:07.220 DEBUG:boto:Token: None 2023-10-12T00:27:07.221 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy 2023-10-12T00:27:07.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6k6PCM+Urtyc9n6Er6dF2QHq2wg= 2023-10-12T00:27:07.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6k6PCM+Urtyc9n6Er6dF2QHq2wg=', 'Content-Length': '0'} 2023-10-12T00:27:07.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba23ea9eaf73fb15-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.225 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.225 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.225 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.226 DEBUG:boto:Path: /wafmoa-13/dummy 2023-10-12T00:27:07.226 DEBUG:boto:Data: 2023-10-12T00:27:07.227 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.228 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.228 DEBUG:boto:Params: {} 2023-10-12T00:27:07.228 DEBUG:boto:Token: None 2023-10-12T00:27:07.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy 2023-10-12T00:27:07.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6k6PCM+Urtyc9n6Er6dF2QHq2wg= 2023-10-12T00:27:07.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6k6PCM+Urtyc9n6Er6dF2QHq2wg=', 'Content-Length': '0'} 2023-10-12T00:27:07.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b189fe73ef1bacd4-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T00:27:07.233 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.233 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.234 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.234 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.235 DEBUG:boto:Method: GET 2023-10-12T00:27:07.235 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.236 DEBUG:boto:Data: 2023-10-12T00:27:07.236 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.237 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.237 DEBUG:boto:Params: {} 2023-10-12T00:27:07.238 DEBUG:boto:Token: None 2023-10-12T00:27:07.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdd722cb7701013d-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.242 DEBUG:boto:path=/wafmoa-13/dummy 2023-10-12T00:27:07.243 DEBUG:boto:auth_path=/wafmoa-13/dummy 2023-10-12T00:27:07.243 DEBUG:boto:path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.243 DEBUG:boto:auth_path=/wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.244 DEBUG:boto:Method: GET 2023-10-12T00:27:07.244 DEBUG:boto:Path: /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.245 DEBUG:boto:Data: 2023-10-12T00:27:07.245 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.246 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.246 DEBUG:boto:Params: {} 2023-10-12T00:27:07.247 DEBUG:boto:Token: None 2023-10-12T00:27:07.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/dummy?versionId=VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv 2023-10-12T00:27:07.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4= 2023-10-12T00:27:07.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pyHRwnW7gktQ00IUfP1X50gHfc4=', 'Content-Length': '0'} 2023-10-12T00:27:07.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:25:45 GMT'), ('x-amz-version-id', 'VELXn-JFmtILsDuZBAM5MxEQ-hyl6Iv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c517fde7bfe03612-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.251 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.251 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.251 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.252 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.252 DEBUG:boto:Method: GET 2023-10-12T00:27:07.253 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.253 DEBUG:boto:Data: 2023-10-12T00:27:07.253 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.254 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.255 DEBUG:boto:Params: {} 2023-10-12T00:27:07.255 DEBUG:boto:Token: None 2023-10-12T00:27:07.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdae189feaf0d498-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.259 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.260 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.261 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.261 DEBUG:boto:path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.261 DEBUG:boto:auth_path=/wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.262 DEBUG:boto:Method: GET 2023-10-12T00:27:07.262 DEBUG:boto:Path: /wafmoa-13/?prefix=dummy 2023-10-12T00:27:07.263 DEBUG:boto:Data: 2023-10-12T00:27:07.263 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.264 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.264 DEBUG:boto:Params: {} 2023-10-12T00:27:07.265 DEBUG:boto:Token: None 2023-10-12T00:27:07.265 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac8228b4789d98b-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.269 DEBUG:boto:b'wafmoa-13dummy1000falsedummy2023-10-12T00:25:45.592Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.270 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.271 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.271 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.272 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.272 DEBUG:boto:Method: GET 2023-10-12T00:27:07.273 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.273 DEBUG:boto:Data: 2023-10-12T00:27:07.274 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.274 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.275 DEBUG:boto:Params: {} 2023-10-12T00:27:07.275 DEBUG:boto:Token: None 2023-10-12T00:27:07.276 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a3d0b4692874a3-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.280 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.280 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.280 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.281 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.281 DEBUG:boto:Method: GET 2023-10-12T00:27:07.282 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.282 DEBUG:boto:Data: 2023-10-12T00:27:07.283 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.283 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.284 DEBUG:boto:Params: {} 2023-10-12T00:27:07.284 DEBUG:boto:Token: None 2023-10-12T00:27:07.285 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa4ea77e9a78062-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.288 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.289 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.289 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.289 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.290 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.290 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.291 DEBUG:boto:Data: 2023-10-12T00:27:07.291 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.292 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.292 DEBUG:boto:Params: {} 2023-10-12T00:27:07.292 DEBUG:boto:Token: None 2023-10-12T00:27:07.293 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K1BXIOeRs8eY9OPuwNqHo0TN/kk= 2023-10-12T00:27:07.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K1BXIOeRs8eY9OPuwNqHo0TN/kk=', 'Content-Length': '0'} 2023-10-12T00:27:07.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000567a30978630f197-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.296 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.296 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.297 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.297 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.298 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.298 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.298 DEBUG:boto:Data: 2023-10-12T00:27:07.299 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.299 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.300 DEBUG:boto:Params: {} 2023-10-12T00:27:07.300 DEBUG:boto:Token: None 2023-10-12T00:27:07.301 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K1BXIOeRs8eY9OPuwNqHo0TN/kk= 2023-10-12T00:27:07.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K1BXIOeRs8eY9OPuwNqHo0TN/kk=', 'Content-Length': '0'} 2023-10-12T00:27:07.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f468a8123494d5-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.304 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.305 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.305 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.305 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.306 DEBUG:boto:Method: GET 2023-10-12T00:27:07.306 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.307 DEBUG:boto:Data: 2023-10-12T00:27:07.307 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.308 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.308 DEBUG:boto:Params: {} 2023-10-12T00:27:07.309 DEBUG:boto:Token: None 2023-10-12T00:27:07.309 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.310 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1811109db0ad4bf-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.354 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.355 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.355 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.355 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.356 DEBUG:boto:Method: GET 2023-10-12T00:27:07.356 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.357 DEBUG:boto:Data: 2023-10-12T00:27:07.357 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.358 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.358 DEBUG:boto:Params: {} 2023-10-12T00:27:07.359 DEBUG:boto:Token: None 2023-10-12T00:27:07.359 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fb108d920b5f8a8-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.362 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.363 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.363 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.364 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.364 DEBUG:boto:Method: GET 2023-10-12T00:27:07.364 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.365 DEBUG:boto:Data: 2023-10-12T00:27:07.365 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.366 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.366 DEBUG:boto:Params: {} 2023-10-12T00:27:07.367 DEBUG:boto:Token: None 2023-10-12T00:27:07.367 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000047ce9c737bcaec6-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.371 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.372 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.372 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.372 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.373 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.373 DEBUG:boto:Method: GET 2023-10-12T00:27:07.374 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.374 DEBUG:boto:Data: 2023-10-12T00:27:07.374 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.375 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.376 DEBUG:boto:Params: {} 2023-10-12T00:27:07.376 DEBUG:boto:Token: None 2023-10-12T00:27:07.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d924c5d695f521a-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.380 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.381 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.381 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.382 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.382 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.382 DEBUG:boto:Data: 2023-10-12T00:27:07.383 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.384 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.384 DEBUG:boto:Params: {} 2023-10-12T00:27:07.384 DEBUG:boto:Token: None 2023-10-12T00:27:07.385 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026bcfb00e712bc52-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.389 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.389 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.390 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.390 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.391 DEBUG:boto:Data: 2023-10-12T00:27:07.391 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.392 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.392 DEBUG:boto:Params: {} 2023-10-12T00:27:07.392 DEBUG:boto:Token: None 2023-10-12T00:27:07.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4fd7e72893b770-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.397 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.398 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.398 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.399 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.399 DEBUG:boto:Method: GET 2023-10-12T00:27:07.399 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.400 DEBUG:boto:Data: 2023-10-12T00:27:07.400 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.401 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.401 DEBUG:boto:Params: {} 2023-10-12T00:27:07.402 DEBUG:boto:Token: None 2023-10-12T00:27:07.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094e9f295c2c545a6-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.405 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.406 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.406 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.407 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.407 DEBUG:boto:Method: GET 2023-10-12T00:27:07.408 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.408 DEBUG:boto:Data: 2023-10-12T00:27:07.408 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.409 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.409 DEBUG:boto:Params: {} 2023-10-12T00:27:07.410 DEBUG:boto:Token: None 2023-10-12T00:27:07.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aec3478e72135b24-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.413 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.414 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.414 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.415 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.415 DEBUG:boto:Method: GET 2023-10-12T00:27:07.416 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.416 DEBUG:boto:Data: 2023-10-12T00:27:07.416 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.417 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.418 DEBUG:boto:Params: {} 2023-10-12T00:27:07.418 DEBUG:boto:Token: None 2023-10-12T00:27:07.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b64c6f2dcf235c4-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.422 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.423 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.423 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.423 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.424 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.424 DEBUG:boto:Method: GET 2023-10-12T00:27:07.425 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.425 DEBUG:boto:Data: 2023-10-12T00:27:07.425 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.426 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.427 DEBUG:boto:Params: {} 2023-10-12T00:27:07.427 DEBUG:boto:Token: None 2023-10-12T00:27:07.428 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d88e28f75ac921-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.431 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.432 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.433 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.433 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.434 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.434 DEBUG:boto:Method: GET 2023-10-12T00:27:07.434 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.435 DEBUG:boto:Data: 2023-10-12T00:27:07.435 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.436 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.436 DEBUG:boto:Params: {} 2023-10-12T00:27:07.437 DEBUG:boto:Token: None 2023-10-12T00:27:07.437 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc= 2023-10-12T00:27:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc=', 'Content-Length': '0'} 2023-10-12T00:27:07.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029bfe6184437790e-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.441 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.441 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.441 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.442 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.442 DEBUG:boto:Method: GET 2023-10-12T00:27:07.443 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.443 DEBUG:boto:Data: 2023-10-12T00:27:07.444 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.444 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.445 DEBUG:boto:Params: {} 2023-10-12T00:27:07.445 DEBUG:boto:Token: None 2023-10-12T00:27:07.445 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc= 2023-10-12T00:27:07.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc=', 'Content-Length': '0'} 2023-10-12T00:27:07.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cf421c68bdc1f4a-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.449 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.450 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.450 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.451 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.451 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.451 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.452 DEBUG:boto:Data: 2023-10-12T00:27:07.452 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.453 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.453 DEBUG:boto:Params: {} 2023-10-12T00:27:07.454 DEBUG:boto:Token: None 2023-10-12T00:27:07.454 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SbCEzUQFvnfO3c8NkIO/oIQX8Qc= 2023-10-12T00:27:07.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SbCEzUQFvnfO3c8NkIO/oIQX8Qc=', 'Content-Length': '0'} 2023-10-12T00:27:07.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008abb2a0c60a0e6f5-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.457 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.458 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.458 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.459 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.459 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.459 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.460 DEBUG:boto:Data: 2023-10-12T00:27:07.460 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.461 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.461 DEBUG:boto:Params: {} 2023-10-12T00:27:07.461 DEBUG:boto:Token: None 2023-10-12T00:27:07.462 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.462 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SbCEzUQFvnfO3c8NkIO/oIQX8Qc= 2023-10-12T00:27:07.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SbCEzUQFvnfO3c8NkIO/oIQX8Qc=', 'Content-Length': '0'} 2023-10-12T00:27:07.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d2bf2552bb795da-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.466 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.466 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.466 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.467 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.467 DEBUG:boto:Method: GET 2023-10-12T00:27:07.468 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.468 DEBUG:boto:Data: 2023-10-12T00:27:07.468 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.469 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.470 DEBUG:boto:Params: {} 2023-10-12T00:27:07.470 DEBUG:boto:Token: None 2023-10-12T00:27:07.471 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc= 2023-10-12T00:27:07.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc=', 'Content-Length': '0'} 2023-10-12T00:27:07.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c48d6f7b30fccd2-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.474 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.474 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.475 DEBUG:boto:path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.475 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.475 DEBUG:boto:Method: GET 2023-10-12T00:27:07.476 DEBUG:boto:Path: /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.476 DEBUG:boto:Data: 2023-10-12T00:27:07.476 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.477 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.478 DEBUG:boto:Params: {} 2023-10-12T00:27:07.478 DEBUG:boto:Token: None 2023-10-12T00:27:07.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9 2023-10-12T00:27:07.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc= 2023-10-12T00:27:07.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zP1KjJVn4+2JmVweHVDXrrO+jc=', 'Content-Length': '0'} 2023-10-12T00:27:07.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'yVNgX6SP1gcFgv9bLUxNJ3F1B08BFi9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005276edbd3209cad0-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.482 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.483 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.483 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.483 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.484 DEBUG:boto:Method: GET 2023-10-12T00:27:07.484 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.485 DEBUG:boto:Data: 2023-10-12T00:27:07.485 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.486 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.486 DEBUG:boto:Params: {} 2023-10-12T00:27:07.487 DEBUG:boto:Token: None 2023-10-12T00:27:07.487 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005076b264ff9a9953-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.491 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.491 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.492 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.492 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.493 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.493 DEBUG:boto:Method: GET 2023-10-12T00:27:07.493 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.494 DEBUG:boto:Data: 2023-10-12T00:27:07.494 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.495 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.495 DEBUG:boto:Params: {} 2023-10-12T00:27:07.496 DEBUG:boto:Token: None 2023-10-12T00:27:07.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b341414fb54b962f-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.500 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.500 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.501 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.501 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.501 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.502 DEBUG:boto:Data: 2023-10-12T00:27:07.502 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.503 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.503 DEBUG:boto:Params: {} 2023-10-12T00:27:07.504 DEBUG:boto:Token: None 2023-10-12T00:27:07.504 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b3671f9fa10a04c-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.508 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.509 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.509 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.510 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.510 DEBUG:boto:Data: 2023-10-12T00:27:07.510 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.511 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.512 DEBUG:boto:Params: {} 2023-10-12T00:27:07.512 DEBUG:boto:Token: None 2023-10-12T00:27:07.512 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da4b5c682a15a01c-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.517 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.517 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.518 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.518 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.519 DEBUG:boto:Method: GET 2023-10-12T00:27:07.519 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.519 DEBUG:boto:Data: 2023-10-12T00:27:07.520 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.521 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.521 DEBUG:boto:Params: {} 2023-10-12T00:27:07.521 DEBUG:boto:Token: None 2023-10-12T00:27:07.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000863752c11b590400-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.566 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.567 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.567 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.567 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.568 DEBUG:boto:Method: GET 2023-10-12T00:27:07.568 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.569 DEBUG:boto:Data: 2023-10-12T00:27:07.569 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.570 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.570 DEBUG:boto:Params: {} 2023-10-12T00:27:07.571 DEBUG:boto:Token: None 2023-10-12T00:27:07.571 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.572 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000975618454d50fb5d-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.574 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.575 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.575 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.576 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.576 DEBUG:boto:Method: GET 2023-10-12T00:27:07.576 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.577 DEBUG:boto:Data: 2023-10-12T00:27:07.577 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.578 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.579 DEBUG:boto:Params: {} 2023-10-12T00:27:07.579 DEBUG:boto:Token: None 2023-10-12T00:27:07.579 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.580 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc17cd772d421c7-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.583 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.584 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.584 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.585 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.585 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.586 DEBUG:boto:Method: GET 2023-10-12T00:27:07.586 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.586 DEBUG:boto:Data: 2023-10-12T00:27:07.587 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.588 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.588 DEBUG:boto:Params: {} 2023-10-12T00:27:07.588 DEBUG:boto:Token: None 2023-10-12T00:27:07.589 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd84366d701e10b-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.593 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.594 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.594 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.595 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.595 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.595 DEBUG:boto:Method: GET 2023-10-12T00:27:07.596 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.596 DEBUG:boto:Data: 2023-10-12T00:27:07.596 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.597 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.598 DEBUG:boto:Params: {} 2023-10-12T00:27:07.598 DEBUG:boto:Token: None 2023-10-12T00:27:07.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ= 2023-10-12T00:27:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ=', 'Content-Length': '0'} 2023-10-12T00:27:07.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057f3c281a9002064-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.602 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.603 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.603 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.603 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.604 DEBUG:boto:Method: GET 2023-10-12T00:27:07.604 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.604 DEBUG:boto:Data: 2023-10-12T00:27:07.605 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.606 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.606 DEBUG:boto:Params: {} 2023-10-12T00:27:07.606 DEBUG:boto:Token: None 2023-10-12T00:27:07.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ= 2023-10-12T00:27:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ=', 'Content-Length': '0'} 2023-10-12T00:27:07.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c087816a7e683e42-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.610 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.611 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.611 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.612 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.612 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.612 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.613 DEBUG:boto:Data: 2023-10-12T00:27:07.613 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.614 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.614 DEBUG:boto:Params: {} 2023-10-12T00:27:07.615 DEBUG:boto:Token: None 2023-10-12T00:27:07.615 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71rTf1thhUPH0ngELDuXolBa3/g= 2023-10-12T00:27:07.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71rTf1thhUPH0ngELDuXolBa3/g=', 'Content-Length': '0'} 2023-10-12T00:27:07.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aade8a1c4502fc8c-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.618 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.619 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.619 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.620 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.620 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.620 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.621 DEBUG:boto:Data: 2023-10-12T00:27:07.621 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.622 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.622 DEBUG:boto:Params: {} 2023-10-12T00:27:07.623 DEBUG:boto:Token: None 2023-10-12T00:27:07.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71rTf1thhUPH0ngELDuXolBa3/g= 2023-10-12T00:27:07.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71rTf1thhUPH0ngELDuXolBa3/g=', 'Content-Length': '0'} 2023-10-12T00:27:07.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041674212bea9da87-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.627 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.627 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.628 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.628 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.628 DEBUG:boto:Method: GET 2023-10-12T00:27:07.629 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.629 DEBUG:boto:Data: 2023-10-12T00:27:07.630 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.630 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.631 DEBUG:boto:Params: {} 2023-10-12T00:27:07.631 DEBUG:boto:Token: None 2023-10-12T00:27:07.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ= 2023-10-12T00:27:07.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ=', 'Content-Length': '0'} 2023-10-12T00:27:07.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5eb3fb9f7a06826-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.635 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.636 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.636 DEBUG:boto:path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.636 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.637 DEBUG:boto:Method: GET 2023-10-12T00:27:07.637 DEBUG:boto:Path: /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.638 DEBUG:boto:Data: 2023-10-12T00:27:07.638 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.639 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.639 DEBUG:boto:Params: {} 2023-10-12T00:27:07.640 DEBUG:boto:Token: None 2023-10-12T00:27:07.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=KzSVGht51uCwIbejDUKpHTrvMCcWbAV 2023-10-12T00:27:07.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ= 2023-10-12T00:27:07.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1iETFtM9fqKou5PFGcFO6Ju2fEQ=', 'Content-Length': '0'} 2023-10-12T00:27:07.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'KzSVGht51uCwIbejDUKpHTrvMCcWbAV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d09ba98ff4df350-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.643 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.644 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.644 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.644 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.645 DEBUG:boto:Method: GET 2023-10-12T00:27:07.645 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.646 DEBUG:boto:Data: 2023-10-12T00:27:07.646 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.647 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.647 DEBUG:boto:Params: {} 2023-10-12T00:27:07.648 DEBUG:boto:Token: None 2023-10-12T00:27:07.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e60b2f703885c479-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.694 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.695 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.695 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.696 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.696 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.696 DEBUG:boto:Method: GET 2023-10-12T00:27:07.697 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.697 DEBUG:boto:Data: 2023-10-12T00:27:07.698 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.698 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.699 DEBUG:boto:Params: {} 2023-10-12T00:27:07.699 DEBUG:boto:Token: None 2023-10-12T00:27:07.700 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf0a34d97233e94-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.703 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.704 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.704 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.705 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.705 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.706 DEBUG:boto:Data: 2023-10-12T00:27:07.706 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.707 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.707 DEBUG:boto:Params: {} 2023-10-12T00:27:07.708 DEBUG:boto:Token: None 2023-10-12T00:27:07.708 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094a3a7b53622bef9-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.713 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.714 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.714 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.714 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.715 DEBUG:boto:Data: 2023-10-12T00:27:07.715 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.716 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.716 DEBUG:boto:Params: {} 2023-10-12T00:27:07.717 DEBUG:boto:Token: None 2023-10-12T00:27:07.717 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005df43d99a421037c-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.722 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.723 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.723 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.724 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.724 DEBUG:boto:Method: GET 2023-10-12T00:27:07.724 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.725 DEBUG:boto:Data: 2023-10-12T00:27:07.725 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.726 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.727 DEBUG:boto:Params: {} 2023-10-12T00:27:07.727 DEBUG:boto:Token: None 2023-10-12T00:27:07.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000702eaee8d10185aa-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.731 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.732 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.732 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.733 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.733 DEBUG:boto:Method: GET 2023-10-12T00:27:07.733 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.734 DEBUG:boto:Data: 2023-10-12T00:27:07.734 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.735 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.736 DEBUG:boto:Params: {} 2023-10-12T00:27:07.736 DEBUG:boto:Token: None 2023-10-12T00:27:07.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e4329031d691364-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.740 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.741 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.741 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.741 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.742 DEBUG:boto:Method: GET 2023-10-12T00:27:07.742 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.743 DEBUG:boto:Data: 2023-10-12T00:27:07.743 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.744 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.745 DEBUG:boto:Params: {} 2023-10-12T00:27:07.745 DEBUG:boto:Token: None 2023-10-12T00:27:07.746 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f912932d00b263a0-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.750 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.751 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.751 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.751 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.752 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.752 DEBUG:boto:Method: GET 2023-10-12T00:27:07.753 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.753 DEBUG:boto:Data: 2023-10-12T00:27:07.754 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.754 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.755 DEBUG:boto:Params: {} 2023-10-12T00:27:07.755 DEBUG:boto:Token: None 2023-10-12T00:27:07.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030da82786fee15bd-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.760 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.761 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.761 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.762 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.762 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.763 DEBUG:boto:Method: GET 2023-10-12T00:27:07.763 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.764 DEBUG:boto:Data: 2023-10-12T00:27:07.764 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.765 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.765 DEBUG:boto:Params: {} 2023-10-12T00:27:07.765 DEBUG:boto:Token: None 2023-10-12T00:27:07.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo= 2023-10-12T00:27:07.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo=', 'Content-Length': '0'} 2023-10-12T00:27:07.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e40615c586ea179e-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.770 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.770 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.771 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.771 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.772 DEBUG:boto:Method: GET 2023-10-12T00:27:07.772 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.772 DEBUG:boto:Data: 2023-10-12T00:27:07.773 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.774 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.774 DEBUG:boto:Params: {} 2023-10-12T00:27:07.775 DEBUG:boto:Token: None 2023-10-12T00:27:07.775 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo= 2023-10-12T00:27:07.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo=', 'Content-Length': '0'} 2023-10-12T00:27:07.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db72284f59a3c710-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.779 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.779 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.780 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.780 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.780 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.781 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.781 DEBUG:boto:Data: 2023-10-12T00:27:07.782 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.783 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.783 DEBUG:boto:Params: {} 2023-10-12T00:27:07.784 DEBUG:boto:Token: None 2023-10-12T00:27:07.784 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iVhSEGNbZ1mDD3RicYXC9ON6tuY= 2023-10-12T00:27:07.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iVhSEGNbZ1mDD3RicYXC9ON6tuY=', 'Content-Length': '0'} 2023-10-12T00:27:07.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a478efdd6dcc308-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.787 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.788 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.788 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.789 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.789 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.789 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.790 DEBUG:boto:Data: 2023-10-12T00:27:07.790 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.791 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.791 DEBUG:boto:Params: {} 2023-10-12T00:27:07.792 DEBUG:boto:Token: None 2023-10-12T00:27:07.792 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iVhSEGNbZ1mDD3RicYXC9ON6tuY= 2023-10-12T00:27:07.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iVhSEGNbZ1mDD3RicYXC9ON6tuY=', 'Content-Length': '0'} 2023-10-12T00:27:07.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba937bf86714cbe6-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.796 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.796 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.797 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.797 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.797 DEBUG:boto:Method: GET 2023-10-12T00:27:07.798 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.798 DEBUG:boto:Data: 2023-10-12T00:27:07.799 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.799 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.800 DEBUG:boto:Params: {} 2023-10-12T00:27:07.800 DEBUG:boto:Token: None 2023-10-12T00:27:07.801 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo= 2023-10-12T00:27:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo=', 'Content-Length': '0'} 2023-10-12T00:27:07.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8f22bc6626b0c5e-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.846 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.847 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.847 DEBUG:boto:path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.848 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.848 DEBUG:boto:Method: GET 2023-10-12T00:27:07.849 DEBUG:boto:Path: /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.849 DEBUG:boto:Data: 2023-10-12T00:27:07.850 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.851 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.852 DEBUG:boto:Params: {} 2023-10-12T00:27:07.852 DEBUG:boto:Token: None 2023-10-12T00:27:07.853 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS 2023-10-12T00:27:07.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo= 2023-10-12T00:27:07.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mxz3CALn8+WhA3z11z4xgV5ltMo=', 'Content-Length': '0'} 2023-10-12T00:27:07.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'ih9Ogh.lZQJwApzn2a1cuUYUks7.GhS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e63e4fda70eeeb7-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.857 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.858 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.858 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.859 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.859 DEBUG:boto:Method: GET 2023-10-12T00:27:07.859 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.860 DEBUG:boto:Data: 2023-10-12T00:27:07.860 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.861 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.861 DEBUG:boto:Params: {} 2023-10-12T00:27:07.862 DEBUG:boto:Token: None 2023-10-12T00:27:07.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc99074572f80cf1-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.866 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.867 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.867 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.868 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.868 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.869 DEBUG:boto:Method: GET 2023-10-12T00:27:07.869 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.870 DEBUG:boto:Data: 2023-10-12T00:27:07.870 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.871 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.872 DEBUG:boto:Params: {} 2023-10-12T00:27:07.872 DEBUG:boto:Token: None 2023-10-12T00:27:07.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf1d9ec286d0d0a8-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.877 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.878 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.879 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.879 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.880 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.880 DEBUG:boto:Data: 2023-10-12T00:27:07.881 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.882 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.882 DEBUG:boto:Params: {} 2023-10-12T00:27:07.883 DEBUG:boto:Token: None 2023-10-12T00:27:07.883 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c27777e559a706-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.888 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.889 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.889 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.890 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:07.890 DEBUG:boto:Data: 2023-10-12T00:27:07.890 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.891 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.891 DEBUG:boto:Params: {} 2023-10-12T00:27:07.892 DEBUG:boto:Token: None 2023-10-12T00:27:07.892 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj 2023-10-12T00:27:07.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI= 2023-10-12T00:27:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A/BWONXvIdpILBd0vJIVD6mbUZI=', 'Content-Length': '0'} 2023-10-12T00:27:07.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044ba5ddcbbe84454-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.897 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.898 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.898 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.898 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.899 DEBUG:boto:Method: GET 2023-10-12T00:27:07.899 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.900 DEBUG:boto:Data: 2023-10-12T00:27:07.900 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.901 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.901 DEBUG:boto:Params: {} 2023-10-12T00:27:07.902 DEBUG:boto:Token: None 2023-10-12T00:27:07.902 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099341637706f3b3b-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.906 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.907 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.907 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.908 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.908 DEBUG:boto:Method: GET 2023-10-12T00:27:07.909 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.909 DEBUG:boto:Data: 2023-10-12T00:27:07.910 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.911 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.911 DEBUG:boto:Params: {} 2023-10-12T00:27:07.912 DEBUG:boto:Token: None 2023-10-12T00:27:07.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:07.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY= 2023-10-12T00:27:07.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVs8JmYJ+v++6b/rvvjneZQ/FdY=', 'Content-Length': '0'} 2023-10-12T00:27:07.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4ad65cff53ba4a4-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.916 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.917 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.917 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.918 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.918 DEBUG:boto:Method: GET 2023-10-12T00:27:07.919 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.919 DEBUG:boto:Data: 2023-10-12T00:27:07.920 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.921 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.921 DEBUG:boto:Params: {} 2023-10-12T00:27:07.922 DEBUG:boto:Token: None 2023-10-12T00:27:07.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002882e731b225d5a8-0065273d5b-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.926 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.927 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:07.927 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:07.928 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.928 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:07.929 DEBUG:boto:Method: GET 2023-10-12T00:27:07.929 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:07.930 DEBUG:boto:Data: 2023-10-12T00:27:07.930 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.931 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.932 DEBUG:boto:Params: {} 2023-10-12T00:27:07.932 DEBUG:boto:Token: None 2023-10-12T00:27:07.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/ 2023-10-12T00:27:07.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE= 2023-10-12T00:27:07.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKaoM/l0D9gKVSg+93fh/DLa4AE=', 'Content-Length': '0'} 2023-10-12T00:27:07.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001273f37bab1ce8c6-0065273d5b-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.936 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:07.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.937 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.938 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.938 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.939 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.939 DEBUG:boto:Method: GET 2023-10-12T00:27:07.940 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.940 DEBUG:boto:Data: 2023-10-12T00:27:07.941 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.942 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.942 DEBUG:boto:Params: {} 2023-10-12T00:27:07.943 DEBUG:boto:Token: None 2023-10-12T00:27:07.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI= 2023-10-12T00:27:07.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI=', 'Content-Length': '0'} 2023-10-12T00:27:07.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5c706df040f74d1-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.947 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.948 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.948 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.949 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.949 DEBUG:boto:Method: GET 2023-10-12T00:27:07.950 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.950 DEBUG:boto:Data: 2023-10-12T00:27:07.951 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.952 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.952 DEBUG:boto:Params: {} 2023-10-12T00:27:07.953 DEBUG:boto:Token: None 2023-10-12T00:27:07.954 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI= 2023-10-12T00:27:07.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI=', 'Content-Length': '0'} 2023-10-12T00:27:07.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e30a635bf1f253b-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.958 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.959 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.959 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.960 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.960 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.961 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.961 DEBUG:boto:Data: 2023-10-12T00:27:07.962 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.963 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.964 DEBUG:boto:Params: {} 2023-10-12T00:27:07.964 DEBUG:boto:Token: None 2023-10-12T00:27:07.965 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fvd2lZg3ygrmtV1AzE7w8cx14b4= 2023-10-12T00:27:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fvd2lZg3ygrmtV1AzE7w8cx14b4=', 'Content-Length': '0'} 2023-10-12T00:27:07.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d0dff42efac6800-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.969 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.970 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.970 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.971 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.971 DEBUG:boto:Method: HEAD 2023-10-12T00:27:07.972 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.972 DEBUG:boto:Data: 2023-10-12T00:27:07.973 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:07.974 DEBUG:boto:Port: 8000 2023-10-12T00:27:07.975 DEBUG:boto:Params: {} 2023-10-12T00:27:07.975 DEBUG:boto:Token: None 2023-10-12T00:27:07.976 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fvd2lZg3ygrmtV1AzE7w8cx14b4= 2023-10-12T00:27:07.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fvd2lZg3ygrmtV1AzE7w8cx14b4=', 'Content-Length': '0'} 2023-10-12T00:27:07.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2c486083d68e31c-0065273d5b-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:07.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:07.980 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:07.981 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:07.981 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.982 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.983 DEBUG:boto:Method: GET 2023-10-12T00:27:07.983 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.984 DEBUG:boto:Data: 2023-10-12T00:27:07.984 DEBUG:boto:Headers: {} 2023-10-12T00:27:07.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:07.986 DEBUG:boto:Port: 8001 2023-10-12T00:27:07.986 DEBUG:boto:Params: {} 2023-10-12T00:27:07.987 DEBUG:boto:Token: None 2023-10-12T00:27:07.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:07 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:07.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI= 2023-10-12T00:27:07.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pSHXb/Q5Aej9jjtwZ9VyJC4kzmI=', 'Content-Length': '0'} 2023-10-12T00:27:07.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f85fdc44d98a2f0-0065273d5b-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.034 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.035 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.035 DEBUG:boto:path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:08.036 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:08.036 DEBUG:boto:Method: GET 2023-10-12T00:27:08.037 DEBUG:boto:Path: /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:08.037 DEBUG:boto:Data: 2023-10-12T00:27:08.038 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.039 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.040 DEBUG:boto:Params: {} 2023-10-12T00:27:08.041 DEBUG:boto:Token: None 2023-10-12T00:27:08.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh 2023-10-12T00:27:08.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3Xn2ezZzOyaQwMQL1EGYfe6WdsI= 2023-10-12T00:27:08.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3Xn2ezZzOyaQwMQL1EGYfe6WdsI=', 'Content-Length': '0'} 2023-10-12T00:27:08.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'YGWhHG0Ib6FCdA6at2MMDfIDcgO2wxh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d52228bf96bd30f6-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.086 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.087 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.087 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.088 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.088 DEBUG:boto:Method: GET 2023-10-12T00:27:08.088 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.089 DEBUG:boto:Data: 2023-10-12T00:27:08.089 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.090 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.090 DEBUG:boto:Params: {} 2023-10-12T00:27:08.091 DEBUG:boto:Token: None 2023-10-12T00:27:08.091 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea5fde84ce9bfcb-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.095 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.096 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.096 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.097 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.097 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.098 DEBUG:boto:Method: GET 2023-10-12T00:27:08.098 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.098 DEBUG:boto:Data: 2023-10-12T00:27:08.099 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.100 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.100 DEBUG:boto:Params: {} 2023-10-12T00:27:08.100 DEBUG:boto:Token: None 2023-10-12T00:27:08.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014436d6cdc7cbef1-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.105 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.106 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.106 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.107 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.107 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.107 DEBUG:boto:Data: 2023-10-12T00:27:08.108 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.108 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.109 DEBUG:boto:Params: {} 2023-10-12T00:27:08.109 DEBUG:boto:Token: None 2023-10-12T00:27:08.110 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e45501210029f2e2-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.115 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.115 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.116 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.116 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.116 DEBUG:boto:Data: 2023-10-12T00:27:08.117 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.117 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.118 DEBUG:boto:Params: {} 2023-10-12T00:27:08.118 DEBUG:boto:Token: None 2023-10-12T00:27:08.119 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1fd89a170bad511-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.123 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.124 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.124 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.124 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.125 DEBUG:boto:Method: GET 2023-10-12T00:27:08.125 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.126 DEBUG:boto:Data: 2023-10-12T00:27:08.126 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.127 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.127 DEBUG:boto:Params: {} 2023-10-12T00:27:08.128 DEBUG:boto:Token: None 2023-10-12T00:27:08.128 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d268229a1be8449-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.132 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.132 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.133 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.133 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.133 DEBUG:boto:Method: GET 2023-10-12T00:27:08.134 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.134 DEBUG:boto:Data: 2023-10-12T00:27:08.135 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.135 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.136 DEBUG:boto:Params: {} 2023-10-12T00:27:08.136 DEBUG:boto:Token: None 2023-10-12T00:27:08.137 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099c248ebdd7c417a-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.182 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.183 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.183 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.184 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.184 DEBUG:boto:Method: GET 2023-10-12T00:27:08.184 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.185 DEBUG:boto:Data: 2023-10-12T00:27:08.185 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.186 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.186 DEBUG:boto:Params: {} 2023-10-12T00:27:08.187 DEBUG:boto:Token: None 2023-10-12T00:27:08.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb8685240b07bc8a-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.191 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.192 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.192 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.193 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.193 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.193 DEBUG:boto:Method: GET 2023-10-12T00:27:08.194 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.194 DEBUG:boto:Data: 2023-10-12T00:27:08.195 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.195 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.196 DEBUG:boto:Params: {} 2023-10-12T00:27:08.196 DEBUG:boto:Token: None 2023-10-12T00:27:08.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023330d5d4c991b48-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.201 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.202 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.202 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.202 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.203 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.203 DEBUG:boto:Method: GET 2023-10-12T00:27:08.204 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.204 DEBUG:boto:Data: 2023-10-12T00:27:08.204 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.205 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.205 DEBUG:boto:Params: {} 2023-10-12T00:27:08.206 DEBUG:boto:Token: None 2023-10-12T00:27:08.206 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.207 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8= 2023-10-12T00:27:08.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8=', 'Content-Length': '0'} 2023-10-12T00:27:08.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b50df4d87a5f50a-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.250 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.251 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.251 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.252 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.252 DEBUG:boto:Method: GET 2023-10-12T00:27:08.252 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.253 DEBUG:boto:Data: 2023-10-12T00:27:08.253 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.254 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.254 DEBUG:boto:Params: {} 2023-10-12T00:27:08.255 DEBUG:boto:Token: None 2023-10-12T00:27:08.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8= 2023-10-12T00:27:08.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8=', 'Content-Length': '0'} 2023-10-12T00:27:08.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004168bd730139e5ee-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.259 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.260 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.260 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.261 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.261 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.261 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.262 DEBUG:boto:Data: 2023-10-12T00:27:08.262 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.263 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.263 DEBUG:boto:Params: {} 2023-10-12T00:27:08.264 DEBUG:boto:Token: None 2023-10-12T00:27:08.264 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+Zxuk0oIKweeLr6hw7gAViHOZ8= 2023-10-12T00:27:08.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+Zxuk0oIKweeLr6hw7gAViHOZ8=', 'Content-Length': '0'} 2023-10-12T00:27:08.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f1ddfb4481d6fd6-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.267 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.268 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.268 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.268 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.269 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.269 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.270 DEBUG:boto:Data: 2023-10-12T00:27:08.270 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.271 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.271 DEBUG:boto:Params: {} 2023-10-12T00:27:08.272 DEBUG:boto:Token: None 2023-10-12T00:27:08.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+Zxuk0oIKweeLr6hw7gAViHOZ8= 2023-10-12T00:27:08.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+Zxuk0oIKweeLr6hw7gAViHOZ8=', 'Content-Length': '0'} 2023-10-12T00:27:08.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003243e2824488f814-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.276 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.276 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.276 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.277 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.277 DEBUG:boto:Method: GET 2023-10-12T00:27:08.278 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.278 DEBUG:boto:Data: 2023-10-12T00:27:08.278 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.279 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.280 DEBUG:boto:Params: {} 2023-10-12T00:27:08.280 DEBUG:boto:Token: None 2023-10-12T00:27:08.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8= 2023-10-12T00:27:08.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8=', 'Content-Length': '0'} 2023-10-12T00:27:08.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000904008e0770fd1e6-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.284 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.284 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.285 DEBUG:boto:path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.285 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.286 DEBUG:boto:Method: GET 2023-10-12T00:27:08.286 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.286 DEBUG:boto:Data: 2023-10-12T00:27:08.287 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.288 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.288 DEBUG:boto:Params: {} 2023-10-12T00:27:08.288 DEBUG:boto:Token: None 2023-10-12T00:27:08.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qOdYv7tgNZreFYF8opEwj3jAqY-OBoC 2023-10-12T00:27:08.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8= 2023-10-12T00:27:08.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vZjvYJZ2Pnpv6xVJS8xp9bz9i8=', 'Content-Length': '0'} 2023-10-12T00:27:08.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qOdYv7tgNZreFYF8opEwj3jAqY-OBoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5ee48ff463f4448-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.334 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.335 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.335 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.336 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.336 DEBUG:boto:Method: GET 2023-10-12T00:27:08.337 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.337 DEBUG:boto:Data: 2023-10-12T00:27:08.337 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.338 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.338 DEBUG:boto:Params: {} 2023-10-12T00:27:08.339 DEBUG:boto:Token: None 2023-10-12T00:27:08.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1d84fcd997b2c3-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.343 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.344 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.344 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.345 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.345 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.345 DEBUG:boto:Method: GET 2023-10-12T00:27:08.346 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.346 DEBUG:boto:Data: 2023-10-12T00:27:08.347 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.347 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.348 DEBUG:boto:Params: {} 2023-10-12T00:27:08.348 DEBUG:boto:Token: None 2023-10-12T00:27:08.349 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c7c8fb9b99253d-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.352 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.353 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.353 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.354 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.354 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.354 DEBUG:boto:Data: 2023-10-12T00:27:08.355 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.355 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.356 DEBUG:boto:Params: {} 2023-10-12T00:27:08.356 DEBUG:boto:Token: None 2023-10-12T00:27:08.357 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc176dfcdc2b7a0-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.361 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.361 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.362 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.362 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.363 DEBUG:boto:Data: 2023-10-12T00:27:08.363 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.364 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.364 DEBUG:boto:Params: {} 2023-10-12T00:27:08.364 DEBUG:boto:Token: None 2023-10-12T00:27:08.365 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3810b6e6c299ec1-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.369 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.370 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.370 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.371 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.371 DEBUG:boto:Method: GET 2023-10-12T00:27:08.372 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.372 DEBUG:boto:Data: 2023-10-12T00:27:08.372 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.373 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.373 DEBUG:boto:Params: {} 2023-10-12T00:27:08.374 DEBUG:boto:Token: None 2023-10-12T00:27:08.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2306f496af5ddc6-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.418 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.419 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.420 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.420 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.420 DEBUG:boto:Method: GET 2023-10-12T00:27:08.421 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.421 DEBUG:boto:Data: 2023-10-12T00:27:08.422 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.423 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.423 DEBUG:boto:Params: {} 2023-10-12T00:27:08.423 DEBUG:boto:Token: None 2023-10-12T00:27:08.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a93ad7b807683df6-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.470 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.471 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.472 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.472 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.473 DEBUG:boto:Method: GET 2023-10-12T00:27:08.473 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.474 DEBUG:boto:Data: 2023-10-12T00:27:08.474 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.475 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.476 DEBUG:boto:Params: {} 2023-10-12T00:27:08.476 DEBUG:boto:Token: None 2023-10-12T00:27:08.477 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3b6f31f3160f3e-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.482 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.483 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.483 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.484 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.484 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.484 DEBUG:boto:Method: GET 2023-10-12T00:27:08.485 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.485 DEBUG:boto:Data: 2023-10-12T00:27:08.486 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.487 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.487 DEBUG:boto:Params: {} 2023-10-12T00:27:08.487 DEBUG:boto:Token: None 2023-10-12T00:27:08.488 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bebc13d7e0881545-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.492 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.493 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.494 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.494 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.495 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.495 DEBUG:boto:Method: GET 2023-10-12T00:27:08.496 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.496 DEBUG:boto:Data: 2023-10-12T00:27:08.497 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.497 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.498 DEBUG:boto:Params: {} 2023-10-12T00:27:08.498 DEBUG:boto:Token: None 2023-10-12T00:27:08.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI= 2023-10-12T00:27:08.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI=', 'Content-Length': '0'} 2023-10-12T00:27:08.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fd7dac7695b1f8b-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.503 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.503 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.504 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.504 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.505 DEBUG:boto:Method: GET 2023-10-12T00:27:08.505 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.506 DEBUG:boto:Data: 2023-10-12T00:27:08.506 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.507 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.508 DEBUG:boto:Params: {} 2023-10-12T00:27:08.508 DEBUG:boto:Token: None 2023-10-12T00:27:08.509 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI= 2023-10-12T00:27:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI=', 'Content-Length': '0'} 2023-10-12T00:27:08.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004f62939e6409ddb-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.554 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.555 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.555 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.556 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.556 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.557 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.557 DEBUG:boto:Data: 2023-10-12T00:27:08.558 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.559 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.560 DEBUG:boto:Params: {} 2023-10-12T00:27:08.560 DEBUG:boto:Token: None 2023-10-12T00:27:08.561 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.562 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lDwRd0hIaAktG3uo7TyjO8RQvyQ= 2023-10-12T00:27:08.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lDwRd0hIaAktG3uo7TyjO8RQvyQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000684170819f7e7d1d-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.565 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.565 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.566 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.566 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.567 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.567 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.568 DEBUG:boto:Data: 2023-10-12T00:27:08.568 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.569 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.570 DEBUG:boto:Params: {} 2023-10-12T00:27:08.570 DEBUG:boto:Token: None 2023-10-12T00:27:08.571 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.571 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lDwRd0hIaAktG3uo7TyjO8RQvyQ= 2023-10-12T00:27:08.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lDwRd0hIaAktG3uo7TyjO8RQvyQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca4d0edbe8fa4d5-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.575 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.575 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.575 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.576 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.576 DEBUG:boto:Method: GET 2023-10-12T00:27:08.577 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.577 DEBUG:boto:Data: 2023-10-12T00:27:08.577 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.578 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.579 DEBUG:boto:Params: {} 2023-10-12T00:27:08.579 DEBUG:boto:Token: None 2023-10-12T00:27:08.580 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.580 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI= 2023-10-12T00:27:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI=', 'Content-Length': '0'} 2023-10-12T00:27:08.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087679bb6a8a3bcaf-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.583 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.584 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.584 DEBUG:boto:path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.584 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.585 DEBUG:boto:Method: GET 2023-10-12T00:27:08.585 DEBUG:boto:Path: /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.586 DEBUG:boto:Data: 2023-10-12T00:27:08.586 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.587 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.588 DEBUG:boto:Params: {} 2023-10-12T00:27:08.588 DEBUG:boto:Token: None 2023-10-12T00:27:08.589 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq. 2023-10-12T00:27:08.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI= 2023-10-12T00:27:08.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBOChCP/3bp9oxeBfm2GbA+CiMI=', 'Content-Length': '0'} 2023-10-12T00:27:08.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'q262BVRKd1Ns9Gz.m.CDC1a1lY13Tq.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000415c27e60d0f841f-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.593 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.593 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.594 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.594 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.595 DEBUG:boto:Method: GET 2023-10-12T00:27:08.595 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.596 DEBUG:boto:Data: 2023-10-12T00:27:08.596 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.598 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.598 DEBUG:boto:Params: {} 2023-10-12T00:27:08.599 DEBUG:boto:Token: None 2023-10-12T00:27:08.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54c900415d6744f-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.604 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.604 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.605 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.605 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.606 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.606 DEBUG:boto:Method: GET 2023-10-12T00:27:08.607 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.607 DEBUG:boto:Data: 2023-10-12T00:27:08.607 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.608 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.609 DEBUG:boto:Params: {} 2023-10-12T00:27:08.609 DEBUG:boto:Token: None 2023-10-12T00:27:08.610 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d8936cc60dc6ad-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.613 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.614 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.615 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.615 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.615 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.616 DEBUG:boto:Data: 2023-10-12T00:27:08.616 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.617 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.617 DEBUG:boto:Params: {} 2023-10-12T00:27:08.618 DEBUG:boto:Token: None 2023-10-12T00:27:08.618 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f0ccfd468102f33-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.623 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.624 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.624 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.625 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.625 DEBUG:boto:Data: 2023-10-12T00:27:08.626 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.627 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.627 DEBUG:boto:Params: {} 2023-10-12T00:27:08.628 DEBUG:boto:Token: None 2023-10-12T00:27:08.628 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.629 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096b7616567db7dfa-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.634 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.634 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.635 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.636 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.637 DEBUG:boto:Method: GET 2023-10-12T00:27:08.637 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.638 DEBUG:boto:Data: 2023-10-12T00:27:08.638 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.639 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.640 DEBUG:boto:Params: {} 2023-10-12T00:27:08.640 DEBUG:boto:Token: None 2023-10-12T00:27:08.641 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060b3477931673f68-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.686 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.687 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.687 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.688 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.688 DEBUG:boto:Method: GET 2023-10-12T00:27:08.689 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.689 DEBUG:boto:Data: 2023-10-12T00:27:08.690 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.691 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.691 DEBUG:boto:Params: {} 2023-10-12T00:27:08.692 DEBUG:boto:Token: None 2023-10-12T00:27:08.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bac38433892d5e25-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.696 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.697 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.698 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.698 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.699 DEBUG:boto:Method: GET 2023-10-12T00:27:08.699 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.700 DEBUG:boto:Data: 2023-10-12T00:27:08.700 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.701 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.701 DEBUG:boto:Params: {} 2023-10-12T00:27:08.702 DEBUG:boto:Token: None 2023-10-12T00:27:08.702 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5a17393fed31d8-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.707 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.708 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.708 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.708 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.709 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.709 DEBUG:boto:Method: GET 2023-10-12T00:27:08.710 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.710 DEBUG:boto:Data: 2023-10-12T00:27:08.710 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.711 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.712 DEBUG:boto:Params: {} 2023-10-12T00:27:08.712 DEBUG:boto:Token: None 2023-10-12T00:27:08.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1682beb96ba0c6-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.717 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.718 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.718 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.718 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.719 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.719 DEBUG:boto:Method: GET 2023-10-12T00:27:08.720 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.720 DEBUG:boto:Data: 2023-10-12T00:27:08.721 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.722 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.722 DEBUG:boto:Params: {} 2023-10-12T00:27:08.723 DEBUG:boto:Token: None 2023-10-12T00:27:08.723 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M= 2023-10-12T00:27:08.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M=', 'Content-Length': '0'} 2023-10-12T00:27:08.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000129662c9d406411b-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.728 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.728 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.729 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.729 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.730 DEBUG:boto:Method: GET 2023-10-12T00:27:08.730 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.731 DEBUG:boto:Data: 2023-10-12T00:27:08.731 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.732 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.733 DEBUG:boto:Params: {} 2023-10-12T00:27:08.733 DEBUG:boto:Token: None 2023-10-12T00:27:08.734 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M= 2023-10-12T00:27:08.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M=', 'Content-Length': '0'} 2023-10-12T00:27:08.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000927878517c66ddd7-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.738 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.739 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.739 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.740 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.740 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.740 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.741 DEBUG:boto:Data: 2023-10-12T00:27:08.741 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.742 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.742 DEBUG:boto:Params: {} 2023-10-12T00:27:08.743 DEBUG:boto:Token: None 2023-10-12T00:27:08.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.743 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zd9SbFEfhR+W8V+qPnnJU+eKKWQ= 2023-10-12T00:27:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zd9SbFEfhR+W8V+qPnnJU+eKKWQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbfde06e613ecd2c-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.746 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.747 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.747 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.748 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.748 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.749 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.749 DEBUG:boto:Data: 2023-10-12T00:27:08.749 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.750 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.750 DEBUG:boto:Params: {} 2023-10-12T00:27:08.751 DEBUG:boto:Token: None 2023-10-12T00:27:08.751 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zd9SbFEfhR+W8V+qPnnJU+eKKWQ= 2023-10-12T00:27:08.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zd9SbFEfhR+W8V+qPnnJU+eKKWQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b6b13d9d81e0171-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.755 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.756 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.756 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.757 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.757 DEBUG:boto:Method: GET 2023-10-12T00:27:08.758 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.758 DEBUG:boto:Data: 2023-10-12T00:27:08.758 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.759 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.760 DEBUG:boto:Params: {} 2023-10-12T00:27:08.760 DEBUG:boto:Token: None 2023-10-12T00:27:08.761 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M= 2023-10-12T00:27:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M=', 'Content-Length': '0'} 2023-10-12T00:27:08.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036e43d8e297a724b-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.806 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.807 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.807 DEBUG:boto:path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.808 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.808 DEBUG:boto:Method: GET 2023-10-12T00:27:08.808 DEBUG:boto:Path: /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.809 DEBUG:boto:Data: 2023-10-12T00:27:08.809 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.810 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.810 DEBUG:boto:Params: {} 2023-10-12T00:27:08.811 DEBUG:boto:Token: None 2023-10-12T00:27:08.811 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=qdRaE86GREIsJ3Eg774dLKCsX96HV3G 2023-10-12T00:27:08.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M= 2023-10-12T00:27:08.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+y4Ibf6/i3kzVKjca7JjKwxQ1+M=', 'Content-Length': '0'} 2023-10-12T00:27:08.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'qdRaE86GREIsJ3Eg774dLKCsX96HV3G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001de05c817231623a-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.815 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.815 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.816 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.816 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.817 DEBUG:boto:Method: GET 2023-10-12T00:27:08.817 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.818 DEBUG:boto:Data: 2023-10-12T00:27:08.818 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.819 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.820 DEBUG:boto:Params: {} 2023-10-12T00:27:08.820 DEBUG:boto:Token: None 2023-10-12T00:27:08.821 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b35550a27ef5218-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.825 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.826 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.827 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.827 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.828 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.828 DEBUG:boto:Method: GET 2023-10-12T00:27:08.829 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.829 DEBUG:boto:Data: 2023-10-12T00:27:08.829 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.830 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.831 DEBUG:boto:Params: {} 2023-10-12T00:27:08.832 DEBUG:boto:Token: None 2023-10-12T00:27:08.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3278da106eca26-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.836 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.837 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.837 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.838 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.838 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.838 DEBUG:boto:Data: 2023-10-12T00:27:08.839 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.839 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.840 DEBUG:boto:Params: {} 2023-10-12T00:27:08.840 DEBUG:boto:Token: None 2023-10-12T00:27:08.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3030f6fafcc3fb8-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.845 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.846 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.846 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.847 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:08.847 DEBUG:boto:Data: 2023-10-12T00:27:08.848 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.849 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.850 DEBUG:boto:Params: {} 2023-10-12T00:27:08.850 DEBUG:boto:Token: None 2023-10-12T00:27:08.851 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj 2023-10-12T00:27:08.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew= 2023-10-12T00:27:08.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MU3dMdsOi4+T4sIjDjDbWbTF4Ew=', 'Content-Length': '0'} 2023-10-12T00:27:08.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dbf0f317e02719f-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.857 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.858 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.859 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.859 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.860 DEBUG:boto:Method: GET 2023-10-12T00:27:08.860 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.861 DEBUG:boto:Data: 2023-10-12T00:27:08.862 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.863 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.864 DEBUG:boto:Params: {} 2023-10-12T00:27:08.864 DEBUG:boto:Token: None 2023-10-12T00:27:08.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a01e924bedb59dc0-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.868 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.869 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.869 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.870 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.870 DEBUG:boto:Method: GET 2023-10-12T00:27:08.871 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.871 DEBUG:boto:Data: 2023-10-12T00:27:08.872 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.872 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.873 DEBUG:boto:Params: {} 2023-10-12T00:27:08.873 DEBUG:boto:Token: None 2023-10-12T00:27:08.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:08.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ= 2023-10-12T00:27:08.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4v1cke8j1B/ImEbwD4+a6i+asVQ=', 'Content-Length': '0'} 2023-10-12T00:27:08.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000803a9b9d8e94e319-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.918 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.919 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.919 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.919 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.920 DEBUG:boto:Method: GET 2023-10-12T00:27:08.920 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.921 DEBUG:boto:Data: 2023-10-12T00:27:08.921 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.922 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.922 DEBUG:boto:Params: {} 2023-10-12T00:27:08.923 DEBUG:boto:Token: None 2023-10-12T00:27:08.923 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b3dfeccb95002db-0065273d5c-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.927 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.928 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:08.928 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:08.929 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.929 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:08.930 DEBUG:boto:Method: GET 2023-10-12T00:27:08.930 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:08.930 DEBUG:boto:Data: 2023-10-12T00:27:08.931 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.932 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.932 DEBUG:boto:Params: {} 2023-10-12T00:27:08.932 DEBUG:boto:Token: None 2023-10-12T00:27:08.933 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/ 2023-10-12T00:27:08.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE= 2023-10-12T00:27:08.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCxpI4PKm12/XgIWf2ptautyoJE=', 'Content-Length': '0'} 2023-10-12T00:27:08.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1649aef02607d5b-0065273d5c-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.937 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:08.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.938 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.938 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.939 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.939 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.939 DEBUG:boto:Method: GET 2023-10-12T00:27:08.940 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.940 DEBUG:boto:Data: 2023-10-12T00:27:08.941 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.941 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.942 DEBUG:boto:Params: {} 2023-10-12T00:27:08.942 DEBUG:boto:Token: None 2023-10-12T00:27:08.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M= 2023-10-12T00:27:08.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M=', 'Content-Length': '0'} 2023-10-12T00:27:08.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004454684af1c5d35d-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.947 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.947 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.947 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.948 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.948 DEBUG:boto:Method: GET 2023-10-12T00:27:08.949 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.949 DEBUG:boto:Data: 2023-10-12T00:27:08.949 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.950 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.951 DEBUG:boto:Params: {} 2023-10-12T00:27:08.951 DEBUG:boto:Token: None 2023-10-12T00:27:08.952 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M= 2023-10-12T00:27:08.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M=', 'Content-Length': '0'} 2023-10-12T00:27:08.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abef6df9b87f342b-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.955 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.955 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.956 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.956 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.956 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.957 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.957 DEBUG:boto:Data: 2023-10-12T00:27:08.958 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.958 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.959 DEBUG:boto:Params: {} 2023-10-12T00:27:08.959 DEBUG:boto:Token: None 2023-10-12T00:27:08.960 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0dq5LvxVHEn6pbWOOqwzUgdXEhk= 2023-10-12T00:27:08.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0dq5LvxVHEn6pbWOOqwzUgdXEhk=', 'Content-Length': '0'} 2023-10-12T00:27:08.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3ded32ae47a7f0f-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.963 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.964 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.964 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.964 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.965 DEBUG:boto:Method: HEAD 2023-10-12T00:27:08.965 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.966 DEBUG:boto:Data: 2023-10-12T00:27:08.966 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:08.967 DEBUG:boto:Port: 8000 2023-10-12T00:27:08.967 DEBUG:boto:Params: {} 2023-10-12T00:27:08.968 DEBUG:boto:Token: None 2023-10-12T00:27:08.968 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0dq5LvxVHEn6pbWOOqwzUgdXEhk= 2023-10-12T00:27:08.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0dq5LvxVHEn6pbWOOqwzUgdXEhk=', 'Content-Length': '0'} 2023-10-12T00:27:08.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000142f5be011b4851a-0065273d5c-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:08.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:08.972 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:08.972 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:08.973 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.973 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.973 DEBUG:boto:Method: GET 2023-10-12T00:27:08.974 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.974 DEBUG:boto:Data: 2023-10-12T00:27:08.975 DEBUG:boto:Headers: {} 2023-10-12T00:27:08.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:08.976 DEBUG:boto:Port: 8001 2023-10-12T00:27:08.976 DEBUG:boto:Params: {} 2023-10-12T00:27:08.976 DEBUG:boto:Token: None 2023-10-12T00:27:08.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:08 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:08.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M= 2023-10-12T00:27:08.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IowkWuc4rqpoY+Bol8DD2M4Cm1M=', 'Content-Length': '0'} 2023-10-12T00:27:08.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c95869a871bc031-0065273d5c-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.026 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.027 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.027 DEBUG:boto:path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:09.028 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:09.028 DEBUG:boto:Method: GET 2023-10-12T00:27:09.028 DEBUG:boto:Path: /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:09.029 DEBUG:boto:Data: 2023-10-12T00:27:09.029 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.030 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.030 DEBUG:boto:Params: {} 2023-10-12T00:27:09.031 DEBUG:boto:Token: None 2023-10-12T00:27:09.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp 2023-10-12T00:27:09.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K5YxfGopmzIBWwhLtmeBBrWt0FU= 2023-10-12T00:27:09.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K5YxfGopmzIBWwhLtmeBBrWt0FU=', 'Content-Length': '0'} 2023-10-12T00:27:09.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '6-YU8dqB3d5GRTg1gBQXBUl7EWHtNqp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006007edfc51146996-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.035 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.035 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.036 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.036 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.036 DEBUG:boto:Method: GET 2023-10-12T00:27:09.037 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.037 DEBUG:boto:Data: 2023-10-12T00:27:09.038 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.038 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.039 DEBUG:boto:Params: {} 2023-10-12T00:27:09.039 DEBUG:boto:Token: None 2023-10-12T00:27:09.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef256f2d48ba3317-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.044 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.044 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.045 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.045 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.045 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.046 DEBUG:boto:Method: GET 2023-10-12T00:27:09.046 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.047 DEBUG:boto:Data: 2023-10-12T00:27:09.047 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.048 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.048 DEBUG:boto:Params: {} 2023-10-12T00:27:09.049 DEBUG:boto:Token: None 2023-10-12T00:27:09.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae32ca225858fe5-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.094 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.095 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.095 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.096 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.096 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.096 DEBUG:boto:Data: 2023-10-12T00:27:09.097 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.097 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.098 DEBUG:boto:Params: {} 2023-10-12T00:27:09.098 DEBUG:boto:Token: None 2023-10-12T00:27:09.099 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.099 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8a8cd62a1e78029-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.104 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.104 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.104 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.105 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.105 DEBUG:boto:Data: 2023-10-12T00:27:09.105 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.106 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.107 DEBUG:boto:Params: {} 2023-10-12T00:27:09.107 DEBUG:boto:Token: None 2023-10-12T00:27:09.107 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000554f18f3a2a32efc-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.113 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.113 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.113 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.114 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.114 DEBUG:boto:Method: GET 2023-10-12T00:27:09.115 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.115 DEBUG:boto:Data: 2023-10-12T00:27:09.115 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.116 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.116 DEBUG:boto:Params: {} 2023-10-12T00:27:09.117 DEBUG:boto:Token: None 2023-10-12T00:27:09.117 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.118 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000859da0b148cd5ac2-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.162 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.163 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.163 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.164 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.164 DEBUG:boto:Method: GET 2023-10-12T00:27:09.164 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.165 DEBUG:boto:Data: 2023-10-12T00:27:09.165 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.166 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.167 DEBUG:boto:Params: {} 2023-10-12T00:27:09.167 DEBUG:boto:Token: None 2023-10-12T00:27:09.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7f0585088ae78d0-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.214 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.215 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.215 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.215 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.216 DEBUG:boto:Method: GET 2023-10-12T00:27:09.216 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.217 DEBUG:boto:Data: 2023-10-12T00:27:09.217 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.218 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.219 DEBUG:boto:Params: {} 2023-10-12T00:27:09.219 DEBUG:boto:Token: None 2023-10-12T00:27:09.219 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f88049017d2a9c-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.224 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.224 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.225 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.225 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.225 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.226 DEBUG:boto:Method: GET 2023-10-12T00:27:09.226 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.227 DEBUG:boto:Data: 2023-10-12T00:27:09.227 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.228 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.229 DEBUG:boto:Params: {} 2023-10-12T00:27:09.230 DEBUG:boto:Token: None 2023-10-12T00:27:09.230 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.231 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d25d1720736cb1-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.235 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.236 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.236 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.237 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.237 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.238 DEBUG:boto:Method: GET 2023-10-12T00:27:09.241 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.242 DEBUG:boto:Data: 2023-10-12T00:27:09.243 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.244 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.245 DEBUG:boto:Params: {} 2023-10-12T00:27:09.246 DEBUG:boto:Token: None 2023-10-12T00:27:09.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg= 2023-10-12T00:27:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg=', 'Content-Length': '0'} 2023-10-12T00:27:09.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bc38dbb18105604-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.251 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.251 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.252 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.252 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.252 DEBUG:boto:Method: GET 2023-10-12T00:27:09.253 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.253 DEBUG:boto:Data: 2023-10-12T00:27:09.254 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.255 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.255 DEBUG:boto:Params: {} 2023-10-12T00:27:09.256 DEBUG:boto:Token: None 2023-10-12T00:27:09.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg= 2023-10-12T00:27:09.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg=', 'Content-Length': '0'} 2023-10-12T00:27:09.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e03c2c7b9c2bc657-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.302 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.303 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.303 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.304 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.304 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.305 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.305 DEBUG:boto:Data: 2023-10-12T00:27:09.305 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.306 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.307 DEBUG:boto:Params: {} 2023-10-12T00:27:09.307 DEBUG:boto:Token: None 2023-10-12T00:27:09.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PG4CVJiyBiDrEmwsCpj7EKh7D2Y= 2023-10-12T00:27:09.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PG4CVJiyBiDrEmwsCpj7EKh7D2Y=', 'Content-Length': '0'} 2023-10-12T00:27:09.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020d1e4fee4db1206-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.312 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.312 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.313 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.313 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.314 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.314 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.314 DEBUG:boto:Data: 2023-10-12T00:27:09.315 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.316 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.316 DEBUG:boto:Params: {} 2023-10-12T00:27:09.317 DEBUG:boto:Token: None 2023-10-12T00:27:09.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PG4CVJiyBiDrEmwsCpj7EKh7D2Y= 2023-10-12T00:27:09.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PG4CVJiyBiDrEmwsCpj7EKh7D2Y=', 'Content-Length': '0'} 2023-10-12T00:27:09.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005febba4bda7a26f4-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.321 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.322 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.322 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.323 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.323 DEBUG:boto:Method: GET 2023-10-12T00:27:09.323 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.324 DEBUG:boto:Data: 2023-10-12T00:27:09.324 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.325 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.325 DEBUG:boto:Params: {} 2023-10-12T00:27:09.326 DEBUG:boto:Token: None 2023-10-12T00:27:09.326 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg= 2023-10-12T00:27:09.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg=', 'Content-Length': '0'} 2023-10-12T00:27:09.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000253154b5fe9c480e-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.330 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.331 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.331 DEBUG:boto:path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.332 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.332 DEBUG:boto:Method: GET 2023-10-12T00:27:09.333 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.333 DEBUG:boto:Data: 2023-10-12T00:27:09.333 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.334 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.335 DEBUG:boto:Params: {} 2023-10-12T00:27:09.335 DEBUG:boto:Token: None 2023-10-12T00:27:09.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=aREAq6CYlwscobe6hxBhqtoXwgZf5Ia 2023-10-12T00:27:09.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg= 2023-10-12T00:27:09.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:evbu3Qrh7BxeBCzdY51sp+kw7kg=', 'Content-Length': '0'} 2023-10-12T00:27:09.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aREAq6CYlwscobe6hxBhqtoXwgZf5Ia'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001f5e2f7e43885b3-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.382 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.383 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.383 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.384 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.384 DEBUG:boto:Method: GET 2023-10-12T00:27:09.384 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.385 DEBUG:boto:Data: 2023-10-12T00:27:09.385 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.386 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.387 DEBUG:boto:Params: {} 2023-10-12T00:27:09.387 DEBUG:boto:Token: None 2023-10-12T00:27:09.387 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e44ff4f8e981dd3-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.392 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.392 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.393 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.393 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.393 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.394 DEBUG:boto:Method: GET 2023-10-12T00:27:09.394 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.395 DEBUG:boto:Data: 2023-10-12T00:27:09.395 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.396 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.396 DEBUG:boto:Params: {} 2023-10-12T00:27:09.397 DEBUG:boto:Token: None 2023-10-12T00:27:09.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773ebdda134b3192-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.401 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.402 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.402 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.403 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.403 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.404 DEBUG:boto:Data: 2023-10-12T00:27:09.404 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.405 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.405 DEBUG:boto:Params: {} 2023-10-12T00:27:09.406 DEBUG:boto:Token: None 2023-10-12T00:27:09.406 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4c4efb6879b7b75-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.411 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.411 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.411 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.412 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.412 DEBUG:boto:Data: 2023-10-12T00:27:09.413 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.413 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.414 DEBUG:boto:Params: {} 2023-10-12T00:27:09.414 DEBUG:boto:Token: None 2023-10-12T00:27:09.415 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8bc0ce6a0f9c10c-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.420 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.421 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.421 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.421 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.422 DEBUG:boto:Method: GET 2023-10-12T00:27:09.422 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.423 DEBUG:boto:Data: 2023-10-12T00:27:09.423 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.424 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.425 DEBUG:boto:Params: {} 2023-10-12T00:27:09.425 DEBUG:boto:Token: None 2023-10-12T00:27:09.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.426 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a146607821b7fd4d-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.470 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.471 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.471 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.472 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.472 DEBUG:boto:Method: GET 2023-10-12T00:27:09.472 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.473 DEBUG:boto:Data: 2023-10-12T00:27:09.473 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.474 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.475 DEBUG:boto:Params: {} 2023-10-12T00:27:09.475 DEBUG:boto:Token: None 2023-10-12T00:27:09.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005fbcb0c93b05ee6-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.522 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.523 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.523 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.524 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.524 DEBUG:boto:Method: GET 2023-10-12T00:27:09.525 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.525 DEBUG:boto:Data: 2023-10-12T00:27:09.526 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.527 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.527 DEBUG:boto:Params: {} 2023-10-12T00:27:09.528 DEBUG:boto:Token: None 2023-10-12T00:27:09.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9214d036e0ccec7-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.532 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.533 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.534 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.534 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.534 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.535 DEBUG:boto:Method: GET 2023-10-12T00:27:09.535 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.536 DEBUG:boto:Data: 2023-10-12T00:27:09.536 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.537 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.537 DEBUG:boto:Params: {} 2023-10-12T00:27:09.538 DEBUG:boto:Token: None 2023-10-12T00:27:09.538 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c8fc035111691d-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.582 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.583 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.584 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.584 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.585 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.585 DEBUG:boto:Method: GET 2023-10-12T00:27:09.585 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.586 DEBUG:boto:Data: 2023-10-12T00:27:09.586 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.587 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.588 DEBUG:boto:Params: {} 2023-10-12T00:27:09.588 DEBUG:boto:Token: None 2023-10-12T00:27:09.589 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM= 2023-10-12T00:27:09.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM=', 'Content-Length': '0'} 2023-10-12T00:27:09.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095cafd80ac68af7b-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.593 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.593 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.594 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.594 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.595 DEBUG:boto:Method: GET 2023-10-12T00:27:09.595 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.596 DEBUG:boto:Data: 2023-10-12T00:27:09.596 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.597 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.597 DEBUG:boto:Params: {} 2023-10-12T00:27:09.598 DEBUG:boto:Token: None 2023-10-12T00:27:09.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM= 2023-10-12T00:27:09.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM=', 'Content-Length': '0'} 2023-10-12T00:27:09.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f88c31fa161d3158-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.602 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.602 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.603 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.603 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.604 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.604 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.605 DEBUG:boto:Data: 2023-10-12T00:27:09.605 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.606 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.606 DEBUG:boto:Params: {} 2023-10-12T00:27:09.607 DEBUG:boto:Token: None 2023-10-12T00:27:09.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dKLVAmBMkUGd/LwwhltIoTgRvws= 2023-10-12T00:27:09.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dKLVAmBMkUGd/LwwhltIoTgRvws=', 'Content-Length': '0'} 2023-10-12T00:27:09.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdf1d27ca69f9f28-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.611 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.611 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.611 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.612 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.612 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.613 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.613 DEBUG:boto:Data: 2023-10-12T00:27:09.613 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.614 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.615 DEBUG:boto:Params: {} 2023-10-12T00:27:09.615 DEBUG:boto:Token: None 2023-10-12T00:27:09.616 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dKLVAmBMkUGd/LwwhltIoTgRvws= 2023-10-12T00:27:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dKLVAmBMkUGd/LwwhltIoTgRvws=', 'Content-Length': '0'} 2023-10-12T00:27:09.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e0e5c30895dd84-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.620 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.620 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.620 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.621 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.621 DEBUG:boto:Method: GET 2023-10-12T00:27:09.622 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.622 DEBUG:boto:Data: 2023-10-12T00:27:09.622 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.623 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.624 DEBUG:boto:Params: {} 2023-10-12T00:27:09.624 DEBUG:boto:Token: None 2023-10-12T00:27:09.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM= 2023-10-12T00:27:09.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM=', 'Content-Length': '0'} 2023-10-12T00:27:09.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006044acf4f85cc729-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.628 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.629 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.629 DEBUG:boto:path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.629 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.630 DEBUG:boto:Method: GET 2023-10-12T00:27:09.630 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.631 DEBUG:boto:Data: 2023-10-12T00:27:09.631 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.632 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.633 DEBUG:boto:Params: {} 2023-10-12T00:27:09.633 DEBUG:boto:Token: None 2023-10-12T00:27:09.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8YF.vOWyk5Y0FN1lc5uqCulDrladR24 2023-10-12T00:27:09.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM= 2023-10-12T00:27:09.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkvzzR4xyL4Hwv7JAgPJb4GL9dM=', 'Content-Length': '0'} 2023-10-12T00:27:09.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8YF.vOWyk5Y0FN1lc5uqCulDrladR24'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e77ca508b27ecd89-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.678 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.679 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.679 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.680 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.680 DEBUG:boto:Method: GET 2023-10-12T00:27:09.681 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.681 DEBUG:boto:Data: 2023-10-12T00:27:09.681 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.682 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.683 DEBUG:boto:Params: {} 2023-10-12T00:27:09.683 DEBUG:boto:Token: None 2023-10-12T00:27:09.684 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bd851777e397a51-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.688 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.689 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.689 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.690 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.690 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.690 DEBUG:boto:Method: GET 2023-10-12T00:27:09.691 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.691 DEBUG:boto:Data: 2023-10-12T00:27:09.692 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.693 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.693 DEBUG:boto:Params: {} 2023-10-12T00:27:09.694 DEBUG:boto:Token: None 2023-10-12T00:27:09.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e9452309d094de-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.738 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.739 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.740 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.740 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.740 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.741 DEBUG:boto:Data: 2023-10-12T00:27:09.741 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.742 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.742 DEBUG:boto:Params: {} 2023-10-12T00:27:09.743 DEBUG:boto:Token: None 2023-10-12T00:27:09.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a46584a178401a3-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.748 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.749 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.749 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.750 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.750 DEBUG:boto:Data: 2023-10-12T00:27:09.750 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.751 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.751 DEBUG:boto:Params: {} 2023-10-12T00:27:09.752 DEBUG:boto:Token: None 2023-10-12T00:27:09.752 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bea0fb19fb09b43d-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.757 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.758 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.758 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.759 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.759 DEBUG:boto:Method: GET 2023-10-12T00:27:09.759 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.760 DEBUG:boto:Data: 2023-10-12T00:27:09.760 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.761 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.762 DEBUG:boto:Params: {} 2023-10-12T00:27:09.762 DEBUG:boto:Token: None 2023-10-12T00:27:09.763 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d4d8850454051b-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.806 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.807 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.807 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.808 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.808 DEBUG:boto:Method: GET 2023-10-12T00:27:09.809 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.809 DEBUG:boto:Data: 2023-10-12T00:27:09.809 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.810 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.811 DEBUG:boto:Params: {} 2023-10-12T00:27:09.811 DEBUG:boto:Token: None 2023-10-12T00:27:09.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003073848c1e38a976-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.858 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.859 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.859 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.860 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.860 DEBUG:boto:Method: GET 2023-10-12T00:27:09.861 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.861 DEBUG:boto:Data: 2023-10-12T00:27:09.861 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.862 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.863 DEBUG:boto:Params: {} 2023-10-12T00:27:09.863 DEBUG:boto:Token: None 2023-10-12T00:27:09.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001afad9fcaa0fd775-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.868 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.868 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.869 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.869 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.870 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.870 DEBUG:boto:Method: GET 2023-10-12T00:27:09.871 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.871 DEBUG:boto:Data: 2023-10-12T00:27:09.871 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.872 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.873 DEBUG:boto:Params: {} 2023-10-12T00:27:09.873 DEBUG:boto:Token: None 2023-10-12T00:27:09.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f3b77fe3598af39-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.878 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.879 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.880 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.880 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.880 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.881 DEBUG:boto:Method: GET 2023-10-12T00:27:09.881 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.882 DEBUG:boto:Data: 2023-10-12T00:27:09.882 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.883 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.883 DEBUG:boto:Params: {} 2023-10-12T00:27:09.884 DEBUG:boto:Token: None 2023-10-12T00:27:09.884 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM= 2023-10-12T00:27:09.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM=', 'Content-Length': '0'} 2023-10-12T00:27:09.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056e42455dc8be295-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.888 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.889 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.889 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.890 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.890 DEBUG:boto:Method: GET 2023-10-12T00:27:09.891 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.891 DEBUG:boto:Data: 2023-10-12T00:27:09.891 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.892 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.893 DEBUG:boto:Params: {} 2023-10-12T00:27:09.893 DEBUG:boto:Token: None 2023-10-12T00:27:09.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM= 2023-10-12T00:27:09.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM=', 'Content-Length': '0'} 2023-10-12T00:27:09.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2d7c7c49d9c10ef-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.897 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.898 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.898 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.899 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.899 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.900 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.900 DEBUG:boto:Data: 2023-10-12T00:27:09.900 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.901 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.901 DEBUG:boto:Params: {} 2023-10-12T00:27:09.902 DEBUG:boto:Token: None 2023-10-12T00:27:09.902 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:imsGC6oJcB8KgDz6NusSJeqMHB0= 2023-10-12T00:27:09.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:imsGC6oJcB8KgDz6NusSJeqMHB0=', 'Content-Length': '0'} 2023-10-12T00:27:09.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003283e7f019d1927c-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.906 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.906 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.907 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.907 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.908 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.908 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.908 DEBUG:boto:Data: 2023-10-12T00:27:09.909 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.909 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.910 DEBUG:boto:Params: {} 2023-10-12T00:27:09.910 DEBUG:boto:Token: None 2023-10-12T00:27:09.911 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:imsGC6oJcB8KgDz6NusSJeqMHB0= 2023-10-12T00:27:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:imsGC6oJcB8KgDz6NusSJeqMHB0=', 'Content-Length': '0'} 2023-10-12T00:27:09.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000654bd7e41767c40d-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.915 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.915 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.916 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.916 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.917 DEBUG:boto:Method: GET 2023-10-12T00:27:09.917 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.917 DEBUG:boto:Data: 2023-10-12T00:27:09.918 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.919 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.919 DEBUG:boto:Params: {} 2023-10-12T00:27:09.920 DEBUG:boto:Token: None 2023-10-12T00:27:09.920 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM= 2023-10-12T00:27:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM=', 'Content-Length': '0'} 2023-10-12T00:27:09.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e3a73df26334af3-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.924 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.924 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.925 DEBUG:boto:path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.925 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.926 DEBUG:boto:Method: GET 2023-10-12T00:27:09.926 DEBUG:boto:Path: /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.927 DEBUG:boto:Data: 2023-10-12T00:27:09.927 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.928 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.928 DEBUG:boto:Params: {} 2023-10-12T00:27:09.929 DEBUG:boto:Token: None 2023-10-12T00:27:09.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd 2023-10-12T00:27:09.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM= 2023-10-12T00:27:09.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLhznH3SDURtvxiqQCqJLOoI6qM=', 'Content-Length': '0'} 2023-10-12T00:27:09.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'cW5Ye2CI-TLxfd9wO06BlCDsesBA4Pd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059f99f49beac4c47-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.933 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.933 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.934 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.934 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.935 DEBUG:boto:Method: GET 2023-10-12T00:27:09.935 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.936 DEBUG:boto:Data: 2023-10-12T00:27:09.936 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.937 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.937 DEBUG:boto:Params: {} 2023-10-12T00:27:09.938 DEBUG:boto:Token: None 2023-10-12T00:27:09.938 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a4baca1133708a1-0065273d5d-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.943 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.943 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:09.944 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:09.944 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.945 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:09.945 DEBUG:boto:Method: GET 2023-10-12T00:27:09.945 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:09.946 DEBUG:boto:Data: 2023-10-12T00:27:09.946 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.947 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.947 DEBUG:boto:Params: {} 2023-10-12T00:27:09.948 DEBUG:boto:Token: None 2023-10-12T00:27:09.948 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/ 2023-10-12T00:27:09.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs= 2023-10-12T00:27:09.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wn2RQKjc0FZVy4SkNYqTq2cEKNs=', 'Content-Length': '0'} 2023-10-12T00:27:09.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4ec680325a9a63-0065273d5d-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.952 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:09.953 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.953 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.954 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.954 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.955 DEBUG:boto:Data: 2023-10-12T00:27:09.955 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.956 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.956 DEBUG:boto:Params: {} 2023-10-12T00:27:09.957 DEBUG:boto:Token: None 2023-10-12T00:27:09.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000372b8f7ebd063399-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.962 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.962 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.963 DEBUG:boto:Method: HEAD 2023-10-12T00:27:09.963 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:09.964 DEBUG:boto:Data: 2023-10-12T00:27:09.964 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:09.965 DEBUG:boto:Port: 8000 2023-10-12T00:27:09.965 DEBUG:boto:Params: {} 2023-10-12T00:27:09.966 DEBUG:boto:Token: None 2023-10-12T00:27:09.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj 2023-10-12T00:27:09.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ= 2023-10-12T00:27:09.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pcRdf9sKZBoIOn+s3ocomUTKFUQ=', 'Content-Length': '0'} 2023-10-12T00:27:09.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e126988fd53ecaf-0065273d5d-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:09.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:09.971 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:09.972 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:09.972 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.973 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.973 DEBUG:boto:Method: GET 2023-10-12T00:27:09.973 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.974 DEBUG:boto:Data: 2023-10-12T00:27:09.974 DEBUG:boto:Headers: {} 2023-10-12T00:27:09.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:09.975 DEBUG:boto:Port: 8001 2023-10-12T00:27:09.976 DEBUG:boto:Params: {} 2023-10-12T00:27:09.976 DEBUG:boto:Token: None 2023-10-12T00:27:09.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:09 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:09.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA= 2023-10-12T00:27:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C5ZHi3MtNducWig7PNU1YybZDqA=', 'Content-Length': '0'} 2023-10-12T00:27:09.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d702e49ca1d4d2d-0065273d5d-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.022 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.023 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.023 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.024 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.024 DEBUG:boto:Method: GET 2023-10-12T00:27:10.025 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.025 DEBUG:boto:Data: 2023-10-12T00:27:10.025 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.026 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.027 DEBUG:boto:Params: {} 2023-10-12T00:27:10.027 DEBUG:boto:Token: None 2023-10-12T00:27:10.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ffb253282a3a543-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.032 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.032 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.033 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.033 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.033 DEBUG:boto:Method: GET 2023-10-12T00:27:10.034 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.034 DEBUG:boto:Data: 2023-10-12T00:27:10.035 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.036 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.036 DEBUG:boto:Params: {} 2023-10-12T00:27:10.036 DEBUG:boto:Token: None 2023-10-12T00:27:10.037 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276b74c5e1d17d51-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.041 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.042 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.042 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.043 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.043 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.044 DEBUG:boto:Method: GET 2023-10-12T00:27:10.044 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.045 DEBUG:boto:Data: 2023-10-12T00:27:10.045 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.046 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.046 DEBUG:boto:Params: {} 2023-10-12T00:27:10.047 DEBUG:boto:Token: None 2023-10-12T00:27:10.047 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005bb347e73fa2f98-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.051 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.052 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.052 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.053 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.053 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.054 DEBUG:boto:Method: GET 2023-10-12T00:27:10.054 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.055 DEBUG:boto:Data: 2023-10-12T00:27:10.055 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.056 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.056 DEBUG:boto:Params: {} 2023-10-12T00:27:10.057 DEBUG:boto:Token: None 2023-10-12T00:27:10.057 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk= 2023-10-12T00:27:10.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk=', 'Content-Length': '0'} 2023-10-12T00:27:10.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000faf4be7fc83e648d-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.061 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.061 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.062 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.062 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.062 DEBUG:boto:Method: GET 2023-10-12T00:27:10.063 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.063 DEBUG:boto:Data: 2023-10-12T00:27:10.064 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.064 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.065 DEBUG:boto:Params: {} 2023-10-12T00:27:10.065 DEBUG:boto:Token: None 2023-10-12T00:27:10.066 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk= 2023-10-12T00:27:10.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk=', 'Content-Length': '0'} 2023-10-12T00:27:10.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000062970dd7cbc407f-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.070 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.070 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.071 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.071 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.071 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.072 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.072 DEBUG:boto:Data: 2023-10-12T00:27:10.073 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.073 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.074 DEBUG:boto:Params: {} 2023-10-12T00:27:10.074 DEBUG:boto:Token: None 2023-10-12T00:27:10.075 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gFGgv5oLdI+DZA49PIzSBuZcQm8= 2023-10-12T00:27:10.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gFGgv5oLdI+DZA49PIzSBuZcQm8=', 'Content-Length': '0'} 2023-10-12T00:27:10.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069f0367a0b51e22c-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.078 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.079 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.079 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.079 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.080 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.080 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.081 DEBUG:boto:Data: 2023-10-12T00:27:10.081 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.082 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.082 DEBUG:boto:Params: {} 2023-10-12T00:27:10.083 DEBUG:boto:Token: None 2023-10-12T00:27:10.083 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gFGgv5oLdI+DZA49PIzSBuZcQm8= 2023-10-12T00:27:10.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gFGgv5oLdI+DZA49PIzSBuZcQm8=', 'Content-Length': '0'} 2023-10-12T00:27:10.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaa8ac4e24f074c9-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.087 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.087 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.088 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.088 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.089 DEBUG:boto:Method: GET 2023-10-12T00:27:10.089 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.090 DEBUG:boto:Data: 2023-10-12T00:27:10.090 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.091 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.091 DEBUG:boto:Params: {} 2023-10-12T00:27:10.092 DEBUG:boto:Token: None 2023-10-12T00:27:10.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk= 2023-10-12T00:27:10.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk=', 'Content-Length': '0'} 2023-10-12T00:27:10.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe3b38ed4b43d410-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.095 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.096 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.096 DEBUG:boto:path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.097 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.097 DEBUG:boto:Method: GET 2023-10-12T00:27:10.098 DEBUG:boto:Path: /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.098 DEBUG:boto:Data: 2023-10-12T00:27:10.099 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.099 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.100 DEBUG:boto:Params: {} 2023-10-12T00:27:10.100 DEBUG:boto:Token: None 2023-10-12T00:27:10.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E 2023-10-12T00:27:10.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk= 2023-10-12T00:27:10.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AYY2hdLXz8Gi/a0cze9tDdpBETk=', 'Content-Length': '0'} 2023-10-12T00:27:10.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'AwL75t2G8eZE-9BVIOBGa7w0CBFJ56E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5168b93a1f43e32-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.104 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.105 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.105 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.106 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.106 DEBUG:boto:Method: GET 2023-10-12T00:27:10.107 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.107 DEBUG:boto:Data: 2023-10-12T00:27:10.107 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.108 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.109 DEBUG:boto:Params: {} 2023-10-12T00:27:10.109 DEBUG:boto:Token: None 2023-10-12T00:27:10.110 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005603a60dbb0888df-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.114 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.114 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.115 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.115 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.116 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.116 DEBUG:boto:Method: GET 2023-10-12T00:27:10.117 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.117 DEBUG:boto:Data: 2023-10-12T00:27:10.117 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.118 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.119 DEBUG:boto:Params: {} 2023-10-12T00:27:10.119 DEBUG:boto:Token: None 2023-10-12T00:27:10.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caecb980cb5dc147-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.124 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.125 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.125 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.126 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.126 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.126 DEBUG:boto:Data: 2023-10-12T00:27:10.127 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.128 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.128 DEBUG:boto:Params: {} 2023-10-12T00:27:10.129 DEBUG:boto:Token: None 2023-10-12T00:27:10.129 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b5d7a8c9ebca603-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.134 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.134 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.135 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.135 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.135 DEBUG:boto:Data: 2023-10-12T00:27:10.136 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.137 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.137 DEBUG:boto:Params: {} 2023-10-12T00:27:10.138 DEBUG:boto:Token: None 2023-10-12T00:27:10.138 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ac24afa6889a212-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.143 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.144 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.144 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.144 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.145 DEBUG:boto:Method: GET 2023-10-12T00:27:10.145 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.146 DEBUG:boto:Data: 2023-10-12T00:27:10.146 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.147 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.147 DEBUG:boto:Params: {} 2023-10-12T00:27:10.148 DEBUG:boto:Token: None 2023-10-12T00:27:10.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005511d35fb652979b-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.152 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.152 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.153 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.153 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.154 DEBUG:boto:Method: GET 2023-10-12T00:27:10.154 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.155 DEBUG:boto:Data: 2023-10-12T00:27:10.155 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.156 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.156 DEBUG:boto:Params: {} 2023-10-12T00:27:10.157 DEBUG:boto:Token: None 2023-10-12T00:27:10.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b2a1002005b5cbe-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.202 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.203 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.203 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.204 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.204 DEBUG:boto:Method: GET 2023-10-12T00:27:10.204 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.205 DEBUG:boto:Data: 2023-10-12T00:27:10.205 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.206 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.207 DEBUG:boto:Params: {} 2023-10-12T00:27:10.207 DEBUG:boto:Token: None 2023-10-12T00:27:10.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b7a5958ed4202b-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.212 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.212 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.213 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.213 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.214 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.214 DEBUG:boto:Method: GET 2023-10-12T00:27:10.215 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.215 DEBUG:boto:Data: 2023-10-12T00:27:10.215 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.216 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.217 DEBUG:boto:Params: {} 2023-10-12T00:27:10.217 DEBUG:boto:Token: None 2023-10-12T00:27:10.218 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6ae4003d92ecf7-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.222 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.223 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.223 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.223 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.224 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.224 DEBUG:boto:Method: GET 2023-10-12T00:27:10.225 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.225 DEBUG:boto:Data: 2023-10-12T00:27:10.226 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.226 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.227 DEBUG:boto:Params: {} 2023-10-12T00:27:10.227 DEBUG:boto:Token: None 2023-10-12T00:27:10.228 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c= 2023-10-12T00:27:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c=', 'Content-Length': '0'} 2023-10-12T00:27:10.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb3143b951e84d52-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.274 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.275 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.275 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.276 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.276 DEBUG:boto:Method: GET 2023-10-12T00:27:10.277 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.277 DEBUG:boto:Data: 2023-10-12T00:27:10.277 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.278 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.279 DEBUG:boto:Params: {} 2023-10-12T00:27:10.279 DEBUG:boto:Token: None 2023-10-12T00:27:10.279 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c= 2023-10-12T00:27:10.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c=', 'Content-Length': '0'} 2023-10-12T00:27:10.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004615d22f6cde5eb7-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.283 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.284 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.284 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.284 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.285 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.285 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.286 DEBUG:boto:Data: 2023-10-12T00:27:10.286 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.287 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.287 DEBUG:boto:Params: {} 2023-10-12T00:27:10.288 DEBUG:boto:Token: None 2023-10-12T00:27:10.288 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/3MQyUoB+yWYrgNshlUh0cHKfyQ= 2023-10-12T00:27:10.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/3MQyUoB+yWYrgNshlUh0cHKfyQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f761b3408101c99-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.291 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.292 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.292 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.292 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.293 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.293 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.294 DEBUG:boto:Data: 2023-10-12T00:27:10.294 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.295 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.295 DEBUG:boto:Params: {} 2023-10-12T00:27:10.296 DEBUG:boto:Token: None 2023-10-12T00:27:10.296 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/3MQyUoB+yWYrgNshlUh0cHKfyQ= 2023-10-12T00:27:10.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/3MQyUoB+yWYrgNshlUh0cHKfyQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e931216b8c677aa-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.300 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.300 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.301 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.301 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.302 DEBUG:boto:Method: GET 2023-10-12T00:27:10.302 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.303 DEBUG:boto:Data: 2023-10-12T00:27:10.303 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.304 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.304 DEBUG:boto:Params: {} 2023-10-12T00:27:10.305 DEBUG:boto:Token: None 2023-10-12T00:27:10.305 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.306 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c= 2023-10-12T00:27:10.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c=', 'Content-Length': '0'} 2023-10-12T00:27:10.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd6df66706e3c4dc-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.309 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.309 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.310 DEBUG:boto:path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.310 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.311 DEBUG:boto:Method: GET 2023-10-12T00:27:10.311 DEBUG:boto:Path: /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.311 DEBUG:boto:Data: 2023-10-12T00:27:10.312 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.313 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.313 DEBUG:boto:Params: {} 2023-10-12T00:27:10.313 DEBUG:boto:Token: None 2023-10-12T00:27:10.314 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=UKcEH2brpi8Ybi0biABxFN1eyql3Efg 2023-10-12T00:27:10.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c= 2023-10-12T00:27:10.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnTJtvlQzwFLPk6NjIGXIrfzH4c=', 'Content-Length': '0'} 2023-10-12T00:27:10.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'UKcEH2brpi8Ybi0biABxFN1eyql3Efg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f756915ccb75656-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.358 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.359 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.359 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.360 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.360 DEBUG:boto:Method: GET 2023-10-12T00:27:10.360 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.361 DEBUG:boto:Data: 2023-10-12T00:27:10.361 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.362 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.362 DEBUG:boto:Params: {} 2023-10-12T00:27:10.363 DEBUG:boto:Token: None 2023-10-12T00:27:10.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf3fd6a8f367742-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.368 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.369 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.369 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.370 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.370 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.370 DEBUG:boto:Method: GET 2023-10-12T00:27:10.371 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.371 DEBUG:boto:Data: 2023-10-12T00:27:10.372 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.373 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.373 DEBUG:boto:Params: {} 2023-10-12T00:27:10.373 DEBUG:boto:Token: None 2023-10-12T00:27:10.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a696f8096beb04-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.378 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.379 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.379 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.380 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.380 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.380 DEBUG:boto:Data: 2023-10-12T00:27:10.381 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.382 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.382 DEBUG:boto:Params: {} 2023-10-12T00:27:10.383 DEBUG:boto:Token: None 2023-10-12T00:27:10.383 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b97d74d5d1101350-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.388 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.388 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.389 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.389 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.389 DEBUG:boto:Data: 2023-10-12T00:27:10.390 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.391 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.391 DEBUG:boto:Params: {} 2023-10-12T00:27:10.392 DEBUG:boto:Token: None 2023-10-12T00:27:10.392 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000552f758268db1d32-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.397 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.397 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.398 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.398 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.398 DEBUG:boto:Method: GET 2023-10-12T00:27:10.399 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.399 DEBUG:boto:Data: 2023-10-12T00:27:10.400 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.401 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.401 DEBUG:boto:Params: {} 2023-10-12T00:27:10.401 DEBUG:boto:Token: None 2023-10-12T00:27:10.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bfba3397b444b88-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.446 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.447 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.447 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.448 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.448 DEBUG:boto:Method: GET 2023-10-12T00:27:10.449 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.449 DEBUG:boto:Data: 2023-10-12T00:27:10.449 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.450 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.451 DEBUG:boto:Params: {} 2023-10-12T00:27:10.451 DEBUG:boto:Token: None 2023-10-12T00:27:10.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080d85114f0073d20-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.455 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.456 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.456 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.457 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.457 DEBUG:boto:Method: GET 2023-10-12T00:27:10.457 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.458 DEBUG:boto:Data: 2023-10-12T00:27:10.458 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.459 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.459 DEBUG:boto:Params: {} 2023-10-12T00:27:10.460 DEBUG:boto:Token: None 2023-10-12T00:27:10.460 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499bead7103f393f-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.466 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.466 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.467 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.467 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.468 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.468 DEBUG:boto:Method: GET 2023-10-12T00:27:10.469 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.469 DEBUG:boto:Data: 2023-10-12T00:27:10.469 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.470 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.471 DEBUG:boto:Params: {} 2023-10-12T00:27:10.471 DEBUG:boto:Token: None 2023-10-12T00:27:10.471 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.472 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005444452f4b24ab43-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.475 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.476 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.477 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.477 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.477 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.478 DEBUG:boto:Method: GET 2023-10-12T00:27:10.478 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.479 DEBUG:boto:Data: 2023-10-12T00:27:10.479 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.480 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.480 DEBUG:boto:Params: {} 2023-10-12T00:27:10.481 DEBUG:boto:Token: None 2023-10-12T00:27:10.481 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g= 2023-10-12T00:27:10.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g=', 'Content-Length': '0'} 2023-10-12T00:27:10.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002408a10cd4e89b5e-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.485 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.485 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.486 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.486 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.487 DEBUG:boto:Method: GET 2023-10-12T00:27:10.487 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.487 DEBUG:boto:Data: 2023-10-12T00:27:10.488 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.489 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.489 DEBUG:boto:Params: {} 2023-10-12T00:27:10.489 DEBUG:boto:Token: None 2023-10-12T00:27:10.490 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g= 2023-10-12T00:27:10.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g=', 'Content-Length': '0'} 2023-10-12T00:27:10.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dc78657356e092d-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.534 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.535 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.535 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.535 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.536 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.536 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.537 DEBUG:boto:Data: 2023-10-12T00:27:10.537 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.538 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.538 DEBUG:boto:Params: {} 2023-10-12T00:27:10.539 DEBUG:boto:Token: None 2023-10-12T00:27:10.539 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1CrkWPCceQ4lyRPocJfhizt4GSo= 2023-10-12T00:27:10.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1CrkWPCceQ4lyRPocJfhizt4GSo=', 'Content-Length': '0'} 2023-10-12T00:27:10.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aaea4cdd4d1cd09-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.543 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.543 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.543 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.544 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.544 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.544 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.545 DEBUG:boto:Data: 2023-10-12T00:27:10.545 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.546 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.546 DEBUG:boto:Params: {} 2023-10-12T00:27:10.547 DEBUG:boto:Token: None 2023-10-12T00:27:10.547 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1CrkWPCceQ4lyRPocJfhizt4GSo= 2023-10-12T00:27:10.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1CrkWPCceQ4lyRPocJfhizt4GSo=', 'Content-Length': '0'} 2023-10-12T00:27:10.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cf213f69e3344de-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.551 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.551 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.552 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.552 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.553 DEBUG:boto:Method: GET 2023-10-12T00:27:10.553 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.553 DEBUG:boto:Data: 2023-10-12T00:27:10.554 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.554 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.555 DEBUG:boto:Params: {} 2023-10-12T00:27:10.555 DEBUG:boto:Token: None 2023-10-12T00:27:10.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g= 2023-10-12T00:27:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g=', 'Content-Length': '0'} 2023-10-12T00:27:10.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3caa361ea31ca0f-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.559 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.560 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.560 DEBUG:boto:path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.560 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.561 DEBUG:boto:Method: GET 2023-10-12T00:27:10.561 DEBUG:boto:Path: /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.561 DEBUG:boto:Data: 2023-10-12T00:27:10.562 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.563 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.563 DEBUG:boto:Params: {} 2023-10-12T00:27:10.563 DEBUG:boto:Token: None 2023-10-12T00:27:10.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX 2023-10-12T00:27:10.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g= 2023-10-12T00:27:10.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8+ZnffE+5I0ALIaXj/sIoW767g=', 'Content-Length': '0'} 2023-10-12T00:27:10.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '-9jPLZgkVg9RO4DZDUc3t-6tLUzBAoX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000027bf3b02587c512-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.567 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.568 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.568 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.569 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.569 DEBUG:boto:Method: GET 2023-10-12T00:27:10.569 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.570 DEBUG:boto:Data: 2023-10-12T00:27:10.570 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.571 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.572 DEBUG:boto:Params: {} 2023-10-12T00:27:10.572 DEBUG:boto:Token: None 2023-10-12T00:27:10.572 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.573 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596cad9c7642a70f-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.618 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.619 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.619 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.620 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.620 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.620 DEBUG:boto:Method: GET 2023-10-12T00:27:10.621 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.621 DEBUG:boto:Data: 2023-10-12T00:27:10.622 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.622 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.623 DEBUG:boto:Params: {} 2023-10-12T00:27:10.623 DEBUG:boto:Token: None 2023-10-12T00:27:10.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5fcdc43ffd13f84-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.628 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.629 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.629 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.629 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.630 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.630 DEBUG:boto:Data: 2023-10-12T00:27:10.630 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.631 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.632 DEBUG:boto:Params: {} 2023-10-12T00:27:10.632 DEBUG:boto:Token: None 2023-10-12T00:27:10.632 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec28fa713c03fc09-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.637 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.637 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.638 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.638 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.638 DEBUG:boto:Data: 2023-10-12T00:27:10.639 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.640 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.640 DEBUG:boto:Params: {} 2023-10-12T00:27:10.640 DEBUG:boto:Token: None 2023-10-12T00:27:10.641 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000537e8eff65c37098-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.646 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.646 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.647 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.647 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.647 DEBUG:boto:Method: GET 2023-10-12T00:27:10.648 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.648 DEBUG:boto:Data: 2023-10-12T00:27:10.648 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.649 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.650 DEBUG:boto:Params: {} 2023-10-12T00:27:10.650 DEBUG:boto:Token: None 2023-10-12T00:27:10.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000536d8453a86056ce-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.654 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.654 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.655 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.655 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.656 DEBUG:boto:Method: GET 2023-10-12T00:27:10.656 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.656 DEBUG:boto:Data: 2023-10-12T00:27:10.657 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.658 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.658 DEBUG:boto:Params: {} 2023-10-12T00:27:10.658 DEBUG:boto:Token: None 2023-10-12T00:27:10.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc701f887890a16-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.663 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.663 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.663 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.664 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.664 DEBUG:boto:Method: GET 2023-10-12T00:27:10.664 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.665 DEBUG:boto:Data: 2023-10-12T00:27:10.665 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.666 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.666 DEBUG:boto:Params: {} 2023-10-12T00:27:10.667 DEBUG:boto:Token: None 2023-10-12T00:27:10.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287405db10896f18-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.673 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.673 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.674 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.674 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.675 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.675 DEBUG:boto:Method: GET 2023-10-12T00:27:10.675 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.676 DEBUG:boto:Data: 2023-10-12T00:27:10.676 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.677 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.677 DEBUG:boto:Params: {} 2023-10-12T00:27:10.678 DEBUG:boto:Token: None 2023-10-12T00:27:10.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6802e8c38ed5a9-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.682 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.683 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.684 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.684 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.684 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.685 DEBUG:boto:Method: GET 2023-10-12T00:27:10.685 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.686 DEBUG:boto:Data: 2023-10-12T00:27:10.686 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.687 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.687 DEBUG:boto:Params: {} 2023-10-12T00:27:10.688 DEBUG:boto:Token: None 2023-10-12T00:27:10.688 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA= 2023-10-12T00:27:10.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA=', 'Content-Length': '0'} 2023-10-12T00:27:10.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009012cb72438cc5ec-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.692 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.693 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.693 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.694 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.694 DEBUG:boto:Method: GET 2023-10-12T00:27:10.695 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.695 DEBUG:boto:Data: 2023-10-12T00:27:10.696 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.697 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.697 DEBUG:boto:Params: {} 2023-10-12T00:27:10.698 DEBUG:boto:Token: None 2023-10-12T00:27:10.698 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA= 2023-10-12T00:27:10.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA=', 'Content-Length': '0'} 2023-10-12T00:27:10.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029e9137d62598910-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.702 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.702 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.703 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.703 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.704 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.704 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.704 DEBUG:boto:Data: 2023-10-12T00:27:10.705 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.705 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.706 DEBUG:boto:Params: {} 2023-10-12T00:27:10.706 DEBUG:boto:Token: None 2023-10-12T00:27:10.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fY018EEa3Cj3BbYiYGJ+xBLft5Y= 2023-10-12T00:27:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fY018EEa3Cj3BbYiYGJ+xBLft5Y=', 'Content-Length': '0'} 2023-10-12T00:27:10.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000591a5b6dbe6919ca-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.710 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.711 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.711 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.711 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.712 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.712 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.713 DEBUG:boto:Data: 2023-10-12T00:27:10.713 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.714 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.714 DEBUG:boto:Params: {} 2023-10-12T00:27:10.715 DEBUG:boto:Token: None 2023-10-12T00:27:10.715 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.716 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fY018EEa3Cj3BbYiYGJ+xBLft5Y= 2023-10-12T00:27:10.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fY018EEa3Cj3BbYiYGJ+xBLft5Y=', 'Content-Length': '0'} 2023-10-12T00:27:10.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009511fb82cae92151-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.719 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.719 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.720 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.720 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.721 DEBUG:boto:Method: GET 2023-10-12T00:27:10.721 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.721 DEBUG:boto:Data: 2023-10-12T00:27:10.722 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.722 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.723 DEBUG:boto:Params: {} 2023-10-12T00:27:10.723 DEBUG:boto:Token: None 2023-10-12T00:27:10.724 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA= 2023-10-12T00:27:10.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA=', 'Content-Length': '0'} 2023-10-12T00:27:10.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdaa70280aabf4f2-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.727 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.727 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.728 DEBUG:boto:path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.728 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.729 DEBUG:boto:Method: GET 2023-10-12T00:27:10.729 DEBUG:boto:Path: /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.729 DEBUG:boto:Data: 2023-10-12T00:27:10.730 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.731 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.731 DEBUG:boto:Params: {} 2023-10-12T00:27:10.732 DEBUG:boto:Token: None 2023-10-12T00:27:10.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=bcLIwmkxUK6LImzWGZigFgqMDTZIkF3 2023-10-12T00:27:10.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA= 2023-10-12T00:27:10.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:31zWhaYeWoTV6oxRymNhIN7HaQA=', 'Content-Length': '0'} 2023-10-12T00:27:10.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'bcLIwmkxUK6LImzWGZigFgqMDTZIkF3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c76f3be191cf10bd-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.735 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.736 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.736 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.736 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.737 DEBUG:boto:Method: GET 2023-10-12T00:27:10.737 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.738 DEBUG:boto:Data: 2023-10-12T00:27:10.738 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.739 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.739 DEBUG:boto:Params: {} 2023-10-12T00:27:10.740 DEBUG:boto:Token: None 2023-10-12T00:27:10.740 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ba0dc9b0d9d892-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.786 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.787 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.787 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.788 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.788 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.789 DEBUG:boto:Method: GET 2023-10-12T00:27:10.789 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.789 DEBUG:boto:Data: 2023-10-12T00:27:10.790 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.791 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.791 DEBUG:boto:Params: {} 2023-10-12T00:27:10.791 DEBUG:boto:Token: None 2023-10-12T00:27:10.792 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8e813363522515-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.796 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.796 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.797 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.797 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.798 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.798 DEBUG:boto:Data: 2023-10-12T00:27:10.799 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.799 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.800 DEBUG:boto:Params: {} 2023-10-12T00:27:10.800 DEBUG:boto:Token: None 2023-10-12T00:27:10.800 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003be5cfe1e9cf7e10-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.805 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.805 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.806 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.806 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.806 DEBUG:boto:Data: 2023-10-12T00:27:10.807 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.808 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.808 DEBUG:boto:Params: {} 2023-10-12T00:27:10.809 DEBUG:boto:Token: None 2023-10-12T00:27:10.809 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c84a30d7dc758e4-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.814 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.814 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.814 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.815 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.815 DEBUG:boto:Method: GET 2023-10-12T00:27:10.816 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.816 DEBUG:boto:Data: 2023-10-12T00:27:10.817 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.817 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.818 DEBUG:boto:Params: {} 2023-10-12T00:27:10.818 DEBUG:boto:Token: None 2023-10-12T00:27:10.819 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081548e1ea72d0084-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.822 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.822 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.823 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.823 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.824 DEBUG:boto:Method: GET 2023-10-12T00:27:10.824 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.824 DEBUG:boto:Data: 2023-10-12T00:27:10.825 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.826 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.826 DEBUG:boto:Params: {} 2023-10-12T00:27:10.826 DEBUG:boto:Token: None 2023-10-12T00:27:10.827 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec4ca68c07658cfc-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.870 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.871 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.873 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.874 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.874 DEBUG:boto:Method: GET 2023-10-12T00:27:10.875 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.875 DEBUG:boto:Data: 2023-10-12T00:27:10.875 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.876 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.877 DEBUG:boto:Params: {} 2023-10-12T00:27:10.877 DEBUG:boto:Token: None 2023-10-12T00:27:10.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000940664050f09dbd2-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.882 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.882 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.883 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.883 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.884 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.884 DEBUG:boto:Method: GET 2023-10-12T00:27:10.884 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.885 DEBUG:boto:Data: 2023-10-12T00:27:10.885 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.886 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.886 DEBUG:boto:Params: {} 2023-10-12T00:27:10.887 DEBUG:boto:Token: None 2023-10-12T00:27:10.887 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e0592068782055-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.891 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.892 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.893 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.893 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.893 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.894 DEBUG:boto:Method: GET 2023-10-12T00:27:10.894 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.895 DEBUG:boto:Data: 2023-10-12T00:27:10.895 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.896 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.896 DEBUG:boto:Params: {} 2023-10-12T00:27:10.897 DEBUG:boto:Token: None 2023-10-12T00:27:10.897 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE= 2023-10-12T00:27:10.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE=', 'Content-Length': '0'} 2023-10-12T00:27:10.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cca931693d3465c4-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.901 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.902 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.902 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.902 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.903 DEBUG:boto:Method: GET 2023-10-12T00:27:10.903 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.904 DEBUG:boto:Data: 2023-10-12T00:27:10.904 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.905 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.905 DEBUG:boto:Params: {} 2023-10-12T00:27:10.905 DEBUG:boto:Token: None 2023-10-12T00:27:10.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE= 2023-10-12T00:27:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE=', 'Content-Length': '0'} 2023-10-12T00:27:10.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7aac3ca99909554-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.909 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.910 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.910 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.910 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.911 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.911 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.912 DEBUG:boto:Data: 2023-10-12T00:27:10.912 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.913 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.913 DEBUG:boto:Params: {} 2023-10-12T00:27:10.914 DEBUG:boto:Token: None 2023-10-12T00:27:10.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXKE8YRWR97RIa6W7kt8NtUWmsc= 2023-10-12T00:27:10.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXKE8YRWR97RIa6W7kt8NtUWmsc=', 'Content-Length': '0'} 2023-10-12T00:27:10.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093e4b18fc6fa66de-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.917 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.918 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.918 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.919 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.919 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.919 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.920 DEBUG:boto:Data: 2023-10-12T00:27:10.920 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.921 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.921 DEBUG:boto:Params: {} 2023-10-12T00:27:10.922 DEBUG:boto:Token: None 2023-10-12T00:27:10.922 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXKE8YRWR97RIa6W7kt8NtUWmsc= 2023-10-12T00:27:10.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXKE8YRWR97RIa6W7kt8NtUWmsc=', 'Content-Length': '0'} 2023-10-12T00:27:10.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094f3ce36d102253a-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.926 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.926 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.926 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.927 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.927 DEBUG:boto:Method: GET 2023-10-12T00:27:10.928 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.928 DEBUG:boto:Data: 2023-10-12T00:27:10.928 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.929 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.930 DEBUG:boto:Params: {} 2023-10-12T00:27:10.930 DEBUG:boto:Token: None 2023-10-12T00:27:10.931 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE= 2023-10-12T00:27:10.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE=', 'Content-Length': '0'} 2023-10-12T00:27:10.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087f73dc15213f3cb-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.934 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.934 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.935 DEBUG:boto:path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.935 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.936 DEBUG:boto:Method: GET 2023-10-12T00:27:10.936 DEBUG:boto:Path: /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.936 DEBUG:boto:Data: 2023-10-12T00:27:10.937 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.938 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.938 DEBUG:boto:Params: {} 2023-10-12T00:27:10.938 DEBUG:boto:Token: None 2023-10-12T00:27:10.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=aeNQEx4UDrOBog-IThXBKYcMJlbKvu3 2023-10-12T00:27:10.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE= 2023-10-12T00:27:10.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IDmOlo+QbGD3XZte3W9z0YnW3mE=', 'Content-Length': '0'} 2023-10-12T00:27:10.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'aeNQEx4UDrOBog-IThXBKYcMJlbKvu3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5fb735f86bca415-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.942 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.943 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.943 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.944 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.944 DEBUG:boto:Method: GET 2023-10-12T00:27:10.944 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.945 DEBUG:boto:Data: 2023-10-12T00:27:10.945 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.946 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.946 DEBUG:boto:Params: {} 2023-10-12T00:27:10.947 DEBUG:boto:Token: None 2023-10-12T00:27:10.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862a5b35f4ed54c5-0065273d5e-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.951 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.952 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:10.952 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:10.953 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.953 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:10.954 DEBUG:boto:Method: GET 2023-10-12T00:27:10.954 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:10.955 DEBUG:boto:Data: 2023-10-12T00:27:10.955 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.956 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.956 DEBUG:boto:Params: {} 2023-10-12T00:27:10.956 DEBUG:boto:Token: None 2023-10-12T00:27:10.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/ 2023-10-12T00:27:10.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso= 2023-10-12T00:27:10.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R5XTc3IQ9rZBehUjIIxVV6/nTso=', 'Content-Length': '0'} 2023-10-12T00:27:10.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a888e178c2f49064-0065273d5e-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.961 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:10.961 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.962 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.962 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.963 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.963 DEBUG:boto:Data: 2023-10-12T00:27:10.963 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.964 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.964 DEBUG:boto:Params: {} 2023-10-12T00:27:10.965 DEBUG:boto:Token: None 2023-10-12T00:27:10.965 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e0e3918b75615c5-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.970 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.970 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.971 DEBUG:boto:Method: HEAD 2023-10-12T00:27:10.971 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:10.972 DEBUG:boto:Data: 2023-10-12T00:27:10.972 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:10.973 DEBUG:boto:Port: 8000 2023-10-12T00:27:10.973 DEBUG:boto:Params: {} 2023-10-12T00:27:10.974 DEBUG:boto:Token: None 2023-10-12T00:27:10.974 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj 2023-10-12T00:27:10.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ= 2023-10-12T00:27:10.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNWpbra8QV1sztL+nFCPSlNqppQ=', 'Content-Length': '0'} 2023-10-12T00:27:10.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011229e572b089d5f-0065273d5e-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:10.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:10.979 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:10.980 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:10.980 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.980 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.981 DEBUG:boto:Method: GET 2023-10-12T00:27:10.981 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.982 DEBUG:boto:Data: 2023-10-12T00:27:10.982 DEBUG:boto:Headers: {} 2023-10-12T00:27:10.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:10.983 DEBUG:boto:Port: 8001 2023-10-12T00:27:10.983 DEBUG:boto:Params: {} 2023-10-12T00:27:10.984 DEBUG:boto:Token: None 2023-10-12T00:27:10.984 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:10 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:10.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo= 2023-10-12T00:27:10.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8pS0qs4pi8mt0AfyFNCwX69lqo=', 'Content-Length': '0'} 2023-10-12T00:27:10.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c40fc7aebfe585f-0065273d5e-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.030 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.031 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.031 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.032 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.032 DEBUG:boto:Method: GET 2023-10-12T00:27:11.033 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.033 DEBUG:boto:Data: 2023-10-12T00:27:11.033 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.034 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.035 DEBUG:boto:Params: {} 2023-10-12T00:27:11.035 DEBUG:boto:Token: None 2023-10-12T00:27:11.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000218a295c651aae4a-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.040 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.040 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.040 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.041 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.041 DEBUG:boto:Method: GET 2023-10-12T00:27:11.042 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.042 DEBUG:boto:Data: 2023-10-12T00:27:11.042 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.043 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.044 DEBUG:boto:Params: {} 2023-10-12T00:27:11.044 DEBUG:boto:Token: None 2023-10-12T00:27:11.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4996ea8b7fe44e2-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.049 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.049 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.050 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.050 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.050 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.051 DEBUG:boto:Method: GET 2023-10-12T00:27:11.051 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.051 DEBUG:boto:Data: 2023-10-12T00:27:11.052 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.053 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.053 DEBUG:boto:Params: {} 2023-10-12T00:27:11.053 DEBUG:boto:Token: None 2023-10-12T00:27:11.054 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca67546f6216d6d5-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.098 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.099 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.100 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.100 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.100 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.101 DEBUG:boto:Method: GET 2023-10-12T00:27:11.101 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.102 DEBUG:boto:Data: 2023-10-12T00:27:11.102 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.103 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.103 DEBUG:boto:Params: {} 2023-10-12T00:27:11.104 DEBUG:boto:Token: None 2023-10-12T00:27:11.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0= 2023-10-12T00:27:11.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0=', 'Content-Length': '0'} 2023-10-12T00:27:11.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b777826cb04c43e3-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.108 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.109 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.109 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.110 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.110 DEBUG:boto:Method: GET 2023-10-12T00:27:11.110 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.111 DEBUG:boto:Data: 2023-10-12T00:27:11.111 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.112 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.112 DEBUG:boto:Params: {} 2023-10-12T00:27:11.113 DEBUG:boto:Token: None 2023-10-12T00:27:11.113 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0= 2023-10-12T00:27:11.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0=', 'Content-Length': '0'} 2023-10-12T00:27:11.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e17ea40ffc93560f-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.117 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.117 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.118 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.118 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.119 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.119 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.119 DEBUG:boto:Data: 2023-10-12T00:27:11.120 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.121 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.121 DEBUG:boto:Params: {} 2023-10-12T00:27:11.121 DEBUG:boto:Token: None 2023-10-12T00:27:11.122 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZR8Gwu85dO5CR6/GuzZcfl7rUEE= 2023-10-12T00:27:11.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZR8Gwu85dO5CR6/GuzZcfl7rUEE=', 'Content-Length': '0'} 2023-10-12T00:27:11.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d46bd704964cab4-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.125 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.126 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.126 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.126 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.127 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.127 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.128 DEBUG:boto:Data: 2023-10-12T00:27:11.128 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.129 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.129 DEBUG:boto:Params: {} 2023-10-12T00:27:11.129 DEBUG:boto:Token: None 2023-10-12T00:27:11.130 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.130 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZR8Gwu85dO5CR6/GuzZcfl7rUEE= 2023-10-12T00:27:11.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZR8Gwu85dO5CR6/GuzZcfl7rUEE=', 'Content-Length': '0'} 2023-10-12T00:27:11.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000089457b663c48c0-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.134 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.134 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.135 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.135 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.136 DEBUG:boto:Method: GET 2023-10-12T00:27:11.136 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.137 DEBUG:boto:Data: 2023-10-12T00:27:11.137 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.138 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.138 DEBUG:boto:Params: {} 2023-10-12T00:27:11.139 DEBUG:boto:Token: None 2023-10-12T00:27:11.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0= 2023-10-12T00:27:11.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0=', 'Content-Length': '0'} 2023-10-12T00:27:11.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d29a5d41de7c79e-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.143 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.143 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.143 DEBUG:boto:path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.144 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.144 DEBUG:boto:Method: GET 2023-10-12T00:27:11.145 DEBUG:boto:Path: /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.145 DEBUG:boto:Data: 2023-10-12T00:27:11.145 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.146 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.147 DEBUG:boto:Params: {} 2023-10-12T00:27:11.147 DEBUG:boto:Token: None 2023-10-12T00:27:11.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=l1NxRoVSzbfCB0g0y82beIxhCWs2wv3 2023-10-12T00:27:11.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0= 2023-10-12T00:27:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hxNnZh+I1DQ5ADCdufg30hUttF0=', 'Content-Length': '0'} 2023-10-12T00:27:11.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'l1NxRoVSzbfCB0g0y82beIxhCWs2wv3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003cf9ac95637dea8-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.151 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.152 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.152 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.153 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.153 DEBUG:boto:Method: GET 2023-10-12T00:27:11.153 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.154 DEBUG:boto:Data: 2023-10-12T00:27:11.154 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.155 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.155 DEBUG:boto:Params: {} 2023-10-12T00:27:11.156 DEBUG:boto:Token: None 2023-10-12T00:27:11.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005190c3ac27faedfa-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.202 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.203 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.203 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.204 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.204 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.204 DEBUG:boto:Method: GET 2023-10-12T00:27:11.205 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.205 DEBUG:boto:Data: 2023-10-12T00:27:11.206 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.207 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.207 DEBUG:boto:Params: {} 2023-10-12T00:27:11.207 DEBUG:boto:Token: None 2023-10-12T00:27:11.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa232b844d733483-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.212 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.213 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.213 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.213 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.214 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.214 DEBUG:boto:Data: 2023-10-12T00:27:11.215 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.215 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.216 DEBUG:boto:Params: {} 2023-10-12T00:27:11.216 DEBUG:boto:Token: None 2023-10-12T00:27:11.217 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099933860ba6a581c-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.221 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.222 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.222 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.222 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.223 DEBUG:boto:Data: 2023-10-12T00:27:11.223 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.224 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.224 DEBUG:boto:Params: {} 2023-10-12T00:27:11.225 DEBUG:boto:Token: None 2023-10-12T00:27:11.225 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007867ffc305f27b1d-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.230 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.230 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.231 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.231 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.232 DEBUG:boto:Method: GET 2023-10-12T00:27:11.232 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.232 DEBUG:boto:Data: 2023-10-12T00:27:11.233 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.234 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.234 DEBUG:boto:Params: {} 2023-10-12T00:27:11.234 DEBUG:boto:Token: None 2023-10-12T00:27:11.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005812722c65c3274c-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.239 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.239 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.239 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.240 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.240 DEBUG:boto:Method: GET 2023-10-12T00:27:11.241 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.241 DEBUG:boto:Data: 2023-10-12T00:27:11.242 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.243 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.243 DEBUG:boto:Params: {} 2023-10-12T00:27:11.244 DEBUG:boto:Token: None 2023-10-12T00:27:11.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041bdde7aa9bc48f0-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.248 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.248 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.249 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.249 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.249 DEBUG:boto:Method: GET 2023-10-12T00:27:11.250 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.250 DEBUG:boto:Data: 2023-10-12T00:27:11.251 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.252 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.252 DEBUG:boto:Params: {} 2023-10-12T00:27:11.252 DEBUG:boto:Token: None 2023-10-12T00:27:11.253 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000140265218271b65b-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.257 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.258 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.258 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.259 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.259 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.260 DEBUG:boto:Method: GET 2023-10-12T00:27:11.260 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.260 DEBUG:boto:Data: 2023-10-12T00:27:11.261 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.262 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.262 DEBUG:boto:Params: {} 2023-10-12T00:27:11.263 DEBUG:boto:Token: None 2023-10-12T00:27:11.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c0a56172510cb2-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.310 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.311 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.312 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.312 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.313 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.313 DEBUG:boto:Method: GET 2023-10-12T00:27:11.314 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.314 DEBUG:boto:Data: 2023-10-12T00:27:11.315 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.316 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.316 DEBUG:boto:Params: {} 2023-10-12T00:27:11.317 DEBUG:boto:Token: None 2023-10-12T00:27:11.317 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU= 2023-10-12T00:27:11.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU=', 'Content-Length': '0'} 2023-10-12T00:27:11.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000375ed562889bd210-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.321 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.322 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.322 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.322 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.323 DEBUG:boto:Method: GET 2023-10-12T00:27:11.323 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.324 DEBUG:boto:Data: 2023-10-12T00:27:11.324 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.325 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.326 DEBUG:boto:Params: {} 2023-10-12T00:27:11.326 DEBUG:boto:Token: None 2023-10-12T00:27:11.327 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU= 2023-10-12T00:27:11.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU=', 'Content-Length': '0'} 2023-10-12T00:27:11.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003b151eb4e2d345d-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.331 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.331 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.331 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.332 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.332 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.333 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.333 DEBUG:boto:Data: 2023-10-12T00:27:11.333 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.334 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.334 DEBUG:boto:Params: {} 2023-10-12T00:27:11.335 DEBUG:boto:Token: None 2023-10-12T00:27:11.335 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZC/NK3g0ZgH5nJI2dOzHdu2DUDg= 2023-10-12T00:27:11.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZC/NK3g0ZgH5nJI2dOzHdu2DUDg=', 'Content-Length': '0'} 2023-10-12T00:27:11.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008bfdad1346e7e09-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.339 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.339 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.340 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.340 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.341 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.341 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.341 DEBUG:boto:Data: 2023-10-12T00:27:11.342 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.343 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.343 DEBUG:boto:Params: {} 2023-10-12T00:27:11.343 DEBUG:boto:Token: None 2023-10-12T00:27:11.344 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZC/NK3g0ZgH5nJI2dOzHdu2DUDg= 2023-10-12T00:27:11.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZC/NK3g0ZgH5nJI2dOzHdu2DUDg=', 'Content-Length': '0'} 2023-10-12T00:27:11.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a612c7941d912451-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.347 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.348 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.348 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.349 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.349 DEBUG:boto:Method: GET 2023-10-12T00:27:11.349 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.350 DEBUG:boto:Data: 2023-10-12T00:27:11.350 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.351 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.351 DEBUG:boto:Params: {} 2023-10-12T00:27:11.352 DEBUG:boto:Token: None 2023-10-12T00:27:11.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU= 2023-10-12T00:27:11.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU=', 'Content-Length': '0'} 2023-10-12T00:27:11.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003187e0785925426f-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.356 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.356 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.357 DEBUG:boto:path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.357 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.357 DEBUG:boto:Method: GET 2023-10-12T00:27:11.358 DEBUG:boto:Path: /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.358 DEBUG:boto:Data: 2023-10-12T00:27:11.359 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.359 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.360 DEBUG:boto:Params: {} 2023-10-12T00:27:11.360 DEBUG:boto:Token: None 2023-10-12T00:27:11.361 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=A1GhQw.uNyVkATOZA23KGcLCah.1lK7 2023-10-12T00:27:11.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU= 2023-10-12T00:27:11.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d3sIacNBcEs0JKNbrtdiaTheupU=', 'Content-Length': '0'} 2023-10-12T00:27:11.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'A1GhQw.uNyVkATOZA23KGcLCah.1lK7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a8b4cea14cb7bd-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.364 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.364 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.365 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.365 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.365 DEBUG:boto:Method: GET 2023-10-12T00:27:11.366 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.366 DEBUG:boto:Data: 2023-10-12T00:27:11.367 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.368 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.368 DEBUG:boto:Params: {} 2023-10-12T00:27:11.368 DEBUG:boto:Token: None 2023-10-12T00:27:11.369 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c31c0961bf0142b-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.414 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.415 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.415 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.416 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.416 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.417 DEBUG:boto:Method: GET 2023-10-12T00:27:11.417 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.417 DEBUG:boto:Data: 2023-10-12T00:27:11.418 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.419 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.419 DEBUG:boto:Params: {} 2023-10-12T00:27:11.419 DEBUG:boto:Token: None 2023-10-12T00:27:11.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034db328b723ab856-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.424 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.424 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.425 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.425 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.426 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.426 DEBUG:boto:Data: 2023-10-12T00:27:11.426 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.427 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.428 DEBUG:boto:Params: {} 2023-10-12T00:27:11.428 DEBUG:boto:Token: None 2023-10-12T00:27:11.429 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005433be9c51808bb4-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.433 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.433 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.434 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.434 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.435 DEBUG:boto:Data: 2023-10-12T00:27:11.435 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.436 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.436 DEBUG:boto:Params: {} 2023-10-12T00:27:11.437 DEBUG:boto:Token: None 2023-10-12T00:27:11.437 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea95bb14ee0795d4-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.442 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.443 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.443 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.444 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.444 DEBUG:boto:Method: GET 2023-10-12T00:27:11.444 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.445 DEBUG:boto:Data: 2023-10-12T00:27:11.445 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.446 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.447 DEBUG:boto:Params: {} 2023-10-12T00:27:11.447 DEBUG:boto:Token: None 2023-10-12T00:27:11.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c35a47a22ac3f3b2-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.451 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.452 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.452 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.453 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.453 DEBUG:boto:Method: GET 2023-10-12T00:27:11.453 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.454 DEBUG:boto:Data: 2023-10-12T00:27:11.454 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.455 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.455 DEBUG:boto:Params: {} 2023-10-12T00:27:11.456 DEBUG:boto:Token: None 2023-10-12T00:27:11.456 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007034f8f38ab64c39-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.502 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.503 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.503 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.503 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.504 DEBUG:boto:Method: GET 2023-10-12T00:27:11.504 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.505 DEBUG:boto:Data: 2023-10-12T00:27:11.505 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.506 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.506 DEBUG:boto:Params: {} 2023-10-12T00:27:11.507 DEBUG:boto:Token: None 2023-10-12T00:27:11.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003564621ecd3af406-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.511 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.512 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.512 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.513 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.513 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.513 DEBUG:boto:Method: GET 2023-10-12T00:27:11.514 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.514 DEBUG:boto:Data: 2023-10-12T00:27:11.515 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.518 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.518 DEBUG:boto:Params: {} 2023-10-12T00:27:11.519 DEBUG:boto:Token: None 2023-10-12T00:27:11.519 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d10af5df09237d2-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.523 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.524 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.524 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.525 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.525 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.526 DEBUG:boto:Method: GET 2023-10-12T00:27:11.526 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.527 DEBUG:boto:Data: 2023-10-12T00:27:11.527 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.528 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.528 DEBUG:boto:Params: {} 2023-10-12T00:27:11.529 DEBUG:boto:Token: None 2023-10-12T00:27:11.529 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw= 2023-10-12T00:27:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw=', 'Content-Length': '0'} 2023-10-12T00:27:11.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c86f6196ef3e76f-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.533 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.533 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.534 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.534 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.535 DEBUG:boto:Method: GET 2023-10-12T00:27:11.535 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.535 DEBUG:boto:Data: 2023-10-12T00:27:11.536 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.537 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.537 DEBUG:boto:Params: {} 2023-10-12T00:27:11.537 DEBUG:boto:Token: None 2023-10-12T00:27:11.538 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw= 2023-10-12T00:27:11.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw=', 'Content-Length': '0'} 2023-10-12T00:27:11.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c128f1bfc98e3679-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.542 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.542 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.542 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.543 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.543 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.544 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.544 DEBUG:boto:Data: 2023-10-12T00:27:11.544 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.545 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.545 DEBUG:boto:Params: {} 2023-10-12T00:27:11.546 DEBUG:boto:Token: None 2023-10-12T00:27:11.546 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7TsXsmCC70gaN1f311adBhlcCrw= 2023-10-12T00:27:11.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7TsXsmCC70gaN1f311adBhlcCrw=', 'Content-Length': '0'} 2023-10-12T00:27:11.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d1ffdbb8a181349-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.550 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.550 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.551 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.551 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.552 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.552 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.552 DEBUG:boto:Data: 2023-10-12T00:27:11.553 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.553 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.554 DEBUG:boto:Params: {} 2023-10-12T00:27:11.554 DEBUG:boto:Token: None 2023-10-12T00:27:11.555 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.555 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7TsXsmCC70gaN1f311adBhlcCrw= 2023-10-12T00:27:11.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7TsXsmCC70gaN1f311adBhlcCrw=', 'Content-Length': '0'} 2023-10-12T00:27:11.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fdde1f880c78861-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.558 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.559 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.559 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.560 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.560 DEBUG:boto:Method: GET 2023-10-12T00:27:11.560 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.561 DEBUG:boto:Data: 2023-10-12T00:27:11.561 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.562 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.562 DEBUG:boto:Params: {} 2023-10-12T00:27:11.563 DEBUG:boto:Token: None 2023-10-12T00:27:11.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw= 2023-10-12T00:27:11.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw=', 'Content-Length': '0'} 2023-10-12T00:27:11.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1f0d09b2a5f681b-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.567 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.567 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.568 DEBUG:boto:path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.568 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.569 DEBUG:boto:Method: GET 2023-10-12T00:27:11.569 DEBUG:boto:Path: /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.569 DEBUG:boto:Data: 2023-10-12T00:27:11.570 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.571 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.571 DEBUG:boto:Params: {} 2023-10-12T00:27:11.572 DEBUG:boto:Token: None 2023-10-12T00:27:11.572 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq 2023-10-12T00:27:11.572 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw= 2023-10-12T00:27:11.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owMuN1t26nbEswQs4POjCEQKtzw=', 'Content-Length': '0'} 2023-10-12T00:27:11.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', 'x3qNoJF7sAKv4z2L03tHhlOnXeMpWhq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042635a8d880fe018-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.576 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.576 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.576 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.577 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.577 DEBUG:boto:Method: GET 2023-10-12T00:27:11.578 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.578 DEBUG:boto:Data: 2023-10-12T00:27:11.579 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.579 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.580 DEBUG:boto:Params: {} 2023-10-12T00:27:11.580 DEBUG:boto:Token: None 2023-10-12T00:27:11.581 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f5e13f4b570371-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.626 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.627 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.627 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.628 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.628 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.629 DEBUG:boto:Method: GET 2023-10-12T00:27:11.629 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.629 DEBUG:boto:Data: 2023-10-12T00:27:11.630 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.631 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.631 DEBUG:boto:Params: {} 2023-10-12T00:27:11.632 DEBUG:boto:Token: None 2023-10-12T00:27:11.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f57b1df6f613817-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.636 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.637 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.637 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.638 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.638 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.638 DEBUG:boto:Data: 2023-10-12T00:27:11.639 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.640 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.640 DEBUG:boto:Params: {} 2023-10-12T00:27:11.640 DEBUG:boto:Token: None 2023-10-12T00:27:11.641 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fa082f9a76ce0e4-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.645 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.646 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.646 DEBUG:boto:Method: HEAD 2023-10-12T00:27:11.647 DEBUG:boto:Path: /wafmoa-13/obj 2023-10-12T00:27:11.647 DEBUG:boto:Data: 2023-10-12T00:27:11.648 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.648 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.649 DEBUG:boto:Params: {} 2023-10-12T00:27:11.649 DEBUG:boto:Token: None 2023-10-12T00:27:11.650 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj 2023-10-12T00:27:11.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM= 2023-10-12T00:27:11.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1TcjMQ78FjljURE1v6RPmzn/7SM=', 'Content-Length': '0'} 2023-10-12T00:27:11.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ebaadb188d8bd09-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:11.655 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.655 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.656 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.656 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.657 DEBUG:boto:Method: GET 2023-10-12T00:27:11.657 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.657 DEBUG:boto:Data: 2023-10-12T00:27:11.658 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.659 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.659 DEBUG:boto:Params: {} 2023-10-12T00:27:11.660 DEBUG:boto:Token: None 2023-10-12T00:27:11.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5d9ade7f4edcb60-0065273d5f-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.663 DEBUG:boto:path=/wafmoa-13/obj 2023-10-12T00:27:11.664 DEBUG:boto:auth_path=/wafmoa-13/obj 2023-10-12T00:27:11.664 DEBUG:boto:path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.665 DEBUG:boto:auth_path=/wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.665 DEBUG:boto:Method: GET 2023-10-12T00:27:11.665 DEBUG:boto:Path: /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.666 DEBUG:boto:Data: 2023-10-12T00:27:11.666 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.667 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.668 DEBUG:boto:Params: {} 2023-10-12T00:27:11.668 DEBUG:boto:Token: None 2023-10-12T00:27:11.668 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/obj?versionId=8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC 2023-10-12T00:27:11.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg= 2023-10-12T00:27:11.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX8osBWn13+ZHvHSZ3+q3EC57Pg=', 'Content-Length': '0'} 2023-10-12T00:27:11.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:26:26 GMT'), ('x-amz-version-id', '8iO9WVhOTn5yHQyoYcTMJRuEPOy5rcC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cef556bb5e506979-0065273d5f-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.714 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.715 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.715 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.715 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.716 DEBUG:boto:Method: GET 2023-10-12T00:27:11.716 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.717 DEBUG:boto:Data: 2023-10-12T00:27:11.717 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:11.718 DEBUG:boto:Port: 8001 2023-10-12T00:27:11.718 DEBUG:boto:Params: {} 2023-10-12T00:27:11.719 DEBUG:boto:Token: None 2023-10-12T00:27:11.719 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1953bd132eac6d-0065273d5f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.723 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.724 DEBUG:boto:path=/wafmoa-13/ 2023-10-12T00:27:11.724 DEBUG:boto:auth_path=/wafmoa-13/ 2023-10-12T00:27:11.725 DEBUG:boto:path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.725 DEBUG:boto:auth_path=/wafmoa-13/?prefix=obj 2023-10-12T00:27:11.726 DEBUG:boto:Method: GET 2023-10-12T00:27:11.726 DEBUG:boto:Path: /wafmoa-13/?prefix=obj 2023-10-12T00:27:11.727 DEBUG:boto:Data: 2023-10-12T00:27:11.727 DEBUG:boto:Headers: {} 2023-10-12T00:27:11.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:11.728 DEBUG:boto:Port: 8000 2023-10-12T00:27:11.728 DEBUG:boto:Params: {} 2023-10-12T00:27:11.729 DEBUG:boto:Token: None 2023-10-12T00:27:11.729 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:11 GMT /wafmoa-13/ 2023-10-12T00:27:11.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ= 2023-10-12T00:27:11.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HwuAbtX0E/e0BLnBwlk5Do2/3iQ=', 'Content-Length': '0'} 2023-10-12T00:27:11.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab30872451313930-0065273d5f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:11.733 DEBUG:boto:b'wafmoa-13obj1000falseobj2023-10-12T00:26:26.793Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:11.734 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-13 zones={a2, a1} 2023-10-12T00:27:11.734 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T00:27:11.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.736 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:11.792 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:11.792 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:11.843 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:27:11.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:27:11.844 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:27:11.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001491", 2023-10-12T00:27:11.845 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:48.058428Z" 2023-10-12T00:27:11.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.845 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:27:11.846 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:27:11.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-12T00:27:11.846 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:03.293604Z" 2023-10-12T00:27:11.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.847 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:27:11.848 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:11.848 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:27:11.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.849 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:11.902 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:11.902 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:11.942 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:11.943 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:11.943 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:11.943 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:11.943 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:11.944 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:27:11.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.944 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:11.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.944 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:11.945 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:11.945 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:11.945 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:27:11.945 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:11.945 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:11.946 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:11.946 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:27:11.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.947 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:11.947 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:11.947 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:11.947 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001491", 2023-10-12T00:27:11.947 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:11.948 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:11.948 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:11.948 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:48.058428Z" 2023-10-12T00:27:11.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.949 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:11.949 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:11.949 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:11.949 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:27:11.950 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:11.950 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:11.950 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:11.950 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:27:11.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:11.951 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:11.951 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:11.952 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:11.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-12T00:27:11.952 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:11.952 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.293604Z" 2023-10-12T00:27:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:27:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:27:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:11Z" 2023-10-12T00:27:11.955 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:11.956 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:23.283784Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001491", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:48.058428Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:23.327950Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.293604Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:27:11Z" } 2023-10-12T00:27:11.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:23.283784Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001491', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:48.058428Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:23.327950Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.293604Z'}}] 2023-10-12T00:27:11.957 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:11.957 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000001491', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:11.958 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:27:11.958 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T00:27:11.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:11.959 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:12.007 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:12.007 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:12.052 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:27:12.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.052 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:27:12.053 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:27:12.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.053 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-10-12T00:27:12.053 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:48.022636Z" 2023-10-12T00:27:12.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.054 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:27:12.054 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:27:12.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.055 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-12T00:27:12.055 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:43.251081Z" 2023-10-12T00:27:12.055 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.056 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:27:12.057 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002196', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:12.057 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:27:12.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.058 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:12.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:12.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:12.152 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:27:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:27:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:27:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-10-12T00:27:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:48.022636Z" 2023-10-12T00:27:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:27:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:27:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:27:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-12T00:27:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:43.251081Z" 2023-10-12T00:27:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:27:12.164 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:27:12.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.164 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:12Z" 2023-10-12T00:27:12.164 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:12.165 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002196", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:48.022636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:43.251081Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:27:12Z" } 2023-10-12T00:27:12.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002196', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:48.022636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:43.251081Z'}}] 2023-10-12T00:27:12.166 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002196', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:12.167 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000002196', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:27:12.167 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:27:12.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-10-12T00:27:12.169 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:12.170 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:12.170 DEBUG:boto:Method: PUT 2023-10-12T00:27:12.171 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:27:12.171 DEBUG:boto:Data: 2023-10-12T00:27:12.171 DEBUG:boto:Headers: {} 2023-10-12T00:27:12.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:12.172 DEBUG:boto:Port: 8000 2023-10-12T00:27:12.173 DEBUG:boto:Params: {} 2023-10-12T00:27:12.173 DEBUG:boto:Token: None 2023-10-12T00:27:12.173 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:27:12 GMT /wafmoa-14/ 2023-10-12T00:27:12.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kMuUabpjvXviQa5zfh9o3m6pd6k= 2023-10-12T00:27:12.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kMuUabpjvXviQa5zfh9o3m6pd6k=', 'Content-Length': '0'} 2023-10-12T00:27:12.190 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ccb96d20108747b-0065273d60-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:12.191 DEBUG:rgw_multi.tests:created bucket=wafmoa-14 2023-10-12T00:27:12.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.192 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:12.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:12.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:12.276 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:27:12.287 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:27:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:12.287 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:27:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:12.288 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:27:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070432.189919_115.1", 2023-10-12T00:27:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:27:12.189919Z" 2023-10-12T00:27:12.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:27:12.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.291 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:27:12.292 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070432.189919_115.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:12.292 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:27:12.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:12.294 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:12.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:12.370+0000 7f9e25c28780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:12.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:12.371+0000 7f9e25c28780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:12.382 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:12.383 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:12.383 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:12.383 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:12.383 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:12.383 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:27:12.384 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:12.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.384 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:12.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.385 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:12.385 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.385 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:12.385 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:12.385 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.386 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:27:12.386 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.386 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:27:12.386 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:12.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.387 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:12.387 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.387 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:12.388 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:12.388 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.388 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.388 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.389 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:27:12.389 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:12.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.390 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:12.390 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.390 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:12.390 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070432.183057_112.1", 2023-10-12T00:27:12.390 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.391 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.391 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.391 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:27:12.183057Z", 2023-10-12T00:27:12.391 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:12.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:12.392 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:12.392 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:12.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:12.393 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:12.393 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:12.393 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:12.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:12.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:27:12.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:12.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:12.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:12.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.395 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:12.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:27:12.395 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:27:12.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:12.395 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:12Z" 2023-10-12T00:27:12.396 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:12.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070432.183057_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:27:12.183057Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:27:12Z" } 2023-10-12T00:27:12.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070432.183057_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:27:12.183057Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:27:12.398 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070432.189919_115.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:12.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070432.183057_112.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:12.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070432.189919_115.1 target=1_1697070432.183057_112.1 2023-10-12T00:27:17.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:17.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:17.401 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:17.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:17.458 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:17.487+0000 7fd14e46a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:17.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:17.489+0000 7fd14e46a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:17.501 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:17.502 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:17.502 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:17.502 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:17.502 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:17.503 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:27:17.503 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:17.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.503 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:17.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:17.504 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:17.504 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:17.504 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:17.504 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:17.505 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:17.505 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:27:17.505 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:17.505 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:27:17.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:17.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:17.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:17.506 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:17.507 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:17.507 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:17.507 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:17.507 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:17.508 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:17.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:17.508 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:27:17.508 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:17.509 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:17.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:17.509 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:17.510 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:17.510 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:17.510 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070432.189919_115.1", 2023-10-12T00:27:17.510 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:17.511 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:17.511 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:17.511 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:27:12.189919Z", 2023-10-12T00:27:17.511 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:17.511 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:17.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:17.512 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:17.512 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:17.512 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:17.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:17.513 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:17.513 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:17.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:17.513 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:27:17.514 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:17.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:17.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:17.514 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:17.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.515 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:17.515 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:27:17.516 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:27:17.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:17.516 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:17Z" 2023-10-12T00:27:17.516 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:17.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070432.189919_115.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:27:12.189919Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:27:17Z" } 2023-10-12T00:27:17.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070432.189919_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:27:12.189919Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:27:17.518 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070432.189919_115.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:17.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070432.189919_115.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:17.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:27:17.520 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:17.520 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:17.520 DEBUG:boto:Method: HEAD 2023-10-12T00:27:17.521 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:27:17.521 DEBUG:boto:Data: 2023-10-12T00:27:17.522 DEBUG:boto:Headers: {} 2023-10-12T00:27:17.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:17.522 DEBUG:boto:Port: 8000 2023-10-12T00:27:17.523 DEBUG:boto:Params: {} 2023-10-12T00:27:17.523 DEBUG:boto:Token: None 2023-10-12T00:27:17.524 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:17 GMT /wafmoa-14/ 2023-10-12T00:27:17.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UTASi0aRIftApMb2rypQGomJn6w= 2023-10-12T00:27:17.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UTASi0aRIftApMb2rypQGomJn6w=', 'Content-Length': '0'} 2023-10-12T00:27:17.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de83766397a39ed-0065273d65-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:17.528 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:17.528 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:17.528 DEBUG:boto:Method: PUT 2023-10-12T00:27:17.529 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:17.529 DEBUG:boto:Data: 2023-10-12T00:27:17.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:27:17.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:17.530 DEBUG:boto:Port: 8000 2023-10-12T00:27:17.531 DEBUG:boto:Params: {} 2023-10-12T00:27:17.531 DEBUG:boto:Token: None 2023-10-12T00:27:17.532 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:27:17 GMT /wafmoa-14/obj 2023-10-12T00:27:17.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nQNxfxxgWpp4+8YZ/6BPvd1N3WQ= 2023-10-12T00:27:17.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:27:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nQNxfxxgWpp4+8YZ/6BPvd1N3WQ='} 2023-10-12T00:27:17.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084ba88dfd1f842ca-0065273d65-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:17.539 DEBUG:rgw_multi.tests:created initial version id=None 2023-10-12T00:27:17.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-14 2023-10-12T00:27:17.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:17.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:17.540 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:27:17.598 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:17.598 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:17.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:17.633+0000 7fdaf1546780 1 waiting to reach incremental sync.. 2023-10-12T00:27:22.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:22.635+0000 7fdaf1546780 1 waiting to reach incremental sync.. 2023-10-12T00:27:27.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:27.636+0000 7fdaf1546780 1 bucket sync caught up with source: 2023-10-12T00:27:27.637 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.8.6, , ] 2023-10-12T00:27:27.637 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.8.6, , ] 2023-10-12T00:27:27.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:27.637+0000 7fdaf1546780 0 bucket checkpoint complete 2023-10-12T00:27:27.652 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-14 2023-10-12T00:27:27.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:27.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:27.654 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:27:27.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:27.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:27.753+0000 7f8833e2f780 1 waiting to reach incremental sync.. 2023-10-12T00:27:32.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:32.753+0000 7f8833e2f780 1 waiting to reach incremental sync.. 2023-10-12T00:27:37.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:37.753+0000 7f8833e2f780 1 waiting to reach incremental sync.. 2023-10-12T00:27:42.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:42.754+0000 7f8833e2f780 1 waiting to reach incremental sync.. 2023-10-12T00:27:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:47.756+0000 7f8833e2f780 1 bucket sync caught up with source: 2023-10-12T00:27:47.757 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:27:47.757 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:27:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:47.756+0000 7f8833e2f780 0 bucket checkpoint complete 2023-10-12T00:27:47.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:27:47.772 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.773 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.773 DEBUG:boto:Method: HEAD 2023-10-12T00:27:47.774 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:27:47.774 DEBUG:boto:Data: 2023-10-12T00:27:47.774 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.775 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.776 DEBUG:boto:Params: {} 2023-10-12T00:27:47.776 DEBUG:boto:Token: None 2023-10-12T00:27:47.777 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/ 2023-10-12T00:27:47.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QTjXSyCc8JJlnbz3ZD1qqV/21E= 2023-10-12T00:27:47.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QTjXSyCc8JJlnbz3ZD1qqV/21E=', 'Content-Length': '0'} 2023-10-12T00:27:47.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f376d0662d13fa4-0065273d83-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.781 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.781 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.782 DEBUG:boto:Method: HEAD 2023-10-12T00:27:47.782 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:27:47.783 DEBUG:boto:Data: 2023-10-12T00:27:47.783 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.784 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.784 DEBUG:boto:Params: {} 2023-10-12T00:27:47.785 DEBUG:boto:Token: None 2023-10-12T00:27:47.785 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/ 2023-10-12T00:27:47.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QTjXSyCc8JJlnbz3ZD1qqV/21E= 2023-10-12T00:27:47.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QTjXSyCc8JJlnbz3ZD1qqV/21E=', 'Content-Length': '0'} 2023-10-12T00:27:47.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010f11cdb4776fcfa-0065273d83-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.789 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:27:47.790 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.790 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.790 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.791 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.791 DEBUG:boto:Method: GET 2023-10-12T00:27:47.792 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.792 DEBUG:boto:Data: 2023-10-12T00:27:47.792 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.793 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.794 DEBUG:boto:Params: {} 2023-10-12T00:27:47.794 DEBUG:boto:Token: None 2023-10-12T00:27:47.795 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/?versions 2023-10-12T00:27:47.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI= 2023-10-12T00:27:47.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI=', 'Content-Length': '0'} 2023-10-12T00:27:47.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17f409b8cdb966a-0065273d83-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.799 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:47.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:27:47.800 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.801 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.801 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.802 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.802 DEBUG:boto:Method: GET 2023-10-12T00:27:47.803 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.803 DEBUG:boto:Data: 2023-10-12T00:27:47.803 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.804 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.805 DEBUG:boto:Params: {} 2023-10-12T00:27:47.805 DEBUG:boto:Token: None 2023-10-12T00:27:47.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/?versions 2023-10-12T00:27:47.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI= 2023-10-12T00:27:47.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI=', 'Content-Length': '0'} 2023-10-12T00:27:47.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb350307bde2c587-0065273d83-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.813 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:27:47.814 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.814 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.814 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.815 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.815 DEBUG:boto:Method: GET 2023-10-12T00:27:47.816 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.816 DEBUG:boto:Data: 2023-10-12T00:27:47.816 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.817 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.818 DEBUG:boto:Params: {} 2023-10-12T00:27:47.818 DEBUG:boto:Token: None 2023-10-12T00:27:47.819 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/?versions 2023-10-12T00:27:47.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI= 2023-10-12T00:27:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI=', 'Content-Length': '0'} 2023-10-12T00:27:47.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d7ecf8baae40fb-0065273d83-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.823 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.824 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.824 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.825 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.825 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.826 DEBUG:boto:Method: GET 2023-10-12T00:27:47.826 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:27:47.827 DEBUG:boto:Data: 2023-10-12T00:27:47.827 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.828 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.828 DEBUG:boto:Params: {} 2023-10-12T00:27:47.829 DEBUG:boto:Token: None 2023-10-12T00:27:47.829 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/?versions 2023-10-12T00:27:47.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI= 2023-10-12T00:27:47.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kkjcso0JTWzNKC/oBWPF627qeoI=', 'Content-Length': '0'} 2023-10-12T00:27:47.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c08749fce1fa298-0065273d83-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.874 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:47.876 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.876 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.877 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.877 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.877 DEBUG:boto:Method: GET 2023-10-12T00:27:47.878 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.878 DEBUG:boto:Data: 2023-10-12T00:27:47.879 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.880 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.880 DEBUG:boto:Params: {} 2023-10-12T00:27:47.880 DEBUG:boto:Token: None 2023-10-12T00:27:47.881 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE= 2023-10-12T00:27:47.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE=', 'Content-Length': '0'} 2023-10-12T00:27:47.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1b2b02c6dd0c7e3-0065273d83-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.885 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.885 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.886 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.886 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.887 DEBUG:boto:Method: GET 2023-10-12T00:27:47.887 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.888 DEBUG:boto:Data: 2023-10-12T00:27:47.888 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.889 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.890 DEBUG:boto:Params: {} 2023-10-12T00:27:47.890 DEBUG:boto:Token: None 2023-10-12T00:27:47.891 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE= 2023-10-12T00:27:47.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE=', 'Content-Length': '0'} 2023-10-12T00:27:47.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df25f9f76fe75d6c-0065273d83-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.894 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.895 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.895 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.896 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.896 DEBUG:boto:Method: HEAD 2023-10-12T00:27:47.897 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.897 DEBUG:boto:Data: 2023-10-12T00:27:47.897 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.898 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.899 DEBUG:boto:Params: {} 2023-10-12T00:27:47.899 DEBUG:boto:Token: None 2023-10-12T00:27:47.900 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CSHPKQqx+ZcKtt+VQCkoEkyTiV8= 2023-10-12T00:27:47.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CSHPKQqx+ZcKtt+VQCkoEkyTiV8=', 'Content-Length': '0'} 2023-10-12T00:27:47.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000459545647fa9ca85-0065273d83-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.903 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.903 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.904 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.904 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.905 DEBUG:boto:Method: HEAD 2023-10-12T00:27:47.905 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.905 DEBUG:boto:Data: 2023-10-12T00:27:47.906 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.907 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.907 DEBUG:boto:Params: {} 2023-10-12T00:27:47.908 DEBUG:boto:Token: None 2023-10-12T00:27:47.908 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.909 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CSHPKQqx+ZcKtt+VQCkoEkyTiV8= 2023-10-12T00:27:47.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CSHPKQqx+ZcKtt+VQCkoEkyTiV8=', 'Content-Length': '0'} 2023-10-12T00:27:47.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a5a506fd544ed0e-0065273d83-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:47.912 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.912 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.913 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.913 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.914 DEBUG:boto:Method: GET 2023-10-12T00:27:47.914 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.915 DEBUG:boto:Data: 2023-10-12T00:27:47.915 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.916 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.916 DEBUG:boto:Params: {} 2023-10-12T00:27:47.917 DEBUG:boto:Token: None 2023-10-12T00:27:47.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.918 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE= 2023-10-12T00:27:47.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE=', 'Content-Length': '0'} 2023-10-12T00:27:47.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4a86f91e9cabcb3-0065273d83-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.921 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.921 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.922 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.922 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.923 DEBUG:boto:Method: GET 2023-10-12T00:27:47.923 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.923 DEBUG:boto:Data: 2023-10-12T00:27:47.924 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.925 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.925 DEBUG:boto:Params: {} 2023-10-12T00:27:47.926 DEBUG:boto:Token: None 2023-10-12T00:27:47.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:27:47.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE= 2023-10-12T00:27:47.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0dfiDdvZrbO8Ck1YhnljOc4iNE=', 'Content-Length': '0'} 2023-10-12T00:27:47.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009227d2f6a0798036-0065273d83-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.930 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.930 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.931 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:47.931 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:47.932 DEBUG:boto:Method: GET 2023-10-12T00:27:47.932 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:27:47.933 DEBUG:boto:Data: 2023-10-12T00:27:47.933 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.934 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.934 DEBUG:boto:Params: {} 2023-10-12T00:27:47.935 DEBUG:boto:Token: None 2023-10-12T00:27:47.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/ 2023-10-12T00:27:47.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWrlIkGEEvHgH93zao/K0MHB4oI= 2023-10-12T00:27:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWrlIkGEEvHgH93zao/K0MHB4oI=', 'Content-Length': '0'} 2023-10-12T00:27:47.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001896db594dda8f3e-0065273d83-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.982 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.983 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:47.984 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:47.984 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:47.984 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:47.985 DEBUG:boto:Method: GET 2023-10-12T00:27:47.985 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:27:47.986 DEBUG:boto:Data: 2023-10-12T00:27:47.986 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:47.987 DEBUG:boto:Port: 8000 2023-10-12T00:27:47.988 DEBUG:boto:Params: {} 2023-10-12T00:27:47.988 DEBUG:boto:Token: None 2023-10-12T00:27:47.989 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/ 2023-10-12T00:27:47.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWrlIkGEEvHgH93zao/K0MHB4oI= 2023-10-12T00:27:47.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWrlIkGEEvHgH93zao/K0MHB4oI=', 'Content-Length': '0'} 2023-10-12T00:27:47.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088a2bbdac52e7af-0065273d83-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:47.993 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:47.993 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:47.994 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:47.994 DEBUG:boto:Method: HEAD 2023-10-12T00:27:47.994 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:47.995 DEBUG:boto:Data: 2023-10-12T00:27:47.995 DEBUG:boto:Headers: {} 2023-10-12T00:27:47.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:47.996 DEBUG:boto:Port: 8001 2023-10-12T00:27:47.996 DEBUG:boto:Params: {} 2023-10-12T00:27:47.997 DEBUG:boto:Token: None 2023-10-12T00:27:47.997 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:47 GMT /wafmoa-14/obj 2023-10-12T00:27:47.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5WcQKVCYIGgBStxhIEcA4xBRDZE= 2023-10-12T00:27:47.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5WcQKVCYIGgBStxhIEcA4xBRDZE=', 'Content-Length': '0'} 2023-10-12T00:27:48.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b4d145815b1eb35-0065273d83-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.001 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:48.001 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:48.002 DEBUG:boto:Method: HEAD 2023-10-12T00:27:48.002 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:48.002 DEBUG:boto:Data: 2023-10-12T00:27:48.003 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:48.004 DEBUG:boto:Port: 8000 2023-10-12T00:27:48.004 DEBUG:boto:Params: {} 2023-10-12T00:27:48.004 DEBUG:boto:Token: None 2023-10-12T00:27:48.005 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/obj 2023-10-12T00:27:48.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YREz2IZI9zor+ifJzzyDj6VHt+w= 2023-10-12T00:27:48.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YREz2IZI9zor+ifJzzyDj6VHt+w=', 'Content-Length': '0'} 2023-10-12T00:27:48.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007652795867f11eb1-0065273d84-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:27:48.009 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:48.009 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:48.010 DEBUG:boto:Method: GET 2023-10-12T00:27:48.010 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:48.010 DEBUG:boto:Data: 2023-10-12T00:27:48.011 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:48.012 DEBUG:boto:Port: 8001 2023-10-12T00:27:48.012 DEBUG:boto:Params: {} 2023-10-12T00:27:48.013 DEBUG:boto:Token: None 2023-10-12T00:27:48.013 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/obj 2023-10-12T00:27:48.014 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ykDh+bSSYdLhNAvMnVY7XTGs3M= 2023-10-12T00:27:48.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ykDh+bSSYdLhNAvMnVY7XTGs3M=', 'Content-Length': '0'} 2023-10-12T00:27:48.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6ef02824f629c1e-0065273d84-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.017 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:48.017 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:48.018 DEBUG:boto:Method: GET 2023-10-12T00:27:48.018 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:48.018 DEBUG:boto:Data: 2023-10-12T00:27:48.019 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:48.020 DEBUG:boto:Port: 8000 2023-10-12T00:27:48.020 DEBUG:boto:Params: {} 2023-10-12T00:27:48.021 DEBUG:boto:Token: None 2023-10-12T00:27:48.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/obj 2023-10-12T00:27:48.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ykDh+bSSYdLhNAvMnVY7XTGs3M= 2023-10-12T00:27:48.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ykDh+bSSYdLhNAvMnVY7XTGs3M=', 'Content-Length': '0'} 2023-10-12T00:27:48.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cea3c2de88119443-0065273d84-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.025 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:48.025 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:48.026 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:48.026 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:48.027 DEBUG:boto:Method: GET 2023-10-12T00:27:48.027 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:27:48.028 DEBUG:boto:Data: 2023-10-12T00:27:48.028 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:27:48.029 DEBUG:boto:Port: 8001 2023-10-12T00:27:48.029 DEBUG:boto:Params: {} 2023-10-12T00:27:48.030 DEBUG:boto:Token: None 2023-10-12T00:27:48.030 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/ 2023-10-12T00:27:48.031 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:68s6l0TtygDm0xcaiur5yhJtYcQ= 2023-10-12T00:27:48.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:68s6l0TtygDm0xcaiur5yhJtYcQ=', 'Content-Length': '0'} 2023-10-12T00:27:48.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af8428995d1a045-0065273d84-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.034 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:48.035 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:48.035 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:48.036 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:48.036 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:27:48.037 DEBUG:boto:Method: GET 2023-10-12T00:27:48.037 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:27:48.037 DEBUG:boto:Data: 2023-10-12T00:27:48.038 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:48.039 DEBUG:boto:Port: 8000 2023-10-12T00:27:48.039 DEBUG:boto:Params: {} 2023-10-12T00:27:48.040 DEBUG:boto:Token: None 2023-10-12T00:27:48.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/ 2023-10-12T00:27:48.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:68s6l0TtygDm0xcaiur5yhJtYcQ= 2023-10-12T00:27:48.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:68s6l0TtygDm0xcaiur5yhJtYcQ=', 'Content-Length': '0'} 2023-10-12T00:27:48.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d047998cd86294b-0065273d84-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.044 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:27:48.045 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:27:48.045 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:48.046 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:48.046 DEBUG:boto:path=/wafmoa-14/?versioning 2023-10-12T00:27:48.046 DEBUG:boto:auth_path=/wafmoa-14/?versioning 2023-10-12T00:27:48.047 DEBUG:boto:Method: PUT 2023-10-12T00:27:48.047 DEBUG:boto:Path: /wafmoa-14/?versioning 2023-10-12T00:27:48.048 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:27:48.048 DEBUG:boto:Headers: {} 2023-10-12T00:27:48.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:48.049 DEBUG:boto:Port: 8000 2023-10-12T00:27:48.050 DEBUG:boto:Params: {} 2023-10-12T00:27:48.050 DEBUG:boto:Token: None 2023-10-12T00:27:48.051 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:27:48 GMT /wafmoa-14/?versioning 2023-10-12T00:27:48.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TM8Xvjp9jUMg8ySvsA4O3vHWC30= 2023-10-12T00:27:48.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TM8Xvjp9jUMg8ySvsA4O3vHWC30=', 'Content-Length': '228'} 2023-10-12T00:27:48.057 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2be5e6a555b32f1-0065273d84-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:48.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:48.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:48.059 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:48.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:48.149 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:27:48.161 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:27:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:27:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:27:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070468.057718_117.1", 2023-10-12T00:27:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:27:48.057718Z" 2023-10-12T00:27:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:27:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.164 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:27:48.165 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.057718_117.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:48.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:27:48.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:48.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:48.167 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:48.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:48.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:48.249+0000 7f4c2e623780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:48.250+0000 7f4c2e623780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:48.262 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:48.262 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:48.263 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:48.263 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:48.263 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:48.263 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:27:48.263 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:48.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.264 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:48.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:48.265 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:48.265 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:48.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:48.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:48.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:27:48.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:48.266 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:27:48.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:48.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.267 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:48.267 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:48.267 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:48.268 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:48.268 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:48.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:48.268 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:48.268 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:27:48.269 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:48.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:48.270 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:48.270 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:48.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070468.054253_116.1", 2023-10-12T00:27:48.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:48.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:48.271 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:48.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:27:48.054253Z", 2023-10-12T00:27:48.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:48.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:48.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:48.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:48.272 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:48.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:48.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:48.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:48.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:48.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:27:48.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:48.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:48.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:48.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.275 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:48.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:27:48.275 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:27:48.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:48.275 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:48Z" 2023-10-12T00:27:48.276 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:48.277 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070468.054253_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:27:48.054253Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:27:48Z" } 2023-10-12T00:27:48.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070468.054253_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:27:48.054253Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:27:48.278 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.057718_117.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:48.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.054253_116.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:48.279 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070468.057718_117.1 target=1_1697070468.054253_116.1 2023-10-12T00:27:53.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:53.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:27:53.281 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:27:53.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:53.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:53.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:53.365+0000 7f4086da5780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:53.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:53.367+0000 7f4086da5780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:27:53.381 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:27:53.381 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:27:53.381 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:27:53.382 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:27:53.382 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:27:53.382 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:27:53.382 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:27:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:27:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:53.384 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:53.384 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:27:53.384 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:53.384 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:27:53.384 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:53.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:27:53.385 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:53.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:53.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:53.386 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:27:53.386 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:53.386 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:53.386 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:27:53.387 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:53.387 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:53.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:53.388 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:27:53.388 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:53.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:53.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:53.389 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:27:53.389 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:53.389 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:53.389 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070468.057718_117.1", 2023-10-12T00:27:53.390 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:53.390 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:53.390 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:53.390 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:27:48.057718Z", 2023-10-12T00:27:53.390 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:53.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:53.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:27:53.391 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:27:53.391 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:27:53.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:27:53.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:27:53.392 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:27:53.392 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:27:53.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:27:53.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:27:53.393 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:27:53.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:53.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:27:53.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:27:53.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.394 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:27:53.394 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:27:53.394 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:27:53.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:27:53.395 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:27:53Z" 2023-10-12T00:27:53.395 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:27:53.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070468.057718_117.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:27:48.057718Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:27:53Z" } 2023-10-12T00:27:53.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070468.057718_117.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:27:48.057718Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:27:53.397 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.057718_117.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:53.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.057718_117.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:27:53.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:27:53.398 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:27:53.398 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:27:53.399 DEBUG:boto:Method: HEAD 2023-10-12T00:27:53.399 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:27:53.399 DEBUG:boto:Data: 2023-10-12T00:27:53.400 DEBUG:boto:Headers: {} 2023-10-12T00:27:53.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:53.401 DEBUG:boto:Port: 8000 2023-10-12T00:27:53.401 DEBUG:boto:Params: {} 2023-10-12T00:27:53.402 DEBUG:boto:Token: None 2023-10-12T00:27:53.402 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:27:53 GMT /wafmoa-14/ 2023-10-12T00:27:53.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rB8648MoF3gD0Iuew6MBRZ37heE= 2023-10-12T00:27:53.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:27:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rB8648MoF3gD0Iuew6MBRZ37heE=', 'Content-Length': '0'} 2023-10-12T00:27:53.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc90202ec2252fc9-0065273d89-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:53.406 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:27:53.407 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:27:53.407 DEBUG:boto:Method: PUT 2023-10-12T00:27:53.408 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:27:53.408 DEBUG:boto:Data: 2023-10-12T00:27:53.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:27:53.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:27:53.409 DEBUG:boto:Port: 8000 2023-10-12T00:27:53.410 DEBUG:boto:Params: {} 2023-10-12T00:27:53.410 DEBUG:boto:Token: None 2023-10-12T00:27:53.411 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:27:53 GMT /wafmoa-14/obj 2023-10-12T00:27:53.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NjnSDOa6sRT19uH+4rwVZ+yclUY= 2023-10-12T00:27:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:27:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NjnSDOa6sRT19uH+4rwVZ+yclUY='} 2023-10-12T00:27:53.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-amz-request-id', 'tx000005d620d252b1363b9-0065273d89-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:27:53.423 DEBUG:rgw_multi.tests:created new version id=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:27:53.423 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-14 2023-10-12T00:27:53.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:53.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:27:53.425 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:27:53.480 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:27:53.480 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:27:53.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:53.514+0000 7f5609c7c780 1 waiting for incremental sync to catch up: 2023-10-12T00:27:53.515 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.8.6, , ] 2023-10-12T00:27:53.515 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:27:58.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:27:58.516+0000 7f5609c7c780 1 waiting for incremental sync to catch up: 2023-10-12T00:27:58.517 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.8.6, , ] 2023-10-12T00:27:58.517 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:03.518+0000 7f5609c7c780 1 bucket sync caught up with source: 2023-10-12T00:28:03.518 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:03.518 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:03.518+0000 7f5609c7c780 0 bucket checkpoint complete 2023-10-12T00:28:03.533 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-14 2023-10-12T00:28:03.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:03.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:03.535 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:28:03.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:03.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:03.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:03.631+0000 7f845d0d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:03.631 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:28:03.631 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:08.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:08.632+0000 7f845d0d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:08.633 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:28:08.633 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:13.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:13.633+0000 7f845d0d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:13.634 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:28:13.634 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:18.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:18.634+0000 7f845d0d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:18.635 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.24.6, , ] 2023-10-12T00:28:18.635 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:23.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:23.635+0000 7f845d0d9780 1 bucket sync caught up with source: 2023-10-12T00:28:23.636 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:23.636 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:23.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:23.635+0000 7f845d0d9780 0 bucket checkpoint complete 2023-10-12T00:28:23.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:28:23.648 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.649 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.649 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.650 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:28:23.650 DEBUG:boto:Data: 2023-10-12T00:28:23.651 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.651 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.652 DEBUG:boto:Params: {} 2023-10-12T00:28:23.652 DEBUG:boto:Token: None 2023-10-12T00:28:23.653 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.653 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGrH3yZh0yAoPOFFkeYXYNpPRzM= 2023-10-12T00:28:23.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGrH3yZh0yAoPOFFkeYXYNpPRzM=', 'Content-Length': '0'} 2023-10-12T00:28:23.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057648d7eb77a43f3-0065273da7-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.657 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.658 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.658 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.659 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:28:23.659 DEBUG:boto:Data: 2023-10-12T00:28:23.659 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.660 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.661 DEBUG:boto:Params: {} 2023-10-12T00:28:23.661 DEBUG:boto:Token: None 2023-10-12T00:28:23.661 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.662 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGrH3yZh0yAoPOFFkeYXYNpPRzM= 2023-10-12T00:28:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGrH3yZh0yAoPOFFkeYXYNpPRzM=', 'Content-Length': '0'} 2023-10-12T00:28:23.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000102189e25341152b-0065273da7-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.665 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:28:23.666 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.666 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.667 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.667 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.667 DEBUG:boto:Method: GET 2023-10-12T00:28:23.668 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.668 DEBUG:boto:Data: 2023-10-12T00:28:23.669 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.669 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.670 DEBUG:boto:Params: {} 2023-10-12T00:28:23.670 DEBUG:boto:Token: None 2023-10-12T00:28:23.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/?versions 2023-10-12T00:28:23.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ= 2023-10-12T00:28:23.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ=', 'Content-Length': '0'} 2023-10-12T00:28:23.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e044c4a584854084-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.674 DEBUG:boto:b'wafmoa-14999falseobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZltrue2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:28:23.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:28:23.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:28:23.676 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.677 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.677 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.678 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.678 DEBUG:boto:Method: GET 2023-10-12T00:28:23.678 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.679 DEBUG:boto:Data: 2023-10-12T00:28:23.679 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.680 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.681 DEBUG:boto:Params: {} 2023-10-12T00:28:23.681 DEBUG:boto:Token: None 2023-10-12T00:28:23.681 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/?versions 2023-10-12T00:28:23.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ= 2023-10-12T00:28:23.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ=', 'Content-Length': '0'} 2023-10-12T00:28:23.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b129c780972d2f8e-0065273da7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.685 DEBUG:boto:b'wafmoa-14999falseobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZltrue2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:28:23.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:28:23.687 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.688 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.688 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.688 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.689 DEBUG:boto:Method: GET 2023-10-12T00:28:23.689 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.690 DEBUG:boto:Data: 2023-10-12T00:28:23.690 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.691 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.691 DEBUG:boto:Params: {} 2023-10-12T00:28:23.692 DEBUG:boto:Token: None 2023-10-12T00:28:23.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/?versions 2023-10-12T00:28:23.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ= 2023-10-12T00:28:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ=', 'Content-Length': '0'} 2023-10-12T00:28:23.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d002f72fa57fcd3-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.696 DEBUG:boto:b'wafmoa-14999falseobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZltrue2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.697 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.697 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.698 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.698 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.699 DEBUG:boto:Method: GET 2023-10-12T00:28:23.699 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:28:23.699 DEBUG:boto:Data: 2023-10-12T00:28:23.700 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.701 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.701 DEBUG:boto:Params: {} 2023-10-12T00:28:23.701 DEBUG:boto:Token: None 2023-10-12T00:28:23.702 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/?versions 2023-10-12T00:28:23.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ= 2023-10-12T00:28:23.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXwDY2l+ZWDwHpt/8oNznehESSQ=', 'Content-Length': '0'} 2023-10-12T00:28:23.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ba88f85202bdfa5-0065273da7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.706 DEBUG:boto:b'wafmoa-14999falseobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZltrue2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-12T00:27:17.535Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.707 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.707 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.708 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.708 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.708 DEBUG:boto:Method: GET 2023-10-12T00:28:23.709 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.709 DEBUG:boto:Data: 2023-10-12T00:28:23.710 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.711 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.711 DEBUG:boto:Params: {} 2023-10-12T00:28:23.712 DEBUG:boto:Token: None 2023-10-12T00:28:23.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b63fd3525799157a-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.715 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.716 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.716 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.717 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.717 DEBUG:boto:Method: GET 2023-10-12T00:28:23.718 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.718 DEBUG:boto:Data: 2023-10-12T00:28:23.718 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.719 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.720 DEBUG:boto:Params: {} 2023-10-12T00:28:23.720 DEBUG:boto:Token: None 2023-10-12T00:28:23.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000377e183d90db367e-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.724 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.724 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.725 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.725 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.726 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.726 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.726 DEBUG:boto:Data: 2023-10-12T00:28:23.727 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.728 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.728 DEBUG:boto:Params: {} 2023-10-12T00:28:23.729 DEBUG:boto:Token: None 2023-10-12T00:28:23.729 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:geHQUJLOFyscfjLBAt3UYKm25Z0= 2023-10-12T00:28:23.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:geHQUJLOFyscfjLBAt3UYKm25Z0=', 'Content-Length': '0'} 2023-10-12T00:28:23.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000707bc008d70bd939-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.733 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.733 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.734 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.734 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.735 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.735 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.736 DEBUG:boto:Data: 2023-10-12T00:28:23.736 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.737 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.737 DEBUG:boto:Params: {} 2023-10-12T00:28:23.738 DEBUG:boto:Token: None 2023-10-12T00:28:23.738 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:geHQUJLOFyscfjLBAt3UYKm25Z0= 2023-10-12T00:28:23.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:geHQUJLOFyscfjLBAt3UYKm25Z0=', 'Content-Length': '0'} 2023-10-12T00:28:23.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018aebd5a0d6824fb-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.742 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.742 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.743 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.743 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.744 DEBUG:boto:Method: GET 2023-10-12T00:28:23.744 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.745 DEBUG:boto:Data: 2023-10-12T00:28:23.745 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.746 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.747 DEBUG:boto:Params: {} 2023-10-12T00:28:23.747 DEBUG:boto:Token: None 2023-10-12T00:28:23.747 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a896f717fc1a82d-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.751 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.752 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.752 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.752 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.753 DEBUG:boto:Method: GET 2023-10-12T00:28:23.753 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.754 DEBUG:boto:Data: 2023-10-12T00:28:23.754 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.755 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.756 DEBUG:boto:Params: {} 2023-10-12T00:28:23.756 DEBUG:boto:Token: None 2023-10-12T00:28:23.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a0303bb4fc8c580-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.760 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.760 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.761 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.761 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.762 DEBUG:boto:Method: GET 2023-10-12T00:28:23.762 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.763 DEBUG:boto:Data: 2023-10-12T00:28:23.763 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.764 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.764 DEBUG:boto:Params: {} 2023-10-12T00:28:23.765 DEBUG:boto:Token: None 2023-10-12T00:28:23.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4dd85bee3956f4-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.814 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.815 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.816 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.816 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.817 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.817 DEBUG:boto:Method: GET 2023-10-12T00:28:23.817 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.818 DEBUG:boto:Data: 2023-10-12T00:28:23.818 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.819 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.820 DEBUG:boto:Params: {} 2023-10-12T00:28:23.820 DEBUG:boto:Token: None 2023-10-12T00:28:23.821 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c0890a51ca0c8c-0065273da7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.825 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.825 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.826 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.826 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.827 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:28:23.827 DEBUG:boto:Data: 2023-10-12T00:28:23.827 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.828 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.829 DEBUG:boto:Params: {} 2023-10-12T00:28:23.829 DEBUG:boto:Token: None 2023-10-12T00:28:23.830 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj 2023-10-12T00:28:23.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k= 2023-10-12T00:28:23.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k=', 'Content-Length': '0'} 2023-10-12T00:28:23.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046bcc95d8d9e5cae-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.833 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.834 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.834 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.835 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:28:23.835 DEBUG:boto:Data: 2023-10-12T00:28:23.835 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.836 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.837 DEBUG:boto:Params: {} 2023-10-12T00:28:23.837 DEBUG:boto:Token: None 2023-10-12T00:28:23.837 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj 2023-10-12T00:28:23.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k= 2023-10-12T00:28:23.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k=', 'Content-Length': '0'} 2023-10-12T00:28:23.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec66a44c5bc2683c-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.842 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.842 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.843 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.843 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.843 DEBUG:boto:Method: GET 2023-10-12T00:28:23.844 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.844 DEBUG:boto:Data: 2023-10-12T00:28:23.845 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.846 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.846 DEBUG:boto:Params: {} 2023-10-12T00:28:23.847 DEBUG:boto:Token: None 2023-10-12T00:28:23.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ad46a36d3c2f132-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.850 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.851 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.851 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.852 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.852 DEBUG:boto:Method: GET 2023-10-12T00:28:23.853 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.853 DEBUG:boto:Data: 2023-10-12T00:28:23.854 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.855 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.855 DEBUG:boto:Params: {} 2023-10-12T00:28:23.855 DEBUG:boto:Token: None 2023-10-12T00:28:23.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008dbff9e23660868-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.859 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.860 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.860 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.861 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.861 DEBUG:boto:Method: GET 2023-10-12T00:28:23.861 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.862 DEBUG:boto:Data: 2023-10-12T00:28:23.862 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.863 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.864 DEBUG:boto:Params: {} 2023-10-12T00:28:23.864 DEBUG:boto:Token: None 2023-10-12T00:28:23.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095161cfee93d432d-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.869 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.869 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.870 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.870 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.871 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.871 DEBUG:boto:Method: GET 2023-10-12T00:28:23.871 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.872 DEBUG:boto:Data: 2023-10-12T00:28:23.872 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.873 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.874 DEBUG:boto:Params: {} 2023-10-12T00:28:23.874 DEBUG:boto:Token: None 2023-10-12T00:28:23.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51cc62e13bea0da-0065273da7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.878 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.879 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.880 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.880 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.881 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.881 DEBUG:boto:Method: GET 2023-10-12T00:28:23.882 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.882 DEBUG:boto:Data: 2023-10-12T00:28:23.883 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.883 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.884 DEBUG:boto:Params: {} 2023-10-12T00:28:23.884 DEBUG:boto:Token: None 2023-10-12T00:28:23.885 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c= 2023-10-12T00:28:23.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c=', 'Content-Length': '0'} 2023-10-12T00:28:23.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ab52190288964c9-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.888 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.889 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.889 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.890 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.890 DEBUG:boto:Method: GET 2023-10-12T00:28:23.890 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.891 DEBUG:boto:Data: 2023-10-12T00:28:23.891 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.892 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.893 DEBUG:boto:Params: {} 2023-10-12T00:28:23.893 DEBUG:boto:Token: None 2023-10-12T00:28:23.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c= 2023-10-12T00:28:23.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c=', 'Content-Length': '0'} 2023-10-12T00:28:23.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be0327bc0a41c215-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.897 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.897 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.898 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.898 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.899 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.899 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.899 DEBUG:boto:Data: 2023-10-12T00:28:23.900 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.901 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.901 DEBUG:boto:Params: {} 2023-10-12T00:28:23.901 DEBUG:boto:Token: None 2023-10-12T00:28:23.902 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rlxKHkJCMS8oYKnU0gsrzLb8mAM= 2023-10-12T00:28:23.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rlxKHkJCMS8oYKnU0gsrzLb8mAM=', 'Content-Length': '0'} 2023-10-12T00:28:23.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045ac9037f1afa640-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.905 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.906 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.906 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.906 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.907 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.907 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.908 DEBUG:boto:Data: 2023-10-12T00:28:23.908 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.909 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.909 DEBUG:boto:Params: {} 2023-10-12T00:28:23.910 DEBUG:boto:Token: None 2023-10-12T00:28:23.910 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rlxKHkJCMS8oYKnU0gsrzLb8mAM= 2023-10-12T00:28:23.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rlxKHkJCMS8oYKnU0gsrzLb8mAM=', 'Content-Length': '0'} 2023-10-12T00:28:23.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070519cbcbacd7f70-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.914 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.915 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.915 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.916 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.916 DEBUG:boto:Method: GET 2023-10-12T00:28:23.917 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.917 DEBUG:boto:Data: 2023-10-12T00:28:23.918 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.918 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.919 DEBUG:boto:Params: {} 2023-10-12T00:28:23.919 DEBUG:boto:Token: None 2023-10-12T00:28:23.920 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c= 2023-10-12T00:28:23.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c=', 'Content-Length': '0'} 2023-10-12T00:28:23.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001fb16e6374c7356b-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.923 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.924 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.924 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.924 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.925 DEBUG:boto:Method: GET 2023-10-12T00:28:23.925 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.926 DEBUG:boto:Data: 2023-10-12T00:28:23.926 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.927 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.927 DEBUG:boto:Params: {} 2023-10-12T00:28:23.928 DEBUG:boto:Token: None 2023-10-12T00:28:23.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:28:23.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c= 2023-10-12T00:28:23.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x86QHX0UUnKzBdiqS3QEPcjTd9c=', 'Content-Length': '0'} 2023-10-12T00:28:23.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000952ec7f28fd94829-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.932 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.932 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.933 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.933 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.934 DEBUG:boto:Method: GET 2023-10-12T00:28:23.934 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.935 DEBUG:boto:Data: 2023-10-12T00:28:23.935 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.936 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.936 DEBUG:boto:Params: {} 2023-10-12T00:28:23.937 DEBUG:boto:Token: None 2023-10-12T00:28:23.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006918687fde89690b-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.941 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.942 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.942 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.943 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.943 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.944 DEBUG:boto:Method: GET 2023-10-12T00:28:23.944 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.944 DEBUG:boto:Data: 2023-10-12T00:28:23.945 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.946 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.946 DEBUG:boto:Params: {} 2023-10-12T00:28:23.947 DEBUG:boto:Token: None 2023-10-12T00:28:23.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f7242218edfec54-0065273da7-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.951 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:23.952 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.952 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.953 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.953 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:28:23.954 DEBUG:boto:Data: 2023-10-12T00:28:23.954 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.955 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.955 DEBUG:boto:Params: {} 2023-10-12T00:28:23.956 DEBUG:boto:Token: None 2023-10-12T00:28:23.956 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj 2023-10-12T00:28:23.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k= 2023-10-12T00:28:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k=', 'Content-Length': '0'} 2023-10-12T00:28:23.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005501f9ab95564bdd-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.960 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.960 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.961 DEBUG:boto:Method: HEAD 2023-10-12T00:28:23.961 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:28:23.961 DEBUG:boto:Data: 2023-10-12T00:28:23.962 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.963 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.963 DEBUG:boto:Params: {} 2023-10-12T00:28:23.964 DEBUG:boto:Token: None 2023-10-12T00:28:23.964 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj 2023-10-12T00:28:23.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k= 2023-10-12T00:28:23.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RoZCTYZ3nBCujQOJCfbhCwCAi+k=', 'Content-Length': '0'} 2023-10-12T00:28:23.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a11b855ca73594ae-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:28:23.968 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.968 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.969 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.969 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.970 DEBUG:boto:Method: GET 2023-10-12T00:28:23.970 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.970 DEBUG:boto:Data: 2023-10-12T00:28:23.971 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.972 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.972 DEBUG:boto:Params: {} 2023-10-12T00:28:23.973 DEBUG:boto:Token: None 2023-10-12T00:28:23.973 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.974 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067834abc1b3380c3-0065273da7-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.976 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:23.977 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:23.977 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.978 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.978 DEBUG:boto:Method: GET 2023-10-12T00:28:23.979 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.979 DEBUG:boto:Data: 2023-10-12T00:28:23.980 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:23.980 DEBUG:boto:Port: 8000 2023-10-12T00:28:23.981 DEBUG:boto:Params: {} 2023-10-12T00:28:23.981 DEBUG:boto:Token: None 2023-10-12T00:28:23.982 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:28:23.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg= 2023-10-12T00:28:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0saizdqtsVXAPVxpgLbp57vdgqg=', 'Content-Length': '0'} 2023-10-12T00:28:23.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bba7562d5f7aff53-0065273da7-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:23.985 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:23.986 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:23.986 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.987 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:23.987 DEBUG:boto:Method: GET 2023-10-12T00:28:23.987 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:23.988 DEBUG:boto:Data: 2023-10-12T00:28:23.988 DEBUG:boto:Headers: {} 2023-10-12T00:28:23.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:28:23.989 DEBUG:boto:Port: 8001 2023-10-12T00:28:23.990 DEBUG:boto:Params: {} 2023-10-12T00:28:23.990 DEBUG:boto:Token: None 2023-10-12T00:28:23.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:23 GMT /wafmoa-14/ 2023-10-12T00:28:23.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk= 2023-10-12T00:28:23.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pWyhIG0HmNd6K1O8hr8y+73viAk=', 'Content-Length': '0'} 2023-10-12T00:28:23.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def385878fbca67e-0065273da7-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:24.038 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:24.039 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:24.039 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:24.040 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:24.040 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:28:24.041 DEBUG:boto:Method: GET 2023-10-12T00:28:24.041 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:28:24.042 DEBUG:boto:Data: 2023-10-12T00:28:24.042 DEBUG:boto:Headers: {} 2023-10-12T00:28:24.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:24.043 DEBUG:boto:Port: 8000 2023-10-12T00:28:24.043 DEBUG:boto:Params: {} 2023-10-12T00:28:24.044 DEBUG:boto:Token: None 2023-10-12T00:28:24.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:28:24 GMT /wafmoa-14/ 2023-10-12T00:28:24.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xQMrDEGhuLTBKlmK7HCNiIJwW0c= 2023-10-12T00:28:24.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xQMrDEGhuLTBKlmK7HCNiIJwW0c=', 'Content-Length': '0'} 2023-10-12T00:28:24.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0fd34f33b4c69f9-0065273da8-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:24.048 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:28:24.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:28:24.049 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:24.050 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:24.050 DEBUG:boto:path=/wafmoa-14/?versioning 2023-10-12T00:28:24.050 DEBUG:boto:auth_path=/wafmoa-14/?versioning 2023-10-12T00:28:24.051 DEBUG:boto:Method: PUT 2023-10-12T00:28:24.051 DEBUG:boto:Path: /wafmoa-14/?versioning 2023-10-12T00:28:24.052 DEBUG:boto:Data: Suspended Disabled 2023-10-12T00:28:24.052 DEBUG:boto:Headers: {} 2023-10-12T00:28:24.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:24.053 DEBUG:boto:Port: 8000 2023-10-12T00:28:24.053 DEBUG:boto:Params: {} 2023-10-12T00:28:24.053 DEBUG:boto:Token: None 2023-10-12T00:28:24.054 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:28:24 GMT /wafmoa-14/?versioning 2023-10-12T00:28:24.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:99Tcy0fGm/b9WVbpSSpPqXtbLMk= 2023-10-12T00:28:24.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:99Tcy0fGm/b9WVbpSSpPqXtbLMk=', 'Content-Length': '230'} 2023-10-12T00:28:24.060 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3b3e87401f859a4-0065273da8-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:24.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:24.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:24.061 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:28:24.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:24.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:24.138 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:28:24.149 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:28:24.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:28:24.150 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:28:24.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:28:24.151 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:28:24.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070504.060486_125.1", 2023-10-12T00:28:24.152 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:28:24.060486Z" 2023-10-12T00:28:24.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.152 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:28:24.152 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:28:24.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.153 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:28:24.154 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070504.060486_125.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:28:24.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:28:24.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:24.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:24.156 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:28:24.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:24.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:24.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:24.232+0000 7fd302541780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:28:24.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:24.233+0000 7fd302541780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:28:24.244 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:28:24.244 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:28:24.245 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:28:24.245 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:28:24.245 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:28:24.245 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:28:24.246 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:24.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.246 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:28:24.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.246 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:28:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:28:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:24.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:28:24.248 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:24.248 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:28:24.248 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:24.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.249 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:28:24.249 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:24.249 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:24.250 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:28:24.250 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:24.250 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:24.250 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:24.250 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:28:24.251 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:24.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:28:24.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:24.252 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:24.252 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070468.057718_117.1", 2023-10-12T00:28:24.252 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:24.253 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:24.253 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:24.253 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:27:48.057718Z", 2023-10-12T00:28:24.253 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:24.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:24.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:28:24.254 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:24.254 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:24.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:28:24.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:24.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:24.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:24.255 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:28:24.256 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:24.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:24.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:28:24.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.257 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:28:24.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:28:24.257 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:28:24.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:24.257 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:28:24Z" 2023-10-12T00:28:24.258 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:28:24.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070468.057718_117.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:27:48.057718Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:28:24Z" } 2023-10-12T00:28:24.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070468.057718_117.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:27:48.057718Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:28:24.260 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070504.060486_125.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:28:24.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070468.057718_117.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:28:24.261 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070504.060486_125.1 target=1_1697070468.057718_117.1 2023-10-12T00:28:29.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:29.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:28:29.263 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:28:29.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:29.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:29.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:29.352+0000 7f76ed0cb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:28:29.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:29.353+0000 7f76ed0cb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:28:29.366 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:28:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:28:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:28:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:28:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:28:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:28:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:28:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:28:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:28:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:29.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:28:29.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:29.370 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:28:29.370 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:29.371 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:29.372 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:28:29.372 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:29.372 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:29.372 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:29.372 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:28:29.373 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:29.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:29.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:29.373 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:28:29.374 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:29.374 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:29.374 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070504.060486_125.1", 2023-10-12T00:28:29.374 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:29.374 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:29.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:29.375 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:28:24.060486Z", 2023-10-12T00:28:29.375 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:29.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:29.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:28:29.376 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:28:29.376 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:28:29.376 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:28:29.377 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:28:29.377 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:28:29.377 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:28:29.377 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:28:29.377 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:28:29.378 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:28:29.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:29.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:28:29.378 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:28:29.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.379 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:28:29.379 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:28:29.379 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:28:29.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:28:29.380 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:28:29Z" 2023-10-12T00:28:29.380 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:28:29.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070504.060486_125.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:28:24.060486Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:28:29Z" } 2023-10-12T00:28:29.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070504.060486_125.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:28:24.060486Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:28:29.382 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070504.060486_125.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:28:29.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070504.060486_125.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:28:29.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:28:29.383 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:28:29.384 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:28:29.384 DEBUG:boto:Method: HEAD 2023-10-12T00:28:29.384 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:28:29.385 DEBUG:boto:Data: 2023-10-12T00:28:29.385 DEBUG:boto:Headers: {} 2023-10-12T00:28:29.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:29.386 DEBUG:boto:Port: 8000 2023-10-12T00:28:29.386 DEBUG:boto:Params: {} 2023-10-12T00:28:29.387 DEBUG:boto:Token: None 2023-10-12T00:28:29.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:28:29 GMT /wafmoa-14/ 2023-10-12T00:28:29.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dPFi3KSWvVSIcDCOkYQWO54XR9g= 2023-10-12T00:28:29.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:28:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dPFi3KSWvVSIcDCOkYQWO54XR9g=', 'Content-Length': '0'} 2023-10-12T00:28:29.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3328250e54c8789-0065273dad-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:29.391 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:28:29.392 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:28:29.392 DEBUG:boto:Method: PUT 2023-10-12T00:28:29.393 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:28:29.393 DEBUG:boto:Data: 2023-10-12T00:28:29.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:28:29.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:28:29.394 DEBUG:boto:Port: 8000 2023-10-12T00:28:29.395 DEBUG:boto:Params: {} 2023-10-12T00:28:29.395 DEBUG:boto:Token: None 2023-10-12T00:28:29.396 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:28:29 GMT /wafmoa-14/obj 2023-10-12T00:28:29.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CjXkYMCpG4XaAorWop8zKOnLSkk= 2023-10-12T00:28:29.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:28:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CjXkYMCpG4XaAorWop8zKOnLSkk='} 2023-10-12T00:28:29.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015a7e0f1aaa333d1-0065273dad-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:28:29.408 DEBUG:rgw_multi.tests:created null version id=None 2023-10-12T00:28:29.409 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-14 2023-10-12T00:28:29.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:29.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:29.410 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:28:29.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:29.468 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:29.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:29.500+0000 7f16c69ab780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:29.501 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:29.501 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-12T00:28:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:34.501+0000 7f16c69ab780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:34.503 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:34.503 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-12T00:28:39.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:39.504+0000 7f16c69ab780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:39.504 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-10-12T00:28:39.504 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-12T00:28:44.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:44.505+0000 7f16c69ab780 1 bucket sync caught up with source: 2023-10-12T00:28:44.505 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000005.15.18, , ] 2023-10-12T00:28:44.505 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-12T00:28:44.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:44.505+0000 7f16c69ab780 0 bucket checkpoint complete 2023-10-12T00:28:44.520 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-14 2023-10-12T00:28:44.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:44.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:28:44.522 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:28:44.585 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:28:44.585 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:28:44.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:44.625+0000 7fd383af8780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:44.625 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:44.626 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:28:49.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:49.627+0000 7fd383af8780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:49.627 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:49.627 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:28:54.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:54.627+0000 7fd383af8780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:54.628 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:54.628 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:28:59.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:28:59.628+0000 7fd383af8780 1 waiting for incremental sync to catch up: 2023-10-12T00:28:59.629 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.28.13, , ] 2023-10-12T00:28:59.629 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:29:04.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:04.629+0000 7fd383af8780 1 bucket sync caught up with source: 2023-10-12T00:29:04.631 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:29:04.631 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2023-10-12T00:29:04.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:04.629+0000 7fd383af8780 0 bucket checkpoint complete 2023-10-12T00:29:04.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:29:04.647 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.647 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.648 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.648 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:29:04.649 DEBUG:boto:Data: 2023-10-12T00:29:04.649 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.650 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.650 DEBUG:boto:Params: {} 2023-10-12T00:29:04.651 DEBUG:boto:Token: None 2023-10-12T00:29:04.651 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9maUafYEZrwJHo4G46Zz6d1eonQ= 2023-10-12T00:29:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9maUafYEZrwJHo4G46Zz6d1eonQ=', 'Content-Length': '0'} 2023-10-12T00:29:04.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000364c9bf065f1e003-0065273dd0-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.656 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.656 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.657 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.657 DEBUG:boto:Path: /wafmoa-14/ 2023-10-12T00:29:04.658 DEBUG:boto:Data: 2023-10-12T00:29:04.658 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.659 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.659 DEBUG:boto:Params: {} 2023-10-12T00:29:04.660 DEBUG:boto:Token: None 2023-10-12T00:29:04.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9maUafYEZrwJHo4G46Zz6d1eonQ= 2023-10-12T00:29:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9maUafYEZrwJHo4G46Zz6d1eonQ=', 'Content-Length': '0'} 2023-10-12T00:29:04.664 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d5542213b7cb255-0065273dd0-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.664 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:29:04.665 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.665 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.666 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.666 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.666 DEBUG:boto:Method: GET 2023-10-12T00:29:04.667 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.667 DEBUG:boto:Data: 2023-10-12T00:29:04.668 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.669 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.669 DEBUG:boto:Params: {} 2023-10-12T00:29:04.670 DEBUG:boto:Token: None 2023-10-12T00:29:04.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/?versions 2023-10-12T00:29:04.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk= 2023-10-12T00:29:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk=', 'Content-Length': '0'} 2023-10-12T00:29:04.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007376d813b0ee2ed0-0065273dd0-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.674 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZlfalse2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:04.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:04.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:29:04.676 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.677 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.677 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.677 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.678 DEBUG:boto:Method: GET 2023-10-12T00:29:04.678 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.679 DEBUG:boto:Data: 2023-10-12T00:29:04.679 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.680 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.680 DEBUG:boto:Params: {} 2023-10-12T00:29:04.681 DEBUG:boto:Token: None 2023-10-12T00:29:04.681 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/?versions 2023-10-12T00:29:04.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk= 2023-10-12T00:29:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk=', 'Content-Length': '0'} 2023-10-12T00:29:04.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008921d4c9f3ad3898-0065273dd0-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.687 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZlfalse2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:04.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:04.689 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.689 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.690 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.690 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.691 DEBUG:boto:Method: GET 2023-10-12T00:29:04.691 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.692 DEBUG:boto:Data: 2023-10-12T00:29:04.692 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.693 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.693 DEBUG:boto:Params: {} 2023-10-12T00:29:04.694 DEBUG:boto:Token: None 2023-10-12T00:29:04.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/?versions 2023-10-12T00:29:04.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk= 2023-10-12T00:29:04.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk=', 'Content-Length': '0'} 2023-10-12T00:29:04.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c31a09cebb1555-0065273dd0-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.698 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZlfalse2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.699 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.699 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.700 DEBUG:boto:path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.700 DEBUG:boto:auth_path=/wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.701 DEBUG:boto:Method: GET 2023-10-12T00:29:04.701 DEBUG:boto:Path: /wafmoa-14/?versions&max-keys=999 2023-10-12T00:29:04.702 DEBUG:boto:Data: 2023-10-12T00:29:04.702 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.703 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.703 DEBUG:boto:Params: {} 2023-10-12T00:29:04.704 DEBUG:boto:Token: None 2023-10-12T00:29:04.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/?versions 2023-10-12T00:29:04.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk= 2023-10-12T00:29:04.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KQzGgmHAeLQBJE4k+AiH5w3NvDk=', 'Content-Length': '0'} 2023-10-12T00:29:04.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416f1de5c66c5e8d-0065273dd0-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.750 DEBUG:boto:b'wafmoa-14999falseobjnulltrue2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPBAv5kjwU60pT6bTeN9k83y.1e9pVZlfalse2023-10-12T00:27:53.414Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:04.752 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.752 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.753 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.753 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.753 DEBUG:boto:Method: GET 2023-10-12T00:29:04.754 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.754 DEBUG:boto:Data: 2023-10-12T00:29:04.755 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.756 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.756 DEBUG:boto:Params: {} 2023-10-12T00:29:04.757 DEBUG:boto:Token: None 2023-10-12T00:29:04.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I= 2023-10-12T00:29:04.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I=', 'Content-Length': '0'} 2023-10-12T00:29:04.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049aefe9f58e24922-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.761 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.762 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.762 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.762 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.763 DEBUG:boto:Method: GET 2023-10-12T00:29:04.763 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.764 DEBUG:boto:Data: 2023-10-12T00:29:04.764 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.765 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.765 DEBUG:boto:Params: {} 2023-10-12T00:29:04.766 DEBUG:boto:Token: None 2023-10-12T00:29:04.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I= 2023-10-12T00:29:04.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I=', 'Content-Length': '0'} 2023-10-12T00:29:04.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d2a0c04524f1c8c-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.773 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.773 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.773 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.774 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.774 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.775 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.775 DEBUG:boto:Data: 2023-10-12T00:29:04.776 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.777 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.777 DEBUG:boto:Params: {} 2023-10-12T00:29:04.778 DEBUG:boto:Token: None 2023-10-12T00:29:04.778 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ofsMDlLP+cJiKnhUb6vsjo1tSzU= 2023-10-12T00:29:04.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ofsMDlLP+cJiKnhUb6vsjo1tSzU=', 'Content-Length': '0'} 2023-10-12T00:29:04.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e66899cf20f5214f-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.782 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.782 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.783 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.783 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.784 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.784 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.784 DEBUG:boto:Data: 2023-10-12T00:29:04.785 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.786 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.786 DEBUG:boto:Params: {} 2023-10-12T00:29:04.786 DEBUG:boto:Token: None 2023-10-12T00:29:04.787 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ofsMDlLP+cJiKnhUb6vsjo1tSzU= 2023-10-12T00:29:04.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ofsMDlLP+cJiKnhUb6vsjo1tSzU=', 'Content-Length': '0'} 2023-10-12T00:29:04.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1dfacb7400d4bb8-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:04.791 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.791 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.792 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.792 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.793 DEBUG:boto:Method: GET 2023-10-12T00:29:04.793 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.793 DEBUG:boto:Data: 2023-10-12T00:29:04.794 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.795 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.795 DEBUG:boto:Params: {} 2023-10-12T00:29:04.796 DEBUG:boto:Token: None 2023-10-12T00:29:04.796 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I= 2023-10-12T00:29:04.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I=', 'Content-Length': '0'} 2023-10-12T00:29:04.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc7a36c0a55bb75f-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.800 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.800 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.801 DEBUG:boto:path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.801 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.802 DEBUG:boto:Method: GET 2023-10-12T00:29:04.802 DEBUG:boto:Path: /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.802 DEBUG:boto:Data: 2023-10-12T00:29:04.803 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.804 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.804 DEBUG:boto:Params: {} 2023-10-12T00:29:04.805 DEBUG:boto:Token: None 2023-10-12T00:29:04.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=null 2023-10-12T00:29:04.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I= 2023-10-12T00:29:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rWD/Sab8V0llJRmI3OpIt9CAh7I=', 'Content-Length': '0'} 2023-10-12T00:29:04.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d34379a098c1b315-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.809 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.809 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.810 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.810 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.811 DEBUG:boto:Method: GET 2023-10-12T00:29:04.811 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.811 DEBUG:boto:Data: 2023-10-12T00:29:04.812 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.813 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.813 DEBUG:boto:Params: {} 2023-10-12T00:29:04.814 DEBUG:boto:Token: None 2023-10-12T00:29:04.814 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f073c3e0bf6ff2-0065273dd0-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.818 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.819 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.820 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.820 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.820 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.821 DEBUG:boto:Method: GET 2023-10-12T00:29:04.821 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.822 DEBUG:boto:Data: 2023-10-12T00:29:04.822 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.823 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.823 DEBUG:boto:Params: {} 2023-10-12T00:29:04.824 DEBUG:boto:Token: None 2023-10-12T00:29:04.824 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000703ccf8c8fca9f7b-0065273dd0-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.828 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.829 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.829 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.830 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.830 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:04.831 DEBUG:boto:Data: 2023-10-12T00:29:04.831 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.832 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.832 DEBUG:boto:Params: {} 2023-10-12T00:29:04.833 DEBUG:boto:Token: None 2023-10-12T00:29:04.833 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj 2023-10-12T00:29:04.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YZamDBjBTXlmBQDMIqI0D3DVUIE= 2023-10-12T00:29:04.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YZamDBjBTXlmBQDMIqI0D3DVUIE=', 'Content-Length': '0'} 2023-10-12T00:29:04.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073fba09449ea27a3-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.836 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.837 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.837 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.838 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:04.838 DEBUG:boto:Data: 2023-10-12T00:29:04.839 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.839 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.840 DEBUG:boto:Params: {} 2023-10-12T00:29:04.840 DEBUG:boto:Token: None 2023-10-12T00:29:04.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj 2023-10-12T00:29:04.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YZamDBjBTXlmBQDMIqI0D3DVUIE= 2023-10-12T00:29:04.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YZamDBjBTXlmBQDMIqI0D3DVUIE=', 'Content-Length': '0'} 2023-10-12T00:29:04.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022e4ad499ac75ad5-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:04.845 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.845 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.846 DEBUG:boto:Method: GET 2023-10-12T00:29:04.846 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:04.847 DEBUG:boto:Data: 2023-10-12T00:29:04.847 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.848 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.849 DEBUG:boto:Params: {} 2023-10-12T00:29:04.849 DEBUG:boto:Token: None 2023-10-12T00:29:04.849 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj 2023-10-12T00:29:04.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jdqwAtKjxgzB3ROc4BRONYaRToA= 2023-10-12T00:29:04.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jdqwAtKjxgzB3ROc4BRONYaRToA=', 'Content-Length': '0'} 2023-10-12T00:29:04.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1112b0c780099ed-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.853 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.854 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.855 DEBUG:boto:Method: GET 2023-10-12T00:29:04.856 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:04.856 DEBUG:boto:Data: 2023-10-12T00:29:04.856 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.857 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.858 DEBUG:boto:Params: {} 2023-10-12T00:29:04.858 DEBUG:boto:Token: None 2023-10-12T00:29:04.859 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj 2023-10-12T00:29:04.859 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jdqwAtKjxgzB3ROc4BRONYaRToA= 2023-10-12T00:29:04.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jdqwAtKjxgzB3ROc4BRONYaRToA=', 'Content-Length': '0'} 2023-10-12T00:29:04.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4180d7bddde265b-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.862 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.863 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.863 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.864 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.864 DEBUG:boto:Method: GET 2023-10-12T00:29:04.864 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.865 DEBUG:boto:Data: 2023-10-12T00:29:04.865 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.866 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.867 DEBUG:boto:Params: {} 2023-10-12T00:29:04.867 DEBUG:boto:Token: None 2023-10-12T00:29:04.868 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7efcf21625978e-0065273dd0-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.872 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.872 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.873 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.873 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.874 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.874 DEBUG:boto:Method: GET 2023-10-12T00:29:04.875 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.875 DEBUG:boto:Data: 2023-10-12T00:29:04.876 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.876 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.877 DEBUG:boto:Params: {} 2023-10-12T00:29:04.877 DEBUG:boto:Token: None 2023-10-12T00:29:04.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a94c553d012b93-0065273dd0-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.882 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:04.883 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.883 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.884 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.884 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.884 DEBUG:boto:Method: GET 2023-10-12T00:29:04.885 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.885 DEBUG:boto:Data: 2023-10-12T00:29:04.886 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.887 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.887 DEBUG:boto:Params: {} 2023-10-12T00:29:04.888 DEBUG:boto:Token: None 2023-10-12T00:29:04.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc= 2023-10-12T00:29:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc=', 'Content-Length': '0'} 2023-10-12T00:29:04.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000347d205133b5c532-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.892 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.892 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.893 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.893 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.893 DEBUG:boto:Method: GET 2023-10-12T00:29:04.894 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.894 DEBUG:boto:Data: 2023-10-12T00:29:04.895 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.896 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.896 DEBUG:boto:Params: {} 2023-10-12T00:29:04.897 DEBUG:boto:Token: None 2023-10-12T00:29:04.897 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc= 2023-10-12T00:29:04.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc=', 'Content-Length': '0'} 2023-10-12T00:29:04.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adeb324520b89687-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.901 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.901 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.902 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.902 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.902 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.903 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.903 DEBUG:boto:Data: 2023-10-12T00:29:04.904 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.905 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.905 DEBUG:boto:Params: {} 2023-10-12T00:29:04.906 DEBUG:boto:Token: None 2023-10-12T00:29:04.906 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KaQ7fjIR10ND9OCljCi+6hNXpvE= 2023-10-12T00:29:04.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KaQ7fjIR10ND9OCljCi+6hNXpvE=', 'Content-Length': '0'} 2023-10-12T00:29:04.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004632645b13ece1e7-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.910 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.910 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.911 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.911 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.912 DEBUG:boto:Method: HEAD 2023-10-12T00:29:04.912 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.912 DEBUG:boto:Data: 2023-10-12T00:29:04.913 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.914 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.914 DEBUG:boto:Params: {} 2023-10-12T00:29:04.915 DEBUG:boto:Token: None 2023-10-12T00:29:04.915 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KaQ7fjIR10ND9OCljCi+6hNXpvE= 2023-10-12T00:29:04.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KaQ7fjIR10ND9OCljCi+6hNXpvE=', 'Content-Length': '0'} 2023-10-12T00:29:04.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019b6e64806b5c4d6-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:04.919 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.919 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.920 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.920 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.921 DEBUG:boto:Method: GET 2023-10-12T00:29:04.921 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.922 DEBUG:boto:Data: 2023-10-12T00:29:04.922 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.923 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.923 DEBUG:boto:Params: {} 2023-10-12T00:29:04.924 DEBUG:boto:Token: None 2023-10-12T00:29:04.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc= 2023-10-12T00:29:04.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc=', 'Content-Length': '0'} 2023-10-12T00:29:04.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f8a14580654f626-0065273dd0-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.928 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.928 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:04.929 DEBUG:boto:path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.929 DEBUG:boto:auth_path=/wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.929 DEBUG:boto:Method: GET 2023-10-12T00:29:04.930 DEBUG:boto:Path: /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.930 DEBUG:boto:Data: 2023-10-12T00:29:04.931 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.932 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.932 DEBUG:boto:Params: {} 2023-10-12T00:29:04.933 DEBUG:boto:Token: None 2023-10-12T00:29:04.933 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/obj?versionId=PBAv5kjwU60pT6bTeN9k83y.1e9pVZl 2023-10-12T00:29:04.934 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc= 2023-10-12T00:29:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/EpTp0IFiXmXvGh47UHAiCHnzc=', 'Content-Length': '0'} 2023-10-12T00:29:04.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:27:53 GMT'), ('x-amz-version-id', 'PBAv5kjwU60pT6bTeN9k83y.1e9pVZl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a7ab7b6f37d234d-0065273dd0-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.937 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.937 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.938 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.938 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.939 DEBUG:boto:Method: GET 2023-10-12T00:29:04.939 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.939 DEBUG:boto:Data: 2023-10-12T00:29:04.940 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:04.941 DEBUG:boto:Port: 8001 2023-10-12T00:29:04.941 DEBUG:boto:Params: {} 2023-10-12T00:29:04.942 DEBUG:boto:Token: None 2023-10-12T00:29:04.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c50b3a73b4eae9-0065273dd0-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.987 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.988 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:04.988 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:04.989 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.989 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:04.990 DEBUG:boto:Method: GET 2023-10-12T00:29:04.990 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:04.991 DEBUG:boto:Data: 2023-10-12T00:29:04.991 DEBUG:boto:Headers: {} 2023-10-12T00:29:04.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:04.992 DEBUG:boto:Port: 8000 2023-10-12T00:29:04.992 DEBUG:boto:Params: {} 2023-10-12T00:29:04.993 DEBUG:boto:Token: None 2023-10-12T00:29:04.994 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:04 GMT /wafmoa-14/ 2023-10-12T00:29:04.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs= 2023-10-12T00:29:04.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXXQ21zTyAF5YMIxyLw64jlBWZs=', 'Content-Length': '0'} 2023-10-12T00:29:04.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77f26d59ba44a37-0065273dd0-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:04.998 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:04.999 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:04.999 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:05.000 DEBUG:boto:Method: HEAD 2023-10-12T00:29:05.000 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:05.001 DEBUG:boto:Data: 2023-10-12T00:29:05.001 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:05.002 DEBUG:boto:Port: 8001 2023-10-12T00:29:05.002 DEBUG:boto:Params: {} 2023-10-12T00:29:05.003 DEBUG:boto:Token: None 2023-10-12T00:29:05.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/obj 2023-10-12T00:29:05.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sj7Iau19c6Fe/FHfkG2DPcClMJI= 2023-10-12T00:29:05.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sj7Iau19c6Fe/FHfkG2DPcClMJI=', 'Content-Length': '0'} 2023-10-12T00:29:05.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001862301bcb0096c1-0065273dd1-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.007 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:05.007 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:05.008 DEBUG:boto:Method: HEAD 2023-10-12T00:29:05.008 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:05.008 DEBUG:boto:Data: 2023-10-12T00:29:05.009 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:05.010 DEBUG:boto:Port: 8000 2023-10-12T00:29:05.010 DEBUG:boto:Params: {} 2023-10-12T00:29:05.010 DEBUG:boto:Token: None 2023-10-12T00:29:05.011 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/obj 2023-10-12T00:29:05.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sj7Iau19c6Fe/FHfkG2DPcClMJI= 2023-10-12T00:29:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sj7Iau19c6Fe/FHfkG2DPcClMJI=', 'Content-Length': '0'} 2023-10-12T00:29:05.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029afb4e6f65d2dd4-0065273dd1-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:05.015 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:05.015 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:05.016 DEBUG:boto:Method: GET 2023-10-12T00:29:05.016 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:05.017 DEBUG:boto:Data: 2023-10-12T00:29:05.017 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:05.018 DEBUG:boto:Port: 8001 2023-10-12T00:29:05.018 DEBUG:boto:Params: {} 2023-10-12T00:29:05.019 DEBUG:boto:Token: None 2023-10-12T00:29:05.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/obj 2023-10-12T00:29:05.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NKpc19GXhganVWTWXa56N1Z9A1c= 2023-10-12T00:29:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NKpc19GXhganVWTWXa56N1Z9A1c=', 'Content-Length': '0'} 2023-10-12T00:29:05.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4915fa775fb12e9-0065273dd1-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.023 DEBUG:boto:path=/wafmoa-14/obj 2023-10-12T00:29:05.023 DEBUG:boto:auth_path=/wafmoa-14/obj 2023-10-12T00:29:05.024 DEBUG:boto:Method: GET 2023-10-12T00:29:05.024 DEBUG:boto:Path: /wafmoa-14/obj 2023-10-12T00:29:05.024 DEBUG:boto:Data: 2023-10-12T00:29:05.025 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:05.026 DEBUG:boto:Port: 8000 2023-10-12T00:29:05.026 DEBUG:boto:Params: {} 2023-10-12T00:29:05.027 DEBUG:boto:Token: None 2023-10-12T00:29:05.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/obj 2023-10-12T00:29:05.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NKpc19GXhganVWTWXa56N1Z9A1c= 2023-10-12T00:29:05.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NKpc19GXhganVWTWXa56N1Z9A1c=', 'Content-Length': '0'} 2023-10-12T00:29:05.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:28:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bc2a616e142883e-0065273dd1-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.031 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:05.031 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:05.032 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:05.032 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:05.032 DEBUG:boto:Method: GET 2023-10-12T00:29:05.033 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:05.033 DEBUG:boto:Data: 2023-10-12T00:29:05.034 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:05.035 DEBUG:boto:Port: 8001 2023-10-12T00:29:05.035 DEBUG:boto:Params: {} 2023-10-12T00:29:05.036 DEBUG:boto:Token: None 2023-10-12T00:29:05.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/ 2023-10-12T00:29:05.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADRp5OJbhFCGEte0d9LVOH8aUXw= 2023-10-12T00:29:05.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADRp5OJbhFCGEte0d9LVOH8aUXw=', 'Content-Length': '0'} 2023-10-12T00:29:05.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be0a17f71931642a-0065273dd1-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.040 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:05.041 DEBUG:boto:path=/wafmoa-14/ 2023-10-12T00:29:05.041 DEBUG:boto:auth_path=/wafmoa-14/ 2023-10-12T00:29:05.042 DEBUG:boto:path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:05.042 DEBUG:boto:auth_path=/wafmoa-14/?prefix=obj 2023-10-12T00:29:05.042 DEBUG:boto:Method: GET 2023-10-12T00:29:05.043 DEBUG:boto:Path: /wafmoa-14/?prefix=obj 2023-10-12T00:29:05.043 DEBUG:boto:Data: 2023-10-12T00:29:05.044 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:05.045 DEBUG:boto:Port: 8000 2023-10-12T00:29:05.045 DEBUG:boto:Params: {} 2023-10-12T00:29:05.046 DEBUG:boto:Token: None 2023-10-12T00:29:05.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-14/ 2023-10-12T00:29:05.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADRp5OJbhFCGEte0d9LVOH8aUXw= 2023-10-12T00:29:05.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADRp5OJbhFCGEte0d9LVOH8aUXw=', 'Content-Length': '0'} 2023-10-12T00:29:05.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b89cb0a78fb073-0065273dd1-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.050 DEBUG:boto:b'wafmoa-14obj1000falseobj2023-10-12T00:28:29.399Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:05.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-14 zones={a2, a1} 2023-10-12T00:29:05.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-10-12T00:29:05.053 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-15 2023-10-12T00:29:05.053 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:05.053 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:05.054 DEBUG:boto:Method: PUT 2023-10-12T00:29:05.054 DEBUG:boto:Path: /wafmoa-15/ 2023-10-12T00:29:05.055 DEBUG:boto:Data: 2023-10-12T00:29:05.055 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:05.056 DEBUG:boto:Port: 8000 2023-10-12T00:29:05.056 DEBUG:boto:Params: {} 2023-10-12T00:29:05.057 DEBUG:boto:Token: None 2023-10-12T00:29:05.057 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-15/ 2023-10-12T00:29:05.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QtLY8WJFBiONTi3y7KwytJE5UrM= 2023-10-12T00:29:05.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QtLY8WJFBiONTi3y7KwytJE5UrM=', 'Content-Length': '0'} 2023-10-12T00:29:05.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000004edded8a5cb7ac9-0065273dd1-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.075 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-16 2023-10-12T00:29:05.075 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:05.076 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:05.076 DEBUG:boto:Method: PUT 2023-10-12T00:29:05.077 DEBUG:boto:Path: /wafmoa-16/ 2023-10-12T00:29:05.077 DEBUG:boto:Data: 2023-10-12T00:29:05.077 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:05.078 DEBUG:boto:Port: 8001 2023-10-12T00:29:05.079 DEBUG:boto:Params: {} 2023-10-12T00:29:05.079 DEBUG:boto:Token: None 2023-10-12T00:29:05.080 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-16/ 2023-10-12T00:29:05.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bJIscROUd+94W+PX5fFpFANcuUA= 2023-10-12T00:29:05.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bJIscROUd+94W+PX5fFpFANcuUA=', 'Content-Length': '0'} 2023-10-12T00:29:05.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002e133b5928f1791-0065273dd1-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.111 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:05.112 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:05.112 DEBUG:boto:path=/wafmoa-15/?versioning 2023-10-12T00:29:05.112 DEBUG:boto:auth_path=/wafmoa-15/?versioning 2023-10-12T00:29:05.113 DEBUG:boto:Method: PUT 2023-10-12T00:29:05.113 DEBUG:boto:Path: /wafmoa-15/?versioning 2023-10-12T00:29:05.114 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:29:05.114 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:05.115 DEBUG:boto:Port: 8000 2023-10-12T00:29:05.115 DEBUG:boto:Params: {} 2023-10-12T00:29:05.116 DEBUG:boto:Token: None 2023-10-12T00:29:05.116 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-15/?versioning 2023-10-12T00:29:05.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rpt5pyxvxj9JjF70HiAqhCdCxBU= 2023-10-12T00:29:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rpt5pyxvxj9JjF70HiAqhCdCxBU=', 'Content-Length': '228'} 2023-10-12T00:29:05.129 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000767c1d1742c53112-0065273dd1-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.130 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:05.130 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:05.131 DEBUG:boto:path=/wafmoa-16/?versioning 2023-10-12T00:29:05.131 DEBUG:boto:auth_path=/wafmoa-16/?versioning 2023-10-12T00:29:05.131 DEBUG:boto:Method: PUT 2023-10-12T00:29:05.132 DEBUG:boto:Path: /wafmoa-16/?versioning 2023-10-12T00:29:05.132 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:29:05.132 DEBUG:boto:Headers: {} 2023-10-12T00:29:05.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:05.133 DEBUG:boto:Port: 8001 2023-10-12T00:29:05.134 DEBUG:boto:Params: {} 2023-10-12T00:29:05.134 DEBUG:boto:Token: None 2023-10-12T00:29:05.135 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:05 GMT /wafmoa-16/?versioning 2023-10-12T00:29:05.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jaYbb+gf959oetiCoVXN6RqAW2s= 2023-10-12T00:29:05.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jaYbb+gf959oetiCoVXN6RqAW2s=', 'Content-Length': '228'} 2023-10-12T00:29:05.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e68606d338448fe-0065273dd1-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:05.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:05.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:05.144 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:05.225 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:29:05.237 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:29:05.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.237 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:05.237 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:29:05.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:05.238 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:29:05.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.239 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070545.141648_137.1", 2023-10-12T00:29:05.239 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:29:05.141648Z" 2023-10-12T00:29:05.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.240 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:05.240 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:29:05.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.240 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:29:05.241 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070545.141648_137.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:05.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:29:05.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:05.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:05.243 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:05.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:05.298 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:05.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:05.332+0000 7fef2582c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:05.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:05.333+0000 7fef2582c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:05.344 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:29:05.344 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:29:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:29:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:29:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:29:05.345 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:29:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:29:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:29:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:29:05.348 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:29:05.349 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:29:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:05.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:05.352 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:29:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.354 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:29:05.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:05.354 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:05.354 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070545.069942_128.1", 2023-10-12T00:29:05.355 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:05.355 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:05.355 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:05.355 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:29:05.069942Z", 2023-10-12T00:29:05.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:05.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:05.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:29:05.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:05.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:05.358 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:05.358 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:05.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:05.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:05.359 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:29:05.359 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:05.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:05.360 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:29:05.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.361 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:29:05.361 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:29:05.361 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:29:05.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:05.362 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:29:05Z" 2023-10-12T00:29:05.362 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:29:05.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070545.069942_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:29:05.069942Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:29:05Z" } 2023-10-12T00:29:05.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070545.069942_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:29:05.069942Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:29:05.364 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070545.141648_137.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:05.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070545.069942_128.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:05.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070545.141648_137.1 target=1_1697070545.069942_128.1 2023-10-12T00:29:10.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:10.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:10.368 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:10.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:10.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:10.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:10.454+0000 7f45fe95c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:10.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:10.455+0000 7f45fe95c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:10.468 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:29:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:29:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:29:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:29:10.469 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:29:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:29:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.470 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:29:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:29:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:10.471 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:29:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:10.472 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:29:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:10.473 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:29:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:10.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:10.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:10.475 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:29:10.475 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:10.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:10.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:10.476 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:29:10.476 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:10.476 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:10.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070545.141648_137.1", 2023-10-12T00:29:10.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:10.477 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:10.478 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:10.478 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:29:05.141648Z", 2023-10-12T00:29:10.478 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:10.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:10.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:10.479 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:29:10.479 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:10.479 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:10.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:10.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:10.480 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:10.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:10.480 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:29:10.480 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:10.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:10.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:10.481 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:29:10.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.482 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:29:10.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:29:10.482 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:29:10.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:10.483 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:29:10Z" 2023-10-12T00:29:10.483 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:29:10.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070545.141648_137.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:29:05.141648Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:29:10Z" } 2023-10-12T00:29:10.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070545.141648_137.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:29:05.141648Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:29:10.485 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070545.141648_137.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:10.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070545.141648_137.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:10.486 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:29:10.486 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:10.487 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:10.487 DEBUG:boto:Method: HEAD 2023-10-12T00:29:10.488 DEBUG:boto:Path: /wafmoa-15/ 2023-10-12T00:29:10.488 DEBUG:boto:Data: 2023-10-12T00:29:10.489 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:10.489 DEBUG:boto:Port: 8000 2023-10-12T00:29:10.490 DEBUG:boto:Params: {} 2023-10-12T00:29:10.490 DEBUG:boto:Token: None 2023-10-12T00:29:10.491 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-15/ 2023-10-12T00:29:10.491 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JFdtCmKsAcJJuY5F0U5LUxreLHM= 2023-10-12T00:29:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JFdtCmKsAcJJuY5F0U5LUxreLHM=', 'Content-Length': '0'} 2023-10-12T00:29:10.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa3afce8e47b1227-0065273dd6-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.495 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:10.495 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:10.496 DEBUG:boto:Method: PUT 2023-10-12T00:29:10.496 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:10.497 DEBUG:boto:Data: 2023-10-12T00:29:10.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:29:10.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:10.498 DEBUG:boto:Port: 8000 2023-10-12T00:29:10.498 DEBUG:boto:Params: {} 2023-10-12T00:29:10.499 DEBUG:boto:Token: None 2023-10-12T00:29:10.499 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-15/obj 2023-10-12T00:29:10.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oj7aNkj3SPKCkQnrCvGPWG0iBkw= 2023-10-12T00:29:10.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oj7aNkj3SPKCkQnrCvGPWG0iBkw='} 2023-10-12T00:29:10.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-amz-request-id', 'tx0000054d3d1d2e99afd99-0065273dd6-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.515 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:10.515 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:10.516 DEBUG:boto:Method: HEAD 2023-10-12T00:29:10.516 DEBUG:boto:Path: /wafmoa-15/ 2023-10-12T00:29:10.517 DEBUG:boto:Data: 2023-10-12T00:29:10.517 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:10.518 DEBUG:boto:Port: 8000 2023-10-12T00:29:10.518 DEBUG:boto:Params: {} 2023-10-12T00:29:10.519 DEBUG:boto:Token: None 2023-10-12T00:29:10.519 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-15/ 2023-10-12T00:29:10.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JFdtCmKsAcJJuY5F0U5LUxreLHM= 2023-10-12T00:29:10.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JFdtCmKsAcJJuY5F0U5LUxreLHM=', 'Content-Length': '0'} 2023-10-12T00:29:10.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001228d696259d1da2-0065273dd6-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.523 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:10.523 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:10.523 DEBUG:boto:Method: DELETE 2023-10-12T00:29:10.524 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:10.524 DEBUG:boto:Data: 2023-10-12T00:29:10.525 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:10.525 DEBUG:boto:Port: 8000 2023-10-12T00:29:10.526 DEBUG:boto:Params: {} 2023-10-12T00:29:10.526 DEBUG:boto:Token: None 2023-10-12T00:29:10.527 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-15/obj 2023-10-12T00:29:10.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oPuHyl3kOcISJ8X5I54itZ7Sscs= 2023-10-12T00:29:10.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oPuHyl3kOcISJ8X5I54itZ7Sscs=', 'Content-Length': '0'} 2023-10-12T00:29:10.534 DEBUG:boto:Response headers: [('x-amz-version-id', 't5SR6vaZB9BGSIgVHptqPYgPYivS65P'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000067a018e061038348-0065273dd6-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.535 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:10.536 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:10.536 DEBUG:boto:Method: HEAD 2023-10-12T00:29:10.537 DEBUG:boto:Path: /wafmoa-16/ 2023-10-12T00:29:10.537 DEBUG:boto:Data: 2023-10-12T00:29:10.538 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:10.539 DEBUG:boto:Port: 8001 2023-10-12T00:29:10.540 DEBUG:boto:Params: {} 2023-10-12T00:29:10.541 DEBUG:boto:Token: None 2023-10-12T00:29:10.541 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-16/ 2023-10-12T00:29:10.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AqP4ylSgoWfJOKNoAiex5oufJig= 2023-10-12T00:29:10.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AqP4ylSgoWfJOKNoAiex5oufJig=', 'Content-Length': '0'} 2023-10-12T00:29:10.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebf4d40f7d4cff25-0065273dd6-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.546 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:10.546 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:10.547 DEBUG:boto:Method: PUT 2023-10-12T00:29:10.547 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:10.548 DEBUG:boto:Data: 2023-10-12T00:29:10.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:29:10.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:10.549 DEBUG:boto:Port: 8001 2023-10-12T00:29:10.549 DEBUG:boto:Params: {} 2023-10-12T00:29:10.550 DEBUG:boto:Token: None 2023-10-12T00:29:10.550 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-16/obj 2023-10-12T00:29:10.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NunQjBP9eCXx8ybE9JNengg3SwE= 2023-10-12T00:29:10.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NunQjBP9eCXx8ybE9JNengg3SwE='} 2023-10-12T00:29:10.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-amz-request-id', 'tx00000eb3ce231275c3897-0065273dd6-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.564 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:10.565 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:10.565 DEBUG:boto:Method: HEAD 2023-10-12T00:29:10.566 DEBUG:boto:Path: /wafmoa-16/ 2023-10-12T00:29:10.566 DEBUG:boto:Data: 2023-10-12T00:29:10.566 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:10.567 DEBUG:boto:Port: 8001 2023-10-12T00:29:10.568 DEBUG:boto:Params: {} 2023-10-12T00:29:10.568 DEBUG:boto:Token: None 2023-10-12T00:29:10.569 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-16/ 2023-10-12T00:29:10.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AqP4ylSgoWfJOKNoAiex5oufJig= 2023-10-12T00:29:10.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AqP4ylSgoWfJOKNoAiex5oufJig=', 'Content-Length': '0'} 2023-10-12T00:29:10.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4ffa8f3855e2752-0065273dd6-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.572 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:10.573 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:10.573 DEBUG:boto:Method: DELETE 2023-10-12T00:29:10.574 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:10.574 DEBUG:boto:Data: 2023-10-12T00:29:10.575 DEBUG:boto:Headers: {} 2023-10-12T00:29:10.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:10.575 DEBUG:boto:Port: 8001 2023-10-12T00:29:10.576 DEBUG:boto:Params: {} 2023-10-12T00:29:10.576 DEBUG:boto:Token: None 2023-10-12T00:29:10.577 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:29:10 GMT /wafmoa-16/obj 2023-10-12T00:29:10.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/Yzh+2I+GCHL0Ru+1b+J4Lw9baU= 2023-10-12T00:29:10.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/Yzh+2I+GCHL0Ru+1b+J4Lw9baU=', 'Content-Length': '0'} 2023-10-12T00:29:10.584 DEBUG:boto:Response headers: [('x-amz-version-id', '3AT8yxdo-n6LquStIzynduaCF5EM5nA'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000dfb0cfac2364c34d-0065273dd6-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:10.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-15 2023-10-12T00:29:10.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:10.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:10.586 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:29:10.643 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:10.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:10.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:10.676+0000 7f2f4e257780 1 waiting to reach incremental sync.. 2023-10-12T00:29:15.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:15.677+0000 7f2f4e257780 1 waiting to reach incremental sync.. 2023-10-12T00:29:20.678 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:20.678+0000 7f2f4e257780 1 waiting to reach incremental sync.. 2023-10-12T00:29:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:25.680+0000 7f2f4e257780 1 bucket sync caught up with source: 2023-10-12T00:29:25.680 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.84.12, , ] 2023-10-12T00:29:25.680 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.84.12, , ] 2023-10-12T00:29:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:25.680+0000 7f2f4e257780 0 bucket checkpoint complete 2023-10-12T00:29:25.695 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-15 2023-10-12T00:29:25.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:25.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:25.697 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:29:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:25.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:25.797+0000 7f4c6e375780 1 waiting to reach incremental sync.. 2023-10-12T00:29:30.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:30.795+0000 7f4c6e375780 1 waiting to reach incremental sync.. 2023-10-12T00:29:35.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:35.796+0000 7f4c6e375780 1 waiting to reach incremental sync.. 2023-10-12T00:29:40.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:40.797+0000 7f4c6e375780 1 waiting to reach incremental sync.. 2023-10-12T00:29:45.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:45.798+0000 7f4c6e375780 1 bucket sync caught up with source: 2023-10-12T00:29:45.799 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.118.6, , ] 2023-10-12T00:29:45.799 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000002.118.6, , ] 2023-10-12T00:29:45.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:45.798+0000 7f4c6e375780 0 bucket checkpoint complete 2023-10-12T00:29:45.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-15 zones={a2, a1} 2023-10-12T00:29:45.816 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.816 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.817 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.817 DEBUG:boto:Path: /wafmoa-15/ 2023-10-12T00:29:45.818 DEBUG:boto:Data: 2023-10-12T00:29:45.818 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.819 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.820 DEBUG:boto:Params: {} 2023-10-12T00:29:45.820 DEBUG:boto:Token: None 2023-10-12T00:29:45.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/ 2023-10-12T00:29:45.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+YXB8UgaCBlgAm7sGRaw4kugMPQ= 2023-10-12T00:29:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+YXB8UgaCBlgAm7sGRaw4kugMPQ=', 'Content-Length': '0'} 2023-10-12T00:29:45.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062f6be4cdeb5d55d-0065273df9-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.826 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.827 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.827 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.828 DEBUG:boto:Path: /wafmoa-15/ 2023-10-12T00:29:45.828 DEBUG:boto:Data: 2023-10-12T00:29:45.829 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.830 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.831 DEBUG:boto:Params: {} 2023-10-12T00:29:45.831 DEBUG:boto:Token: None 2023-10-12T00:29:45.832 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/ 2023-10-12T00:29:45.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+YXB8UgaCBlgAm7sGRaw4kugMPQ= 2023-10-12T00:29:45.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+YXB8UgaCBlgAm7sGRaw4kugMPQ=', 'Content-Length': '0'} 2023-10-12T00:29:45.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d96ab8f97584941f-0065273df9-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:29:45.838 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.839 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.839 DEBUG:boto:path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.840 DEBUG:boto:auth_path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.840 DEBUG:boto:Method: GET 2023-10-12T00:29:45.841 DEBUG:boto:Path: /wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.841 DEBUG:boto:Data: 2023-10-12T00:29:45.842 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.843 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.844 DEBUG:boto:Params: {} 2023-10-12T00:29:45.844 DEBUG:boto:Token: None 2023-10-12T00:29:45.845 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/?versions 2023-10-12T00:29:45.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU= 2023-10-12T00:29:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU=', 'Content-Length': '0'} 2023-10-12T00:29:45.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdbfbe607819cb6-0065273df9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.850 DEBUG:boto:b'wafmoa-15999falseobjt5SR6vaZB9BGSIgVHptqPYgPYivS65Ptrue2023-10-12T00:29:10.531Zrgw-multisite-test-userMultisite Test UserNormalobjKJuPuTfyoZsvbfH21w1NqVOwLJt5HKPfalse2023-10-12T00:29:10.503Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:45.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:45.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:45.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:29:45.852 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.853 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.853 DEBUG:boto:path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.854 DEBUG:boto:auth_path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.854 DEBUG:boto:Method: GET 2023-10-12T00:29:45.854 DEBUG:boto:Path: /wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.855 DEBUG:boto:Data: 2023-10-12T00:29:45.855 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.856 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.856 DEBUG:boto:Params: {} 2023-10-12T00:29:45.857 DEBUG:boto:Token: None 2023-10-12T00:29:45.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/?versions 2023-10-12T00:29:45.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU= 2023-10-12T00:29:45.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU=', 'Content-Length': '0'} 2023-10-12T00:29:45.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e971c71eb6825b-0065273df9-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.862 DEBUG:boto:b'wafmoa-15999falseobjt5SR6vaZB9BGSIgVHptqPYgPYivS65Ptrue2023-10-12T00:29:10.531Zrgw-multisite-test-userMultisite Test UserNormalobjKJuPuTfyoZsvbfH21w1NqVOwLJt5HKPfalse2023-10-12T00:29:10.503Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:45.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:45.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:45.863 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.864 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.864 DEBUG:boto:path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.865 DEBUG:boto:auth_path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.865 DEBUG:boto:Method: GET 2023-10-12T00:29:45.866 DEBUG:boto:Path: /wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.866 DEBUG:boto:Data: 2023-10-12T00:29:45.867 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.868 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.868 DEBUG:boto:Params: {} 2023-10-12T00:29:45.869 DEBUG:boto:Token: None 2023-10-12T00:29:45.869 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/?versions 2023-10-12T00:29:45.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU= 2023-10-12T00:29:45.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU=', 'Content-Length': '0'} 2023-10-12T00:29:45.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752370ecff09abe7-0065273df9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.874 DEBUG:boto:b'wafmoa-15999falseobjt5SR6vaZB9BGSIgVHptqPYgPYivS65Ptrue2023-10-12T00:29:10.531Zrgw-multisite-test-userMultisite Test UserNormalobjKJuPuTfyoZsvbfH21w1NqVOwLJt5HKPfalse2023-10-12T00:29:10.503Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:45.874 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.875 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.875 DEBUG:boto:path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.876 DEBUG:boto:auth_path=/wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.876 DEBUG:boto:Method: GET 2023-10-12T00:29:45.876 DEBUG:boto:Path: /wafmoa-15/?versions&max-keys=999 2023-10-12T00:29:45.877 DEBUG:boto:Data: 2023-10-12T00:29:45.877 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.878 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.878 DEBUG:boto:Params: {} 2023-10-12T00:29:45.879 DEBUG:boto:Token: None 2023-10-12T00:29:45.879 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/?versions 2023-10-12T00:29:45.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU= 2023-10-12T00:29:45.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3/hVlpZ6wfafuZjXFxRCsaFvqU=', 'Content-Length': '0'} 2023-10-12T00:29:45.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004abec7f73b7a85a9-0065273df9-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.883 DEBUG:boto:b'wafmoa-15999falseobjt5SR6vaZB9BGSIgVHptqPYgPYivS65Ptrue2023-10-12T00:29:10.531Zrgw-multisite-test-userMultisite Test UserNormalobjKJuPuTfyoZsvbfH21w1NqVOwLJt5HKPfalse2023-10-12T00:29:10.503Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:45.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:45.884 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.885 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.885 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.886 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:45.886 DEBUG:boto:Data: 2023-10-12T00:29:45.886 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.887 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.888 DEBUG:boto:Params: {} 2023-10-12T00:29:45.888 DEBUG:boto:Token: None 2023-10-12T00:29:45.889 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj 2023-10-12T00:29:45.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4= 2023-10-12T00:29:45.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4=', 'Content-Length': '0'} 2023-10-12T00:29:45.892 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bb7a6e74d7ec9a50-0065273df9-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.892 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.893 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.893 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.894 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:45.894 DEBUG:boto:Data: 2023-10-12T00:29:45.894 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.895 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.896 DEBUG:boto:Params: {} 2023-10-12T00:29:45.896 DEBUG:boto:Token: None 2023-10-12T00:29:45.897 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj 2023-10-12T00:29:45.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4= 2023-10-12T00:29:45.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4=', 'Content-Length': '0'} 2023-10-12T00:29:45.900 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007526f8f74eac5e74-0065273df9-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:45.901 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.901 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.902 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.902 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.903 DEBUG:boto:Method: GET 2023-10-12T00:29:45.903 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.903 DEBUG:boto:Data: 2023-10-12T00:29:45.904 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.905 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.905 DEBUG:boto:Params: {} 2023-10-12T00:29:45.906 DEBUG:boto:Token: None 2023-10-12T00:29:45.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0= 2023-10-12T00:29:45.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0=', 'Content-Length': '0'} 2023-10-12T00:29:45.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025fe48279a6dbc2f-0065273df9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.910 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.910 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.911 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.911 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.911 DEBUG:boto:Method: GET 2023-10-12T00:29:45.912 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.912 DEBUG:boto:Data: 2023-10-12T00:29:45.913 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.914 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.914 DEBUG:boto:Params: {} 2023-10-12T00:29:45.914 DEBUG:boto:Token: None 2023-10-12T00:29:45.915 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0= 2023-10-12T00:29:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0=', 'Content-Length': '0'} 2023-10-12T00:29:45.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d57c099c2ee9564-0065273df9-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.918 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.919 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.919 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.920 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.920 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.920 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.921 DEBUG:boto:Data: 2023-10-12T00:29:45.921 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.922 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.922 DEBUG:boto:Params: {} 2023-10-12T00:29:45.923 DEBUG:boto:Token: None 2023-10-12T00:29:45.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ofVAhKV/4j4r7J9fKYgqTx1kZ+0= 2023-10-12T00:29:45.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ofVAhKV/4j4r7J9fKYgqTx1kZ+0=', 'Content-Length': '0'} 2023-10-12T00:29:45.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072881d244243d7c6-0065273df9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.927 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.927 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.928 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.928 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.928 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.929 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.929 DEBUG:boto:Data: 2023-10-12T00:29:45.930 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.930 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.931 DEBUG:boto:Params: {} 2023-10-12T00:29:45.931 DEBUG:boto:Token: None 2023-10-12T00:29:45.932 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ofVAhKV/4j4r7J9fKYgqTx1kZ+0= 2023-10-12T00:29:45.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ofVAhKV/4j4r7J9fKYgqTx1kZ+0=', 'Content-Length': '0'} 2023-10-12T00:29:45.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc8d3f7ce854b370-0065273df9-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:45.935 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.936 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.936 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.937 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.937 DEBUG:boto:Method: GET 2023-10-12T00:29:45.938 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.938 DEBUG:boto:Data: 2023-10-12T00:29:45.939 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.939 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.940 DEBUG:boto:Params: {} 2023-10-12T00:29:45.940 DEBUG:boto:Token: None 2023-10-12T00:29:45.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0= 2023-10-12T00:29:45.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0=', 'Content-Length': '0'} 2023-10-12T00:29:45.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a5403a87a299b0b-0065273df9-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.945 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.945 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.945 DEBUG:boto:path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.946 DEBUG:boto:auth_path=/wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.946 DEBUG:boto:Method: GET 2023-10-12T00:29:45.947 DEBUG:boto:Path: /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.947 DEBUG:boto:Data: 2023-10-12T00:29:45.948 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.948 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.949 DEBUG:boto:Params: {} 2023-10-12T00:29:45.949 DEBUG:boto:Token: None 2023-10-12T00:29:45.950 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj?versionId=KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP 2023-10-12T00:29:45.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0= 2023-10-12T00:29:45.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xe6YnfHzwZRQltXMAzSN7y79tk0=', 'Content-Length': '0'} 2023-10-12T00:29:45.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'KJuPuTfyoZsvbfH21w1NqVOwLJt5HKP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050d5ed64d652a7b5-0065273df9-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.953 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.954 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.954 DEBUG:boto:path=/wafmoa-15/?prefix=obj 2023-10-12T00:29:45.955 DEBUG:boto:auth_path=/wafmoa-15/?prefix=obj 2023-10-12T00:29:45.955 DEBUG:boto:Method: GET 2023-10-12T00:29:45.955 DEBUG:boto:Path: /wafmoa-15/?prefix=obj 2023-10-12T00:29:45.956 DEBUG:boto:Data: 2023-10-12T00:29:45.956 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.957 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.957 DEBUG:boto:Params: {} 2023-10-12T00:29:45.958 DEBUG:boto:Token: None 2023-10-12T00:29:45.958 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/ 2023-10-12T00:29:45.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:83NcPAMigYjx/ER/PnDcPBuJbrQ= 2023-10-12T00:29:45.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:83NcPAMigYjx/ER/PnDcPBuJbrQ=', 'Content-Length': '0'} 2023-10-12T00:29:45.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055c17daa4d3df158-0065273df9-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.964 DEBUG:boto:b'wafmoa-15obj1000false' 2023-10-12T00:29:45.965 DEBUG:boto:path=/wafmoa-15/ 2023-10-12T00:29:45.965 DEBUG:boto:auth_path=/wafmoa-15/ 2023-10-12T00:29:45.965 DEBUG:boto:path=/wafmoa-15/?prefix=obj 2023-10-12T00:29:45.966 DEBUG:boto:auth_path=/wafmoa-15/?prefix=obj 2023-10-12T00:29:45.966 DEBUG:boto:Method: GET 2023-10-12T00:29:45.967 DEBUG:boto:Path: /wafmoa-15/?prefix=obj 2023-10-12T00:29:45.967 DEBUG:boto:Data: 2023-10-12T00:29:45.968 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.968 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.969 DEBUG:boto:Params: {} 2023-10-12T00:29:45.969 DEBUG:boto:Token: None 2023-10-12T00:29:45.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/ 2023-10-12T00:29:45.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:83NcPAMigYjx/ER/PnDcPBuJbrQ= 2023-10-12T00:29:45.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:83NcPAMigYjx/ER/PnDcPBuJbrQ=', 'Content-Length': '0'} 2023-10-12T00:29:45.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e78c106d9af2466-0065273df9-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.974 DEBUG:boto:b'wafmoa-15obj1000false' 2023-10-12T00:29:45.975 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.975 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.975 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.976 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:45.976 DEBUG:boto:Data: 2023-10-12T00:29:45.977 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:45.977 DEBUG:boto:Port: 8001 2023-10-12T00:29:45.978 DEBUG:boto:Params: {} 2023-10-12T00:29:45.978 DEBUG:boto:Token: None 2023-10-12T00:29:45.979 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj 2023-10-12T00:29:45.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4= 2023-10-12T00:29:45.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4=', 'Content-Length': '0'} 2023-10-12T00:29:45.982 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000401ab0202fdba56-0065273df9-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.982 DEBUG:boto:path=/wafmoa-15/obj 2023-10-12T00:29:45.983 DEBUG:boto:auth_path=/wafmoa-15/obj 2023-10-12T00:29:45.983 DEBUG:boto:Method: HEAD 2023-10-12T00:29:45.984 DEBUG:boto:Path: /wafmoa-15/obj 2023-10-12T00:29:45.984 DEBUG:boto:Data: 2023-10-12T00:29:45.984 DEBUG:boto:Headers: {} 2023-10-12T00:29:45.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:45.985 DEBUG:boto:Port: 8000 2023-10-12T00:29:45.986 DEBUG:boto:Params: {} 2023-10-12T00:29:45.986 DEBUG:boto:Token: None 2023-10-12T00:29:45.987 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:45 GMT /wafmoa-15/obj 2023-10-12T00:29:45.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4= 2023-10-12T00:29:45.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g+HkfQ8pSuP/VSRPWnZeXsGgzS4=', 'Content-Length': '0'} 2023-10-12T00:29:45.990 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004eb266bc0a253da9-0065273df9-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:45.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-15 zones={a2, a1} 2023-10-12T00:29:45.991 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-16 2023-10-12T00:29:45.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:45.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:45.992 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:29:46.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:46.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.092+0000 7fb479ad0780 1 bucket sync caught up with source: 2023-10-12T00:29:46.093 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.19.6, , ] 2023-10-12T00:29:46.093 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000002.19.6, , ] 2023-10-12T00:29:46.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.092+0000 7fb479ad0780 0 bucket checkpoint complete 2023-10-12T00:29:46.106 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-16 2023-10-12T00:29:46.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:46.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:46.107 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:29:46.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:46.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:46.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.199+0000 7f082dce4780 1 bucket sync caught up with source: 2023-10-12T00:29:46.201 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.60.12, , ] 2023-10-12T00:29:46.201 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.60.12, , ] 2023-10-12T00:29:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.199+0000 7f082dce4780 0 bucket checkpoint complete 2023-10-12T00:29:46.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-16 zones={a2, a1} 2023-10-12T00:29:46.213 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.213 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.214 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.214 DEBUG:boto:Path: /wafmoa-16/ 2023-10-12T00:29:46.214 DEBUG:boto:Data: 2023-10-12T00:29:46.215 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.216 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.216 DEBUG:boto:Params: {} 2023-10-12T00:29:46.217 DEBUG:boto:Token: None 2023-10-12T00:29:46.217 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/ 2023-10-12T00:29:46.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oSoQtxdCzg3swTOlL7t/EGfT2pg= 2023-10-12T00:29:46.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oSoQtxdCzg3swTOlL7t/EGfT2pg=', 'Content-Length': '0'} 2023-10-12T00:29:46.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7972760cadb8314-0065273dfa-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.221 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.221 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.222 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.222 DEBUG:boto:Path: /wafmoa-16/ 2023-10-12T00:29:46.223 DEBUG:boto:Data: 2023-10-12T00:29:46.223 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.224 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.224 DEBUG:boto:Params: {} 2023-10-12T00:29:46.225 DEBUG:boto:Token: None 2023-10-12T00:29:46.225 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/ 2023-10-12T00:29:46.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oSoQtxdCzg3swTOlL7t/EGfT2pg= 2023-10-12T00:29:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oSoQtxdCzg3swTOlL7t/EGfT2pg=', 'Content-Length': '0'} 2023-10-12T00:29:46.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f344dfde907b5d7b-0065273dfa-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.229 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:29:46.229 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.229 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.230 DEBUG:boto:path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.230 DEBUG:boto:auth_path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.231 DEBUG:boto:Method: GET 2023-10-12T00:29:46.231 DEBUG:boto:Path: /wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.232 DEBUG:boto:Data: 2023-10-12T00:29:46.232 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.233 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.234 DEBUG:boto:Params: {} 2023-10-12T00:29:46.234 DEBUG:boto:Token: None 2023-10-12T00:29:46.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/?versions 2023-10-12T00:29:46.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c= 2023-10-12T00:29:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c=', 'Content-Length': '0'} 2023-10-12T00:29:46.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0c61819d7d8833-0065273dfa-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.239 DEBUG:boto:b'wafmoa-16999falseobj3AT8yxdo-n6LquStIzynduaCF5EM5nAtrue2023-10-12T00:29:10.581Zrgw-multisite-test-userMultisite Test UserNormalobjKy.8eU2AnwJLCkNSE2qYWo.L9qcwrBXfalse2023-10-12T00:29:10.554Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:46.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:46.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:46.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:29:46.241 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.241 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.241 DEBUG:boto:path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.242 DEBUG:boto:auth_path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.242 DEBUG:boto:Method: GET 2023-10-12T00:29:46.243 DEBUG:boto:Path: /wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.243 DEBUG:boto:Data: 2023-10-12T00:29:46.243 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.244 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.245 DEBUG:boto:Params: {} 2023-10-12T00:29:46.245 DEBUG:boto:Token: None 2023-10-12T00:29:46.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/?versions 2023-10-12T00:29:46.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c= 2023-10-12T00:29:46.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c=', 'Content-Length': '0'} 2023-10-12T00:29:46.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11e7d9c8d1acbba-0065273dfa-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.249 DEBUG:boto:b'wafmoa-16999falseobj3AT8yxdo-n6LquStIzynduaCF5EM5nAtrue2023-10-12T00:29:10.581Zrgw-multisite-test-userMultisite Test UserNormalobjKy.8eU2AnwJLCkNSE2qYWo.L9qcwrBXfalse2023-10-12T00:29:10.554Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:46.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:46.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:29:46.251 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.251 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.252 DEBUG:boto:path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.252 DEBUG:boto:auth_path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.252 DEBUG:boto:Method: GET 2023-10-12T00:29:46.253 DEBUG:boto:Path: /wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.253 DEBUG:boto:Data: 2023-10-12T00:29:46.254 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.255 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.255 DEBUG:boto:Params: {} 2023-10-12T00:29:46.255 DEBUG:boto:Token: None 2023-10-12T00:29:46.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/?versions 2023-10-12T00:29:46.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c= 2023-10-12T00:29:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c=', 'Content-Length': '0'} 2023-10-12T00:29:46.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce29addbec07398d-0065273dfa-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.260 DEBUG:boto:b'wafmoa-16999falseobj3AT8yxdo-n6LquStIzynduaCF5EM5nAtrue2023-10-12T00:29:10.581Zrgw-multisite-test-userMultisite Test UserNormalobjKy.8eU2AnwJLCkNSE2qYWo.L9qcwrBXfalse2023-10-12T00:29:10.554Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:46.260 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.261 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.261 DEBUG:boto:path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.262 DEBUG:boto:auth_path=/wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.262 DEBUG:boto:Method: GET 2023-10-12T00:29:46.262 DEBUG:boto:Path: /wafmoa-16/?versions&max-keys=999 2023-10-12T00:29:46.263 DEBUG:boto:Data: 2023-10-12T00:29:46.263 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.264 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.264 DEBUG:boto:Params: {} 2023-10-12T00:29:46.265 DEBUG:boto:Token: None 2023-10-12T00:29:46.265 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/?versions 2023-10-12T00:29:46.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c= 2023-10-12T00:29:46.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgPn0FyVrWZiaHPFXmccy5l3w/c=', 'Content-Length': '0'} 2023-10-12T00:29:46.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012730fb0364a2f18-0065273dfa-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.269 DEBUG:boto:b'wafmoa-16999falseobj3AT8yxdo-n6LquStIzynduaCF5EM5nAtrue2023-10-12T00:29:10.581Zrgw-multisite-test-userMultisite Test UserNormalobjKy.8eU2AnwJLCkNSE2qYWo.L9qcwrBXfalse2023-10-12T00:29:10.554Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:29:46.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:46.270 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.270 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.271 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.271 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:46.271 DEBUG:boto:Data: 2023-10-12T00:29:46.272 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.273 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.273 DEBUG:boto:Params: {} 2023-10-12T00:29:46.274 DEBUG:boto:Token: None 2023-10-12T00:29:46.274 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj 2023-10-12T00:29:46.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc= 2023-10-12T00:29:46.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc=', 'Content-Length': '0'} 2023-10-12T00:29:46.277 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a57058ebcffced75-0065273dfa-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.277 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.278 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.278 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.279 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:46.279 DEBUG:boto:Data: 2023-10-12T00:29:46.280 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.280 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.281 DEBUG:boto:Params: {} 2023-10-12T00:29:46.281 DEBUG:boto:Token: None 2023-10-12T00:29:46.282 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj 2023-10-12T00:29:46.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc= 2023-10-12T00:29:46.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc=', 'Content-Length': '0'} 2023-10-12T00:29:46.284 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000955d40b665147d01-0065273dfa-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:46.285 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.286 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.286 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.287 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.287 DEBUG:boto:Method: GET 2023-10-12T00:29:46.288 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.288 DEBUG:boto:Data: 2023-10-12T00:29:46.289 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.289 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.290 DEBUG:boto:Params: {} 2023-10-12T00:29:46.290 DEBUG:boto:Token: None 2023-10-12T00:29:46.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8= 2023-10-12T00:29:46.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8=', 'Content-Length': '0'} 2023-10-12T00:29:46.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9cfc630e87e1f21-0065273dfa-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.294 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.295 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.295 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.296 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.296 DEBUG:boto:Method: GET 2023-10-12T00:29:46.296 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.297 DEBUG:boto:Data: 2023-10-12T00:29:46.297 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.298 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.299 DEBUG:boto:Params: {} 2023-10-12T00:29:46.299 DEBUG:boto:Token: None 2023-10-12T00:29:46.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8= 2023-10-12T00:29:46.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8=', 'Content-Length': '0'} 2023-10-12T00:29:46.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040b2c3afadd0a830-0065273dfa-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.303 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.303 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.304 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.304 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.304 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.305 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.305 DEBUG:boto:Data: 2023-10-12T00:29:46.306 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.306 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.307 DEBUG:boto:Params: {} 2023-10-12T00:29:46.307 DEBUG:boto:Token: None 2023-10-12T00:29:46.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hduteSiLIPspA5r8vjL8Wf1xeRM= 2023-10-12T00:29:46.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hduteSiLIPspA5r8vjL8Wf1xeRM=', 'Content-Length': '0'} 2023-10-12T00:29:46.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063b7ff1d7abd858e-0065273dfa-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.311 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.311 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.312 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.312 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.312 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.313 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.313 DEBUG:boto:Data: 2023-10-12T00:29:46.314 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.314 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.315 DEBUG:boto:Params: {} 2023-10-12T00:29:46.315 DEBUG:boto:Token: None 2023-10-12T00:29:46.316 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hduteSiLIPspA5r8vjL8Wf1xeRM= 2023-10-12T00:29:46.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hduteSiLIPspA5r8vjL8Wf1xeRM=', 'Content-Length': '0'} 2023-10-12T00:29:46.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009084b47daeba414-0065273dfa-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:29:46.319 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.320 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.320 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.321 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.321 DEBUG:boto:Method: GET 2023-10-12T00:29:46.321 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.322 DEBUG:boto:Data: 2023-10-12T00:29:46.322 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.323 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.324 DEBUG:boto:Params: {} 2023-10-12T00:29:46.324 DEBUG:boto:Token: None 2023-10-12T00:29:46.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8= 2023-10-12T00:29:46.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8=', 'Content-Length': '0'} 2023-10-12T00:29:46.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f224e3f6df6749f-0065273dfa-1111-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.328 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.328 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.329 DEBUG:boto:path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.329 DEBUG:boto:auth_path=/wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.329 DEBUG:boto:Method: GET 2023-10-12T00:29:46.330 DEBUG:boto:Path: /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.330 DEBUG:boto:Data: 2023-10-12T00:29:46.331 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.331 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.332 DEBUG:boto:Params: {} 2023-10-12T00:29:46.332 DEBUG:boto:Token: None 2023-10-12T00:29:46.333 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj?versionId=Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX 2023-10-12T00:29:46.333 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8= 2023-10-12T00:29:46.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OAcinP1bdfWtZD9xfOVxoaISEY8=', 'Content-Length': '0'} 2023-10-12T00:29:46.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:29:10 GMT'), ('x-amz-version-id', 'Ky.8eU2AnwJLCkNSE2qYWo.L9qcwrBX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f461417c68a3704b-0065273dfa-1115-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.336 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.337 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.337 DEBUG:boto:path=/wafmoa-16/?prefix=obj 2023-10-12T00:29:46.337 DEBUG:boto:auth_path=/wafmoa-16/?prefix=obj 2023-10-12T00:29:46.338 DEBUG:boto:Method: GET 2023-10-12T00:29:46.338 DEBUG:boto:Path: /wafmoa-16/?prefix=obj 2023-10-12T00:29:46.339 DEBUG:boto:Data: 2023-10-12T00:29:46.339 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.340 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.341 DEBUG:boto:Params: {} 2023-10-12T00:29:46.341 DEBUG:boto:Token: None 2023-10-12T00:29:46.341 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/ 2023-10-12T00:29:46.342 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oLmeMNeyqNe7vfXdMfZoRc6jn7c= 2023-10-12T00:29:46.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oLmeMNeyqNe7vfXdMfZoRc6jn7c=', 'Content-Length': '0'} 2023-10-12T00:29:46.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d93f5d3a392a666-0065273dfa-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.390 DEBUG:boto:b'wafmoa-16obj1000false' 2023-10-12T00:29:46.391 DEBUG:boto:path=/wafmoa-16/ 2023-10-12T00:29:46.392 DEBUG:boto:auth_path=/wafmoa-16/ 2023-10-12T00:29:46.392 DEBUG:boto:path=/wafmoa-16/?prefix=obj 2023-10-12T00:29:46.393 DEBUG:boto:auth_path=/wafmoa-16/?prefix=obj 2023-10-12T00:29:46.393 DEBUG:boto:Method: GET 2023-10-12T00:29:46.394 DEBUG:boto:Path: /wafmoa-16/?prefix=obj 2023-10-12T00:29:46.394 DEBUG:boto:Data: 2023-10-12T00:29:46.394 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.395 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.396 DEBUG:boto:Params: {} 2023-10-12T00:29:46.396 DEBUG:boto:Token: None 2023-10-12T00:29:46.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/ 2023-10-12T00:29:46.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oLmeMNeyqNe7vfXdMfZoRc6jn7c= 2023-10-12T00:29:46.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oLmeMNeyqNe7vfXdMfZoRc6jn7c=', 'Content-Length': '0'} 2023-10-12T00:29:46.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580d27bbaf241142-0065273dfa-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.400 DEBUG:boto:b'wafmoa-16obj1000false' 2023-10-12T00:29:46.401 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.402 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.402 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.402 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:46.403 DEBUG:boto:Data: 2023-10-12T00:29:46.403 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.404 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.405 DEBUG:boto:Params: {} 2023-10-12T00:29:46.405 DEBUG:boto:Token: None 2023-10-12T00:29:46.405 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj 2023-10-12T00:29:46.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc= 2023-10-12T00:29:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc=', 'Content-Length': '0'} 2023-10-12T00:29:46.408 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d2adeff46814e2c4-0065273dfa-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.409 DEBUG:boto:path=/wafmoa-16/obj 2023-10-12T00:29:46.409 DEBUG:boto:auth_path=/wafmoa-16/obj 2023-10-12T00:29:46.410 DEBUG:boto:Method: HEAD 2023-10-12T00:29:46.410 DEBUG:boto:Path: /wafmoa-16/obj 2023-10-12T00:29:46.411 DEBUG:boto:Data: 2023-10-12T00:29:46.411 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.412 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.412 DEBUG:boto:Params: {} 2023-10-12T00:29:46.413 DEBUG:boto:Token: None 2023-10-12T00:29:46.413 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-16/obj 2023-10-12T00:29:46.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc= 2023-10-12T00:29:46.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HIcCmQ+8K4FfMLMedH4Nc7IyTc=', 'Content-Length': '0'} 2023-10-12T00:29:46.416 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000034b7841525a3a670-0065273dfa-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.417 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-16 zones={a2, a1} 2023-10-12T00:29:46.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-10-12T00:29:46.418 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-17 2023-10-12T00:29:46.419 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:29:46.419 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:29:46.420 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.420 DEBUG:boto:Path: /wafmoa-17/ 2023-10-12T00:29:46.420 DEBUG:boto:Data: 2023-10-12T00:29:46.421 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.422 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.422 DEBUG:boto:Params: {} 2023-10-12T00:29:46.423 DEBUG:boto:Token: None 2023-10-12T00:29:46.423 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-17/ 2023-10-12T00:29:46.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxJBILKc2TR4nBwaWMmtXVl+6jM= 2023-10-12T00:29:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxJBILKc2TR4nBwaWMmtXVl+6jM=', 'Content-Length': '0'} 2023-10-12T00:29:46.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001dc89dee44cf12a4-0065273dfa-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.437 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-18 2023-10-12T00:29:46.438 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:29:46.438 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:29:46.439 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.439 DEBUG:boto:Path: /wafmoa-18/ 2023-10-12T00:29:46.439 DEBUG:boto:Data: 2023-10-12T00:29:46.440 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.441 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.441 DEBUG:boto:Params: {} 2023-10-12T00:29:46.441 DEBUG:boto:Token: None 2023-10-12T00:29:46.442 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-18/ 2023-10-12T00:29:46.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XfzWgpiAzBxYgxGq5wmf6JZd1Es= 2023-10-12T00:29:46.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XfzWgpiAzBxYgxGq5wmf6JZd1Es=', 'Content-Length': '0'} 2023-10-12T00:29:46.465 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5b6cb74f0f76c95-0065273dfa-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.466 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:29:46.466 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:29:46.467 DEBUG:boto:path=/wafmoa-17/?versioning 2023-10-12T00:29:46.467 DEBUG:boto:auth_path=/wafmoa-17/?versioning 2023-10-12T00:29:46.467 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.468 DEBUG:boto:Path: /wafmoa-17/?versioning 2023-10-12T00:29:46.468 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:29:46.469 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.469 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.470 DEBUG:boto:Params: {} 2023-10-12T00:29:46.470 DEBUG:boto:Token: None 2023-10-12T00:29:46.471 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-17/?versioning 2023-10-12T00:29:46.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9YmddIMm/ac+hyA5Bvwxo2u4rCE= 2023-10-12T00:29:46.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9YmddIMm/ac+hyA5Bvwxo2u4rCE=', 'Content-Length': '228'} 2023-10-12T00:29:46.476 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f268c2352dad0d1a-0065273dfa-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.477 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:29:46.477 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:29:46.478 DEBUG:boto:path=/wafmoa-17/?versioning 2023-10-12T00:29:46.478 DEBUG:boto:auth_path=/wafmoa-17/?versioning 2023-10-12T00:29:46.478 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.479 DEBUG:boto:Path: /wafmoa-17/?versioning 2023-10-12T00:29:46.479 DEBUG:boto:Data: Suspended Disabled 2023-10-12T00:29:46.480 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:46.480 DEBUG:boto:Port: 8000 2023-10-12T00:29:46.481 DEBUG:boto:Params: {} 2023-10-12T00:29:46.481 DEBUG:boto:Token: None 2023-10-12T00:29:46.481 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-17/?versioning 2023-10-12T00:29:46.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9YmddIMm/ac+hyA5Bvwxo2u4rCE= 2023-10-12T00:29:46.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9YmddIMm/ac+hyA5Bvwxo2u4rCE=', 'Content-Length': '230'} 2023-10-12T00:29:46.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000074ecc47574faaaa4-0065273dfa-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.487 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:29:46.488 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:29:46.488 DEBUG:boto:path=/wafmoa-18/?versioning 2023-10-12T00:29:46.489 DEBUG:boto:auth_path=/wafmoa-18/?versioning 2023-10-12T00:29:46.489 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.489 DEBUG:boto:Path: /wafmoa-18/?versioning 2023-10-12T00:29:46.490 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:29:46.490 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.491 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.491 DEBUG:boto:Params: {} 2023-10-12T00:29:46.492 DEBUG:boto:Token: None 2023-10-12T00:29:46.492 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-18/?versioning 2023-10-12T00:29:46.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:orvu5aeeniFIYL5vkROpYk5/UiM= 2023-10-12T00:29:46.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:orvu5aeeniFIYL5vkROpYk5/UiM=', 'Content-Length': '228'} 2023-10-12T00:29:46.502 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f1f2e3b13ad4847-0065273dfa-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.503 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:29:46.503 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:29:46.503 DEBUG:boto:path=/wafmoa-18/?versioning 2023-10-12T00:29:46.504 DEBUG:boto:auth_path=/wafmoa-18/?versioning 2023-10-12T00:29:46.504 DEBUG:boto:Method: PUT 2023-10-12T00:29:46.505 DEBUG:boto:Path: /wafmoa-18/?versioning 2023-10-12T00:29:46.505 DEBUG:boto:Data: Suspended Disabled 2023-10-12T00:29:46.505 DEBUG:boto:Headers: {} 2023-10-12T00:29:46.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:46.506 DEBUG:boto:Port: 8001 2023-10-12T00:29:46.507 DEBUG:boto:Params: {} 2023-10-12T00:29:46.507 DEBUG:boto:Token: None 2023-10-12T00:29:46.508 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:29:46 GMT /wafmoa-18/?versioning 2023-10-12T00:29:46.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:orvu5aeeniFIYL5vkROpYk5/UiM= 2023-10-12T00:29:46.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:orvu5aeeniFIYL5vkROpYk5/UiM=', 'Content-Length': '230'} 2023-10-12T00:29:46.516 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000806683a012ff3b7f-0065273dfa-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:46.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:46.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:46.517 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:46.569 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:46.570 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:46.600 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:29:46.611 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:29:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:29:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:29:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070586.514221_154.1", 2023-10-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:29:46.514221Z" 2023-10-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:29:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.615 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:29:46.616 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070586.514221_154.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:46.617 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:29:46.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:46.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:46.618 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:46.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:46.668 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:46.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.699+0000 7f0d18f78780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:46.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:46.700+0000 7f0d18f78780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:46.714 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:29:46.714 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:29:46.714 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:29:46.715 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:29:46.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:29:46.715 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:29:46.715 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:46.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.716 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:29:46.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.716 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:29:46.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:46.718 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:29:46.718 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:46.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.719 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:29:46.719 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:46.719 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:46.719 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:46.721 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:46.722 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070586.485112_149.1", 2023-10-12T00:29:46.722 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:46.722 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:46.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:46.722 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:29:46.485112Z", 2023-10-12T00:29:46.723 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:46.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:46.724 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:29:46.724 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:46.724 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:46.724 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:46.724 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:46.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:46.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:46.725 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:29:46.725 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:46.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:46.726 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:29:46.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.726 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:29:46.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:29:46.727 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:29:46.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:46.727 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:29:46Z" 2023-10-12T00:29:46.727 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:29:46.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070586.485112_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:29:46.485112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:29:46Z" } 2023-10-12T00:29:46.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070586.485112_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:29:46.485112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:29:46.729 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070586.514221_154.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:46.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070586.485112_149.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:46.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070586.514221_154.1 target=1_1697070586.485112_149.1 2023-10-12T00:29:51.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:51.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:29:51.733 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:29:51.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:51.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:51.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:51.822+0000 7ff6d4cbd780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:51.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:51.823+0000 7ff6d4cbd780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:29:51.836 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:29:51.836 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:29:51.836 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:29:51.836 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:29:51.837 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:29:51.837 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:29:51.837 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:51.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.838 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:29:51.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:51.838 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:29:51.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:51.838 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:51.839 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:29:51.839 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:51.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:29:51.839 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:51.840 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:29:51.840 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:51.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:51.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:51.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:29:51.841 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:51.841 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:51.841 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:29:51.842 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:51.842 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:51.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:51.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:29:51.843 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:51.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:51.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:51.844 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:29:51.844 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:51.844 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:51.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070586.514221_154.1", 2023-10-12T00:29:51.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:51.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:51.845 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:51.845 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:29:46.514221Z", 2023-10-12T00:29:51.845 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:51.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:51.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:29:51.846 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:29:51.846 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:29:51.847 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:29:51.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:29:51.847 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:29:51.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:29:51.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:29:51.848 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:29:51.848 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:29:51.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:51.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:29:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:29:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:29:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:29:51.849 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:29:51.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:29:51.850 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:29:51Z" 2023-10-12T00:29:51.850 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:29:51.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070586.514221_154.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:29:46.514221Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:29:51Z" } 2023-10-12T00:29:51.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070586.514221_154.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:29:46.514221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:29:51.852 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070586.514221_154.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:51.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070586.514221_154.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:29:51.853 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:29:51.854 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:29:51.854 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:29:51.855 DEBUG:boto:Method: HEAD 2023-10-12T00:29:51.855 DEBUG:boto:Path: /wafmoa-17/ 2023-10-12T00:29:51.855 DEBUG:boto:Data: 2023-10-12T00:29:51.856 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:51.857 DEBUG:boto:Port: 8000 2023-10-12T00:29:51.857 DEBUG:boto:Params: {} 2023-10-12T00:29:51.857 DEBUG:boto:Token: None 2023-10-12T00:29:51.858 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-17/ 2023-10-12T00:29:51.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rlrdlMOC57sjtFF4/IM4gENGT1o= 2023-10-12T00:29:51.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rlrdlMOC57sjtFF4/IM4gENGT1o=', 'Content-Length': '0'} 2023-10-12T00:29:51.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036375a48ff342f02-0065273dff-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.862 DEBUG:boto:path=/wafmoa-17/obj 2023-10-12T00:29:51.862 DEBUG:boto:auth_path=/wafmoa-17/obj 2023-10-12T00:29:51.863 DEBUG:boto:Method: PUT 2023-10-12T00:29:51.863 DEBUG:boto:Path: /wafmoa-17/obj 2023-10-12T00:29:51.864 DEBUG:boto:Data: 2023-10-12T00:29:51.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:29:51.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:51.865 DEBUG:boto:Port: 8000 2023-10-12T00:29:51.865 DEBUG:boto:Params: {} 2023-10-12T00:29:51.866 DEBUG:boto:Token: None 2023-10-12T00:29:51.867 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-17/obj 2023-10-12T00:29:51.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/MLUtm9nUw0oqCahs83hbkcgvtk= 2023-10-12T00:29:51.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/MLUtm9nUw0oqCahs83hbkcgvtk='} 2023-10-12T00:29:51.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074a8db46a10635c8-0065273dff-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.874 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:29:51.874 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:29:51.875 DEBUG:boto:Method: HEAD 2023-10-12T00:29:51.875 DEBUG:boto:Path: /wafmoa-17/ 2023-10-12T00:29:51.876 DEBUG:boto:Data: 2023-10-12T00:29:51.876 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:51.877 DEBUG:boto:Port: 8000 2023-10-12T00:29:51.877 DEBUG:boto:Params: {} 2023-10-12T00:29:51.878 DEBUG:boto:Token: None 2023-10-12T00:29:51.878 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-17/ 2023-10-12T00:29:51.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rlrdlMOC57sjtFF4/IM4gENGT1o= 2023-10-12T00:29:51.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rlrdlMOC57sjtFF4/IM4gENGT1o=', 'Content-Length': '0'} 2023-10-12T00:29:51.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071b0a231c0103550-0065273dff-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.882 DEBUG:boto:path=/wafmoa-17/obj 2023-10-12T00:29:51.883 DEBUG:boto:auth_path=/wafmoa-17/obj 2023-10-12T00:29:51.883 DEBUG:boto:Method: DELETE 2023-10-12T00:29:51.884 DEBUG:boto:Path: /wafmoa-17/obj 2023-10-12T00:29:51.884 DEBUG:boto:Data: 2023-10-12T00:29:51.885 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:29:51.885 DEBUG:boto:Port: 8000 2023-10-12T00:29:51.886 DEBUG:boto:Params: {} 2023-10-12T00:29:51.886 DEBUG:boto:Token: None 2023-10-12T00:29:51.887 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-17/obj 2023-10-12T00:29:51.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aDNu2R/QUryGTsJfU43XXV0wak4= 2023-10-12T00:29:51.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aDNu2R/QUryGTsJfU43XXV0wak4=', 'Content-Length': '0'} 2023-10-12T00:29:51.897 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000069269adfa66ad734-0065273dff-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.898 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:29:51.898 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:29:51.899 DEBUG:boto:Method: HEAD 2023-10-12T00:29:51.899 DEBUG:boto:Path: /wafmoa-18/ 2023-10-12T00:29:51.900 DEBUG:boto:Data: 2023-10-12T00:29:51.900 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:51.901 DEBUG:boto:Port: 8001 2023-10-12T00:29:51.901 DEBUG:boto:Params: {} 2023-10-12T00:29:51.902 DEBUG:boto:Token: None 2023-10-12T00:29:51.902 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-18/ 2023-10-12T00:29:51.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hEfef6W7Vc2hUct37gAD3oiF6zo= 2023-10-12T00:29:51.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hEfef6W7Vc2hUct37gAD3oiF6zo=', 'Content-Length': '0'} 2023-10-12T00:29:51.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d610b1626ba17ca-0065273dff-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.907 DEBUG:boto:path=/wafmoa-18/obj 2023-10-12T00:29:51.907 DEBUG:boto:auth_path=/wafmoa-18/obj 2023-10-12T00:29:51.907 DEBUG:boto:Method: PUT 2023-10-12T00:29:51.908 DEBUG:boto:Path: /wafmoa-18/obj 2023-10-12T00:29:51.908 DEBUG:boto:Data: 2023-10-12T00:29:51.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-12T00:29:51.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:51.909 DEBUG:boto:Port: 8001 2023-10-12T00:29:51.910 DEBUG:boto:Params: {} 2023-10-12T00:29:51.910 DEBUG:boto:Token: None 2023-10-12T00:29:51.911 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-18/obj 2023-10-12T00:29:51.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eIkXqzyut63eqjnfQdaai/4Zn/Q= 2023-10-12T00:29:51.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eIkXqzyut63eqjnfQdaai/4Zn/Q='} 2023-10-12T00:29:51.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ee14c21d71721ce-0065273dff-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.918 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:29:51.918 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:29:51.919 DEBUG:boto:Method: HEAD 2023-10-12T00:29:51.919 DEBUG:boto:Path: /wafmoa-18/ 2023-10-12T00:29:51.920 DEBUG:boto:Data: 2023-10-12T00:29:51.920 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:51.921 DEBUG:boto:Port: 8001 2023-10-12T00:29:51.922 DEBUG:boto:Params: {} 2023-10-12T00:29:51.922 DEBUG:boto:Token: None 2023-10-12T00:29:51.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-18/ 2023-10-12T00:29:51.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hEfef6W7Vc2hUct37gAD3oiF6zo= 2023-10-12T00:29:51.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hEfef6W7Vc2hUct37gAD3oiF6zo=', 'Content-Length': '0'} 2023-10-12T00:29:51.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cdf8fd8df3c510d-0065273dff-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.926 DEBUG:boto:path=/wafmoa-18/obj 2023-10-12T00:29:51.927 DEBUG:boto:auth_path=/wafmoa-18/obj 2023-10-12T00:29:51.927 DEBUG:boto:Method: DELETE 2023-10-12T00:29:51.928 DEBUG:boto:Path: /wafmoa-18/obj 2023-10-12T00:29:51.928 DEBUG:boto:Data: 2023-10-12T00:29:51.928 DEBUG:boto:Headers: {} 2023-10-12T00:29:51.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:29:51.929 DEBUG:boto:Port: 8001 2023-10-12T00:29:51.930 DEBUG:boto:Params: {} 2023-10-12T00:29:51.930 DEBUG:boto:Token: None 2023-10-12T00:29:51.931 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:29:51 GMT /wafmoa-18/obj 2023-10-12T00:29:51.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xDobzndD4udRm1LnoFiVMsjjneo= 2023-10-12T00:29:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:29:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xDobzndD4udRm1LnoFiVMsjjneo=', 'Content-Length': '0'} 2023-10-12T00:29:51.940 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d58d6b9a98f495b2-0065273dff-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:29:51.940 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-17 2023-10-12T00:29:51.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:51.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:29:51.942 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:29:51.998 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:29:51.998 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:29:52.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:52.031+0000 7f40e280b780 1 waiting to reach incremental sync.. 2023-10-12T00:29:57.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:29:57.032+0000 7f40e280b780 1 waiting to reach incremental sync.. 2023-10-12T00:30:02.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:02.032+0000 7f40e280b780 1 waiting to reach incremental sync.. 2023-10-12T00:30:07.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:07.033+0000 7f40e280b780 1 bucket sync caught up with source: 2023-10-12T00:30:07.034 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.15.7, , ] 2023-10-12T00:30:07.034 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.15.7, , ] 2023-10-12T00:30:07.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:07.033+0000 7f40e280b780 0 bucket checkpoint complete 2023-10-12T00:30:07.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-17 2023-10-12T00:30:07.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:07.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:07.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:07.153+0000 7f571a121780 1 bucket sync caught up with source: 2023-10-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.33.14, , ] 2023-10-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.33.14, , ] 2023-10-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:07.153+0000 7f571a121780 0 bucket checkpoint complete 2023-10-12T00:30:07.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-17 zones={a2, a1} 2023-10-12T00:30:07.167 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.168 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.168 DEBUG:boto:Method: HEAD 2023-10-12T00:30:07.169 DEBUG:boto:Path: /wafmoa-17/ 2023-10-12T00:30:07.169 DEBUG:boto:Data: 2023-10-12T00:30:07.170 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:07.171 DEBUG:boto:Port: 8001 2023-10-12T00:30:07.171 DEBUG:boto:Params: {} 2023-10-12T00:30:07.172 DEBUG:boto:Token: None 2023-10-12T00:30:07.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/ 2023-10-12T00:30:07.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S3FfWRcbmphseoKv5hGTPuQ8Img= 2023-10-12T00:30:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S3FfWRcbmphseoKv5hGTPuQ8Img=', 'Content-Length': '0'} 2023-10-12T00:30:07.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cb73bc0308b36a9-0065273e0f-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.177 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.177 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.178 DEBUG:boto:Method: HEAD 2023-10-12T00:30:07.178 DEBUG:boto:Path: /wafmoa-17/ 2023-10-12T00:30:07.179 DEBUG:boto:Data: 2023-10-12T00:30:07.179 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:07.180 DEBUG:boto:Port: 8000 2023-10-12T00:30:07.180 DEBUG:boto:Params: {} 2023-10-12T00:30:07.181 DEBUG:boto:Token: None 2023-10-12T00:30:07.181 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/ 2023-10-12T00:30:07.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S3FfWRcbmphseoKv5hGTPuQ8Img= 2023-10-12T00:30:07.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S3FfWRcbmphseoKv5hGTPuQ8Img=', 'Content-Length': '0'} 2023-10-12T00:30:07.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5a58f1fba1b86c3-0065273e0f-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:30:07.187 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.187 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.188 DEBUG:boto:path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.188 DEBUG:boto:auth_path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.189 DEBUG:boto:Method: GET 2023-10-12T00:30:07.189 DEBUG:boto:Path: /wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.190 DEBUG:boto:Data: 2023-10-12T00:30:07.190 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:07.191 DEBUG:boto:Port: 8001 2023-10-12T00:30:07.191 DEBUG:boto:Params: {} 2023-10-12T00:30:07.192 DEBUG:boto:Token: None 2023-10-12T00:30:07.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/?versions 2023-10-12T00:30:07.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs= 2023-10-12T00:30:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs=', 'Content-Length': '0'} 2023-10-12T00:30:07.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c605d3c03ba0dd-0065273e0f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.197 DEBUG:boto:b'wafmoa-17999falseobjnulltrue2023-10-12T00:29:51.892Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:07.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:30:07.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:30:07.199 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.199 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.200 DEBUG:boto:path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.200 DEBUG:boto:auth_path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.200 DEBUG:boto:Method: GET 2023-10-12T00:30:07.201 DEBUG:boto:Path: /wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.201 DEBUG:boto:Data: 2023-10-12T00:30:07.202 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:07.203 DEBUG:boto:Port: 8000 2023-10-12T00:30:07.203 DEBUG:boto:Params: {} 2023-10-12T00:30:07.203 DEBUG:boto:Token: None 2023-10-12T00:30:07.204 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/?versions 2023-10-12T00:30:07.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs= 2023-10-12T00:30:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs=', 'Content-Length': '0'} 2023-10-12T00:30:07.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afc2d84a030324dd-0065273e0f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.208 DEBUG:boto:b'wafmoa-17999falseobjnulltrue2023-10-12T00:29:51.892Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:07.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:30:07.209 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.209 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.210 DEBUG:boto:path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.210 DEBUG:boto:auth_path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.211 DEBUG:boto:Method: GET 2023-10-12T00:30:07.211 DEBUG:boto:Path: /wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.211 DEBUG:boto:Data: 2023-10-12T00:30:07.212 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:07.213 DEBUG:boto:Port: 8001 2023-10-12T00:30:07.213 DEBUG:boto:Params: {} 2023-10-12T00:30:07.213 DEBUG:boto:Token: None 2023-10-12T00:30:07.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/?versions 2023-10-12T00:30:07.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs= 2023-10-12T00:30:07.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs=', 'Content-Length': '0'} 2023-10-12T00:30:07.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a6881e51cb066c4-0065273e0f-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.218 DEBUG:boto:b'wafmoa-17999falseobjnulltrue2023-10-12T00:29:51.892Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:07.219 DEBUG:boto:path=/wafmoa-17/ 2023-10-12T00:30:07.219 DEBUG:boto:auth_path=/wafmoa-17/ 2023-10-12T00:30:07.219 DEBUG:boto:path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.220 DEBUG:boto:auth_path=/wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.220 DEBUG:boto:Method: GET 2023-10-12T00:30:07.221 DEBUG:boto:Path: /wafmoa-17/?versions&max-keys=999 2023-10-12T00:30:07.221 DEBUG:boto:Data: 2023-10-12T00:30:07.222 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:07.222 DEBUG:boto:Port: 8000 2023-10-12T00:30:07.223 DEBUG:boto:Params: {} 2023-10-12T00:30:07.223 DEBUG:boto:Token: None 2023-10-12T00:30:07.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/?versions 2023-10-12T00:30:07.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs= 2023-10-12T00:30:07.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yfUxFp7SevQDG9lO8uRFhJOIrs=', 'Content-Length': '0'} 2023-10-12T00:30:07.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031fce61c5cdbd04f-0065273e0f-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.227 DEBUG:boto:b'wafmoa-17999falseobjnulltrue2023-10-12T00:29:51.892Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:07.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:30:07.229 DEBUG:boto:path=/wafmoa-17/obj 2023-10-12T00:30:07.229 DEBUG:boto:auth_path=/wafmoa-17/obj 2023-10-12T00:30:07.229 DEBUG:boto:Method: HEAD 2023-10-12T00:30:07.230 DEBUG:boto:Path: /wafmoa-17/obj 2023-10-12T00:30:07.230 DEBUG:boto:Data: 2023-10-12T00:30:07.231 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:07.232 DEBUG:boto:Port: 8001 2023-10-12T00:30:07.232 DEBUG:boto:Params: {} 2023-10-12T00:30:07.232 DEBUG:boto:Token: None 2023-10-12T00:30:07.233 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/obj 2023-10-12T00:30:07.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ostEHPRGN4b6S2vYy9vFXUmT9ak= 2023-10-12T00:30:07.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ostEHPRGN4b6S2vYy9vFXUmT9ak=', 'Content-Length': '0'} 2023-10-12T00:30:07.235 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000096a7e37375ee3ff6-0065273e0f-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.236 DEBUG:boto:path=/wafmoa-17/obj 2023-10-12T00:30:07.236 DEBUG:boto:auth_path=/wafmoa-17/obj 2023-10-12T00:30:07.237 DEBUG:boto:Method: HEAD 2023-10-12T00:30:07.237 DEBUG:boto:Path: /wafmoa-17/obj 2023-10-12T00:30:07.238 DEBUG:boto:Data: 2023-10-12T00:30:07.238 DEBUG:boto:Headers: {} 2023-10-12T00:30:07.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:07.239 DEBUG:boto:Port: 8000 2023-10-12T00:30:07.239 DEBUG:boto:Params: {} 2023-10-12T00:30:07.240 DEBUG:boto:Token: None 2023-10-12T00:30:07.240 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:07 GMT /wafmoa-17/obj 2023-10-12T00:30:07.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ostEHPRGN4b6S2vYy9vFXUmT9ak= 2023-10-12T00:30:07.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ostEHPRGN4b6S2vYy9vFXUmT9ak=', 'Content-Length': '0'} 2023-10-12T00:30:07.243 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000032ee1c63115b9527-0065273e0f-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:07.243 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-17 zones={a2, a1} 2023-10-12T00:30:07.244 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-18 2023-10-12T00:30:07.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:07.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:07.245 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:30:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:07.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:07.338+0000 7f59c47a5780 1 waiting to reach incremental sync.. 2023-10-12T00:30:12.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:12.338+0000 7f59c47a5780 1 waiting to reach incremental sync.. 2023-10-12T00:30:17.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:17.339+0000 7f59c47a5780 1 waiting to reach incremental sync.. 2023-10-12T00:30:22.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:22.340+0000 7f59c47a5780 1 waiting to reach incremental sync.. 2023-10-12T00:30:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:27.341+0000 7f59c47a5780 1 bucket sync caught up with source: 2023-10-12T00:30:27.343 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.19.14, , ] 2023-10-12T00:30:27.343 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.19.14, , ] 2023-10-12T00:30:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:27.341+0000 7f59c47a5780 0 bucket checkpoint complete 2023-10-12T00:30:27.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-18 2023-10-12T00:30:27.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:27.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:30:27.362 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:30:27.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:27.425 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:27.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:27.464+0000 7f6400a4d780 1 bucket sync caught up with source: 2023-10-12T00:30:27.465 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.46.7, , ] 2023-10-12T00:30:27.465 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.46.7, , ] 2023-10-12T00:30:27.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:27.464+0000 7f6400a4d780 0 bucket checkpoint complete 2023-10-12T00:30:27.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-18 zones={a2, a1} 2023-10-12T00:30:27.478 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.479 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.479 DEBUG:boto:Method: HEAD 2023-10-12T00:30:27.480 DEBUG:boto:Path: /wafmoa-18/ 2023-10-12T00:30:27.480 DEBUG:boto:Data: 2023-10-12T00:30:27.481 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.482 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.482 DEBUG:boto:Params: {} 2023-10-12T00:30:27.483 DEBUG:boto:Token: None 2023-10-12T00:30:27.483 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/ 2023-10-12T00:30:27.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hi2enoO57HM0lbqq/eXxgbjJiAM= 2023-10-12T00:30:27.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hi2enoO57HM0lbqq/eXxgbjJiAM=', 'Content-Length': '0'} 2023-10-12T00:30:27.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d7130394e67b9bf-0065273e23-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.487 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.488 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.488 DEBUG:boto:Method: HEAD 2023-10-12T00:30:27.489 DEBUG:boto:Path: /wafmoa-18/ 2023-10-12T00:30:27.489 DEBUG:boto:Data: 2023-10-12T00:30:27.489 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.490 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.491 DEBUG:boto:Params: {} 2023-10-12T00:30:27.491 DEBUG:boto:Token: None 2023-10-12T00:30:27.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/ 2023-10-12T00:30:27.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hi2enoO57HM0lbqq/eXxgbjJiAM= 2023-10-12T00:30:27.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hi2enoO57HM0lbqq/eXxgbjJiAM=', 'Content-Length': '0'} 2023-10-12T00:30:27.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7a4040ec3804704-0065273e23-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.495 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:30:27.496 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.496 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.497 DEBUG:boto:path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.497 DEBUG:boto:auth_path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.497 DEBUG:boto:Method: GET 2023-10-12T00:30:27.498 DEBUG:boto:Path: /wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.498 DEBUG:boto:Data: 2023-10-12T00:30:27.499 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.500 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.500 DEBUG:boto:Params: {} 2023-10-12T00:30:27.500 DEBUG:boto:Token: None 2023-10-12T00:30:27.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/?versions 2023-10-12T00:30:27.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE= 2023-10-12T00:30:27.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE=', 'Content-Length': '0'} 2023-10-12T00:30:27.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005479c5713b3d848e-0065273e23-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.505 DEBUG:boto:b'wafmoa-18999falseobjnulltrue2023-10-12T00:29:51.934Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:27.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:30:27.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:30:27.506 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.507 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.507 DEBUG:boto:path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.508 DEBUG:boto:auth_path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.508 DEBUG:boto:Method: GET 2023-10-12T00:30:27.509 DEBUG:boto:Path: /wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.509 DEBUG:boto:Data: 2023-10-12T00:30:27.510 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.510 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.511 DEBUG:boto:Params: {} 2023-10-12T00:30:27.511 DEBUG:boto:Token: None 2023-10-12T00:30:27.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/?versions 2023-10-12T00:30:27.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE= 2023-10-12T00:30:27.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE=', 'Content-Length': '0'} 2023-10-12T00:30:27.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f176926f794c268-0065273e23-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.558 DEBUG:boto:b'wafmoa-18999falseobjnulltrue2023-10-12T00:29:51.934Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:27.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-12T00:30:27.560 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.560 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.561 DEBUG:boto:path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.561 DEBUG:boto:auth_path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.562 DEBUG:boto:Method: GET 2023-10-12T00:30:27.562 DEBUG:boto:Path: /wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.562 DEBUG:boto:Data: 2023-10-12T00:30:27.563 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.564 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.564 DEBUG:boto:Params: {} 2023-10-12T00:30:27.565 DEBUG:boto:Token: None 2023-10-12T00:30:27.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/?versions 2023-10-12T00:30:27.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE= 2023-10-12T00:30:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE=', 'Content-Length': '0'} 2023-10-12T00:30:27.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ab308adb55a1bb-0065273e23-1111-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.569 DEBUG:boto:b'wafmoa-18999falseobjnulltrue2023-10-12T00:29:51.934Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:27.570 DEBUG:boto:path=/wafmoa-18/ 2023-10-12T00:30:27.570 DEBUG:boto:auth_path=/wafmoa-18/ 2023-10-12T00:30:27.571 DEBUG:boto:path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.571 DEBUG:boto:auth_path=/wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.572 DEBUG:boto:Method: GET 2023-10-12T00:30:27.572 DEBUG:boto:Path: /wafmoa-18/?versions&max-keys=999 2023-10-12T00:30:27.572 DEBUG:boto:Data: 2023-10-12T00:30:27.573 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.574 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.574 DEBUG:boto:Params: {} 2023-10-12T00:30:27.574 DEBUG:boto:Token: None 2023-10-12T00:30:27.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/?versions 2023-10-12T00:30:27.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE= 2023-10-12T00:30:27.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G71rxkzmf1KjzN2MQdhv1wXQ3zE=', 'Content-Length': '0'} 2023-10-12T00:30:27.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f7828ce95aa283-0065273e23-1115-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.579 DEBUG:boto:b'wafmoa-18999falseobjnulltrue2023-10-12T00:29:51.934Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:30:27.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-12T00:30:27.580 DEBUG:boto:path=/wafmoa-18/obj 2023-10-12T00:30:27.581 DEBUG:boto:auth_path=/wafmoa-18/obj 2023-10-12T00:30:27.581 DEBUG:boto:Method: HEAD 2023-10-12T00:30:27.581 DEBUG:boto:Path: /wafmoa-18/obj 2023-10-12T00:30:27.582 DEBUG:boto:Data: 2023-10-12T00:30:27.582 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.583 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.584 DEBUG:boto:Params: {} 2023-10-12T00:30:27.584 DEBUG:boto:Token: None 2023-10-12T00:30:27.585 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/obj 2023-10-12T00:30:27.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2r0MNTAd8jtVf+PXVuOTkkTIx7Q= 2023-10-12T00:30:27.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2r0MNTAd8jtVf+PXVuOTkkTIx7Q=', 'Content-Length': '0'} 2023-10-12T00:30:27.589 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c891ec1aa6032d61-0065273e23-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.589 DEBUG:boto:path=/wafmoa-18/obj 2023-10-12T00:30:27.590 DEBUG:boto:auth_path=/wafmoa-18/obj 2023-10-12T00:30:27.590 DEBUG:boto:Method: HEAD 2023-10-12T00:30:27.591 DEBUG:boto:Path: /wafmoa-18/obj 2023-10-12T00:30:27.591 DEBUG:boto:Data: 2023-10-12T00:30:27.592 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.592 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.593 DEBUG:boto:Params: {} 2023-10-12T00:30:27.593 DEBUG:boto:Token: None 2023-10-12T00:30:27.594 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-18/obj 2023-10-12T00:30:27.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2r0MNTAd8jtVf+PXVuOTkkTIx7Q= 2023-10-12T00:30:27.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2r0MNTAd8jtVf+PXVuOTkkTIx7Q=', 'Content-Length': '0'} 2023-10-12T00:30:27.596 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c1f6bb9f5357f067-0065273e23-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-18 zones={a2, a1} 2023-10-12T00:30:27.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-10-12T00:30:27.599 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-19 2023-10-12T00:30:27.599 DEBUG:boto:path=/wafmoa-19/ 2023-10-12T00:30:27.599 DEBUG:boto:auth_path=/wafmoa-19/ 2023-10-12T00:30:27.600 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.600 DEBUG:boto:Path: /wafmoa-19/ 2023-10-12T00:30:27.601 DEBUG:boto:Data: 2023-10-12T00:30:27.601 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.602 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.602 DEBUG:boto:Params: {} 2023-10-12T00:30:27.603 DEBUG:boto:Token: None 2023-10-12T00:30:27.603 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-19/ 2023-10-12T00:30:27.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KzYp91cRZj517Mp1RlrP5u1jxWw= 2023-10-12T00:30:27.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KzYp91cRZj517Mp1RlrP5u1jxWw=', 'Content-Length': '0'} 2023-10-12T00:30:27.621 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f37e3bc7fb186bb5-0065273e23-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.621 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-20 2023-10-12T00:30:27.622 DEBUG:boto:path=/wafmoa-20/ 2023-10-12T00:30:27.622 DEBUG:boto:auth_path=/wafmoa-20/ 2023-10-12T00:30:27.623 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.623 DEBUG:boto:Path: /wafmoa-20/ 2023-10-12T00:30:27.624 DEBUG:boto:Data: 2023-10-12T00:30:27.624 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.625 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.625 DEBUG:boto:Params: {} 2023-10-12T00:30:27.626 DEBUG:boto:Token: None 2023-10-12T00:30:27.626 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-20/ 2023-10-12T00:30:27.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DGQ4xp6ni9I1c/V7B4mapvN/EL4= 2023-10-12T00:30:27.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DGQ4xp6ni9I1c/V7B4mapvN/EL4=', 'Content-Length': '0'} 2023-10-12T00:30:27.660 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a986cb64fb24430-0065273e23-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.661 DEBUG:boto:Using access key provided by client. 2023-10-12T00:30:27.662 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:30:27.663 DEBUG:boto:Using access key provided by client. 2023-10-12T00:30:27.663 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:30:27.664 DEBUG:boto:Using access key provided by client. 2023-10-12T00:30:27.664 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:30:27.665 DEBUG:boto:Using access key provided by client. 2023-10-12T00:30:27.666 DEBUG:boto:Using secret key provided by client. 2023-10-12T00:30:27.666 INFO:rgw_multi.tests:create bucket zone=b1 name=wafmoa-21 2023-10-12T00:30:27.667 DEBUG:boto:path=/wafmoa-21/ 2023-10-12T00:30:27.667 DEBUG:boto:auth_path=/wafmoa-21/ 2023-10-12T00:30:27.667 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.668 DEBUG:boto:Path: /wafmoa-21/ 2023-10-12T00:30:27.668 DEBUG:boto:Data: 2023-10-12T00:30:27.669 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.669 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:27.669 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.670 DEBUG:boto:Params: {} 2023-10-12T00:30:27.670 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:30:27.671 DEBUG:boto:Token: None 2023-10-12T00:30:27.671 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-21/ 2023-10-12T00:30:27.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UEIHfeIZU8ikSJTD+N+KMEIuXdw= 2023-10-12T00:30:27.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UEIHfeIZU8ikSJTD+N+KMEIuXdw=', 'Content-Length': '0'} 2023-10-12T00:30:27.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000565b2deb9edc27bf-0065273e23-10fd-b1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.703 INFO:rgw_multi.tests:create bucket zone=b2 name=wafmoa-22 2023-10-12T00:30:27.703 DEBUG:boto:path=/wafmoa-22/ 2023-10-12T00:30:27.704 DEBUG:boto:auth_path=/wafmoa-22/ 2023-10-12T00:30:27.704 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.705 DEBUG:boto:Path: /wafmoa-22/ 2023-10-12T00:30:27.705 DEBUG:boto:Data: 2023-10-12T00:30:27.705 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.706 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:27.706 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.707 DEBUG:boto:Params: {} 2023-10-12T00:30:27.707 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:30:27.707 DEBUG:boto:Token: None 2023-10-12T00:30:27.708 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-22/ 2023-10-12T00:30:27.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oa1YSvLqPutX1ETAKp7Rugt7tmU= 2023-10-12T00:30:27.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oa1YSvLqPutX1ETAKp7Rugt7tmU=', 'Content-Length': '0'} 2023-10-12T00:30:27.739 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ecdd0a46142a8f54-0065273e23-1103-b2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.740 DEBUG:boto:path=/wafmoa-19/ 2023-10-12T00:30:27.741 DEBUG:boto:auth_path=/wafmoa-19/ 2023-10-12T00:30:27.741 DEBUG:boto:path=/wafmoa-19/?versioning 2023-10-12T00:30:27.741 DEBUG:boto:auth_path=/wafmoa-19/?versioning 2023-10-12T00:30:27.742 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.742 DEBUG:boto:Path: /wafmoa-19/?versioning 2023-10-12T00:30:27.743 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:30:27.743 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.744 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.744 DEBUG:boto:Params: {} 2023-10-12T00:30:27.745 DEBUG:boto:Token: None 2023-10-12T00:30:27.745 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-19/?versioning 2023-10-12T00:30:27.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GKB8ocBUpf47mBFeguynWA8fVmI= 2023-10-12T00:30:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GKB8ocBUpf47mBFeguynWA8fVmI=', 'Content-Length': '228'} 2023-10-12T00:30:27.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013345de1592ddd08-0065273e23-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.751 DEBUG:boto:path=/wafmoa-19/ 2023-10-12T00:30:27.752 DEBUG:boto:auth_path=/wafmoa-19/ 2023-10-12T00:30:27.752 DEBUG:boto:path=/wafmoa-19/?versioning 2023-10-12T00:30:27.752 DEBUG:boto:auth_path=/wafmoa-19/?versioning 2023-10-12T00:30:27.753 DEBUG:boto:Method: GET 2023-10-12T00:30:27.753 DEBUG:boto:Path: /wafmoa-19/?versioning 2023-10-12T00:30:27.754 DEBUG:boto:Data: 2023-10-12T00:30:27.754 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.755 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.755 DEBUG:boto:Params: {} 2023-10-12T00:30:27.756 DEBUG:boto:Token: None 2023-10-12T00:30:27.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-19/?versioning 2023-10-12T00:30:27.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Cr1KpojN1usDSV0c9gn8uf0xvA= 2023-10-12T00:30:27.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Cr1KpojN1usDSV0c9gn8uf0xvA=', 'Content-Length': '0'} 2023-10-12T00:30:27.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0d4b64cb6c93e4e-0065273e23-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.806 DEBUG:boto:EnabledDisabled 2023-10-12T00:30:27.807 DEBUG:boto:path=/wafmoa-20/ 2023-10-12T00:30:27.808 DEBUG:boto:auth_path=/wafmoa-20/ 2023-10-12T00:30:27.808 DEBUG:boto:path=/wafmoa-20/?versioning 2023-10-12T00:30:27.809 DEBUG:boto:auth_path=/wafmoa-20/?versioning 2023-10-12T00:30:27.809 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.809 DEBUG:boto:Path: /wafmoa-20/?versioning 2023-10-12T00:30:27.810 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:30:27.810 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.811 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.811 DEBUG:boto:Params: {} 2023-10-12T00:30:27.812 DEBUG:boto:Token: None 2023-10-12T00:30:27.812 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-20/?versioning 2023-10-12T00:30:27.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xm7JDwjjtK3SXUAmEMFRNM95JPc= 2023-10-12T00:30:27.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xm7JDwjjtK3SXUAmEMFRNM95JPc=', 'Content-Length': '228'} 2023-10-12T00:30:27.821 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7e4c48558b43af8-0065273e23-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.821 DEBUG:boto:path=/wafmoa-20/ 2023-10-12T00:30:27.822 DEBUG:boto:auth_path=/wafmoa-20/ 2023-10-12T00:30:27.822 DEBUG:boto:path=/wafmoa-20/?versioning 2023-10-12T00:30:27.822 DEBUG:boto:auth_path=/wafmoa-20/?versioning 2023-10-12T00:30:27.823 DEBUG:boto:Method: GET 2023-10-12T00:30:27.823 DEBUG:boto:Path: /wafmoa-20/?versioning 2023-10-12T00:30:27.824 DEBUG:boto:Data: 2023-10-12T00:30:27.824 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.825 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.825 DEBUG:boto:Params: {} 2023-10-12T00:30:27.826 DEBUG:boto:Token: None 2023-10-12T00:30:27.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-20/?versioning 2023-10-12T00:30:27.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WgvAJ7mqAPyi3sx1mNsLvdG+q7w= 2023-10-12T00:30:27.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WgvAJ7mqAPyi3sx1mNsLvdG+q7w=', 'Content-Length': '0'} 2023-10-12T00:30:27.828 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000701ff4f0c3982ac-0065273e23-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.874 DEBUG:boto:EnabledDisabled 2023-10-12T00:30:27.875 DEBUG:boto:path=/wafmoa-21/ 2023-10-12T00:30:27.875 DEBUG:boto:auth_path=/wafmoa-21/ 2023-10-12T00:30:27.876 DEBUG:boto:path=/wafmoa-21/?versioning 2023-10-12T00:30:27.876 DEBUG:boto:auth_path=/wafmoa-21/?versioning 2023-10-12T00:30:27.876 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.877 DEBUG:boto:Path: /wafmoa-21/?versioning 2023-10-12T00:30:27.877 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:30:27.878 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.878 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:27.878 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.879 DEBUG:boto:Params: {} 2023-10-12T00:30:27.879 DEBUG:boto:Token: None 2023-10-12T00:30:27.880 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-21/?versioning 2023-10-12T00:30:27.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kth860ATqfePfWtskwy1jOHw1a8= 2023-10-12T00:30:27.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kth860ATqfePfWtskwy1jOHw1a8=', 'Content-Length': '228'} 2023-10-12T00:30:27.889 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a3810e79cdb91085-0065273e23-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.889 DEBUG:boto:path=/wafmoa-21/ 2023-10-12T00:30:27.890 DEBUG:boto:auth_path=/wafmoa-21/ 2023-10-12T00:30:27.890 DEBUG:boto:path=/wafmoa-21/?versioning 2023-10-12T00:30:27.890 DEBUG:boto:auth_path=/wafmoa-21/?versioning 2023-10-12T00:30:27.891 DEBUG:boto:Method: GET 2023-10-12T00:30:27.891 DEBUG:boto:Path: /wafmoa-21/?versioning 2023-10-12T00:30:27.892 DEBUG:boto:Data: 2023-10-12T00:30:27.892 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.892 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:27.893 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.893 DEBUG:boto:Params: {} 2023-10-12T00:30:27.894 DEBUG:boto:Token: None 2023-10-12T00:30:27.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-21/?versioning 2023-10-12T00:30:27.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+AFqtlMpJNkbF4DEHOvpHgJgdHw= 2023-10-12T00:30:27.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+AFqtlMpJNkbF4DEHOvpHgJgdHw=', 'Content-Length': '0'} 2023-10-12T00:30:27.897 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005afd071647c5f3ee-0065273e23-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.897 DEBUG:boto:EnabledDisabled 2023-10-12T00:30:27.898 DEBUG:boto:path=/wafmoa-22/ 2023-10-12T00:30:27.898 DEBUG:boto:auth_path=/wafmoa-22/ 2023-10-12T00:30:27.899 DEBUG:boto:path=/wafmoa-22/?versioning 2023-10-12T00:30:27.899 DEBUG:boto:auth_path=/wafmoa-22/?versioning 2023-10-12T00:30:27.899 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.900 DEBUG:boto:Path: /wafmoa-22/?versioning 2023-10-12T00:30:27.900 DEBUG:boto:Data: Enabled Disabled 2023-10-12T00:30:27.901 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.901 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:27.901 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.902 DEBUG:boto:Params: {} 2023-10-12T00:30:27.902 DEBUG:boto:Token: None 2023-10-12T00:30:27.903 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-22/?versioning 2023-10-12T00:30:27.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MKrDeRZgtV+htHqEp+DTutPo5A8= 2023-10-12T00:30:27.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MKrDeRZgtV+htHqEp+DTutPo5A8=', 'Content-Length': '228'} 2023-10-12T00:30:27.912 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b26db129b58251f-0065273e23-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.913 DEBUG:boto:path=/wafmoa-22/ 2023-10-12T00:30:27.914 DEBUG:boto:auth_path=/wafmoa-22/ 2023-10-12T00:30:27.915 DEBUG:boto:path=/wafmoa-22/?versioning 2023-10-12T00:30:27.915 DEBUG:boto:auth_path=/wafmoa-22/?versioning 2023-10-12T00:30:27.916 DEBUG:boto:Method: GET 2023-10-12T00:30:27.917 DEBUG:boto:Path: /wafmoa-22/?versioning 2023-10-12T00:30:27.918 DEBUG:boto:Data: 2023-10-12T00:30:27.918 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.919 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:27.920 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.920 DEBUG:boto:Params: {} 2023-10-12T00:30:27.921 DEBUG:boto:Token: None 2023-10-12T00:30:27.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-22/?versioning 2023-10-12T00:30:27.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4DmhL/MIknZjGOtU2auHbtwgrzw= 2023-10-12T00:30:27.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4DmhL/MIknZjGOtU2auHbtwgrzw=', 'Content-Length': '0'} 2023-10-12T00:30:27.926 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b5182c3db7ad8cc-0065273e23-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.926 DEBUG:boto:EnabledDisabled 2023-10-12T00:30:27.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-10-12T00:30:27.929 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-23 2023-10-12T00:30:27.929 DEBUG:boto:path=/wafmoa-23/ 2023-10-12T00:30:27.930 DEBUG:boto:auth_path=/wafmoa-23/ 2023-10-12T00:30:27.931 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.931 DEBUG:boto:Path: /wafmoa-23/ 2023-10-12T00:30:27.932 DEBUG:boto:Data: 2023-10-12T00:30:27.933 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:27.934 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.934 DEBUG:boto:Params: {} 2023-10-12T00:30:27.935 DEBUG:boto:Token: None 2023-10-12T00:30:27.936 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-23/ 2023-10-12T00:30:27.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFYhPULN1DoP5apu/iTCNPoMnF4= 2023-10-12T00:30:27.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFYhPULN1DoP5apu/iTCNPoMnF4=', 'Content-Length': '0'} 2023-10-12T00:30:27.952 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a09e3fcb686cae7-0065273e23-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.953 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-24 2023-10-12T00:30:27.953 DEBUG:boto:path=/wafmoa-24/ 2023-10-12T00:30:27.954 DEBUG:boto:auth_path=/wafmoa-24/ 2023-10-12T00:30:27.955 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.955 DEBUG:boto:Path: /wafmoa-24/ 2023-10-12T00:30:27.956 DEBUG:boto:Data: 2023-10-12T00:30:27.956 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:27.958 DEBUG:boto:Port: 8001 2023-10-12T00:30:27.958 DEBUG:boto:Params: {} 2023-10-12T00:30:27.959 DEBUG:boto:Token: None 2023-10-12T00:30:27.959 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-24/ 2023-10-12T00:30:27.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2QeK5Ndk47u+zCyWsTjMiPcQ7ec= 2023-10-12T00:30:27.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2QeK5Ndk47u+zCyWsTjMiPcQ7ec=', 'Content-Length': '0'} 2023-10-12T00:30:27.989 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ea54efac33360e37-0065273e23-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:27.990 INFO:rgw_multi.tests:create bucket zone=b1 name=wafmoa-25 2023-10-12T00:30:27.991 DEBUG:boto:path=/wafmoa-25/ 2023-10-12T00:30:27.991 DEBUG:boto:auth_path=/wafmoa-25/ 2023-10-12T00:30:27.991 DEBUG:boto:Method: PUT 2023-10-12T00:30:27.992 DEBUG:boto:Path: /wafmoa-25/ 2023-10-12T00:30:27.992 DEBUG:boto:Data: 2023-10-12T00:30:27.993 DEBUG:boto:Headers: {} 2023-10-12T00:30:27.993 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:27.993 DEBUG:boto:Port: 8000 2023-10-12T00:30:27.994 DEBUG:boto:Params: {} 2023-10-12T00:30:27.994 DEBUG:boto:Token: None 2023-10-12T00:30:27.995 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:27 GMT /wafmoa-25/ 2023-10-12T00:30:27.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6BVnrKnF8pljJu5oZuLxti62wwc= 2023-10-12T00:30:27.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6BVnrKnF8pljJu5oZuLxti62wwc=', 'Content-Length': '0'} 2023-10-12T00:30:28.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072cf248eaf973ab2-0065273e23-10fd-b1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.026 INFO:rgw_multi.tests:create bucket zone=b2 name=wafmoa-26 2023-10-12T00:30:28.027 DEBUG:boto:path=/wafmoa-26/ 2023-10-12T00:30:28.027 DEBUG:boto:auth_path=/wafmoa-26/ 2023-10-12T00:30:28.028 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.028 DEBUG:boto:Path: /wafmoa-26/ 2023-10-12T00:30:28.029 DEBUG:boto:Data: 2023-10-12T00:30:28.029 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.029 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:28.030 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.031 DEBUG:boto:Params: {} 2023-10-12T00:30:28.032 DEBUG:boto:Token: None 2023-10-12T00:30:28.032 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-26/ 2023-10-12T00:30:28.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:scNVf+nlecnThRtYt8QKFV6ppqk= 2023-10-12T00:30:28.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:scNVf+nlecnThRtYt8QKFV6ppqk=', 'Content-Length': '0'} 2023-10-12T00:30:28.057 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d754db0aaaa7871-0065273e24-1103-b2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.058 DEBUG:boto:path=/wafmoa-23/ 2023-10-12T00:30:28.058 DEBUG:boto:auth_path=/wafmoa-23/ 2023-10-12T00:30:28.059 DEBUG:boto:path=/wafmoa-23/?acl 2023-10-12T00:30:28.060 DEBUG:boto:auth_path=/wafmoa-23/?acl 2023-10-12T00:30:28.060 DEBUG:boto:Method: GET 2023-10-12T00:30:28.060 DEBUG:boto:Path: /wafmoa-23/?acl 2023-10-12T00:30:28.061 DEBUG:boto:Data: 2023-10-12T00:30:28.061 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.062 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.063 DEBUG:boto:Params: {} 2023-10-12T00:30:28.063 DEBUG:boto:Token: None 2023-10-12T00:30:28.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-23/?acl 2023-10-12T00:30:28.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kf/iM8hIjoVDm3wCLX+Nt0njCVA= 2023-10-12T00:30:28.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kf/iM8hIjoVDm3wCLX+Nt0njCVA=', 'Content-Length': '0'} 2023-10-12T00:30:28.066 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001a7d2965b45e4b4-0065273e24-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.067 DEBUG:boto:path=/wafmoa-23/ 2023-10-12T00:30:28.067 DEBUG:boto:auth_path=/wafmoa-23/ 2023-10-12T00:30:28.068 DEBUG:boto:path=/wafmoa-23/?acl 2023-10-12T00:30:28.068 DEBUG:boto:auth_path=/wafmoa-23/?acl 2023-10-12T00:30:28.069 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.069 DEBUG:boto:Path: /wafmoa-23/?acl 2023-10-12T00:30:28.070 DEBUG:boto:Data: 2023-10-12T00:30:28.070 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:30:28.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.072 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.072 DEBUG:boto:Params: {} 2023-10-12T00:30:28.073 DEBUG:boto:Token: None 2023-10-12T00:30:28.074 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT x-amz-acl:public-read /wafmoa-23/?acl 2023-10-12T00:30:28.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bP8ANcxEXU+0hl5BDmzPEv0oC5E= 2023-10-12T00:30:28.075 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bP8ANcxEXU+0hl5BDmzPEv0oC5E=', 'Content-Length': '0'} 2023-10-12T00:30:28.080 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d3be5e259f41d14-0065273e24-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.081 DEBUG:boto:path=/wafmoa-23/ 2023-10-12T00:30:28.082 DEBUG:boto:auth_path=/wafmoa-23/ 2023-10-12T00:30:28.083 DEBUG:boto:path=/wafmoa-23/?acl 2023-10-12T00:30:28.083 DEBUG:boto:auth_path=/wafmoa-23/?acl 2023-10-12T00:30:28.084 DEBUG:boto:Method: GET 2023-10-12T00:30:28.085 DEBUG:boto:Path: /wafmoa-23/?acl 2023-10-12T00:30:28.085 DEBUG:boto:Data: 2023-10-12T00:30:28.086 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.088 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.088 DEBUG:boto:Params: {} 2023-10-12T00:30:28.089 DEBUG:boto:Token: None 2023-10-12T00:30:28.089 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-23/?acl 2023-10-12T00:30:28.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kf/iM8hIjoVDm3wCLX+Nt0njCVA= 2023-10-12T00:30:28.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kf/iM8hIjoVDm3wCLX+Nt0njCVA=', 'Content-Length': '0'} 2023-10-12T00:30:28.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bffccc92dbc52d9c-0065273e24-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.092 DEBUG:boto:path=/wafmoa-24/ 2023-10-12T00:30:28.093 DEBUG:boto:auth_path=/wafmoa-24/ 2023-10-12T00:30:28.093 DEBUG:boto:path=/wafmoa-24/?acl 2023-10-12T00:30:28.094 DEBUG:boto:auth_path=/wafmoa-24/?acl 2023-10-12T00:30:28.094 DEBUG:boto:Method: GET 2023-10-12T00:30:28.095 DEBUG:boto:Path: /wafmoa-24/?acl 2023-10-12T00:30:28.096 DEBUG:boto:Data: 2023-10-12T00:30:28.096 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:28.098 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.098 DEBUG:boto:Params: {} 2023-10-12T00:30:28.099 DEBUG:boto:Token: None 2023-10-12T00:30:28.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-24/?acl 2023-10-12T00:30:28.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNXtg7hvYPytR+lGfURsIrV3Kcw= 2023-10-12T00:30:28.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNXtg7hvYPytR+lGfURsIrV3Kcw=', 'Content-Length': '0'} 2023-10-12T00:30:28.103 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8e19fd4a605a186-0065273e24-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.104 DEBUG:boto:path=/wafmoa-24/ 2023-10-12T00:30:28.105 DEBUG:boto:auth_path=/wafmoa-24/ 2023-10-12T00:30:28.105 DEBUG:boto:path=/wafmoa-24/?acl 2023-10-12T00:30:28.106 DEBUG:boto:auth_path=/wafmoa-24/?acl 2023-10-12T00:30:28.106 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.107 DEBUG:boto:Path: /wafmoa-24/?acl 2023-10-12T00:30:28.108 DEBUG:boto:Data: 2023-10-12T00:30:28.108 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:30:28.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:28.109 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.109 DEBUG:boto:Params: {} 2023-10-12T00:30:28.110 DEBUG:boto:Token: None 2023-10-12T00:30:28.110 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT x-amz-acl:public-read /wafmoa-24/?acl 2023-10-12T00:30:28.111 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L2BxCNrXfeA84hGQTX5dUIa69pE= 2023-10-12T00:30:28.111 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L2BxCNrXfeA84hGQTX5dUIa69pE=', 'Content-Length': '0'} 2023-10-12T00:30:28.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000138e6ecc09dec0ad-0065273e24-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.136 DEBUG:boto:path=/wafmoa-24/ 2023-10-12T00:30:28.136 DEBUG:boto:auth_path=/wafmoa-24/ 2023-10-12T00:30:28.137 DEBUG:boto:path=/wafmoa-24/?acl 2023-10-12T00:30:28.137 DEBUG:boto:auth_path=/wafmoa-24/?acl 2023-10-12T00:30:28.137 DEBUG:boto:Method: GET 2023-10-12T00:30:28.138 DEBUG:boto:Path: /wafmoa-24/?acl 2023-10-12T00:30:28.138 DEBUG:boto:Data: 2023-10-12T00:30:28.139 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:28.139 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.140 DEBUG:boto:Params: {} 2023-10-12T00:30:28.141 DEBUG:boto:Token: None 2023-10-12T00:30:28.141 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-24/?acl 2023-10-12T00:30:28.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNXtg7hvYPytR+lGfURsIrV3Kcw= 2023-10-12T00:30:28.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNXtg7hvYPytR+lGfURsIrV3Kcw=', 'Content-Length': '0'} 2023-10-12T00:30:28.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b28725b7c360d380-0065273e24-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.144 DEBUG:boto:path=/wafmoa-25/ 2023-10-12T00:30:28.145 DEBUG:boto:auth_path=/wafmoa-25/ 2023-10-12T00:30:28.145 DEBUG:boto:path=/wafmoa-25/?acl 2023-10-12T00:30:28.145 DEBUG:boto:auth_path=/wafmoa-25/?acl 2023-10-12T00:30:28.146 DEBUG:boto:Method: GET 2023-10-12T00:30:28.146 DEBUG:boto:Path: /wafmoa-25/?acl 2023-10-12T00:30:28.147 DEBUG:boto:Data: 2023-10-12T00:30:28.147 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.148 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:28.148 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.148 DEBUG:boto:Params: {} 2023-10-12T00:30:28.149 DEBUG:boto:Token: None 2023-10-12T00:30:28.149 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-25/?acl 2023-10-12T00:30:28.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tNv4UHErN/LiuXMpGM8XSpmIi4= 2023-10-12T00:30:28.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tNv4UHErN/LiuXMpGM8XSpmIi4=', 'Content-Length': '0'} 2023-10-12T00:30:28.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000007f98af5803e154b-0065273e24-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.152 DEBUG:boto:path=/wafmoa-25/ 2023-10-12T00:30:28.153 DEBUG:boto:auth_path=/wafmoa-25/ 2023-10-12T00:30:28.153 DEBUG:boto:path=/wafmoa-25/?acl 2023-10-12T00:30:28.153 DEBUG:boto:auth_path=/wafmoa-25/?acl 2023-10-12T00:30:28.154 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.154 DEBUG:boto:Path: /wafmoa-25/?acl 2023-10-12T00:30:28.155 DEBUG:boto:Data: 2023-10-12T00:30:28.155 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:30:28.155 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:28.156 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.156 DEBUG:boto:Params: {} 2023-10-12T00:30:28.156 DEBUG:boto:Token: None 2023-10-12T00:30:28.157 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT x-amz-acl:public-read /wafmoa-25/?acl 2023-10-12T00:30:28.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Md1Wyv8PGi113yJ11pBSNU8pdFE= 2023-10-12T00:30:28.158 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Md1Wyv8PGi113yJ11pBSNU8pdFE=', 'Content-Length': '0'} 2023-10-12T00:30:28.168 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003500d5b4bf9a1efc-0065273e24-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.168 DEBUG:boto:path=/wafmoa-25/ 2023-10-12T00:30:28.168 DEBUG:boto:auth_path=/wafmoa-25/ 2023-10-12T00:30:28.169 DEBUG:boto:path=/wafmoa-25/?acl 2023-10-12T00:30:28.169 DEBUG:boto:auth_path=/wafmoa-25/?acl 2023-10-12T00:30:28.170 DEBUG:boto:Method: GET 2023-10-12T00:30:28.170 DEBUG:boto:Path: /wafmoa-25/?acl 2023-10-12T00:30:28.171 DEBUG:boto:Data: 2023-10-12T00:30:28.171 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.171 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:28.172 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.172 DEBUG:boto:Params: {} 2023-10-12T00:30:28.173 DEBUG:boto:Token: None 2023-10-12T00:30:28.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-25/?acl 2023-10-12T00:30:28.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tNv4UHErN/LiuXMpGM8XSpmIi4= 2023-10-12T00:30:28.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tNv4UHErN/LiuXMpGM8XSpmIi4=', 'Content-Length': '0'} 2023-10-12T00:30:28.175 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b790859d84290181-0065273e24-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.176 DEBUG:boto:path=/wafmoa-26/ 2023-10-12T00:30:28.176 DEBUG:boto:auth_path=/wafmoa-26/ 2023-10-12T00:30:28.177 DEBUG:boto:path=/wafmoa-26/?acl 2023-10-12T00:30:28.177 DEBUG:boto:auth_path=/wafmoa-26/?acl 2023-10-12T00:30:28.177 DEBUG:boto:Method: GET 2023-10-12T00:30:28.178 DEBUG:boto:Path: /wafmoa-26/?acl 2023-10-12T00:30:28.178 DEBUG:boto:Data: 2023-10-12T00:30:28.179 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.179 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:28.179 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.180 DEBUG:boto:Params: {} 2023-10-12T00:30:28.180 DEBUG:boto:Token: None 2023-10-12T00:30:28.181 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-26/?acl 2023-10-12T00:30:28.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9INXlfEk18EJiSId6XXTI11k/2c= 2023-10-12T00:30:28.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9INXlfEk18EJiSId6XXTI11k/2c=', 'Content-Length': '0'} 2023-10-12T00:30:28.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d12a1c4202a5581d-0065273e24-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.184 DEBUG:boto:path=/wafmoa-26/ 2023-10-12T00:30:28.185 DEBUG:boto:auth_path=/wafmoa-26/ 2023-10-12T00:30:28.186 DEBUG:boto:path=/wafmoa-26/?acl 2023-10-12T00:30:28.186 DEBUG:boto:auth_path=/wafmoa-26/?acl 2023-10-12T00:30:28.187 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.188 DEBUG:boto:Path: /wafmoa-26/?acl 2023-10-12T00:30:28.188 DEBUG:boto:Data: 2023-10-12T00:30:28.189 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-12T00:30:28.190 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:28.190 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.191 DEBUG:boto:Params: {} 2023-10-12T00:30:28.191 DEBUG:boto:Token: None 2023-10-12T00:30:28.192 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT x-amz-acl:public-read /wafmoa-26/?acl 2023-10-12T00:30:28.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uu9P207iKiH9bc4GOILad91L/Tg= 2023-10-12T00:30:28.193 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uu9P207iKiH9bc4GOILad91L/Tg=', 'Content-Length': '0'} 2023-10-12T00:30:28.201 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f646ad6bdb3033f8-0065273e24-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.202 DEBUG:boto:path=/wafmoa-26/ 2023-10-12T00:30:28.202 DEBUG:boto:auth_path=/wafmoa-26/ 2023-10-12T00:30:28.202 DEBUG:boto:path=/wafmoa-26/?acl 2023-10-12T00:30:28.203 DEBUG:boto:auth_path=/wafmoa-26/?acl 2023-10-12T00:30:28.203 DEBUG:boto:Method: GET 2023-10-12T00:30:28.204 DEBUG:boto:Path: /wafmoa-26/?acl 2023-10-12T00:30:28.204 DEBUG:boto:Data: 2023-10-12T00:30:28.204 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.205 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:28.205 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.206 DEBUG:boto:Params: {} 2023-10-12T00:30:28.206 DEBUG:boto:Token: None 2023-10-12T00:30:28.207 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-26/?acl 2023-10-12T00:30:28.207 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9INXlfEk18EJiSId6XXTI11k/2c= 2023-10-12T00:30:28.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9INXlfEk18EJiSId6XXTI11k/2c=', 'Content-Length': '0'} 2023-10-12T00:30:28.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028320c97bae9d6d1-0065273e24-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-10-12T00:30:28.211 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-27 2023-10-12T00:30:28.212 DEBUG:boto:path=/wafmoa-27/ 2023-10-12T00:30:28.212 DEBUG:boto:auth_path=/wafmoa-27/ 2023-10-12T00:30:28.213 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.214 DEBUG:boto:Path: /wafmoa-27/ 2023-10-12T00:30:28.215 DEBUG:boto:Data: 2023-10-12T00:30:28.215 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.217 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.217 DEBUG:boto:Params: {} 2023-10-12T00:30:28.218 DEBUG:boto:Token: None 2023-10-12T00:30:28.219 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-27/ 2023-10-12T00:30:28.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YC2C/ODpZbEMQft6AktHB6EN9+Q= 2023-10-12T00:30:28.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YC2C/ODpZbEMQft6AktHB6EN9+Q=', 'Content-Length': '0'} 2023-10-12T00:30:28.233 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9df3e47f76aab40-0065273e24-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.233 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-28 2023-10-12T00:30:28.234 DEBUG:boto:path=/wafmoa-28/ 2023-10-12T00:30:28.234 DEBUG:boto:auth_path=/wafmoa-28/ 2023-10-12T00:30:28.235 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.235 DEBUG:boto:Path: /wafmoa-28/ 2023-10-12T00:30:28.236 DEBUG:boto:Data: 2023-10-12T00:30:28.236 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:28.238 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.238 DEBUG:boto:Params: {} 2023-10-12T00:30:28.239 DEBUG:boto:Token: None 2023-10-12T00:30:28.240 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-28/ 2023-10-12T00:30:28.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:77OMISg9Gg1dXeMsQL7iivpmoQY= 2023-10-12T00:30:28.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:77OMISg9Gg1dXeMsQL7iivpmoQY=', 'Content-Length': '0'} 2023-10-12T00:30:28.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d4fb9c6251c5489-0065273e24-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.276 INFO:rgw_multi.tests:create bucket zone=b1 name=wafmoa-29 2023-10-12T00:30:28.277 DEBUG:boto:path=/wafmoa-29/ 2023-10-12T00:30:28.277 DEBUG:boto:auth_path=/wafmoa-29/ 2023-10-12T00:30:28.278 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.279 DEBUG:boto:Path: /wafmoa-29/ 2023-10-12T00:30:28.279 DEBUG:boto:Data: 2023-10-12T00:30:28.279 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.280 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:28.280 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.281 DEBUG:boto:Params: {} 2023-10-12T00:30:28.281 DEBUG:boto:Token: None 2023-10-12T00:30:28.282 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-29/ 2023-10-12T00:30:28.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EpYp0EblrmX+ZyMhpUhpGegkQE4= 2023-10-12T00:30:28.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EpYp0EblrmX+ZyMhpUhpGegkQE4=', 'Content-Length': '0'} 2023-10-12T00:30:28.317 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c07b02a695ed4d8d-0065273e24-10fd-b1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.317 INFO:rgw_multi.tests:create bucket zone=b2 name=wafmoa-30 2023-10-12T00:30:28.318 DEBUG:boto:path=/wafmoa-30/ 2023-10-12T00:30:28.318 DEBUG:boto:auth_path=/wafmoa-30/ 2023-10-12T00:30:28.319 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.319 DEBUG:boto:Path: /wafmoa-30/ 2023-10-12T00:30:28.319 DEBUG:boto:Data: 2023-10-12T00:30:28.320 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.320 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:28.321 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.321 DEBUG:boto:Params: {} 2023-10-12T00:30:28.321 DEBUG:boto:Token: None 2023-10-12T00:30:28.322 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-30/ 2023-10-12T00:30:28.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5OZk471+kGqG4MI2UFJUUka7EfM= 2023-10-12T00:30:28.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5OZk471+kGqG4MI2UFJUUka7EfM=', 'Content-Length': '0'} 2023-10-12T00:30:28.347 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c59123e16153dd2d-0065273e24-1103-b2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.348 DEBUG:boto:path=/wafmoa-27/ 2023-10-12T00:30:28.349 DEBUG:boto:auth_path=/wafmoa-27/ 2023-10-12T00:30:28.349 DEBUG:boto:path=/wafmoa-27/?cors 2023-10-12T00:30:28.349 DEBUG:boto:auth_path=/wafmoa-27/?cors 2023-10-12T00:30:28.350 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.350 DEBUG:boto:Path: /wafmoa-27/?cors 2023-10-12T00:30:28.351 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-12T00:30:28.351 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-12T00:30:28.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.352 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.352 DEBUG:boto:Params: {} 2023-10-12T00:30:28.353 DEBUG:boto:Token: None 2023-10-12T00:30:28.353 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-27/?cors 2023-10-12T00:30:28.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UgAQ2qOFetB56tpxcEo1DzS5XZs= 2023-10-12T00:30:28.354 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UgAQ2qOFetB56tpxcEo1DzS5XZs=', 'Content-Length': '218'} 2023-10-12T00:30:28.369 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a32fd276a4922e9-0065273e24-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.370 DEBUG:boto:path=/wafmoa-27/ 2023-10-12T00:30:28.370 DEBUG:boto:auth_path=/wafmoa-27/ 2023-10-12T00:30:28.371 DEBUG:boto:path=/wafmoa-27/?cors 2023-10-12T00:30:28.371 DEBUG:boto:auth_path=/wafmoa-27/?cors 2023-10-12T00:30:28.372 DEBUG:boto:Method: GET 2023-10-12T00:30:28.372 DEBUG:boto:Path: /wafmoa-27/?cors 2023-10-12T00:30:28.372 DEBUG:boto:Data: 2023-10-12T00:30:28.373 DEBUG:boto:Headers: {} 2023-10-12T00:30:28.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:28.374 DEBUG:boto:Port: 8000 2023-10-12T00:30:28.374 DEBUG:boto:Params: {} 2023-10-12T00:30:28.374 DEBUG:boto:Token: None 2023-10-12T00:30:28.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-27/?cors 2023-10-12T00:30:28.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D09c8gnUGWvCKuq9YXHHHrXgDQI= 2023-10-12T00:30:28.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D09c8gnUGWvCKuq9YXHHHrXgDQI=', 'Content-Length': '0'} 2023-10-12T00:30:28.377 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c4d6f3bae4de7f7-0065273e24-1115-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 12 Oct 2023 00:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:28.418 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-12T00:30:28.419 DEBUG:boto:path=/wafmoa-28/ 2023-10-12T00:30:28.420 DEBUG:boto:auth_path=/wafmoa-28/ 2023-10-12T00:30:28.420 DEBUG:boto:path=/wafmoa-28/?cors 2023-10-12T00:30:28.420 DEBUG:boto:auth_path=/wafmoa-28/?cors 2023-10-12T00:30:28.421 DEBUG:boto:Method: PUT 2023-10-12T00:30:28.421 DEBUG:boto:Path: /wafmoa-28/?cors 2023-10-12T00:30:28.422 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-12T00:30:28.422 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-12T00:30:28.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:28.423 DEBUG:boto:Port: 8001 2023-10-12T00:30:28.423 DEBUG:boto:Params: {} 2023-10-12T00:30:28.424 DEBUG:boto:Token: None 2023-10-12T00:30:28.424 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 12 Oct 2023 00:30:28 GMT /wafmoa-28/?cors 2023-10-12T00:30:28.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7aR+RHcBuSHvAi0n2oLddMYFHa0= 2023-10-12T00:30:28.425 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7aR+RHcBuSHvAi0n2oLddMYFHa0=', 'Content-Length': '218'} 2023-10-12T00:30:29.435 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048c6b537b82d19c9-0065273e24-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:29.436 DEBUG:boto:path=/wafmoa-28/ 2023-10-12T00:30:29.436 DEBUG:boto:auth_path=/wafmoa-28/ 2023-10-12T00:30:29.437 DEBUG:boto:path=/wafmoa-28/?cors 2023-10-12T00:30:29.437 DEBUG:boto:auth_path=/wafmoa-28/?cors 2023-10-12T00:30:29.437 DEBUG:boto:Method: GET 2023-10-12T00:30:29.438 DEBUG:boto:Path: /wafmoa-28/?cors 2023-10-12T00:30:29.438 DEBUG:boto:Data: 2023-10-12T00:30:29.439 DEBUG:boto:Headers: {} 2023-10-12T00:30:29.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:29.440 DEBUG:boto:Port: 8001 2023-10-12T00:30:29.441 DEBUG:boto:Params: {} 2023-10-12T00:30:29.442 DEBUG:boto:Token: None 2023-10-12T00:30:29.443 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:29 GMT /wafmoa-28/?cors 2023-10-12T00:30:29.444 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oz9+Ff3inQwpsiHGXx32VK+kwy0= 2023-10-12T00:30:29.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oz9+Ff3inQwpsiHGXx32VK+kwy0=', 'Content-Length': '0'} 2023-10-12T00:30:29.446 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003e68a96318efa493-0065273e25-1111-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 12 Oct 2023 00:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:29.447 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-12T00:30:29.447 DEBUG:boto:path=/wafmoa-29/ 2023-10-12T00:30:29.448 DEBUG:boto:auth_path=/wafmoa-29/ 2023-10-12T00:30:29.448 DEBUG:boto:path=/wafmoa-29/?cors 2023-10-12T00:30:29.449 DEBUG:boto:auth_path=/wafmoa-29/?cors 2023-10-12T00:30:29.449 DEBUG:boto:Method: PUT 2023-10-12T00:30:29.449 DEBUG:boto:Path: /wafmoa-29/?cors 2023-10-12T00:30:29.450 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-12T00:30:29.450 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-12T00:30:29.451 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:29.451 DEBUG:boto:Port: 8000 2023-10-12T00:30:29.452 DEBUG:boto:Params: {} 2023-10-12T00:30:29.452 DEBUG:boto:Token: None 2023-10-12T00:30:29.452 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 12 Oct 2023 00:30:29 GMT /wafmoa-29/?cors 2023-10-12T00:30:29.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WZk4hzw/39NbleJn+Sj60u4rz+E= 2023-10-12T00:30:29.453 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WZk4hzw/39NbleJn+Sj60u4rz+E=', 'Content-Length': '218'} 2023-10-12T00:30:30.465 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bb284bb7b867d85a-0065273e25-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:30.465 DEBUG:boto:path=/wafmoa-29/ 2023-10-12T00:30:30.466 DEBUG:boto:auth_path=/wafmoa-29/ 2023-10-12T00:30:30.466 DEBUG:boto:path=/wafmoa-29/?cors 2023-10-12T00:30:30.466 DEBUG:boto:auth_path=/wafmoa-29/?cors 2023-10-12T00:30:30.467 DEBUG:boto:Method: GET 2023-10-12T00:30:30.467 DEBUG:boto:Path: /wafmoa-29/?cors 2023-10-12T00:30:30.468 DEBUG:boto:Data: 2023-10-12T00:30:30.468 DEBUG:boto:Headers: {} 2023-10-12T00:30:30.468 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:30:30.469 DEBUG:boto:Port: 8000 2023-10-12T00:30:30.469 DEBUG:boto:Params: {} 2023-10-12T00:30:30.469 DEBUG:boto:Token: None 2023-10-12T00:30:30.470 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:30 GMT /wafmoa-29/?cors 2023-10-12T00:30:30.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hPZP0wVZK+64PFeh0HNxIXOPNd0= 2023-10-12T00:30:30.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hPZP0wVZK+64PFeh0HNxIXOPNd0=', 'Content-Length': '0'} 2023-10-12T00:30:30.472 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000777929d3096ed2c8-0065273e26-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 12 Oct 2023 00:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:30.514 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-12T00:30:30.515 DEBUG:boto:path=/wafmoa-30/ 2023-10-12T00:30:30.516 DEBUG:boto:auth_path=/wafmoa-30/ 2023-10-12T00:30:30.516 DEBUG:boto:path=/wafmoa-30/?cors 2023-10-12T00:30:30.517 DEBUG:boto:auth_path=/wafmoa-30/?cors 2023-10-12T00:30:30.517 DEBUG:boto:Method: PUT 2023-10-12T00:30:30.517 DEBUG:boto:Path: /wafmoa-30/?cors 2023-10-12T00:30:30.518 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-12T00:30:30.518 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-12T00:30:30.519 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:30.519 DEBUG:boto:Port: 8001 2023-10-12T00:30:30.519 DEBUG:boto:Params: {} 2023-10-12T00:30:30.520 DEBUG:boto:Token: None 2023-10-12T00:30:30.520 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Thu, 12 Oct 2023 00:30:30 GMT /wafmoa-30/?cors 2023-10-12T00:30:30.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6wRGI1xF37dJ1g7uoGwQEUpfR2k= 2023-10-12T00:30:30.521 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6wRGI1xF37dJ1g7uoGwQEUpfR2k=', 'Content-Length': '218'} 2023-10-12T00:30:31.532 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9454f9885c7dd67-0065273e26-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:31.533 DEBUG:boto:path=/wafmoa-30/ 2023-10-12T00:30:31.534 DEBUG:boto:auth_path=/wafmoa-30/ 2023-10-12T00:30:31.534 DEBUG:boto:path=/wafmoa-30/?cors 2023-10-12T00:30:31.535 DEBUG:boto:auth_path=/wafmoa-30/?cors 2023-10-12T00:30:31.536 DEBUG:boto:Method: GET 2023-10-12T00:30:31.537 DEBUG:boto:Path: /wafmoa-30/?cors 2023-10-12T00:30:31.537 DEBUG:boto:Data: 2023-10-12T00:30:31.538 DEBUG:boto:Headers: {} 2023-10-12T00:30:31.539 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:30:31.539 DEBUG:boto:Port: 8001 2023-10-12T00:30:31.540 DEBUG:boto:Params: {} 2023-10-12T00:30:31.541 DEBUG:boto:Token: None 2023-10-12T00:30:31.541 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:30:31 GMT /wafmoa-30/?cors 2023-10-12T00:30:31.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gg8NtKE34uCqxbBQqpd9AE30aMs= 2023-10-12T00:30:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gg8NtKE34uCqxbBQqpd9AE30aMs=', 'Content-Length': '0'} 2023-10-12T00:30:31.545 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a5d4ce37d515303-0065273e27-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Thu, 12 Oct 2023 00:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:31.590 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-12T00:30:31.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-10-12T00:30:31.592 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-31 2023-10-12T00:30:31.592 DEBUG:boto:path=/wafmoa-31/ 2023-10-12T00:30:31.593 DEBUG:boto:auth_path=/wafmoa-31/ 2023-10-12T00:30:31.593 DEBUG:boto:Method: PUT 2023-10-12T00:30:31.594 DEBUG:boto:Path: /wafmoa-31/ 2023-10-12T00:30:31.594 DEBUG:boto:Data: 2023-10-12T00:30:31.595 DEBUG:boto:Headers: {} 2023-10-12T00:30:31.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:31.595 DEBUG:boto:Port: 8000 2023-10-12T00:30:31.596 DEBUG:boto:Params: {} 2023-10-12T00:30:31.596 DEBUG:boto:Token: None 2023-10-12T00:30:31.597 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:31 GMT /wafmoa-31/ 2023-10-12T00:30:31.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0iJVWjqTkGyenAvkIefDgty/9Uo= 2023-10-12T00:30:31.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0iJVWjqTkGyenAvkIefDgty/9Uo=', 'Content-Length': '0'} 2023-10-12T00:30:31.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d1f5797d4e36fa4-0065273e27-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:31.619 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-32 2023-10-12T00:30:31.620 DEBUG:boto:path=/wafmoa-32/ 2023-10-12T00:30:31.620 DEBUG:boto:auth_path=/wafmoa-32/ 2023-10-12T00:30:31.620 DEBUG:boto:Method: PUT 2023-10-12T00:30:31.621 DEBUG:boto:Path: /wafmoa-32/ 2023-10-12T00:30:31.621 DEBUG:boto:Data: 2023-10-12T00:30:31.622 DEBUG:boto:Headers: {} 2023-10-12T00:30:31.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:31.623 DEBUG:boto:Port: 8001 2023-10-12T00:30:31.623 DEBUG:boto:Params: {} 2023-10-12T00:30:31.623 DEBUG:boto:Token: None 2023-10-12T00:30:31.624 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:31 GMT /wafmoa-32/ 2023-10-12T00:30:31.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:slGLks4QS7N+lXhnVbkuaXYy758= 2023-10-12T00:30:31.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:slGLks4QS7N+lXhnVbkuaXYy758=', 'Content-Length': '0'} 2023-10-12T00:30:31.656 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0b176c74316dc8a-0065273e27-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:31.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:31.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:31.658 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:31.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:31.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:30:31.754 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:30:31.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:30:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:30:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070631.641350_235.1", 2023-10-12T00:30:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:31.641350Z" 2023-10-12T00:30:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:30:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.758 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:30:31.759 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070631.641350_235.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:31.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:30:31.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:31.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:31.761 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:31.813 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:31.813 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:31.842+0000 7fa2e9620780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:31.843+0000 7fa2e9620780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:31.854 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:31.854 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:30:31.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:31.856 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:30:31.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:30:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:31.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:30:31.860 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070630.461659_225.1", 2023-10-12T00:30:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:31.862 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:30.461659Z", 2023-10-12T00:30:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:31.864 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:30:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:31Z" 2023-10-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:31.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070630.461659_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:30.461659Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:30:31Z" } 2023-10-12T00:30:31.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070630.461659_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:30.461659Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:30:31.870 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070631.641350_235.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:31.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070630.461659_225.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:31.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070631.641350_235.1 target=1_1697070630.461659_225.1 2023-10-12T00:30:36.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:36.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:36.874 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:36.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:36.931 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:36.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:36.963+0000 7f0bfd14b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:36.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:36.964+0000 7f0bfd14b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:36.976 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:36.976 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:36.977 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:36.977 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:36.977 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:36.977 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:30:36.977 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:36.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.978 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:36.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:36.978 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:36.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:36.979 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:36.979 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:36.979 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:36.979 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:30:36.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:36.980 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:30:36.980 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:36.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:36.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:36.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:36.981 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:36.981 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:30:36.983 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:36.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:36.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:36.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:36.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:36.984 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:36.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070631.641350_235.1", 2023-10-12T00:30:36.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:36.984 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:36.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:36.985 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:31.641350Z", 2023-10-12T00:30:36.985 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:36.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:36.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:36.986 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:36.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:36.986 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:36.987 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:36.987 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:36.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:36.987 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:36.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:30:36.989 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:36.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:36.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:36.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:36.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.990 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:36.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:30:36.991 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:30:36.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:36.991 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:36Z" 2023-10-12T00:30:36.991 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:36.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070631.641350_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:31.641350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:30:36Z" } 2023-10-12T00:30:36.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070631.641350_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:31.641350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:30:36.994 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070631.641350_235.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:36.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070631.641350_235.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:36.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:30:36.995 DEBUG:boto:path=/wafmoa-31/ 2023-10-12T00:30:36.995 DEBUG:boto:auth_path=/wafmoa-31/ 2023-10-12T00:30:36.996 DEBUG:boto:Method: HEAD 2023-10-12T00:30:36.996 DEBUG:boto:Path: /wafmoa-31/ 2023-10-12T00:30:36.996 DEBUG:boto:Data: 2023-10-12T00:30:36.997 DEBUG:boto:Headers: {} 2023-10-12T00:30:36.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:36.998 DEBUG:boto:Port: 8000 2023-10-12T00:30:36.998 DEBUG:boto:Params: {} 2023-10-12T00:30:36.999 DEBUG:boto:Token: None 2023-10-12T00:30:36.999 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:36 GMT /wafmoa-31/ 2023-10-12T00:30:37.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UB6b3ec0ix9XDK+Y1XZ9VMeXISI= 2023-10-12T00:30:37.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UB6b3ec0ix9XDK+Y1XZ9VMeXISI=', 'Content-Length': '0'} 2023-10-12T00:30:37.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6333db8e3695126-0065273e2d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.003 DEBUG:boto:path=/wafmoa-31/foo 2023-10-12T00:30:37.004 DEBUG:boto:auth_path=/wafmoa-31/foo 2023-10-12T00:30:37.004 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.005 DEBUG:boto:Path: /wafmoa-31/foo 2023-10-12T00:30:37.005 DEBUG:boto:Data: 2023-10-12T00:30:37.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:30:37.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.007 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.007 DEBUG:boto:Params: {} 2023-10-12T00:30:37.007 DEBUG:boto:Token: None 2023-10-12T00:30:37.008 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-31/foo 2023-10-12T00:30:37.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qF7/Vu8CfN/5VMs5AIF/IUnxjk4= 2023-10-12T00:30:37.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qF7/Vu8CfN/5VMs5AIF/IUnxjk4='} 2023-10-12T00:30:37.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcd91b5dafc6d8aa-0065273e2d-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.015 DEBUG:boto:path=/wafmoa-31/ 2023-10-12T00:30:37.016 DEBUG:boto:auth_path=/wafmoa-31/ 2023-10-12T00:30:37.016 DEBUG:boto:Method: DELETE 2023-10-12T00:30:37.017 DEBUG:boto:Path: /wafmoa-31/ 2023-10-12T00:30:37.017 DEBUG:boto:Data: 2023-10-12T00:30:37.017 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.018 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.019 DEBUG:boto:Params: {} 2023-10-12T00:30:37.019 DEBUG:boto:Token: None 2023-10-12T00:30:37.020 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-31/ 2023-10-12T00:30:37.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/u0eE2Zf/EcB1bn9QdPGwnfyk24= 2023-10-12T00:30:37.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/u0eE2Zf/EcB1bn9QdPGwnfyk24=', 'Content-Length': '0'} 2023-10-12T00:30:37.027 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx0000069a8b27dde1bc70b-0065273e2d-1115-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.027 DEBUG:boto:path=/wafmoa-32/ 2023-10-12T00:30:37.028 DEBUG:boto:auth_path=/wafmoa-32/ 2023-10-12T00:30:37.028 DEBUG:boto:Method: HEAD 2023-10-12T00:30:37.029 DEBUG:boto:Path: /wafmoa-32/ 2023-10-12T00:30:37.029 DEBUG:boto:Data: 2023-10-12T00:30:37.029 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.030 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.031 DEBUG:boto:Params: {} 2023-10-12T00:30:37.031 DEBUG:boto:Token: None 2023-10-12T00:30:37.032 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-32/ 2023-10-12T00:30:37.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zxdERYJk4kGgHTYxvNVCWCbToVc= 2023-10-12T00:30:37.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zxdERYJk4kGgHTYxvNVCWCbToVc=', 'Content-Length': '0'} 2023-10-12T00:30:37.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000300f2ac027435017-0065273e2d-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.035 DEBUG:boto:path=/wafmoa-32/foo 2023-10-12T00:30:37.036 DEBUG:boto:auth_path=/wafmoa-32/foo 2023-10-12T00:30:37.036 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.037 DEBUG:boto:Path: /wafmoa-32/foo 2023-10-12T00:30:37.037 DEBUG:boto:Data: 2023-10-12T00:30:37.038 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:30:37.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.039 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.039 DEBUG:boto:Params: {} 2023-10-12T00:30:37.039 DEBUG:boto:Token: None 2023-10-12T00:30:37.040 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-32/foo 2023-10-12T00:30:37.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkv11Xfi/1CCaHO5MZy5d3QBJiA= 2023-10-12T00:30:37.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkv11Xfi/1CCaHO5MZy5d3QBJiA='} 2023-10-12T00:30:37.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f204e46f7652e11d-0065273e2d-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.087 DEBUG:boto:path=/wafmoa-32/ 2023-10-12T00:30:37.087 DEBUG:boto:auth_path=/wafmoa-32/ 2023-10-12T00:30:37.088 DEBUG:boto:Method: DELETE 2023-10-12T00:30:37.088 DEBUG:boto:Path: /wafmoa-32/ 2023-10-12T00:30:37.089 DEBUG:boto:Data: 2023-10-12T00:30:37.089 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.090 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.090 DEBUG:boto:Params: {} 2023-10-12T00:30:37.091 DEBUG:boto:Token: None 2023-10-12T00:30:37.091 DEBUG:boto:StringToSign: DELETE Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-32/ 2023-10-12T00:30:37.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IJCIHOS4LIf3ROwaIWCqoeJ0+Nw= 2023-10-12T00:30:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IJCIHOS4LIf3ROwaIWCqoeJ0+Nw=', 'Content-Length': '0'} 2023-10-12T00:30:37.098 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx0000041868161451752d8-0065273e2d-1111-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.099 DEBUG:boto:path=/wafmoa-31/ 2023-10-12T00:30:37.100 DEBUG:boto:auth_path=/wafmoa-31/ 2023-10-12T00:30:37.100 DEBUG:boto:Method: HEAD 2023-10-12T00:30:37.100 DEBUG:boto:Path: /wafmoa-31/ 2023-10-12T00:30:37.101 DEBUG:boto:Data: 2023-10-12T00:30:37.101 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.102 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.103 DEBUG:boto:Params: {} 2023-10-12T00:30:37.103 DEBUG:boto:Token: None 2023-10-12T00:30:37.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-31/ 2023-10-12T00:30:37.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7H5ozn79qNT3+2fq/UR0K3DB90w= 2023-10-12T00:30:37.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7H5ozn79qNT3+2fq/UR0K3DB90w=', 'Content-Length': '0'} 2023-10-12T00:30:37.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032a15b93b481aa5d-0065273e2d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.107 DEBUG:boto:path=/wafmoa-32/ 2023-10-12T00:30:37.108 DEBUG:boto:auth_path=/wafmoa-32/ 2023-10-12T00:30:37.108 DEBUG:boto:Method: HEAD 2023-10-12T00:30:37.108 DEBUG:boto:Path: /wafmoa-32/ 2023-10-12T00:30:37.109 DEBUG:boto:Data: 2023-10-12T00:30:37.109 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.110 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.111 DEBUG:boto:Params: {} 2023-10-12T00:30:37.111 DEBUG:boto:Token: None 2023-10-12T00:30:37.111 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-32/ 2023-10-12T00:30:37.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zxdERYJk4kGgHTYxvNVCWCbToVc= 2023-10-12T00:30:37.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zxdERYJk4kGgHTYxvNVCWCbToVc=', 'Content-Length': '0'} 2023-10-12T00:30:37.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdc35b50f2363bf6-0065273e2d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-10-12T00:30:37.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-10-12T00:30:37.128 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-33 2023-10-12T00:30:37.129 DEBUG:boto:path=/wafmoa-33/ 2023-10-12T00:30:37.129 DEBUG:boto:auth_path=/wafmoa-33/ 2023-10-12T00:30:37.130 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.130 DEBUG:boto:Path: /wafmoa-33/ 2023-10-12T00:30:37.131 DEBUG:boto:Data: 2023-10-12T00:30:37.131 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.132 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.132 DEBUG:boto:Params: {} 2023-10-12T00:30:37.133 DEBUG:boto:Token: None 2023-10-12T00:30:37.133 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-33/ 2023-10-12T00:30:37.134 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Vx28PbiKvJTK7jnfQ4wTiwJBqwE= 2023-10-12T00:30:37.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Vx28PbiKvJTK7jnfQ4wTiwJBqwE=', 'Content-Length': '0'} 2023-10-12T00:30:37.149 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e063f57a03a20f3-0065273e2d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.150 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-34 2023-10-12T00:30:37.150 DEBUG:boto:path=/wafmoa-34/ 2023-10-12T00:30:37.151 DEBUG:boto:auth_path=/wafmoa-34/ 2023-10-12T00:30:37.151 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.151 DEBUG:boto:Path: /wafmoa-34/ 2023-10-12T00:30:37.152 DEBUG:boto:Data: 2023-10-12T00:30:37.152 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.153 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.153 DEBUG:boto:Params: {} 2023-10-12T00:30:37.154 DEBUG:boto:Token: None 2023-10-12T00:30:37.154 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-34/ 2023-10-12T00:30:37.154 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+TMAJtkUD2r+iOfk7uTTklgOS4= 2023-10-12T00:30:37.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+TMAJtkUD2r+iOfk7uTTklgOS4=', 'Content-Length': '0'} 2023-10-12T00:30:37.187 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e52b2c05560505a5-0065273e2d-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.188 DEBUG:boto:path=/wafmoa-33/ 2023-10-12T00:30:37.189 DEBUG:boto:auth_path=/wafmoa-33/ 2023-10-12T00:30:37.189 DEBUG:boto:Method: HEAD 2023-10-12T00:30:37.189 DEBUG:boto:Path: /wafmoa-33/ 2023-10-12T00:30:37.190 DEBUG:boto:Data: 2023-10-12T00:30:37.190 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.191 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.191 DEBUG:boto:Params: {} 2023-10-12T00:30:37.192 DEBUG:boto:Token: None 2023-10-12T00:30:37.192 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-33/ 2023-10-12T00:30:37.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nv6gugQOIzMhT/RXiVi5wtFb2Q0= 2023-10-12T00:30:37.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nv6gugQOIzMhT/RXiVi5wtFb2Q0=', 'Content-Length': '0'} 2023-10-12T00:30:37.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c91cd5006358326c-0065273e2d-1115-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.197 DEBUG:boto:path=/wafmoa-33/key 2023-10-12T00:30:37.197 DEBUG:boto:auth_path=/wafmoa-33/key 2023-10-12T00:30:37.198 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.198 DEBUG:boto:Path: /wafmoa-33/key 2023-10-12T00:30:37.199 DEBUG:boto:Data: 2023-10-12T00:30:37.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-12T00:30:37.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:30:37.200 DEBUG:boto:Port: 8000 2023-10-12T00:30:37.201 DEBUG:boto:Params: {} 2023-10-12T00:30:37.201 DEBUG:boto:Token: None 2023-10-12T00:30:37.202 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-33/key 2023-10-12T00:30:37.202 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5dJf84c13I3ue/AArIEtuhTE/oA= 2023-10-12T00:30:37.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5dJf84c13I3ue/AArIEtuhTE/oA='} 2023-10-12T00:30:37.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a218c5f9887a9a14-0065273e2d-1115-a1'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.209 DEBUG:boto:path=/wafmoa-34/ 2023-10-12T00:30:37.209 DEBUG:boto:auth_path=/wafmoa-34/ 2023-10-12T00:30:37.210 DEBUG:boto:Method: HEAD 2023-10-12T00:30:37.210 DEBUG:boto:Path: /wafmoa-34/ 2023-10-12T00:30:37.211 DEBUG:boto:Data: 2023-10-12T00:30:37.211 DEBUG:boto:Headers: {} 2023-10-12T00:30:37.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.212 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.212 DEBUG:boto:Params: {} 2023-10-12T00:30:37.213 DEBUG:boto:Token: None 2023-10-12T00:30:37.214 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-34/ 2023-10-12T00:30:37.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+gvuG6G98j75zeeWyok7/I1f0k= 2023-10-12T00:30:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+gvuG6G98j75zeeWyok7/I1f0k=', 'Content-Length': '0'} 2023-10-12T00:30:37.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ba7f92b130fb781-0065273e2d-1111-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.218 DEBUG:boto:path=/wafmoa-34/key 2023-10-12T00:30:37.219 DEBUG:boto:auth_path=/wafmoa-34/key 2023-10-12T00:30:37.220 DEBUG:boto:Method: PUT 2023-10-12T00:30:37.220 DEBUG:boto:Path: /wafmoa-34/key 2023-10-12T00:30:37.220 DEBUG:boto:Data: 2023-10-12T00:30:37.221 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-12T00:30:37.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:30:37.222 DEBUG:boto:Port: 8001 2023-10-12T00:30:37.222 DEBUG:boto:Params: {} 2023-10-12T00:30:37.223 DEBUG:boto:Token: None 2023-10-12T00:30:37.223 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Thu, 12 Oct 2023 00:30:37 GMT /wafmoa-34/key 2023-10-12T00:30:37.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BpfiIDdzpEVnbu917OQmPEX1H6E= 2023-10-12T00:30:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:30:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BpfiIDdzpEVnbu917OQmPEX1H6E='} 2023-10-12T00:30:37.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f15aa851285fca0-0065273e2d-1111-a2'), ('Date', 'Thu, 12 Oct 2023 00:30:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:30:37.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:37.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:37.233 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:37.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:37.285 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:37.315 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:30:37.331 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:30:37.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.332 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:37.332 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:30:37.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.333 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:37.333 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:30:37.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.333 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:30:37.334 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:37.173811Z" 2023-10-12T00:30:37.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:37.334 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:30:37.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.335 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:30:37.336 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:37.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:30:37.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:37.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:37.338 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:37.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:37.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:37.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:37.416+0000 7f292a21a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:37.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:37.417+0000 7f292a21a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:37.428 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:30:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:30:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:30:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:30:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070631.641350_235.1", 2023-10-12T00:30:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:31.641350Z", 2023-10-12T00:30:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:30:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:30:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:30:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:37Z" 2023-10-12T00:30:37.442 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:37.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070631.641350_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:31.641350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:30:37Z" } 2023-10-12T00:30:37.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070631.641350_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:31.641350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:30:37.444 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:37.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070631.641350_235.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:37.445 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070637.173811_243.1 target=1_1697070631.641350_235.1 2023-10-12T00:30:42.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.447 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:42.504 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:42.504 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:42.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:42.534+0000 7f8f9c707780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:42.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:30:42.535+0000 7f8f9c707780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:30:42.547 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:42.547 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:42.548 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:42.548 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:42.548 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:42.548 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:30:42.549 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:42.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.549 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:42.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.550 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:42.550 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.550 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:42.550 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:30:42.551 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.551 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:30:42.551 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.552 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:30:42.552 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:42.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.554 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:42.554 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.554 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:42.555 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:30:42.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.606 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.606 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:30:42.606 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:42.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.607 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:42.607 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.607 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:42.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:30:42.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.609 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:30:42.609 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:42.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.610 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:42.610 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.610 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:30:42.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:30:42.611 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:30:42.612 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:30:42.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.612 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:42.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.613 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:42.613 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:30:42.613 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:30:42.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.614 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:42Z" 2023-10-12T00:30:42.614 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:42.615 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:30:42Z" } 2023-10-12T00:30:42.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:30:42.616 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:42.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:30:42.617 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:30:42.617 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T00:30:42.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.619 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:42.715 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:30:42.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:42.716 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:30:42.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.717 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-12T00:30:42.717 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:37.208867Z" 2023-10-12T00:30:42.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.717 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:30:42.718 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:30:42.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-12T00:30:42.718 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:37.015414Z" 2023-10-12T00:30:42.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.719 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:30:42.720 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:42.720 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:30:42.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:42.722 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:42.772 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:42.772 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:42.813 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:42.814 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:42.814 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:42.815 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:42.815 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:42.815 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:30:42.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.816 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:42.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:42.817 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.818 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:42.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:42.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:30:42.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:42.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.821 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:42.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003042", 2023-10-12T00:30:42.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.822 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:06.064253Z" 2023-10-12T00:30:42.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.823 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:42.823 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.824 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:42.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:30:42.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.824 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:30:42.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:42.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:42.827 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:42.827 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:42.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-12T00:30:42.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:42.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:42.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:42.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.293604Z" 2023-10-12T00:30:42.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:42.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:42.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.829 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:42.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:30:42.830 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:30:42.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:42.830 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:42Z" 2023-10-12T00:30:42.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:42.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:23.283784Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003042", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:06.064253Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:23.327950Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.293604Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:30:42Z" } 2023-10-12T00:30:42.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:23.283784Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:06.064253Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:23.327950Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.293604Z'}}] 2023-10-12T00:30:42.832 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:42.833 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003042', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000000929'} 2023-10-12T00:30:42.833 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000003183 target=00000000000000000000:00000000000000003042, shard=3 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-10-12T00:30:47.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:47.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:47.836 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:47.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:47.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:47.942 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:47.942 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:47.943 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:47.943 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:47.943 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:47.943 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:30:47.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.944 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:47.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:47.944 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:47.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:47.945 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:47.945 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:47.945 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:47.945 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:47.946 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:47.946 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:30:47.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:47.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:47.947 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:47.947 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:47.947 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:47.947 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-12T00:30:47.948 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:47.948 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:47.948 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:47.948 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.208867Z" 2023-10-12T00:30:47.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:47.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:47.949 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:47.949 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:47.950 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:47.950 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:30:47.950 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:47.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:47.951 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:47.951 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:30:47.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:47.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:47.952 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:47.952 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:47.953 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:47.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-12T00:30:47.953 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:47.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:47.954 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:47.954 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.015414Z" 2023-10-12T00:30:47.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:47.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:47.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:47.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.955 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:47.955 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:30:47.956 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:30:47.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:47.956 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:47Z" 2023-10-12T00:30:47.956 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:47.958 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:23.283784Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.208867Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:23.327950Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.015414Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:30:47Z" } 2023-10-12T00:30:47.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:23.283784Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.208867Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:23.327950Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.015414Z'}}] 2023-10-12T00:30:47.959 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:47.959 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003183', 2: '00000000000000000000:00000000000000002204', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:47.959 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:30:47.960 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T00:30:47.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:47.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:47.961 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:48.055 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:30:48.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.056 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:48.056 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:30:48.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.056 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:44.418283Z" 2023-10-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:30:48.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.058 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:30:48.058 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:43.347442Z" 2023-10-12T00:30:48.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.059 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:30:48.060 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:30:48.060 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:30:48.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:48.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:48.062 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:48.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:48.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:48.156 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:48.156 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:48.157 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:48.157 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:48.157 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:48.157 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:30:48.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.158 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:48.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.158 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:48.158 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:48.160 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:48.160 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:30:48.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003888", 2023-10-12T00:30:48.161 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.230130Z" 2023-10-12T00:30:48.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:48.163 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:30:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:30:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:48.166 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:48.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-12T00:30:48.166 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:48.166 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:48.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:48.167 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.047179Z" 2023-10-12T00:30:48.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:48.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:48.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.168 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:48.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:30:48.168 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:30:48.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:48.169 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:48Z" 2023-10-12T00:30:48.169 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:48.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003888", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.230130Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.047179Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:30:48Z" } 2023-10-12T00:30:48.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003888', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.230130Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.047179Z'}}] 2023-10-12T00:30:48.171 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:30:48.171 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:48.172 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004029 target=00000000000000000000:00000000000000003888, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-10-12T00:30:53.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:53.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:53.175 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:53.234 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:53.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:54.286 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:54.286 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:30:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:30:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003888", 2023-10-12T00:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.230130Z" 2023-10-12T00:30:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:30:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:30:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.047179Z" 2023-10-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:54Z" 2023-10-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:54.301 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003888", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.230130Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.047179Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:30:54Z" } 2023-10-12T00:30:54.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003888', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.230130Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.047179Z'}}] 2023-10-12T00:30:54.302 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:30:54.302 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:54.303 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004029 target=00000000000000000000:00000000000000003888, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-10-12T00:30:59.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:59.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:30:59.305 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:30:59.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:30:59.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:30:59.408 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:30:59.409 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:30:59.409 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:30:59.409 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:30:59.409 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:30:59.410 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:30:59.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.410 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:30:59.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:59.411 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:30:59.411 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:59.411 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:59.411 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:30:59.411 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:59.412 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:59.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:59.412 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:30:59.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:59.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:59.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:30:59.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:59.413 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:59.413 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003888", 2023-10-12T00:30:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.230130Z" 2023-10-12T00:30:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:30:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:30:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:30:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:30:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:30:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:30:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:30:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-12T00:30:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.047179Z" 2023-10-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:30:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:30:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:30:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:30:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:30:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:30:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:30:59Z" 2023-10-12T00:30:59.421 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:30:59.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003888", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.230130Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.047179Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:30:59Z" } 2023-10-12T00:30:59.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003888', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.230130Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.047179Z'}}] 2023-10-12T00:30:59.423 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:30:59.424 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001068'} 2023-10-12T00:30:59.424 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004029 target=00000000000000000000:00000000000000003888, shard=3 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-10-12T00:31:04.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:04.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:04.426 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:04.487 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:04.487 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:04.532 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:31:04.532 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:31:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:31:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:31:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:31:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:31:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:31:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:31:04.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:31:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:31:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:31:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003888", 2023-10-12T00:31:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.230130Z" 2023-10-12T00:31:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:31:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:31:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:31:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:31:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:31:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:43.347442Z" 2023-10-12T00:31:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:31:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:31:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:31:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:31:04Z" 2023-10-12T00:31:04.546 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:31:04.547 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003888", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.230130Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:43.347442Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:31:04Z" } 2023-10-12T00:31:04.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003888', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.230130Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:43.347442Z'}}] 2023-10-12T00:31:04.548 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:31:04.548 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:31:04.549 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000004029 target=00000000000000000000:00000000000000003888 2023-10-12T00:31:09.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.551 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:09.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:09.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:09.657 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:31:09.658 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:31:09.658 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:31:09.659 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:31:09.659 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:31:09.659 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:31:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:31:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:31:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:31:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:31:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:31:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-12T00:31:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:44.418283Z" 2023-10-12T00:31:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:31:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-12T00:31:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:26:03.338811Z" 2023-10-12T00:31:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:31:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:31:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:31:09.673 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:31:09.673 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:31:09.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:31:09.674 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:43.347442Z" 2023-10-12T00:31:09.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:31:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:31:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:31:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:31:09Z" 2023-10-12T00:31:09.676 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:31:09.677 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:44.418283Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:26:03.338811Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:43.347442Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:31:09Z" } 2023-10-12T00:31:09.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:44.418283Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:26:03.338811Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:43.347442Z'}}] 2023-10-12T00:31:09.678 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:31:09.679 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001499', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:31:09.679 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:31:09.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.681 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:09.774 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:31:09.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:31:09.775 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:31:09.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.776 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-12T00:31:09.776 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:44.490949Z" 2023-10-12T00:31:09.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.777 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:31:09.777 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:31:09.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:09.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:09.778 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:31:09.778 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:43.375485Z" 2023-10-12T00:31:09.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:09.779 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:31:09.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:09.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:09.781 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:31:09.826 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:09.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:09.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.876 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:09.925 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:09.925 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:09.965 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:09.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:09.969 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.057 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.060 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.109 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.109 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.151 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.154 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.246 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.248 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.342 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:31:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:31:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:31:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-12T00:31:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:44.490949Z" 2023-10-12T00:31:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-12T00:31:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:26:23.327950Z" 2023-10-12T00:31:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:31:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:43.375485Z" 2023-10-12T00:31:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:10.346 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:31:10.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:31:10.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.444 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.531 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.534 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.626 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.629 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.719 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:31:10.721 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:31:10.770 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.810 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:31:10.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-10-12T00:31:10.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.816 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:31:10.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.883 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:31:10.883 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:31:10.883 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:31:10.883 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:31:10.884 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:31:10.884 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:31:10.884 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:31:10.884 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:31:10.884 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:31:10.885 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:31:10.885 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:31:10.885 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:31:10.885 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:31:10.886 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:31:10.886 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:31:10.886 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:31:10.886 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:31:10.887 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:31:10.887 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:31:10.887 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:31:10.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:10.888 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:31:10.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:10.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:31:10.888 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:31:10.888 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:31:10.889 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:31:10.889 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:31:10.889 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T00:31:10.889 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:31:10.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:10.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:10.890 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:31:10.890 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:31:10.890 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:31:10.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:10.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:10.891 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:10.891 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:31:10.892 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:31:10.892 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:31:10.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:31:10.894 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:31:10.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:31:10.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:31:10.986 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:31:11.013 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:31:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:31:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5, 2023-10-12T00:31:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:31:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:31:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:31:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:31:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:31:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.016 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:31:11.016 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:31:11.016 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:31:11.016 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:31:11.016 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:31:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:31:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:31:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:31:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:31:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:31:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:31:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:31:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:31:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:31:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:31:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:31:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:31:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:31:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:31:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:31:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:31:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:31:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:31:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:31:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:31:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:31:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:31:11.024 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:31:11.024 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:31:11.024 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:31:11.024 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": false, 2023-10-12T00:31:11.024 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:31:11.025 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:31:11.025 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:31:11.025 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:31:11.025 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:31:11.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.027 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:31:11.027 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:31:11.027 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:31:11.027 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:31:11.027 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.028 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.028 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:31:11.028 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:31:11.028 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:31:11.029 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:31:11.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.029 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:31:11.029 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.030 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:31:11.030 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:31:11.031 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:31:11.031 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:31:11.031 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.031 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:31:11.031 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.032 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:31:11.032 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:31:11.032 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:31:11.032 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:31:11.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.033 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:31:11.033 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:31:11.033 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.033 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:31:11.034 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.034 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:31:11.034 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:31:11.034 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:31:11.035 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:31:11.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:31:11.035 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:31:11.035 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:31:11.035 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:31:11.036 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:31:11.036 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:31:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:31:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:31:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:31:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:31:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:31:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:31:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:31:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:31:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:31:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:31:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:31:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:31:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:31:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:31:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:31:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:31:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:31:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:31:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:31:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:31:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:31:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:31:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:31:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:31:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:31:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:31:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:31:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:31:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:31:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:31:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:31:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:31:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:31:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:31:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:31:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:31:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:31:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:31:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:31:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:31:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:31:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:31:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:31:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:31:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:31:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:31:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:31:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:31:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:31:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:31:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:31:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:31:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:31:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:31:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:31:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:31:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:31:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:31:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:31:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:31:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:31:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:31:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:31:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:31:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:31:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:31:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:31:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:31:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:31:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:31:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:31:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:31:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:31:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:31:11.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:31:11.060 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:31:11.060 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-10-12T00:32:11.062 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-35 2023-10-12T00:32:11.063 DEBUG:boto:path=/wafmoa-35/ 2023-10-12T00:32:11.063 DEBUG:boto:auth_path=/wafmoa-35/ 2023-10-12T00:32:11.064 DEBUG:boto:Method: PUT 2023-10-12T00:32:11.064 DEBUG:boto:Path: /wafmoa-35/ 2023-10-12T00:32:11.065 DEBUG:boto:Data: 2023-10-12T00:32:11.065 DEBUG:boto:Headers: {} 2023-10-12T00:32:11.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:32:11.066 DEBUG:boto:Port: 8000 2023-10-12T00:32:11.066 DEBUG:boto:Params: {} 2023-10-12T00:32:11.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:32:11.068 DEBUG:boto:Token: None 2023-10-12T00:32:11.068 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:32:11 GMT /wafmoa-35/ 2023-10-12T00:32:11.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Npy0cgDsTG3FvabE1issMJS7blA= 2023-10-12T00:32:11.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:32:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Npy0cgDsTG3FvabE1issMJS7blA=', 'Content-Length': '0'} 2023-10-12T00:32:11.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1fbe32bd812f911-0065273e8b-153d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:32:11.092 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:32:11.092 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:32:11.092 DEBUG:boto:Method: PUT 2023-10-12T00:32:11.093 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:32:11.093 DEBUG:boto:Data: 2023-10-12T00:32:11.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:32:11.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:32:11.094 DEBUG:boto:Port: 8000 2023-10-12T00:32:11.095 DEBUG:boto:Params: {} 2023-10-12T00:32:11.095 DEBUG:boto:Token: None 2023-10-12T00:32:11.096 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 12 Oct 2023 00:32:11 GMT /wafmoa-35/testredirect 2023-10-12T00:32:11.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TNmjmaMuQaHOIyD1Tdd5UmF3yTs= 2023-10-12T00:32:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:32:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TNmjmaMuQaHOIyD1Tdd5UmF3yTs='} 2023-10-12T00:32:11.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051f92e8034d44bad-0065273e8b-153d-a1'), ('Date', 'Thu, 12 Oct 2023 00:32:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:32:11.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:11.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:11.107 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:11.195 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:32:11.207 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:32:11.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.208 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:11.208 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:32:11.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:11.209 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:32:11.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:32:11.210 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:32:11.089698Z" 2023-10-12T00:32:11.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:11.211 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:32:11.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.211 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:32:11.214 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:11.214 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:32:11.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:11.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:11.216 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:11.266 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:11.266 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:11.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:11.292+0000 7f0149a92780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:11.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:11.293+0000 7f0149a92780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:11.308 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:11.308 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:11.308 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:11.309 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:11.309 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:11.309 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:11.309 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:11.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.310 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:11.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.310 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:11.310 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:11.311 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:11.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:11.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:11.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:11.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:11.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:11.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:11.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:11.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:11.313 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:11.314 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:11.314 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:11.314 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:11.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:11.315 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:11.315 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:11.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.316 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:11.316 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:11.316 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:11.316 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:11.316 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:11.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:11.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:11.317 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:11.317 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:11.318 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:11.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:11.319 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:11.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:11.320 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:11.320 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:11.320 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:11.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:11.321 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:11.321 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:11.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:11.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:11.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.323 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:11.323 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:11.324 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:11.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:11.324 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:11Z" 2023-10-12T00:32:11.325 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:11.326 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:11Z" } 2023-10-12T00:32:11.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:11.327 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:11.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:11.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:16.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:16.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:16.330 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:16.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:16.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:16.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:16.416+0000 7fad5612c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:16.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:16.417+0000 7fad5612c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:16.430 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:16Z" 2023-10-12T00:32:16.444 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:16.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:16Z" } 2023-10-12T00:32:16.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:16.446 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:16.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:16.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:21.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:21.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:21.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:21.505 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:21.506 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:21.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:21.535+0000 7f0d84af1780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:21.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:21.536+0000 7f0d84af1780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:21.549 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:21.549 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:21.549 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:21.550 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:21.550 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:21.550 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:21.550 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:21.552 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:21.552 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:21.552 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:21.552 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:21.553 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:21.553 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:21.553 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:21.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:21.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:21.554 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:21.554 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:21.554 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:21.554 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:21.555 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:21.555 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:21.555 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:21.555 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:21.555 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:21.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:21.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:21.556 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:21.557 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:21.557 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:21.557 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:21.557 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:21.557 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:21.558 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:21.558 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:21.558 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:21.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:21.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:21.559 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:21.559 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:21.559 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:21.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:21.560 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:21.560 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:21.560 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:21.560 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:21.561 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:21.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:21.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:21.561 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:21.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.562 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:21.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:21.562 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:21.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:21.563 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:21Z" 2023-10-12T00:32:21.563 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:21.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:21Z" } 2023-10-12T00:32:21.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:21.565 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:21.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:21.566 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:26.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:26.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:26.568 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:26.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:26.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:26.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:26.658+0000 7f75aa43c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:26.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:26.659+0000 7f75aa43c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:26.672 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:26.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:26.682 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:26.682 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:26Z" 2023-10-12T00:32:26.686 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:26.687 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:26Z" } 2023-10-12T00:32:26.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:26.688 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:26.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:26.689 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:31.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:31.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:31.692 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:31.781+0000 7f98f4f28780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:31.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:31.783+0000 7f98f4f28780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:31.795 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:31.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:31.797 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:31.798 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:31.799 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:31.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:31.802 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:31.803 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:31.804 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:31.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:31.806 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:31.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:31.808 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:31.810 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:31.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.812 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:31.813 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:31Z" 2023-10-12T00:32:31.814 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:31.815 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:31Z" } 2023-10-12T00:32:31.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:31.817 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:31.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:31.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:36.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:36.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:36.821 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:36.878 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:36.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:36.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:36.909+0000 7ff13c5ed780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:36.910+0000 7ff13c5ed780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:36.923 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:36.923 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:36.923 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:36.923 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:36.924 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:36.924 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:36.924 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:36.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.925 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:36.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:36.925 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:36.925 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:36.926 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:36.926 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:36.926 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:36.926 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:36.927 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:36.927 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:36.927 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:36.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:36.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:36.938 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:36.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:36.938 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:36Z" 2023-10-12T00:32:36.938 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:36.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:36Z" } 2023-10-12T00:32:36.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:36.941 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:36.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:36.942 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:41.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:41.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:41.944 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:42.001 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:42.002 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:42.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:42.032+0000 7f75c6192780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:42.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:42.033+0000 7f75c6192780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:42.046 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:42.046 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:42.047 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:42.047 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:42.047 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:42.047 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:42.048 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:42.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.048 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:42.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:42.048 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:42Z" 2023-10-12T00:32:42.060 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:42.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:42Z" } 2023-10-12T00:32:42.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:42.062 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:42.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:42.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:47.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:47.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:47.066 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:47.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:47.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:47.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:47.154+0000 7fbbbcc0e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:47.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:47.155+0000 7fbbbcc0e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:47.168 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:47.168 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:47.168 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:47.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:47.169 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:47.169 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:47.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:47.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.170 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:47.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:47.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:47.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:47.171 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:47.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:47.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:47.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:47.172 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:47.172 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:47.172 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:47.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:47.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:47.173 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:47.173 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:47.173 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:47.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:47.174 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:47.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:47.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:47.174 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:47.175 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:47.175 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:47.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:47.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:47.176 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:47.176 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:47.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:47.176 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:47.176 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:47.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:47.177 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:47.177 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:47.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:47.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:47.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:47.178 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:47.178 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:47.180 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:47.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:47.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:47.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:47.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.181 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:47.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:47.182 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:47.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:47.182 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:47Z" 2023-10-12T00:32:47.183 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:47.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:47Z" } 2023-10-12T00:32:47.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:47.185 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:47.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:47.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:52.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:52.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:52.188 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:52.245 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:52.246 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:52.277+0000 7ff294cb9780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:52.277+0000 7ff294cb9780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:52Z" 2023-10-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:52.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:52Z" } 2023-10-12T00:32:52.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:52.308 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:52.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:52.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:32:57.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:57.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:32:57.312 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:32:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:32:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:32:57.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:57.399+0000 7f307adc4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:57.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:32:57.400+0000 7f307adc4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:32:57.413 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:32:57.413 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:32:57.414 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:32:57.414 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:32:57.414 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:32:57.414 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:32:57.415 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:57.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.415 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:32:57.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:57.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:32:57.416 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:57.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:57.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:32:57.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:57.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:32:57.417 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:57.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:32:57.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:57.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:57.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:57.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:32:57.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:57.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:57.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:32:57.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:57.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:57.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:57.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:32:57.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:57.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:57.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:57.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:32:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:32:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:32:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:32:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:32:57.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:32:57.424 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:32:57.424 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:32:57.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:32:57.424 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:32:57.425 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:32:57.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:57.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:32:57.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:32:57.427 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:32:57Z" 2023-10-12T00:32:57.427 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:32:57.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:32:57Z" } 2023-10-12T00:32:57.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:32:57.429 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:57.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:32:57.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:33:02.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:02.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:02.432 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:33:02.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:33:02.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:33:02.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:02.519+0000 7f4c4181b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:02.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:02.520+0000 7f4c4181b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:02.532 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:33:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:33:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:33:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:33:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:33:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:33:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:33:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:33:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:33:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:33:02.536 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:02.536 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:33:02.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:02.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:02.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:02.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:33:02.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:02.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:02.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:33:02.538 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:02.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:02.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:02.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:33:02.539 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:02.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:02.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:02.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:33:02.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:02.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:02.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:33:02.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:02.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:02.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:02.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:33:02.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:02.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:02.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:02.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:33:02.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:02.542 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:33:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:33:02.544 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:02.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:02.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:02.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:02.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.545 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:33:02.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:33:02.545 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:33:02.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:02.546 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:33:02Z" 2023-10-12T00:33:02.546 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:33:02.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:33:02Z" } 2023-10-12T00:33:02.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:33:02.548 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:02.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:02.549 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:33:07.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:07.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:07.552 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:33:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:33:07.609 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:33:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:07.640+0000 7fbfba379780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:07.641+0000 7fbfba379780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:07.653 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:33:07.653 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:33:07.653 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:33:07.653 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:33:07.654 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:33:07.654 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:33:07.654 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:07.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.655 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:33:07.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:07.655 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:33:07.655 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:07.655 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:07.656 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:33:07.656 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:07.656 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:33:07.656 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:33:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:07.658 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:33:07.658 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:07.658 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:07.658 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:33:07.658 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:07.659 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:07.659 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:07.659 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:33:07.659 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:07.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:07.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:07.660 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:33:07.660 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:07.660 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:07.661 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070637.173811_243.1", 2023-10-12T00:33:07.661 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:07.661 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:07.661 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:07.662 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:37.173811Z", 2023-10-12T00:33:07.662 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:07.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:07.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:07.663 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:33:07.663 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:07.663 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:07.663 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:33:07.663 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:07.664 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:07.664 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:07.664 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:33:07.664 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:07.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:07.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:07.665 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:07.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.665 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:33:07.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:33:07.666 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:33:07.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:07.666 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:33:07Z" 2023-10-12T00:33:07.666 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:33:07.667 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070637.173811_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:37.173811Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:33:07Z" } 2023-10-12T00:33:07.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070637.173811_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:37.173811Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:33:07.669 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:07.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070637.173811_243.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:07.669 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070731.089698_270.1 target=1_1697070637.173811_243.1 2023-10-12T00:33:12.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:12.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:33:12.671 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:33:12.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:33:12.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:33:12.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:12.759+0000 7f7ce55be780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:12.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:33:12.760+0000 7f7ce55be780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:33:12.772 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:33:12.773 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:33:12.773 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:33:12.773 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:33:12.773 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:33:12.773 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:33:12.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:12.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.774 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:33:12.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:12.775 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:33:12.775 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:12.775 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:12.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:33:12.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:12.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:33:12.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:12.776 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:33:12.776 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:12.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:12.777 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:33:12.777 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:12.778 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:12.778 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:33:12.778 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:12.778 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:12.778 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:12.779 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:33:12.779 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:12.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:12.780 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:33:12.780 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:12.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:12.780 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:33:12.781 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:12.781 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:12.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:12.781 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:33:12.781 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:12.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:12.782 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:33:12.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:12.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:33:12.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:33:12.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:33:12.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:33:12.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:33:12.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:33:12.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:12.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.785 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:12.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.785 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:33:12.785 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:33:12.786 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:33:12.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.786 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:33:12Z" 2023-10-12T00:33:12.786 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:33:12.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:33:12Z" } 2023-10-12T00:33:12.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:33:12.788 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:12.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:33:12.789 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:33:12.790 DEBUG:boto:path=/wafmoa-35/ 2023-10-12T00:33:12.790 DEBUG:boto:auth_path=/wafmoa-35/ 2023-10-12T00:33:12.791 DEBUG:boto:Method: HEAD 2023-10-12T00:33:12.791 DEBUG:boto:Path: /wafmoa-35/ 2023-10-12T00:33:12.791 DEBUG:boto:Data: 2023-10-12T00:33:12.792 DEBUG:boto:Headers: {} 2023-10-12T00:33:12.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:33:12.793 DEBUG:boto:Port: 8001 2023-10-12T00:33:12.793 DEBUG:boto:Params: {} 2023-10-12T00:33:12.794 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:33:12.794 DEBUG:boto:Token: None 2023-10-12T00:33:12.795 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:33:12 GMT /wafmoa-35/ 2023-10-12T00:33:12.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IgV0xu/htMkQ2eZyiM9WSRE6im4= 2023-10-12T00:33:12.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:33:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IgV0xu/htMkQ2eZyiM9WSRE6im4=', 'Content-Length': '0'} 2023-10-12T00:33:12.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034e577e436af4f1b-0065273ec8-1539-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:33:12.802 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:33:12.803 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:33:12.803 DEBUG:boto:Method: HEAD 2023-10-12T00:33:12.803 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:33:12.804 DEBUG:boto:Data: 2023-10-12T00:33:12.804 DEBUG:boto:Headers: {} 2023-10-12T00:33:12.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:33:12.805 DEBUG:boto:Port: 8001 2023-10-12T00:33:12.805 DEBUG:boto:Params: {} 2023-10-12T00:33:12.806 DEBUG:boto:Token: None 2023-10-12T00:33:12.806 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:33:12 GMT /wafmoa-35/testredirect 2023-10-12T00:33:12.807 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TBWAOvlpwINZ5lhlMoUGVKy4qO0= 2023-10-12T00:33:12.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:33:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TBWAOvlpwINZ5lhlMoUGVKy4qO0=', 'Content-Length': '0'} 2023-10-12T00:33:12.809 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000324859de8ee78cb0-0065273ec8-1539-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:33:12.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:33:12.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:33:12.810 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:33:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:33:12.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:33:12.880 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:33:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:33:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:33:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:33:12.881 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:33:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:33:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:33:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:33:12.882 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:33:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:33:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:33:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:33:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:33:12.883 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:33:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:33:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:33:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:12.885 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:33:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:33:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:33:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:33:12.886 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:33:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T00:33:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:33:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:12.887 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:33:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:33:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:33:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:12.888 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:33:12.889 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:33:12.889 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:33:12.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:33:12.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:33:12.891 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:33:12.940 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:33:12.941 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:33:12.980 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:33:13.003 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:33:13.004 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:33:13.004 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6, 2023-10-12T00:33:13.004 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:33:13.004 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:33:13.005 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:33:13.005 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:33:13.005 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:33:13.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.006 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:33:13.006 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:33:13.006 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:33:13.006 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:33:13.006 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.007 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:33:13.007 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.007 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:33:13.007 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:33:13.008 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:33:13.008 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:33:13.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.008 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:33:13.008 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:33:13.009 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.009 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:33:13.009 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.009 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:33:13.009 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:33:13.010 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:33:13.010 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:33:13.010 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:33:13.010 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:33:13.010 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:33:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:33:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:33:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:33:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:33:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:33:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:33:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:33:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:33:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:33:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:33:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:33:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": false, 2023-10-12T00:33:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:33:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:33:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:33:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:33:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:33:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:33:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:33:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:33:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:33:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:33:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:33:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:33:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:33:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:33:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:33:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:33:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:33:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:33:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:33:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:33:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:33:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:33:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:33:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:33:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:33:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:33:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:33:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:33:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:33:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:33:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:33:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:33:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:33:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:33:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:33:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:33:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:33:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:33:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:33:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:33:13.030 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.030 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:33:13.030 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:33:13.030 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:33:13.030 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:33:13.031 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:33:13.031 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:33:13.031 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:33:13.031 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:33:13.032 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:33:13.032 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:33:13.032 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.033 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.033 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:33:13.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.033 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:33:13.033 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:33:13.034 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:33:13.034 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:33:13.034 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.035 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.035 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:33:13.035 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:33:13.035 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:33:13.035 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:33:13.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.036 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:33:13.036 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:33:13.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.037 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:33:13.037 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:33:13.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.037 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:33:13.038 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:33:13.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.038 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:33:13.038 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:33:13.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:33:13.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:33:13.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:33:13.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:33:13.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:33:13.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:33:13.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.043 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:33:13.043 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:33:13.043 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:33:13.044 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:33:13.044 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:33:13.044 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:33:13.044 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:33:13.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:33:13.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:33:13.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.045 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:33:13.045 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:33:13.046 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:33:13.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:33:13.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:33:13.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:33:13.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.047 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:33:13.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:33:13.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:33:13.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:33:13.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:33:13.048 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:33:13.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.048 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:33:13.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:33:13.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:33:13.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:33:13.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:33:13.049 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:33:13.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.049 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:33:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:33:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:33:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:33:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:33:13.050 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:33:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:33:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:33:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:33:13.051 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:33:13.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:33:13.052 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:33:13.052 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T00:34:13.054 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.055 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.055 DEBUG:boto:Method: HEAD 2023-10-12T00:34:13.056 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.056 DEBUG:boto:Data: 2023-10-12T00:34:13.057 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.057 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.058 DEBUG:boto:Params: {} 2023-10-12T00:34:13.058 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.059 DEBUG:boto:Token: None 2023-10-12T00:34:13.060 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eogj+ob/YOYJuL4sxeSDHUM9u3s= 2023-10-12T00:34:13.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eogj+ob/YOYJuL4sxeSDHUM9u3s=', 'Content-Length': '0'} 2023-10-12T00:34:13.066 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx000007cf19c72c0e7daa5-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.067 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-10-12T00:34:13.068 DEBUG:boto:Token: None 2023-10-12T00:34:13.068 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eogj+ob/YOYJuL4sxeSDHUM9u3s= 2023-10-12T00:34:13.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eogj+ob/YOYJuL4sxeSDHUM9u3s=', 'Content-Length': '0'} 2023-10-12T00:34:13.074 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:32:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000002dedb156687538e9-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.075 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.076 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.076 DEBUG:boto:Method: GET 2023-10-12T00:34:13.076 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.077 DEBUG:boto:Data: 2023-10-12T00:34:13.077 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.078 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.079 DEBUG:boto:Params: {} 2023-10-12T00:34:13.079 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.080 DEBUG:boto:Token: None 2023-10-12T00:34:13.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.085 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx00000550050043b0acc96-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.085 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.086 DEBUG:boto:Token: None 2023-10-12T00:34:13.086 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.089 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:32:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000c52118296af1518a-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.090 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.090 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.091 DEBUG:boto:Method: PUT 2023-10-12T00:34:13.091 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.092 DEBUG:boto:Data: 2023-10-12T00:34:13.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:34:13.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:34:13.093 DEBUG:boto:Port: 8000 2023-10-12T00:34:13.093 DEBUG:boto:Params: {} 2023-10-12T00:34:13.094 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:34:13.094 DEBUG:boto:Token: None 2023-10-12T00:34:13.095 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.095 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kQV7cJ/NQFyOrcdDTn61kKxUVKc= 2023-10-12T00:34:13.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kQV7cJ/NQFyOrcdDTn61kKxUVKc='} 2023-10-12T00:34:13.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c929ad7c79fb695-0065273f05-15c1-a1'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.107 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.108 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.108 DEBUG:boto:Method: GET 2023-10-12T00:34:13.109 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.109 DEBUG:boto:Data: 2023-10-12T00:34:13.110 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.110 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.111 DEBUG:boto:Params: {} 2023-10-12T00:34:13.111 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.112 DEBUG:boto:Token: None 2023-10-12T00:34:13.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.117 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx00000e32199055907858e-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.117 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.118 DEBUG:boto:Token: None 2023-10-12T00:34:13.118 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.121 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000a38422aae1f1f9b2-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.122 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.122 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.123 DEBUG:boto:Method: PUT 2023-10-12T00:34:13.123 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.124 DEBUG:boto:Data: 2023-10-12T00:34:13.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:34:13.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:34:13.125 DEBUG:boto:Port: 8000 2023-10-12T00:34:13.125 DEBUG:boto:Params: {} 2023-10-12T00:34:13.126 DEBUG:boto:Token: None 2023-10-12T00:34:13.126 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HdIeh1/Qulmok3aTarPJftByN9o= 2023-10-12T00:34:13.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HdIeh1/Qulmok3aTarPJftByN9o='} 2023-10-12T00:34:13.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cda57f2721ed26e-0065273f05-15c1-a1'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.134 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.134 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.135 DEBUG:boto:Method: GET 2023-10-12T00:34:13.135 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.136 DEBUG:boto:Data: 2023-10-12T00:34:13.136 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.137 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.137 DEBUG:boto:Params: {} 2023-10-12T00:34:13.138 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.138 DEBUG:boto:Token: None 2023-10-12T00:34:13.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.143 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx00000e6ce49b9475eb95b-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.144 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.145 DEBUG:boto:Token: None 2023-10-12T00:34:13.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.148 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000a908435616db336e-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.149 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.150 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.150 DEBUG:boto:Method: PUT 2023-10-12T00:34:13.150 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.151 DEBUG:boto:Data: 2023-10-12T00:34:13.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:34:13.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:34:13.154 DEBUG:boto:Port: 8000 2023-10-12T00:34:13.155 DEBUG:boto:Params: {} 2023-10-12T00:34:13.155 DEBUG:boto:Token: None 2023-10-12T00:34:13.156 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:czedwPJRGY70CjZFDdQVGnVmjlI= 2023-10-12T00:34:13.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:czedwPJRGY70CjZFDdQVGnVmjlI='} 2023-10-12T00:34:13.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e86ca1e2c6417383-0065273f05-15c1-a1'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.163 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.164 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.164 DEBUG:boto:Method: GET 2023-10-12T00:34:13.164 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.165 DEBUG:boto:Data: 2023-10-12T00:34:13.165 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.166 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.167 DEBUG:boto:Params: {} 2023-10-12T00:34:13.167 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.167 DEBUG:boto:Token: None 2023-10-12T00:34:13.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.172 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx00000be807d12d127b116-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.173 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.173 DEBUG:boto:Token: None 2023-10-12T00:34:13.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.177 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx000005c0e3a91e4170939-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.178 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.178 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.179 DEBUG:boto:Method: PUT 2023-10-12T00:34:13.179 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.180 DEBUG:boto:Data: 2023-10-12T00:34:13.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:34:13.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:34:13.181 DEBUG:boto:Port: 8000 2023-10-12T00:34:13.181 DEBUG:boto:Params: {} 2023-10-12T00:34:13.182 DEBUG:boto:Token: None 2023-10-12T00:34:13.182 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qbWzXCY0k5Jsxt8z12U1Z0E5hXA= 2023-10-12T00:34:13.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qbWzXCY0k5Jsxt8z12U1Z0E5hXA='} 2023-10-12T00:34:13.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a657f7b819337b00-0065273f05-15c1-a1'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.228 DEBUG:boto:path=/wafmoa-35/testredirect 2023-10-12T00:34:13.228 DEBUG:boto:auth_path=/wafmoa-35/testredirect 2023-10-12T00:34:13.229 DEBUG:boto:Method: GET 2023-10-12T00:34:13.229 DEBUG:boto:Path: /wafmoa-35/testredirect 2023-10-12T00:34:13.230 DEBUG:boto:Data: 2023-10-12T00:34:13.231 DEBUG:boto:Headers: {} 2023-10-12T00:34:13.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:34:13.232 DEBUG:boto:Port: 8001 2023-10-12T00:34:13.233 DEBUG:boto:Params: {} 2023-10-12T00:34:13.233 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:34:13.234 DEBUG:boto:Token: None 2023-10-12T00:34:13.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.238 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/wafmoa-35/testredirect?'), ('x-amz-request-id', 'tx000001b1ad9b31329e431-0065273f05-15c5-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.239 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/wafmoa-35/testredirect 2023-10-12T00:34:13.239 DEBUG:boto:Token: None 2023-10-12T00:34:13.240 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:34:13 GMT /wafmoa-35/testredirect 2023-10-12T00:34:13.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA= 2023-10-12T00:34:13.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:34:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ph4Kx2M3w7Dp4tuaMRiRedHBUyA=', 'Content-Length': '0'} 2023-10-12T00:34:13.243 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000b369b19ef2747a2e-0065273f05-15c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:34:13.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:34:13.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:34:13.245 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:34:13.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:34:13.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:34:13.323 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:34:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:34:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:34:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:34:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:34:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:34:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:34:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:34:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:34:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:34:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:34:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:34:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:34:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:34:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:34:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:34:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:34:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:34:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:34:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:34:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:34:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:34:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:34:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:34:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:34:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:34:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T00:34:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:34:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:34:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:34:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:34:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:34:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:34:13.333 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:34:13.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:34:13.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:34:13.336 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:34:13.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:34:13.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:34:13.427 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:34:13.449 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:34:13.450 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:34:13.450 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7, 2023-10-12T00:34:13.450 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:34:13.450 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:34:13.451 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:34:13.451 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:34:13.451 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:34:13.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.452 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:34:13.452 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:34:13.452 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:34:13.452 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:34:13.452 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.453 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:34:13.453 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.453 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:34:13.453 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:34:13.453 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:34:13.454 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:34:13.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.454 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:34:13.454 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:34:13.454 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.455 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:34:13.455 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.455 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:34:13.455 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:34:13.455 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:34:13.456 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:34:13.456 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:34:13.456 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:34:13.457 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:34:13.457 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:34:13.457 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:34:13.457 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:34:13.457 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.459 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:34:13.459 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:34:13.459 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.459 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:34:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:34:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:34:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:34:13.460 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:34:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:34:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:34:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:34:13.461 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:34:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:34:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:34:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:34:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:34:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:34:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:34:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:34:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:34:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:34:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:34:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:34:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:34:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:34:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:34:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:34:13.467 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:34:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:34:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.468 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:34:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:34:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:34:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:34:13.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:34:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:34:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:34:13.470 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:34:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:34:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:34:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:34:13.471 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:34:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:34:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:34:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:34:13.472 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:34:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:34:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:34:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:34:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:34:13.475 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:34:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:34:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:34:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:34:13.476 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:34:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:34:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:34:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:34:13.477 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:34:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:34:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:34:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:34:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:34:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:34:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:34:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:34:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:34:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:34:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:34:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:34:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:34:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:34:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:34:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:34:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:34:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:34:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:34:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:34:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:34:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:34:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:34:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:34:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:34:13.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:34:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:34:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:34:13.487 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:34:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:34:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:34:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:34:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:34:13.488 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:34:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:34:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:34:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:34:13.489 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:34:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:34:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:34:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:34:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:34:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:34:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:34:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:34:13.491 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:34:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:34:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:34:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:34:13.492 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:34:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:34:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:34:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:34:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.493 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:34:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:34:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:34:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:34:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:34:13.494 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:34:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:34:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:34:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:34:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:34:13.495 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:34:13.496 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:34:13.496 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-10-12T00:35:13.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:35:13.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:35:13.500 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:35:13.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:35:13.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:35:13.581 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:35:13.582 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:35:13.582 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:35:13.583 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-12T00:35:13.583 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-10-12T00:35:13.583 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-12T00:35:13.584 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-12T00:35:13.584 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-10-12T00:35:13.585 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-12T00:35:13.585 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-12T00:35:13.585 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-12T00:35:13.586 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-12T00:35:13.586 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-12T00:35:13.587 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-12T00:35:13.587 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-12T00:35:13.588 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-12T00:35:13.588 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-10-12T00:35:13.588 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:35:13.589 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:35:13.589 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:35:13.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.590 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:35:13.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:35:13.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:35:13.592 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-12T00:35:13.592 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:35:13.593 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:35:13.593 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-12T00:35:13.593 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-10-12T00:35:13.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.595 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-12T00:35:13.595 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:35:13.595 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:35:13.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.597 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:35:13.597 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-12T00:35:13.598 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:35:13.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:35:13.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:35:13.602 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:35:13.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:35:13.653 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:35:13.692 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:35:13.715 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:35:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:35:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8, 2023-10-12T00:35:13.716 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:35:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:35:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:35:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:35:13.717 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:35:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:35:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:35:13.718 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:35:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:35:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:35:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.719 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:35:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:35:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:35:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:35:13.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:35:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:35:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:35:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:35:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:35:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:35:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:35:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:35:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:35:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:35:13.723 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:35:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:35:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:35:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.724 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:35:13.725 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:35:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:35:13.726 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:35:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:35:13.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:35:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:35:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:35:13.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:35:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:35:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:35:13.729 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:35:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:35:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:35:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:35:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:35:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:35:13.732 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:35:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:35:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:35:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:35:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:35:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.734 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:35:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.735 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:35:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:35:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:35:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:35:13.736 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:35:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.737 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:35:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:35:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:35:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:35:13.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:35:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:35:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:35:13.739 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:35:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:35:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:35:13.740 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:35:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:35:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:35:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:35:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:35:13.741 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:35:13.742 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:35:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:35:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:35:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:35:13.744 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:35:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:35:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:35:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:35:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:35:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:35:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:35:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:35:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:35:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:35:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:35:13.747 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:35:13.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:35:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:35:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:35:13.749 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:35:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:35:13.750 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:35:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:35:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:35:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:35:13.751 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:35:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:35:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:35:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:35:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:35:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:35:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:35:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:35:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:35:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:35:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:35:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:35:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:35:13.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:35:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:35:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:35:13.757 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:35:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:35:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:35:13.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:35:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:35:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:35:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.759 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:35:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:35:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:35:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:35:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:35:13.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:35:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:35:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:35:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:35:13.761 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:35:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:35:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:35:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.762 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:35:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:35:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:35:13.763 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:35:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:35:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:35:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:35:13.764 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:35:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:35:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:35:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:35:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:35:13.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:35:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:35:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:35:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:35:13.766 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:35:13.767 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:35:13.767 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-10-12T00:36:13.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-10-12T00:36:13.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:13.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:13.772 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:13.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:13.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:13.861 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:36:13.873 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:36:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:36:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:36:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:32:11.089698Z" 2023-10-12T00:36:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:36:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.877 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:36:13.878 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:13.879 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:36:13.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:13.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:13.880 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:13.931 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:13.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:13.958+0000 7f4849788780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:13.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:13.959+0000 7f4849788780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:13.970 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:13.970 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:13.971 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:13.971 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:13.971 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:13.971 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:13.971 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:13.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.972 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:13.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.972 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:13.973 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:13.973 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:13.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:13.973 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:13.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:13.974 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:13.974 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:13.974 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:13.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:13.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:13.976 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:13.976 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:13.976 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:13.976 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:13.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:13.977 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:13.977 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:13.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.978 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:13.978 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:13.978 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:13.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:13.979 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:13.979 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:13.979 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:13.979 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:13.979 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:13.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:13.980 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:13.980 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:13.981 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:13.981 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:13.981 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:13.981 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:13.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:13.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:13.982 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:13.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:13.983 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:13.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.983 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:13.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:13.984 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:13.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:13.984 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:13Z" 2023-10-12T00:36:13.984 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:13.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:13Z" } 2023-10-12T00:36:13.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:13.987 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:13.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:13.987 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:36:13.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:13.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'TECD6UO8UM1CNM5T', '--secret', 'TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:13.989 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key TECD6UO8UM1CNM5T --secret TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:14.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:14.040 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=a9c32acc-d391-49fc-954e-a55395784dbf 2023-10-12T00:36:14.067 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:14.067 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "8ec302bd-71c3-4c74-89d6-195cd6bb72fa", 2023-10-12T00:36:14.068 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-10-12T00:36:14.068 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-12T00:36:14.068 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "remove.rgw.control", 2023-10-12T00:36:14.068 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-12T00:36:14.069 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-12T00:36:14.069 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "remove.rgw.log", 2023-10-12T00:36:14.069 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-12T00:36:14.069 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-12T00:36:14.070 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-12T00:36:14.070 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-12T00:36:14.070 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-12T00:36:14.070 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-12T00:36:14.071 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-12T00:36:14.071 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-12T00:36:14.071 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "remove.rgw.otp", 2023-10-12T00:36:14.071 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:36:14.071 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "TECD6UO8UM1CNM5T", 2023-10-12T00:36:14.072 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "TMc8iaFD9uhx9d7NMDrgyj18GIvfGmCb" 2023-10-12T00:36:14.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:14.072 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:36:14.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:14.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:36:14.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:14.073 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-12T00:36:14.073 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:36:14.073 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:36:14.073 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-12T00:36:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-12T00:36:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:36:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:36:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-12T00:36:14.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:14.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:14.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:14.078 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:14.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:14.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:14.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:14.139+0000 7fee08a23780 0 period (f8a42833-a8a4-4b21-a670-8605a7b80b21 does not have zone 8ec302bd-71c3-4c74-89d6-195cd6bb72fa configured 2023-10-12T00:36:20.411 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 677f5a1d-3ab4-4b5f-af15-9890c2c79648 2023-10-12T00:36:20.438 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:20.438 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:20.439 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9, 2023-10-12T00:36:20.439 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:36:20.439 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:36:20.439 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:36:20.440 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:20.440 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:36:20.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.440 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:36:20.441 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:36:20.441 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:20.441 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:36:20.441 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.441 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.442 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.442 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:20.442 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:20.442 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.443 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:20.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.443 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.443 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:36:20.443 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.444 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.444 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.444 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.444 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.444 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.445 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.445 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.445 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.445 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.445 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.446 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.446 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.446 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.446 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.447 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "8ec302bd-71c3-4c74-89d6-195cd6bb72fa", 2023-10-12T00:36:20.447 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-10-12T00:36:20.448 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-10-12T00:36:20.448 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.448 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.449 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.449 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.449 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.450 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.450 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.450 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.451 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.451 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.451 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.453 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:20.453 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:36:20.454 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.454 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:36:20.454 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.455 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.455 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.455 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.456 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.456 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.456 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.457 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.457 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.457 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.458 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.458 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.459 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.459 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:20.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.460 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:20.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:20.461 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:20.461 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:20.461 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.462 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.462 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:20.462 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:36:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:36:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:36:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:36:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:36:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.477 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:20.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.477 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:20.477 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:20.478 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:20.478 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:20.479 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.480 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.480 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:20.480 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:20.481 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:20.481 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:20.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.482 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:20.482 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.482 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.483 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.483 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:36:20.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:20.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:36:20.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:36:20.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.486 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:20.486 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:36:20.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.487 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "8ec302bd-71c3-4c74-89d6-195cd6bb72fa", 2023-10-12T00:36:20.487 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1916520367 2023-10-12T00:36:20.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.488 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:20.489 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:36:20.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.489 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.490 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:36:20.490 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.491 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:36:20.491 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:36:20.492 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:20.492 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:20.492 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:20.493 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:20.493 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:20.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.494 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:36:20.494 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:20.494 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:20.495 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:20.495 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:20.495 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:20.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.495 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:36:20.496 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:20.496 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:20.496 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:20.496 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:36:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:20.497 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:20.498 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:20.498 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:20.498 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:20.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.499 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:36:20.499 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:20.499 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:20.499 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:20.499 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:20.500 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:20.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.500 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:20.502 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:36:20.502 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:20.502 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:20.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.504 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:20.568 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:20.569 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:20.610 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:36:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:36:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:36:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:36:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.615 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.615 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.615 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.615 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.615 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.616 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.616 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.616 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.616 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:36:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:36:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:20.622 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:20.622 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:20.622 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:20.622 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:20.623 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:20.623 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:20.623 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:20.623 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:20.624 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:20.624 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:20.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:20.624 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:20.624 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:20.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:20.625 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:20.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.627 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:20.692 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:20.692 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:20.724 INFO:teuthology.orchestra.run.smithi112.stderr:zone name remove not found in zonegroup a 2023-10-12T00:36:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-12T00:36:20.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.732 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:20.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:20.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:20.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:20.861 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:20.935 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:20.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:20.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:20.956+0000 7f71dcb37780 0 WARNING: can't generate connection for zone 8ec302bd-71c3-4c74-89d6-195cd6bb72fa id remove: no endpoints defined 2023-10-12T00:36:21.021 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:21.022 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:21.022 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10, 2023-10-12T00:36:21.022 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:36:21.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:36:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:36:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:36:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:36:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:36:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:36:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:36:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:36:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:36:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.034 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.034 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.034 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.034 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.035 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.035 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.035 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:21.036 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:21.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.036 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.037 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.037 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.037 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.038 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.039 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.039 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.039 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:36:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:36:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:21.041 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.041 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.041 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.041 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.042 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.042 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.043 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:21.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.043 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:21.044 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:36:21.044 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.044 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:36:21.044 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.044 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.045 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.045 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.046 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.046 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.046 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.046 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.047 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.047 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.047 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.048 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.048 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.049 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.049 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:36:21.050 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.050 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.050 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.050 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.051 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.051 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.051 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.052 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.052 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.052 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.052 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.052 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.053 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.053 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.054 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.054 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.055 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.055 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.056 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:21.056 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:21.056 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.057 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.058 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.058 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.058 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.059 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.059 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.060 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.060 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.061 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.061 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:36:21.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:21.063 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:36:21.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.064 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.064 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:36:21.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.065 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:36:21.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.066 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:21.066 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:36:21.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.067 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:36:21.067 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.067 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:36:21.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:36:21.068 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:21.068 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:21.068 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:21.068 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:21.069 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:21.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.069 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:36:21.069 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:21.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.071 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:36:21.071 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.071 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.071 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.072 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.072 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.072 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:36:21.072 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.074 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:36:21.074 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.074 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.075 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.075 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.075 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:36:21.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:21.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:21.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-10-12T00:36:21.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.079 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:21.163 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:21.164 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "87750559-0cdf-4dc5-b405-7ea7ac1f5394", 2023-10-12T00:36:21.164 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove_zg", 2023-10-12T00:36:21.164 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "remove_zg", 2023-10-12T00:36:21.165 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:21.165 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-10-12T00:36:21.165 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.166 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.166 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "", 2023-10-12T00:36:21.166 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [], 2023-10-12T00:36:21.167 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.167 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.167 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.168 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [] 2023-10-12T00:36:21.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.168 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.169 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.169 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.170 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.170 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.171 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.171 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:21.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.173 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.231 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.231 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:21.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:21.338+0000 7f7fa6fa0780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-10-12T00:36:21.349 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:21.350 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "98b28111-92c0-44f4-b364-de08eaff3628", 2023-10-12T00:36:21.350 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-10-12T00:36:21.351 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-12T00:36:21.351 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "remove.rgw.control", 2023-10-12T00:36:21.352 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-12T00:36:21.352 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-12T00:36:21.352 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "remove.rgw.log", 2023-10-12T00:36:21.353 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-12T00:36:21.353 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-12T00:36:21.353 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-12T00:36:21.354 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-12T00:36:21.354 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-12T00:36:21.354 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-12T00:36:21.354 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-12T00:36:21.355 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-12T00:36:21.355 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "remove.rgw.otp", 2023-10-12T00:36:21.355 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-10-12T00:36:21.355 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-10-12T00:36:21.356 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-10-12T00:36:21.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.356 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-10-12T00:36:21.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-10-12T00:36:21.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:21.358 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-12T00:36:21.358 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-10-12T00:36:21.359 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-10-12T00:36:21.359 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-12T00:36:21.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.360 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-12T00:36:21.361 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-10-12T00:36:21.361 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-10-12T00:36:21.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-12T00:36:21.363 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:21.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.366 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:21.574 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:21.575 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:21.575 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11, 2023-10-12T00:36:21.575 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:36:21.576 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:36:21.576 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:36:21.576 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:21.576 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:36:21.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.577 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "87750559-0cdf-4dc5-b405-7ea7ac1f5394", 2023-10-12T00:36:21.577 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove_zg", 2023-10-12T00:36:21.577 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "remove_zg", 2023-10-12T00:36:21.578 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:21.578 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-10-12T00:36:21.578 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.578 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.578 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "98b28111-92c0-44f4-b364-de08eaff3628", 2023-10-12T00:36:21.579 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:21.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.579 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "98b28111-92c0-44f4-b364-de08eaff3628", 2023-10-12T00:36:21.579 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-10-12T00:36:21.580 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-10-12T00:36:21.580 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.580 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": false, 2023-10-12T00:36:21.581 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.581 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.581 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.581 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.582 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.582 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.582 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.582 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.582 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.583 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.583 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.583 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.584 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.584 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.584 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:21.584 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:21.585 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.585 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.585 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.586 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.586 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.586 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.586 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.587 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.587 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.588 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:36:21.589 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:36:21.589 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:21.589 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:36:21.590 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.590 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.590 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.591 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.591 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.591 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.592 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:21.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.592 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.593 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:36:21.593 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.593 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.594 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.594 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.594 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.595 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.595 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.595 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.595 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.596 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.596 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.596 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.596 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.596 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.597 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:21.597 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:36:21.598 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.598 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:36:21.598 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.598 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.599 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.599 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.599 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.599 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.600 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.600 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.600 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.601 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.601 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.601 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.601 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.603 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.603 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:21.603 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:21.603 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.604 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.604 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.604 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.604 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:36:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:36:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:36:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:36:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:36:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:21.621 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:21.621 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.622 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.623 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.624 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:21.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.624 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:21.625 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:21.625 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:21.626 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:21.626 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.627 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.627 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:21.627 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.628 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:21.628 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:21.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.629 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:21.629 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:21.630 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.631 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:21.631 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:36:21.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.632 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:21.632 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:36:21.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.632 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.633 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:36:21.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.633 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:21.634 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:36:21.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.634 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "98b28111-92c0-44f4-b364-de08eaff3628", 2023-10-12T00:36:21.635 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2515680698 2023-10-12T00:36:21.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:21.636 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:21.636 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:36:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:36:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:36:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:36:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:36:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:36:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:36:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:36:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:36:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:21.650 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:21.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.651 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.710 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:21.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:21.746+0000 7f320b517780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-10-12T00:36:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:21.748+0000 7f320b517780 0 Removed zone from zonegroup remove_zg 2023-10-12T00:36:21.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.761 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:21.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:21.859+0000 7fbc77df0780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-10-12T00:36:21.865 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-12T00:36:21.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:21.867 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:21.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:21.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:22.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:22.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:36:22.004 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:36:22.080 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:22.080 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:22.222 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:22.222 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:22.222 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12, 2023-10-12T00:36:22.222 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T00:36:22.223 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T00:36:22.223 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T00:36:22.223 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:22.223 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T00:36:22.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.224 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T00:36:22.224 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T00:36:22.224 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:22.224 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T00:36:22.225 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.225 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:22.225 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.225 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:22.225 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:22.226 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:22.226 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:22.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.226 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:22.227 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T00:36:22.227 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.227 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T00:36:22.227 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.227 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:22.228 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:22.228 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:22.228 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:22.228 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:22.228 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:22.229 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:22.229 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:22.229 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:22.229 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:22.229 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.230 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.230 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:22.230 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T00:36:22.231 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.231 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T00:36:22.231 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.231 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:22.231 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:22.232 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:22.232 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:22.232 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:22.232 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:22.233 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:22.233 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:22.233 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:22.233 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:22.233 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.234 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.234 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.234 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:22.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.235 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:22.235 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:22.235 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:22.235 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:22.235 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.236 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.236 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:22.236 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:22.237 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:22.237 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:22.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.237 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:22.237 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.238 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T00:36:22.238 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T00:36:22.239 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T00:36:22.239 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T00:36:22.239 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.239 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:22.240 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.240 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T00:36:22.240 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T00:36:22.240 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:22.240 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T00:36:22.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.241 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:22.241 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T00:36:22.241 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.241 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T00:36:22.242 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.242 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:22.242 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:22.242 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:22.243 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:22.243 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:22.243 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:22.243 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:22.243 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:22.244 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:22.244 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:22.245 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.245 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.246 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:22.246 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T00:36:22.246 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T00:36:22.247 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T00:36:22.247 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.247 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T00:36:22.247 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T00:36:22.247 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T00:36:22.248 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T00:36:22.248 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T00:36:22.248 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T00:36:22.248 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T00:36:22.248 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T00:36:22.249 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T00:36:22.249 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T00:36:22.249 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.249 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.250 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.250 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T00:36:22.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.250 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T00:36:22.251 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T00:36:22.251 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T00:36:22.251 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T00:36:22.251 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.252 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.252 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T00:36:22.252 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:22.252 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T00:36:22.252 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T00:36:22.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.253 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T00:36:22.253 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T00:36:22.253 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T00:36:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T00:36:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T00:36:22.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T00:36:22.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:22.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T00:36:22.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T00:36:22.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T00:36:22.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:22.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:36:22.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T00:36:22.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T00:36:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:36:22.258 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T00:36:22.259 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T00:36:22.259 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:22.259 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:22.259 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:22.259 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:22.260 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:22.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.260 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T00:36:22.260 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T00:36:22.260 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T00:36:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T00:36:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T00:36:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T00:36:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T00:36:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T00:36:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:22.264 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:22.264 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:22.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.264 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T00:36:22.265 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T00:36:22.265 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T00:36:22.265 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T00:36:22.265 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T00:36:22.265 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T00:36:22.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:22.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:22.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T00:36:22.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T00:36:22.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:22.267 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:22.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-10-12T00:36:22.268 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-36 2023-10-12T00:36:22.268 DEBUG:boto:path=/wafmoa-36/ 2023-10-12T00:36:22.269 DEBUG:boto:auth_path=/wafmoa-36/ 2023-10-12T00:36:22.269 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.270 DEBUG:boto:Path: /wafmoa-36/ 2023-10-12T00:36:22.271 DEBUG:boto:Data: 2023-10-12T00:36:22.271 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:22.272 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.272 DEBUG:boto:Params: {} 2023-10-12T00:36:22.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:36:22.273 DEBUG:boto:Token: None 2023-10-12T00:36:22.274 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-36/ 2023-10-12T00:36:22.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZNrlftdRIHj/zBxUpdJ+GLtSJmg= 2023-10-12T00:36:22.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZNrlftdRIHj/zBxUpdJ+GLtSJmg=', 'Content-Length': '0'} 2023-10-12T00:36:22.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e54ef4f5b038e4d-0065273f86-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.615 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-37 2023-10-12T00:36:22.615 DEBUG:boto:path=/wafmoa-37/ 2023-10-12T00:36:22.616 DEBUG:boto:auth_path=/wafmoa-37/ 2023-10-12T00:36:22.616 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.617 DEBUG:boto:Path: /wafmoa-37/ 2023-10-12T00:36:22.617 DEBUG:boto:Data: 2023-10-12T00:36:22.617 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:36:22.618 DEBUG:boto:Port: 8001 2023-10-12T00:36:22.619 DEBUG:boto:Params: {} 2023-10-12T00:36:22.619 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:36:22.620 DEBUG:boto:Token: None 2023-10-12T00:36:22.620 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-37/ 2023-10-12T00:36:22.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rtbiFjzDr37RBUh5Ng6iqT8Klis= 2023-10-12T00:36:22.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rtbiFjzDr37RBUh5Ng6iqT8Klis=', 'Content-Length': '0'} 2023-10-12T00:36:22.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b079432d10703436-0065273f86-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.686 INFO:rgw_multi.tests:create bucket zone=b1 name=wafmoa-38 2023-10-12T00:36:22.686 DEBUG:boto:path=/wafmoa-38/ 2023-10-12T00:36:22.687 DEBUG:boto:auth_path=/wafmoa-38/ 2023-10-12T00:36:22.687 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.687 DEBUG:boto:Path: /wafmoa-38/ 2023-10-12T00:36:22.688 DEBUG:boto:Data: 2023-10-12T00:36:22.688 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.689 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:36:22.689 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.690 DEBUG:boto:Params: {} 2023-10-12T00:36:22.690 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:36:22.691 DEBUG:boto:Token: None 2023-10-12T00:36:22.691 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-38/ 2023-10-12T00:36:22.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7crTcxpOK1hI+sXbL5lC5reqp4M= 2023-10-12T00:36:22.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7crTcxpOK1hI+sXbL5lC5reqp4M=', 'Content-Length': '0'} 2023-10-12T00:36:22.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa5dad1cceadb454-0065273f86-1165-b1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.805 INFO:rgw_multi.tests:create bucket zone=b2 name=wafmoa-39 2023-10-12T00:36:22.806 DEBUG:boto:path=/wafmoa-39/ 2023-10-12T00:36:22.806 DEBUG:boto:auth_path=/wafmoa-39/ 2023-10-12T00:36:22.806 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.807 DEBUG:boto:Path: /wafmoa-39/ 2023-10-12T00:36:22.807 DEBUG:boto:Data: 2023-10-12T00:36:22.808 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.808 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:36:22.808 DEBUG:boto:Port: 8001 2023-10-12T00:36:22.809 DEBUG:boto:Params: {} 2023-10-12T00:36:22.809 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:36:22.810 DEBUG:boto:Token: None 2023-10-12T00:36:22.810 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-39/ 2023-10-12T00:36:22.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6DvBxCOYR4akoAuEX2zCDx/Y4g= 2023-10-12T00:36:22.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6DvBxCOYR4akoAuEX2zCDx/Y4g=', 'Content-Length': '0'} 2023-10-12T00:36:22.850 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1dcb825bc0b1e8e-0065273f86-1161-b2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.851 DEBUG:boto:path=/wafmoa-36/ 2023-10-12T00:36:22.851 DEBUG:boto:auth_path=/wafmoa-36/ 2023-10-12T00:36:22.852 DEBUG:boto:path=/wafmoa-36/?website 2023-10-12T00:36:22.852 DEBUG:boto:auth_path=/wafmoa-36/?website 2023-10-12T00:36:22.853 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.853 DEBUG:boto:Path: /wafmoa-36/?website 2023-10-12T00:36:22.854 DEBUG:boto:Data: index.htmlerror.html 2023-10-12T00:36:22.854 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:22.855 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.855 DEBUG:boto:Params: {} 2023-10-12T00:36:22.856 DEBUG:boto:Token: None 2023-10-12T00:36:22.856 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-36/?website 2023-10-12T00:36:22.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z93VZZNbAn4Z91vykqJt1VySNZo= 2023-10-12T00:36:22.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z93VZZNbAn4Z91vykqJt1VySNZo=', 'Content-Length': '241'} 2023-10-12T00:36:22.858 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000876974561b05f94c-0065273f86-168d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-10-12T00:36:22.860 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-40 2023-10-12T00:36:22.861 DEBUG:boto:path=/wafmoa-40/ 2023-10-12T00:36:22.861 DEBUG:boto:auth_path=/wafmoa-40/ 2023-10-12T00:36:22.861 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.862 DEBUG:boto:Path: /wafmoa-40/ 2023-10-12T00:36:22.862 DEBUG:boto:Data: 2023-10-12T00:36:22.863 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:22.863 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.864 DEBUG:boto:Params: {} 2023-10-12T00:36:22.864 DEBUG:boto:Token: None 2023-10-12T00:36:22.865 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-40/ 2023-10-12T00:36:22.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fkdKHturJgIM7vB2W8CC7Yknn4= 2023-10-12T00:36:22.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fkdKHturJgIM7vB2W8CC7Yknn4=', 'Content-Length': '0'} 2023-10-12T00:36:22.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b44a396918d81978-0065273f86-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.883 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-41 2023-10-12T00:36:22.883 DEBUG:boto:path=/wafmoa-41/ 2023-10-12T00:36:22.884 DEBUG:boto:auth_path=/wafmoa-41/ 2023-10-12T00:36:22.884 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.884 DEBUG:boto:Path: /wafmoa-41/ 2023-10-12T00:36:22.885 DEBUG:boto:Data: 2023-10-12T00:36:22.885 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:36:22.886 DEBUG:boto:Port: 8001 2023-10-12T00:36:22.886 DEBUG:boto:Params: {} 2023-10-12T00:36:22.887 DEBUG:boto:Token: None 2023-10-12T00:36:22.887 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-41/ 2023-10-12T00:36:22.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u3OAF+3yuyaZqX+fnlMa4vlRjww= 2023-10-12T00:36:22.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u3OAF+3yuyaZqX+fnlMa4vlRjww=', 'Content-Length': '0'} 2023-10-12T00:36:22.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3e4a0659e2da743-0065273f86-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.917 INFO:rgw_multi.tests:create bucket zone=b1 name=wafmoa-42 2023-10-12T00:36:22.917 DEBUG:boto:path=/wafmoa-42/ 2023-10-12T00:36:22.918 DEBUG:boto:auth_path=/wafmoa-42/ 2023-10-12T00:36:22.918 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.919 DEBUG:boto:Path: /wafmoa-42/ 2023-10-12T00:36:22.919 DEBUG:boto:Data: 2023-10-12T00:36:22.920 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.920 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:36:22.920 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.921 DEBUG:boto:Params: {} 2023-10-12T00:36:22.921 DEBUG:boto:Token: None 2023-10-12T00:36:22.922 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-42/ 2023-10-12T00:36:22.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RumlmzaFthXTkoABTgXgwpjGT7o= 2023-10-12T00:36:22.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RumlmzaFthXTkoABTgXgwpjGT7o=', 'Content-Length': '0'} 2023-10-12T00:36:22.953 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000281e7dba33273236-0065273f86-1165-b1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.954 INFO:rgw_multi.tests:create bucket zone=b2 name=wafmoa-43 2023-10-12T00:36:22.954 DEBUG:boto:path=/wafmoa-43/ 2023-10-12T00:36:22.955 DEBUG:boto:auth_path=/wafmoa-43/ 2023-10-12T00:36:22.956 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.956 DEBUG:boto:Path: /wafmoa-43/ 2023-10-12T00:36:22.956 DEBUG:boto:Data: 2023-10-12T00:36:22.957 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.957 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:36:22.958 DEBUG:boto:Port: 8001 2023-10-12T00:36:22.958 DEBUG:boto:Params: {} 2023-10-12T00:36:22.959 DEBUG:boto:Token: None 2023-10-12T00:36:22.959 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-43/ 2023-10-12T00:36:22.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R4ZR9EjfaaztUILWjuf3VFxFJVk= 2023-10-12T00:36:22.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R4ZR9EjfaaztUILWjuf3VFxFJVk=', 'Content-Length': '0'} 2023-10-12T00:36:22.990 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000043bac2b4259fb019-0065273f86-1161-b2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:22.990 DEBUG:boto:path=/wafmoa-40/ 2023-10-12T00:36:22.991 DEBUG:boto:auth_path=/wafmoa-40/ 2023-10-12T00:36:22.991 DEBUG:boto:path=/wafmoa-40/?policy 2023-10-12T00:36:22.992 DEBUG:boto:auth_path=/wafmoa-40/?policy 2023-10-12T00:36:22.992 DEBUG:boto:Method: PUT 2023-10-12T00:36:22.992 DEBUG:boto:Path: /wafmoa-40/?policy 2023-10-12T00:36:22.993 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-12T00:36:22.993 DEBUG:boto:Headers: {} 2023-10-12T00:36:22.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:22.994 DEBUG:boto:Port: 8000 2023-10-12T00:36:22.994 DEBUG:boto:Params: {} 2023-10-12T00:36:22.995 DEBUG:boto:Token: None 2023-10-12T00:36:22.995 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:22 GMT /wafmoa-40/?policy 2023-10-12T00:36:22.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DWu+6Bp6dAn1GNLUxyGN54JW6EE= 2023-10-12T00:36:22.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DWu+6Bp6dAn1GNLUxyGN54JW6EE=', 'Content-Length': '97'} 2023-10-12T00:36:23.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c12da74833e8aaf-0065273f86-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:23.002 DEBUG:boto:path=/wafmoa-40/ 2023-10-12T00:36:23.003 DEBUG:boto:auth_path=/wafmoa-40/ 2023-10-12T00:36:23.003 DEBUG:boto:path=/wafmoa-40/?policy 2023-10-12T00:36:23.004 DEBUG:boto:auth_path=/wafmoa-40/?policy 2023-10-12T00:36:23.004 DEBUG:boto:Method: GET 2023-10-12T00:36:23.004 DEBUG:boto:Path: /wafmoa-40/?policy 2023-10-12T00:36:23.005 DEBUG:boto:Data: 2023-10-12T00:36:23.005 DEBUG:boto:Headers: {} 2023-10-12T00:36:23.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:23.006 DEBUG:boto:Port: 8000 2023-10-12T00:36:23.006 DEBUG:boto:Params: {} 2023-10-12T00:36:23.007 DEBUG:boto:Token: None 2023-10-12T00:36:23.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:36:23 GMT /wafmoa-40/?policy 2023-10-12T00:36:23.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c+3YUu71JBUHH18uIwMz4CfR28o= 2023-10-12T00:36:23.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c+3YUu71JBUHH18uIwMz4CfR28o=', 'Content-Length': '0'} 2023-10-12T00:36:23.010 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0b568f5c929b017-0065273f87-168d-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 12 Oct 2023 00:36:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:23.010 DEBUG:boto:path=/wafmoa-41/ 2023-10-12T00:36:23.011 DEBUG:boto:auth_path=/wafmoa-41/ 2023-10-12T00:36:23.011 DEBUG:boto:path=/wafmoa-41/?policy 2023-10-12T00:36:23.011 DEBUG:boto:auth_path=/wafmoa-41/?policy 2023-10-12T00:36:23.012 DEBUG:boto:Method: PUT 2023-10-12T00:36:23.012 DEBUG:boto:Path: /wafmoa-41/?policy 2023-10-12T00:36:23.013 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-12T00:36:23.013 DEBUG:boto:Headers: {} 2023-10-12T00:36:23.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:36:23.014 DEBUG:boto:Port: 8001 2023-10-12T00:36:23.014 DEBUG:boto:Params: {} 2023-10-12T00:36:23.015 DEBUG:boto:Token: None 2023-10-12T00:36:23.015 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:23 GMT /wafmoa-41/?policy 2023-10-12T00:36:23.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBQG9oKmpBIFLfu5t60HtDFqp4A= 2023-10-12T00:36:23.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBQG9oKmpBIFLfu5t60HtDFqp4A=', 'Content-Length': '97'} 2023-10-12T00:36:24.026 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e274ac6329f43809-0065273f87-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:24.027 DEBUG:boto:path=/wafmoa-41/ 2023-10-12T00:36:24.027 DEBUG:boto:auth_path=/wafmoa-41/ 2023-10-12T00:36:24.028 DEBUG:boto:path=/wafmoa-41/?policy 2023-10-12T00:36:24.028 DEBUG:boto:auth_path=/wafmoa-41/?policy 2023-10-12T00:36:24.028 DEBUG:boto:Method: GET 2023-10-12T00:36:24.029 DEBUG:boto:Path: /wafmoa-41/?policy 2023-10-12T00:36:24.029 DEBUG:boto:Data: 2023-10-12T00:36:24.029 DEBUG:boto:Headers: {} 2023-10-12T00:36:24.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:36:24.030 DEBUG:boto:Port: 8001 2023-10-12T00:36:24.031 DEBUG:boto:Params: {} 2023-10-12T00:36:24.031 DEBUG:boto:Token: None 2023-10-12T00:36:24.032 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:36:24 GMT /wafmoa-41/?policy 2023-10-12T00:36:24.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8xKRoSN34jglSXPMRMN4sjeAuMI= 2023-10-12T00:36:24.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8xKRoSN34jglSXPMRMN4sjeAuMI=', 'Content-Length': '0'} 2023-10-12T00:36:24.034 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016af887261906184-0065273f88-1689-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 12 Oct 2023 00:36:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:24.035 DEBUG:boto:path=/wafmoa-42/ 2023-10-12T00:36:24.035 DEBUG:boto:auth_path=/wafmoa-42/ 2023-10-12T00:36:24.035 DEBUG:boto:path=/wafmoa-42/?policy 2023-10-12T00:36:24.036 DEBUG:boto:auth_path=/wafmoa-42/?policy 2023-10-12T00:36:24.036 DEBUG:boto:Method: PUT 2023-10-12T00:36:24.037 DEBUG:boto:Path: /wafmoa-42/?policy 2023-10-12T00:36:24.037 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-12T00:36:24.038 DEBUG:boto:Headers: {} 2023-10-12T00:36:24.038 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:36:24.038 DEBUG:boto:Port: 8000 2023-10-12T00:36:24.039 DEBUG:boto:Params: {} 2023-10-12T00:36:24.039 DEBUG:boto:Token: None 2023-10-12T00:36:24.040 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:24 GMT /wafmoa-42/?policy 2023-10-12T00:36:24.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EZJU72EIvqy5syB8ePsutYovHrg= 2023-10-12T00:36:24.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EZJU72EIvqy5syB8ePsutYovHrg=', 'Content-Length': '97'} 2023-10-12T00:36:25.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005f0deed79b440e5-0065273f88-1165-b1'), ('Date', 'Thu, 12 Oct 2023 00:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:25.051 DEBUG:boto:path=/wafmoa-42/ 2023-10-12T00:36:25.051 DEBUG:boto:auth_path=/wafmoa-42/ 2023-10-12T00:36:25.052 DEBUG:boto:path=/wafmoa-42/?policy 2023-10-12T00:36:25.052 DEBUG:boto:auth_path=/wafmoa-42/?policy 2023-10-12T00:36:25.053 DEBUG:boto:Method: GET 2023-10-12T00:36:25.053 DEBUG:boto:Path: /wafmoa-42/?policy 2023-10-12T00:36:25.053 DEBUG:boto:Data: 2023-10-12T00:36:25.054 DEBUG:boto:Headers: {} 2023-10-12T00:36:25.054 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8000 2023-10-12T00:36:25.055 DEBUG:boto:Port: 8000 2023-10-12T00:36:25.055 DEBUG:boto:Params: {} 2023-10-12T00:36:25.056 DEBUG:boto:Token: None 2023-10-12T00:36:25.056 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:36:25 GMT /wafmoa-42/?policy 2023-10-12T00:36:25.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5fNpIOA4mEJY30O65vF7TzEwkcY= 2023-10-12T00:36:25.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5fNpIOA4mEJY30O65vF7TzEwkcY=', 'Content-Length': '0'} 2023-10-12T00:36:25.059 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b0fe0bd82981b72c-0065273f89-1165-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 12 Oct 2023 00:36:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:25.059 DEBUG:boto:path=/wafmoa-43/ 2023-10-12T00:36:25.059 DEBUG:boto:auth_path=/wafmoa-43/ 2023-10-12T00:36:25.060 DEBUG:boto:path=/wafmoa-43/?policy 2023-10-12T00:36:25.060 DEBUG:boto:auth_path=/wafmoa-43/?policy 2023-10-12T00:36:25.061 DEBUG:boto:Method: PUT 2023-10-12T00:36:25.061 DEBUG:boto:Path: /wafmoa-43/?policy 2023-10-12T00:36:25.061 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-12T00:36:25.062 DEBUG:boto:Headers: {} 2023-10-12T00:36:25.062 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:36:25.063 DEBUG:boto:Port: 8001 2023-10-12T00:36:25.063 DEBUG:boto:Params: {} 2023-10-12T00:36:25.063 DEBUG:boto:Token: None 2023-10-12T00:36:25.064 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:25 GMT /wafmoa-43/?policy 2023-10-12T00:36:25.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5qF3xFx3Bz5D6in3cY52/SRFkXw= 2023-10-12T00:36:25.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5qF3xFx3Bz5D6in3cY52/SRFkXw=', 'Content-Length': '97'} 2023-10-12T00:36:26.076 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000030e762ac0a613f2d-0065273f89-1161-b2'), ('Date', 'Thu, 12 Oct 2023 00:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:26.076 DEBUG:boto:path=/wafmoa-43/ 2023-10-12T00:36:26.077 DEBUG:boto:auth_path=/wafmoa-43/ 2023-10-12T00:36:26.077 DEBUG:boto:path=/wafmoa-43/?policy 2023-10-12T00:36:26.078 DEBUG:boto:auth_path=/wafmoa-43/?policy 2023-10-12T00:36:26.078 DEBUG:boto:Method: GET 2023-10-12T00:36:26.078 DEBUG:boto:Path: /wafmoa-43/?policy 2023-10-12T00:36:26.079 DEBUG:boto:Data: 2023-10-12T00:36:26.079 DEBUG:boto:Headers: {} 2023-10-12T00:36:26.079 DEBUG:boto:Host: smithi118.front.sepia.ceph.com:8001 2023-10-12T00:36:26.080 DEBUG:boto:Port: 8001 2023-10-12T00:36:26.080 DEBUG:boto:Params: {} 2023-10-12T00:36:26.081 DEBUG:boto:Token: None 2023-10-12T00:36:26.081 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:36:26 GMT /wafmoa-43/?policy 2023-10-12T00:36:26.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pONgvJmAVHh8YBwc4AzDO+3ilyM= 2023-10-12T00:36:26.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pONgvJmAVHh8YBwc4AzDO+3ilyM=', 'Content-Length': '0'} 2023-10-12T00:36:26.084 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085eefb6e01c04cb8-0065273f8a-1161-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Thu, 12 Oct 2023 00:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:26.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-10-12T00:36:26.085 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-44 2023-10-12T00:36:26.085 DEBUG:boto:path=/wafmoa-44/ 2023-10-12T00:36:26.086 DEBUG:boto:auth_path=/wafmoa-44/ 2023-10-12T00:36:26.086 DEBUG:boto:Method: PUT 2023-10-12T00:36:26.087 DEBUG:boto:Path: /wafmoa-44/ 2023-10-12T00:36:26.087 DEBUG:boto:Data: 2023-10-12T00:36:26.087 DEBUG:boto:Headers: {} 2023-10-12T00:36:26.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:36:26.088 DEBUG:boto:Port: 8000 2023-10-12T00:36:26.088 DEBUG:boto:Params: {} 2023-10-12T00:36:26.089 DEBUG:boto:Token: None 2023-10-12T00:36:26.089 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:26 GMT /wafmoa-44/ 2023-10-12T00:36:26.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pGatkqJRWKxhdwHdtHKlTlf7/Kk= 2023-10-12T00:36:26.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pGatkqJRWKxhdwHdtHKlTlf7/Kk=', 'Content-Length': '0'} 2023-10-12T00:36:26.107 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e52673ebea72c1c1-0065273f8a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:26.107 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-45 2023-10-12T00:36:26.108 DEBUG:boto:path=/wafmoa-45/ 2023-10-12T00:36:26.108 DEBUG:boto:auth_path=/wafmoa-45/ 2023-10-12T00:36:26.109 DEBUG:boto:Method: PUT 2023-10-12T00:36:26.109 DEBUG:boto:Path: /wafmoa-45/ 2023-10-12T00:36:26.109 DEBUG:boto:Data: 2023-10-12T00:36:26.110 DEBUG:boto:Headers: {} 2023-10-12T00:36:26.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:36:26.110 DEBUG:boto:Port: 8001 2023-10-12T00:36:26.111 DEBUG:boto:Params: {} 2023-10-12T00:36:26.111 DEBUG:boto:Token: None 2023-10-12T00:36:26.112 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:36:26 GMT /wafmoa-45/ 2023-10-12T00:36:26.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:azgwdtBljAS6P0Jcr1ZbWMjI69A= 2023-10-12T00:36:26.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:36:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:azgwdtBljAS6P0Jcr1ZbWMjI69A=', 'Content-Length': '0'} 2023-10-12T00:36:26.138 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034e7eeb8fa34660a-0065273f8a-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:36:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:36:26.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:26.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:26.140 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:26.225 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:36:26.236 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:36:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:36:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:36:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070986.129394_466.1", 2023-10-12T00:36:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:36:26.129394Z" 2023-10-12T00:36:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:36:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.240 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:36:26.241 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:26.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:36:26.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:26.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:26.243 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:26.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:26.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:26.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:26.320+0000 7fa3f04ad780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:26.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:26.321+0000 7fa3f04ad780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:26.333 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:26.334 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:26.335 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:26.335 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:26.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.335 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:26.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.336 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:26.336 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:26.336 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:26.336 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:26.336 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:26.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:26.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:26.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:26.337 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:26.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.338 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:26.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:26.339 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:26.339 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:26.339 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:26.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:26.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:26.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:26.343 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:26.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:26.344 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:26.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:26.344 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:26.344 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:26.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:26.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:26.345 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:26.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:26.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:26.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.346 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:26.346 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:26.347 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:26.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:26.347 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:26Z" 2023-10-12T00:36:26.347 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:26.348 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:26Z" } 2023-10-12T00:36:26.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:26.349 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:26.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:26.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:31.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:31.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:31.352 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:31.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:31.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:31.440+0000 7ff5eecd0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:31.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:31.441+0000 7ff5eecd0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:31.454 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:31.455 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:31.455 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:31.455 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:31.456 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:31.456 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:31.456 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:31.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.457 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:31.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:31.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:31.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:31.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:31.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:31.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:31.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:31.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:31.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:31.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:31.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:31.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:31.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:31.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:31.461 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:31.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:31.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:31.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:31.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:31.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:31.463 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:31.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:31.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:31.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:31.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:31.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:31.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:31.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:31.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:31.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:31.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:31.467 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:31.467 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:31.467 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:31.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:31.467 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:31.468 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:31.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:31.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:31.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:31.469 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:31.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.469 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:31.469 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:31.470 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:31.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:31.470 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:31Z" 2023-10-12T00:36:31.470 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:31.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:31Z" } 2023-10-12T00:36:31.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:31.473 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:31.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:31.474 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:36.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:36.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:36.477 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:36.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:36.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:36.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:36.564+0000 7f364dd79780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:36.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:36.565+0000 7f364dd79780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:36.578 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:36.578 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:36.579 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:36.579 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:36.579 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:36.579 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:36.580 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:36.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.580 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:36.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:36.580 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:36.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:36.581 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:36.581 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:36.581 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:36.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:36.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:36.582 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:36.582 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:36.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:36.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:36.583 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:36.583 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:36.584 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:36.584 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:36.584 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:36.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:36.584 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:36.585 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:36.585 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:36.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:36.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:36.586 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:36.586 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:36.586 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:36.586 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:36.587 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:36.587 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:36.587 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:36.587 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:36.587 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:36.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:36.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:36.588 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:36.588 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:36.589 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:36.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:36.589 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:36.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:36.589 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:36.590 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:36.590 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:36.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:36.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:36.591 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:36.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.591 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:36.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:36.591 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:36.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:36.592 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:36Z" 2023-10-12T00:36:36.592 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:36.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:36Z" } 2023-10-12T00:36:36.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:36.594 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:36.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:36.595 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:41.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:41.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:41.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:41.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:41.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:41.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:41.684+0000 7fc594bd3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:41.685+0000 7fc594bd3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:41.698 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:41.705 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:41.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:41.705 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:41.705 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:41.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:41.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:41.707 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:41.707 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:41.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:41.707 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:41.708 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:41.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:41.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:41.708 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:41.709 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:41.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:41.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:41.709 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:41.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:41.710 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:41.710 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:41.711 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:41.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:41.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:41.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:41.711 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:41.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:41.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:41.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:41.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:41Z" 2023-10-12T00:36:41.714 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:41.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:41Z" } 2023-10-12T00:36:41.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:41.716 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:41.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:41.716 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:46.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:46.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:46.719 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:46.776 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:46.777 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:46.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:46.808+0000 7f3c1b9cb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:46.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:46.808+0000 7f3c1b9cb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:46.821 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:46.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:46.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:46.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:46.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:46.829 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:46.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:46.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:46.830 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:46.830 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:46.830 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:46.831 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:46.831 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:46.831 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:46.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:46.833 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:46.833 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:46.834 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:46.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:46.834 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:46.834 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:46.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:46.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:46.835 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:46.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:46.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:46.836 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:46.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.836 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:46.837 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:46Z" 2023-10-12T00:36:46.837 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:46.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:46Z" } 2023-10-12T00:36:46.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:46.840 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:46.840 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:46.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:51.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:51.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:51.845 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:51.902 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:51.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:51.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:51.934+0000 7f143961f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:51.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:51.935+0000 7f143961f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:51.947 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:51.947 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:51.948 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:51.948 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:51.948 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:51.948 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:51.949 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:51.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.949 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:51.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:51.949 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:51.950 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:51.950 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:51.950 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:51.950 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:51.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:51.951 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:51.951 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:51.951 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:51.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:51.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:51.952 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:51.952 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:51.953 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:51.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:51.953 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:51.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:51.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:51.954 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:51.954 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:51.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:51.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:51.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:51.955 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:51.955 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:51.955 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:51.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:51.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:51.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:51.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:51.956 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:51.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:51.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:51.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:51.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:51.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:51.959 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:51.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:51.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:51.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:51.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.960 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:51.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:51.961 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:51.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:51.961 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:51Z" 2023-10-12T00:36:51.961 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:51.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:51Z" } 2023-10-12T00:36:51.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:51.963 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:51.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:51.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:36:56.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:56.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:36:56.967 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:36:57.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:36:57.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:36:57.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:57.055+0000 7f0ea4f84780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:57.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:36:57.056+0000 7f0ea4f84780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:36:57.068 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:36:57.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:36:57.068 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:36:57.068 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:36:57.069 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:36:57.069 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:36:57.069 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:57.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.070 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:36:57.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:57.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:36:57.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:57.070 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:57.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:36:57.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:57.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:36:57.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:57.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:36:57.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:57.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:57.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:57.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:36:57.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:57.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:57.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:36:57.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:57.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:57.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:57.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:36:57.074 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:57.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:57.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:57.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:36:57.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:57.076 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:57.076 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:36:57.076 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:57.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:57.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:57.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:36:57.077 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:57.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:57.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:36:57.078 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:36:57.078 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:36:57.078 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:36:57.078 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:36:57.079 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:36:57.079 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:36:57.079 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:36:57.079 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:36:57.080 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:36:57.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:57.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:36:57.080 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:36:57.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.081 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:36:57.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:36:57.081 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:36:57.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:36:57.082 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:36:57Z" 2023-10-12T00:36:57.082 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:36:57.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:36:57Z" } 2023-10-12T00:36:57.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:36:57.084 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:57.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:36:57.085 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:02.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:02.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:02.087 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:02.146 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:02.146 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:02.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:02.177+0000 7f671589f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:02.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:02.178+0000 7f671589f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:02.191 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:02.191 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:02.191 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:02.191 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:02.192 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:02.192 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:02.192 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:02.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.193 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:02.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:02.193 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:02.193 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:02.193 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:02.194 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:02.194 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:02.194 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:02.194 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:02.194 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:02.197 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:02.197 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:02.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:02.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:02.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:02.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:02.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:02.198 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:02.199 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:02.199 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:02.199 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:02.199 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:02.200 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:02.200 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:02.200 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:02.200 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:02.201 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:02.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:02.201 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:02.201 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:02.202 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:02.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:02.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:02.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:02.206 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:02.206 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:02.206 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:02.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:02.207 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:02.207 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:02.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.207 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:02.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:02.208 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:02.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:02.208 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:02Z" 2023-10-12T00:37:02.208 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:02.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:02Z" } 2023-10-12T00:37:02.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:02.210 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:02.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:02.211 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:07.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:07.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:07.213 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:07.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:07.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:07.302+0000 7f2df78d4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:07.305+0000 7f2df78d4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:07.320 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:07Z" 2023-10-12T00:37:07.334 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:07.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:07Z" } 2023-10-12T00:37:07.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:07.340 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:07.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:07.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:12.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:12.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:12.343 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:12.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:12.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:12.433+0000 7fc89a75b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:12.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:12.434+0000 7fc89a75b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:12.448 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:12.449 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:12.449 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:12.449 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:12.449 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:12.450 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:12.450 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:12.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.450 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:12.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:12.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:12.451 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:12.451 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:12.451 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:12.452 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:12.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:12.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:12.453 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:12.453 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:12.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:12.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:12.454 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:12.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:12.454 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:12.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:12.455 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:12.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:12.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:12.455 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:12.456 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:12.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:12.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:12.456 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:12.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:12.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:12.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:12.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:12.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:12.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:12.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:12.458 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:12.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:12.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:12.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:12.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:12.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:12.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:12.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:12.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:12.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:12.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:12.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:12.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:12.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:12.461 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:12.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.462 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:12.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:12.462 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:12.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:12.463 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:12Z" 2023-10-12T00:37:12.463 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:12.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:12Z" } 2023-10-12T00:37:12.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:12.465 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:12.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:12.466 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:17.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:17.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:17.469 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:17.526 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:17.557+0000 7f1b14ae0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:17.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:17.558+0000 7f1b14ae0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:17.571 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:17.571 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:17.571 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:17.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:17.573 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:17.574 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:17.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:17.574 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:17.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:17.575 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:17.575 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:17.575 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:17.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:17.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:17.576 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:17.576 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:17.576 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:17.577 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:17.577 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:17.577 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:17.577 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:17.577 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:17.578 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:17.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:17.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:17.579 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:17.579 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:17.579 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:17.579 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:17.579 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:17.580 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:17.580 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:17.580 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:17.580 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:17.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:17.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:17.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:17.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:17.582 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:17.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:17.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:17.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:17.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:17.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:17.583 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:17.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:17.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:17.584 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:17.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.584 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:17.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:17.584 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:17.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:17.585 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:17Z" 2023-10-12T00:37:17.585 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:17.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:17Z" } 2023-10-12T00:37:17.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:17.587 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:17.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:17.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:22.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:22.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:22.592 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:22.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:22.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:22.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:22.679+0000 7f3f878d9780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:22.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:22.680+0000 7f3f878d9780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:22.694 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:22.694 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:22.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:22.696 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:22.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:22.698 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:22.699 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:22Z" 2023-10-12T00:37:22.709 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:22.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:22Z" } 2023-10-12T00:37:22.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:22.711 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:22.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:22.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:27.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:27.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:27.714 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:27.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:27.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:27.803+0000 7f1fd030d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:27.804+0000 7f1fd030d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:27.816 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:27.816 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:27.816 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:27.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:27.817 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:27.817 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:27.817 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:27.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.818 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:27.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:27.818 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:27.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:27.819 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:27.819 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:27.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:27.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:27.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:27.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:27.820 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:27.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:27.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:27.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:27.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:27.821 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:27.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:27.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:27.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:27.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:27.823 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:27.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:27.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:27.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:27.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:27.824 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:27.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:27.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:27.825 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:27.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:27.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:27.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:27.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:27.826 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:27.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:27.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:27.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:27.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:27.827 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:27.828 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:27.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:27.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:27.828 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:27.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.829 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:27.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:27.829 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:27.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:27.829 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:27Z" 2023-10-12T00:37:27.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:27.831 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:27Z" } 2023-10-12T00:37:27.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:27.832 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:27.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:27.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:32.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:32.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:32.835 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:32.892 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:32.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:32.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:32.924+0000 7fb86dbee780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:32.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:32.925+0000 7fb86dbee780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:32.937 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:32.938 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:32.938 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:32.938 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:32.938 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:32.939 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:32.939 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:32.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.939 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:32.943 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:32.943 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:32.943 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:32.943 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:32.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:32.944 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:32.944 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:32.944 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:32.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:32.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:32.945 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:32.945 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:32.946 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:32.946 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:32.946 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:32.946 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:32.947 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:32.947 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:32.947 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:32.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:32.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:32.948 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:32.948 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:32.948 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:32.949 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:32.949 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:32.949 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:32.949 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:32.949 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:32.950 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:32.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:32.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:32.950 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:32.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.951 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:32.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:32.951 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:32.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:32.952 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:32Z" 2023-10-12T00:37:32.952 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:32.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:32Z" } 2023-10-12T00:37:32.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:32.954 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:32.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:32.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:37.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:37.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:37.958 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:38.029 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:38.030 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:38.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:38.057+0000 7f6ca1845780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:38.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:38.058+0000 7f6ca1845780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:38Z" 2023-10-12T00:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:38.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:38Z" } 2023-10-12T00:37:38.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:38.087 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:38.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:38.088 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:43.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:43.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:43.090 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:43.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:43.178+0000 7f5cd1c9c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:43.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:43.179+0000 7f5cd1c9c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:43.193 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:43.193 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:43.193 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:43.193 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:43.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:43.202 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:43.202 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:43.202 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:43.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:43.203 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:43.203 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:43.203 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:43.204 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:43.204 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:43.204 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:43.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:43.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:43.205 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:43.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:43.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:43.205 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:43.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.206 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:43.206 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:43.206 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:43.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:43.207 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:43Z" 2023-10-12T00:37:43.207 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:43.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:43Z" } 2023-10-12T00:37:43.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:43.209 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:43.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:43.210 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:48.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:48.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:48.213 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:48.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:48.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:48.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:48.301+0000 7fe1ea8e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:48.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:48.302+0000 7fe1ea8e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:48.316 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:48.318 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:48.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.318 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:48.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:48.319 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:48.319 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:48.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:48.319 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:48.320 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:48.320 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:48.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:48.320 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:48.320 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:48.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:48.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:48.321 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:48.321 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:48.322 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:48.322 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:48.322 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:48.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:48.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:48.323 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:48.323 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:48.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:48.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:48.324 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:48.324 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:48.325 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:48.325 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:48.325 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:48.325 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:48.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:48.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:48.326 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:48.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:48.327 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:48.328 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:48.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:48.330 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:48.331 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:48Z" 2023-10-12T00:37:48.331 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:48.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:48Z" } 2023-10-12T00:37:48.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:48.333 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:48.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:48.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:53.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:53.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:53.336 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:53.395 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:53.396 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:53.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:53.426+0000 7f26c8a20780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:53.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:53.427+0000 7f26c8a20780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:53.440 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:53.441 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:53.441 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:53.441 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:53.441 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:53.442 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:53.442 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:53.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.442 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:53.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:53.443 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:53.443 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:53.443 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:53.443 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:53.444 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:53.444 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:53.444 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:53.444 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:53.445 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:53.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:53.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:53.445 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:53.446 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:53.446 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:53.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:53.446 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:53.446 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:53.447 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:53.447 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:53.447 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:53.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:53.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:53.448 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:53.448 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:53.448 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:53.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:53.449 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:53.449 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:53.449 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:53.449 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:53.450 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:53.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:53.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:53.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:53.451 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:53.451 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:53.451 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:53Z" 2023-10-12T00:37:53.455 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:53.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:53Z" } 2023-10-12T00:37:53.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:53.457 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:53.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:53.458 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:37:58.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:58.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:37:58.460 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:37:58.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:37:58.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:37:58.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:58.546+0000 7f9490c15780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:58.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:37:58.547+0000 7f9490c15780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:37:58.560 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:37:58.561 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:37:58.561 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:37:58.561 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:37:58.561 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:37:58.561 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:37:58.562 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:58.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.562 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:37:58.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:58.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:37:58.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:37:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:37:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:37:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:37:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:37:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:37:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:37:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:37:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:37:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:37:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:37:58.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:37:58.572 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:37:58.572 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:37:58.572 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:37:58.572 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:37:58.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:37:58.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:37:58.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:37:58.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:58.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:37:58.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:37:58.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.574 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:37:58.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:37:58.575 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:37:58.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:37:58.575 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:37:58Z" 2023-10-12T00:37:58.575 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:37:58.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:37:58Z" } 2023-10-12T00:37:58.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:37:58.577 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:58.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:37:58.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:38:03.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:03.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:03.581 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:03.638 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:03.638 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:03.668+0000 7f0c26950780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:03.669+0000 7f0c26950780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:03.682 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:03.683 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:03.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:03.685 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:03.686 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:03.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:03.688 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:03.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:03.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:38:03.692 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:38:03.693 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:03.694 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:03.695 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:03.696 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:03.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:03.698 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:03.699 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:03Z" 2023-10-12T00:38:03.699 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:03.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:03Z" } 2023-10-12T00:38:03.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:03.701 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:03.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:03.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:38:08.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:08.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:08.704 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:08.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:08.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:08.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:08.793+0000 7f1c16c2c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:08.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:08.794+0000 7f1c16c2c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:08.806 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:08.807 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:08.807 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:08.807 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:08.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:08.809 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:08.809 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:08.809 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:08.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:08.810 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:08.810 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:08.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:08.810 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:08.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:08.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:08.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:08.812 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:08.812 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:08.813 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:08.813 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:08.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:08.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:08.814 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:08.814 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:08.814 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:08.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:08.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:08.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:08.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:08.817 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:08.817 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:38:08.817 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:08.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:08.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:08.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:38:08.819 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:08.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:08.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:08.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:08.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:08.821 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:08.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:08.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:08.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:08.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:08.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:08.824 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:08.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:08.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:08.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:08.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.826 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:08.826 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:08.827 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:08.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:08.827 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:08Z" 2023-10-12T00:38:08.828 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:08.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:08Z" } 2023-10-12T00:38:08.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:08.831 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:08.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:08.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:38:13.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:13.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:13.836 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:13.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:13.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:13.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:13.925+0000 7f3c8b48c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:13.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:13.926+0000 7f3c8b48c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:13.938 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:13.939 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:13.939 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:13.939 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:13.939 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:13.940 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:13.940 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:13.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.940 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:13.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:13.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:13.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:13.941 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:13.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:13.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:13.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:13.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:13.942 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:13.942 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:13.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:13.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:13.943 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:13.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:13.944 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:13.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:13.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:13.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:13.945 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:13.945 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:13.945 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:13.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:13.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:13.946 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:13.947 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:38:13.947 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:13.947 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:13.947 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:13.947 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:38:13.949 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:13.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:13.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:13.950 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:13.950 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:13.950 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:13.950 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:13.951 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:13.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:13.951 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:13.951 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:13.951 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:13.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:13.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:13.952 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:13.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.953 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:13.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:13.953 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:13.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:13.953 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:13Z" 2023-10-12T00:38:13.954 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:13.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:13Z" } 2023-10-12T00:38:13.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:13.956 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:13.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:13.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:38:18.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:18.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:18.959 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:19.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:19.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:19.047+0000 7f08ad24a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:19.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:19.048+0000 7f08ad24a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:19.062 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:19.062 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:19.062 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:19.062 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:19.063 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:19.063 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:19.063 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:19.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:19.066 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:19.066 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:19.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:19.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:19.067 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:19.067 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:19.067 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:19.069 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:19.069 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:19.070 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:19.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:19.070 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:19.071 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:19.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:19.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:19.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:19.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:19.072 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:19.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070731.089698_270.1", 2023-10-12T00:38:19.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:19.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:19.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:19.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:32:11.089698Z", 2023-10-12T00:38:19.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:19.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:19.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:19.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:19.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:19.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:19.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:19.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:19.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:19.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:19.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:19.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.077 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:19.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:19.077 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:19.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:19.078 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:19Z" 2023-10-12T00:38:19.078 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:19.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070731.089698_270.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:32:11.089698Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:19Z" } 2023-10-12T00:38:19.080 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070731.089698_270.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:32:11.089698Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:19.081 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:19.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070731.089698_270.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:19.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070731.089698_270.1 2023-10-12T00:38:24.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:24.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:24.084 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:24.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:24.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:24.172+0000 7fa39f9b1780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:24.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:24.174+0000 7fa39f9b1780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:24.190 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:24.198 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:24.198 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070982.589104_398.1", 2023-10-12T00:38:24.198 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:24.198 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:24.198 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:24.199 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:36:22.589104Z", 2023-10-12T00:38:24.199 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:24.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:24.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:24.200 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:24.200 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:24.200 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:24.201 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:24.201 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:24.201 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:24.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:24.203 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:24.203 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:24.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:24.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:24.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:24.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.204 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:24.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:24.205 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:24.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:24.205 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:24Z" 2023-10-12T00:38:24.205 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:24.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070982.589104_398.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:36:22.589104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:24Z" } 2023-10-12T00:38:24.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070982.589104_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:36:22.589104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:24.208 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:24.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070982.589104_398.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:24.209 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070982.589104_398.1 2023-10-12T00:38:29.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:29.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:29.212 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:29.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:29.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:29.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:29.310+0000 7f8e9220b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:29.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:29.312+0000 7f8e9220b780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:29.355 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:29.355 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:29.355 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:29.356 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:29.356 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:29.356 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:29.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:29.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.357 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:29.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:29.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:29.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:29.358 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:29.358 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:29.358 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:29.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:29.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:29.359 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:29.359 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:29.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:29.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:29.360 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:29.360 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:29.360 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:29.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:29.361 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:29.361 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:29.361 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:29.361 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:29.362 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:29.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:29.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:29.362 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:29.363 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:29.363 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:29.363 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070982.731561_417.1", 2023-10-12T00:38:29.363 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:29.364 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:29.364 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:29.364 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:36:22.731561Z", 2023-10-12T00:38:29.364 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:29.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:29.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:29.365 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:29.365 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:29Z" 2023-10-12T00:38:29.369 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:29.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070982.731561_417.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:36:22.731561Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:29Z" } 2023-10-12T00:38:29.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070982.731561_417.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:36:22.731561Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:29.371 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:29.372 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070982.731561_417.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:29.372 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070982.731561_417.1 2023-10-12T00:38:34.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:34.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:34.393 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:34.452 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:34.515+0000 7fce83245780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:34.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:34.516+0000 7fce83245780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:34.528 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:34.528 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:34.529 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:34.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:34.529 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:34.529 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:34.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:34.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.530 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:34.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:34.530 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:34.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:34.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:34.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:34.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:34.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:34.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:34.532 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:34.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:34.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:34.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:34.533 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:34.533 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:34.533 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:34.534 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:34.534 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:34.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:34.534 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070982.647612_409.1", 2023-10-12T00:38:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:36:22.647612Z", 2023-10-12T00:38:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:34Z" 2023-10-12T00:38:34.542 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:34.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070982.647612_409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:36:22.647612Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:34Z" } 2023-10-12T00:38:34.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070982.647612_409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:36:22.647612Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:34.544 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:34.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070982.647612_409.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:34.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697070986.129394_466.1 target=1_1697070982.647612_409.1 2023-10-12T00:38:39.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:39.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:39.548 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:39.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:39.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:39.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:39.633+0000 7fa429b88780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:39.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:39.634+0000 7fa429b88780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:39.652 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070986.129394_466.1", 2023-10-12T00:38:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:36:26.129394Z", 2023-10-12T00:38:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:39Z" 2023-10-12T00:38:39.665 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:39.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697070986.129394_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:36:26.129394Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:39Z" } 2023-10-12T00:38:39.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697070986.129394_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:36:26.129394Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:39.667 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:39.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697070986.129394_466.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:39.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:38:39.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:38:39.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:38:39.669 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:38:39.720 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:39.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:39.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:38:39.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:38:39.784 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:38:39.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:39.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:39.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:39.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:39.900 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:39.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:39.947 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:39.987 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:38:39.987 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000078", 2023-10-12T00:38:39.987 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.history", 2023-10-12T00:38:39.988 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:2", 2023-10-12T00:38:39.988 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000070", 2023-10-12T00:38:39.988 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000104", 2023-10-12T00:38:39.988 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000026", 2023-10-12T00:38:39.988 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:4", 2023-10-12T00:38:39.989 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:2", 2023-10-12T00:38:39.989 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:4", 2023-10-12T00:38:39.989 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:7", 2023-10-12T00:38:39.989 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:3", 2023-10-12T00:38:39.990 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.8", 2023-10-12T00:38:39.990 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000028", 2023-10-12T00:38:39.990 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:4", 2023-10-12T00:38:39.990 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33", 2023-10-12T00:38:39.990 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:4", 2023-10-12T00:38:39.991 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:3", 2023-10-12T00:38:39.991 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:5", 2023-10-12T00:38:39.991 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000040", 2023-10-12T00:38:39.991 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.6", 2023-10-12T00:38:39.992 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:0", 2023-10-12T00:38:39.992 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000015", 2023-10-12T00:38:39.992 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15", 2023-10-12T00:38:39.992 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:1", 2023-10-12T00:38:39.992 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000069", 2023-10-12T00:38:39.993 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000095", 2023-10-12T00:38:39.993 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000003", 2023-10-12T00:38:39.993 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000047", 2023-10-12T00:38:39.993 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:6", 2023-10-12T00:38:39.994 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000035", 2023-10-12T00:38:39.994 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000037", 2023-10-12T00:38:39.994 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:4", 2023-10-12T00:38:39.994 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:0", 2023-10-12T00:38:39.994 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:9", 2023-10-12T00:38:39.995 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:0", 2023-10-12T00:38:39.995 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000024", 2023-10-12T00:38:39.995 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:4", 2023-10-12T00:38:39.995 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000000", 2023-10-12T00:38:39.996 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000031", 2023-10-12T00:38:39.996 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:10", 2023-10-12T00:38:39.996 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.3", 2023-10-12T00:38:39.996 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3", 2023-10-12T00:38:39.996 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:6", 2023-10-12T00:38:39.997 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:1", 2023-10-12T00:38:39.997 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:4", 2023-10-12T00:38:39.997 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:5", 2023-10-12T00:38:39.997 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:3", 2023-10-12T00:38:39.998 INFO:teuthology.orchestra.run.smithi112.stdout: "data_loggenerations_metadata", 2023-10-12T00:38:39.998 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:8", 2023-10-12T00:38:39.998 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:5", 2023-10-12T00:38:39.998 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:1", 2023-10-12T00:38:39.998 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000076", 2023-10-12T00:38:39.999 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000113", 2023-10-12T00:38:39.999 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000005", 2023-10-12T00:38:39.999 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:1", 2023-10-12T00:38:39.999 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:7", 2023-10-12T00:38:40.000 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000011", 2023-10-12T00:38:40.000 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000058", 2023-10-12T00:38:40.000 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000052", 2023-10-12T00:38:40.000 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:0", 2023-10-12T00:38:40.001 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:7", 2023-10-12T00:38:40.001 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:7", 2023-10-12T00:38:40.001 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:10", 2023-10-12T00:38:40.001 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:3", 2023-10-12T00:38:40.001 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:8", 2023-10-12T00:38:40.002 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:0", 2023-10-12T00:38:40.002 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:4", 2023-10-12T00:38:40.002 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:1", 2023-10-12T00:38:40.002 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000088", 2023-10-12T00:38:40.002 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:9", 2023-10-12T00:38:40.003 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000080", 2023-10-12T00:38:40.003 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000090", 2023-10-12T00:38:40.003 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000110", 2023-10-12T00:38:40.003 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:7", 2023-10-12T00:38:40.004 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:3", 2023-10-12T00:38:40.004 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000096", 2023-10-12T00:38:40.004 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:7", 2023-10-12T00:38:40.004 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000087", 2023-10-12T00:38:40.004 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000008", 2023-10-12T00:38:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:7", 2023-10-12T00:38:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:3", 2023-10-12T00:38:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000006", 2023-10-12T00:38:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000029", 2023-10-12T00:38:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:0", 2023-10-12T00:38:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:9", 2023-10-12T00:38:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000089", 2023-10-12T00:38:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000022", 2023-10-12T00:38:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10", 2023-10-12T00:38:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog.trim", 2023-10-12T00:38:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000016", 2023-10-12T00:38:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:3", 2023-10-12T00:38:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:6", 2023-10-12T00:38:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000041", 2023-10-12T00:38:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:6", 2023-10-12T00:38:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.2", 2023-10-12T00:38:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000018", 2023-10-12T00:38:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000092", 2023-10-12T00:38:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12", 2023-10-12T00:38:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:3", 2023-10-12T00:38:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.4", 2023-10-12T00:38:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:0", 2023-10-12T00:38:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000014", 2023-10-12T00:38:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:5", 2023-10-12T00:38:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:9", 2023-10-12T00:38:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000112", 2023-10-12T00:38:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000007", 2023-10-12T00:38:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000021", 2023-10-12T00:38:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:10", 2023-10-12T00:38:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000064", 2023-10-12T00:38:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:2", 2023-10-12T00:38:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000071", 2023-10-12T00:38:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:1", 2023-10-12T00:38:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:8", 2023-10-12T00:38:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:0", 2023-10-12T00:38:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000074", 2023-10-12T00:38:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.0.retry", 2023-10-12T00:38:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34", 2023-10-12T00:38:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:1", 2023-10-12T00:38:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000081", 2023-10-12T00:38:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000009", 2023-10-12T00:38:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:4", 2023-10-12T00:38:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000121", 2023-10-12T00:38:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1", 2023-10-12T00:38:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:10", 2023-10-12T00:38:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:6", 2023-10-12T00:38:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:6", 2023-10-12T00:38:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:4", 2023-10-12T00:38:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000125", 2023-10-12T00:38:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9", 2023-10-12T00:38:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:10", 2023-10-12T00:38:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000082", 2023-10-12T00:38:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17", 2023-10-12T00:38:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000105", 2023-10-12T00:38:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000059", 2023-10-12T00:38:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:0", 2023-10-12T00:38:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000077", 2023-10-12T00:38:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000032", 2023-10-12T00:38:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:0", 2023-10-12T00:38:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16", 2023-10-12T00:38:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:8", 2023-10-12T00:38:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:3", 2023-10-12T00:38:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:1", 2023-10-12T00:38:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:9", 2023-10-12T00:38:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1", 2023-10-12T00:38:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000053", 2023-10-12T00:38:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000091", 2023-10-12T00:38:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:10", 2023-10-12T00:38:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.1", 2023-10-12T00:38:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:10", 2023-10-12T00:38:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:7", 2023-10-12T00:38:40.021 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000065", 2023-10-12T00:38:40.021 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000083", 2023-10-12T00:38:40.021 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000010", 2023-10-12T00:38:40.021 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:4", 2023-10-12T00:38:40.021 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.7", 2023-10-12T00:38:40.022 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:2", 2023-10-12T00:38:40.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.3", 2023-10-12T00:38:40.022 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:10", 2023-10-12T00:38:40.022 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:2", 2023-10-12T00:38:40.023 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:9", 2023-10-12T00:38:40.023 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:10", 2023-10-12T00:38:40.023 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:5", 2023-10-12T00:38:40.023 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:6", 2023-10-12T00:38:40.023 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.2", 2023-10-12T00:38:40.024 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:0", 2023-10-12T00:38:40.024 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000045", 2023-10-12T00:38:40.024 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000002", 2023-10-12T00:38:40.025 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2.BZVTun6qQBj.0", 2023-10-12T00:38:40.025 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:8", 2023-10-12T00:38:40.025 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:8", 2023-10-12T00:38:40.026 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000116", 2023-10-12T00:38:40.026 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:3", 2023-10-12T00:38:40.026 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000034", 2023-10-12T00:38:40.026 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:1", 2023-10-12T00:38:40.027 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000101", 2023-10-12T00:38:40.027 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:2", 2023-10-12T00:38:40.027 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:3", 2023-10-12T00:38:40.027 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1.ifI++sGDr+s.0", 2023-10-12T00:38:40.027 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:3", 2023-10-12T00:38:40.028 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:2", 2023-10-12T00:38:40.028 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000079", 2023-10-12T00:38:40.028 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000049", 2023-10-12T00:38:40.028 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:9", 2023-10-12T00:38:40.028 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000117", 2023-10-12T00:38:40.029 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000044", 2023-10-12T00:38:40.029 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:5", 2023-10-12T00:38:40.029 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:6", 2023-10-12T00:38:40.029 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:10", 2023-10-12T00:38:40.030 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000066", 2023-10-12T00:38:40.030 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000068", 2023-10-12T00:38:40.030 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000085", 2023-10-12T00:38:40.030 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:9", 2023-10-12T00:38:40.030 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:0", 2023-10-12T00:38:40.031 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000073", 2023-10-12T00:38:40.031 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000038", 2023-10-12T00:38:40.031 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11", 2023-10-12T00:38:40.031 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T00:38:40.032 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:6", 2023-10-12T00:38:40.032 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:5", 2023-10-12T00:38:40.032 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:10", 2023-10-12T00:38:40.032 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000118", 2023-10-12T00:38:40.032 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:6", 2023-10-12T00:38:40.033 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:8", 2023-10-12T00:38:40.033 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:9", 2023-10-12T00:38:40.033 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:1", 2023-10-12T00:38:40.033 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000036", 2023-10-12T00:38:40.034 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:2", 2023-10-12T00:38:40.034 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000103", 2023-10-12T00:38:40.034 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:1", 2023-10-12T00:38:40.034 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:2", 2023-10-12T00:38:40.034 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000119", 2023-10-12T00:38:40.035 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:8", 2023-10-12T00:38:40.035 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000098", 2023-10-12T00:38:40.035 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32", 2023-10-12T00:38:40.035 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:2", 2023-10-12T00:38:40.036 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.1.retry", 2023-10-12T00:38:40.036 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:3", 2023-10-12T00:38:40.036 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:3", 2023-10-12T00:38:40.036 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000027", 2023-10-12T00:38:40.036 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:5", 2023-10-12T00:38:40.037 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000019", 2023-10-12T00:38:40.037 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:10", 2023-10-12T00:38:40.037 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:2", 2023-10-12T00:38:40.037 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:7", 2023-10-12T00:38:40.037 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000039", 2023-10-12T00:38:40.038 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:8", 2023-10-12T00:38:40.038 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:1", 2023-10-12T00:38:40.038 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:4", 2023-10-12T00:38:40.038 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:5", 2023-10-12T00:38:40.038 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000100", 2023-10-12T00:38:40.039 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31", 2023-10-12T00:38:40.039 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:2", 2023-10-12T00:38:40.039 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:9", 2023-10-12T00:38:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000093", 2023-10-12T00:38:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:0", 2023-10-12T00:38:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:7", 2023-10-12T00:38:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:6", 2023-10-12T00:38:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:8", 2023-10-12T00:38:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000004", 2023-10-12T00:38:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000063", 2023-10-12T00:38:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000122", 2023-10-12T00:38:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:10", 2023-10-12T00:38:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:7", 2023-10-12T00:38:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000057", 2023-10-12T00:38:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:9", 2023-10-12T00:38:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:8", 2023-10-12T00:38:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:2", 2023-10-12T00:38:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000054", 2023-10-12T00:38:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000114", 2023-10-12T00:38:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.2", 2023-10-12T00:38:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0", 2023-10-12T00:38:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:9", 2023-10-12T00:38:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000012", 2023-10-12T00:38:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:9", 2023-10-12T00:38:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:5", 2023-10-12T00:38:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000084", 2023-10-12T00:38:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000043", 2023-10-12T00:38:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:7", 2023-10-12T00:38:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:2", 2023-10-12T00:38:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:5", 2023-10-12T00:38:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:4", 2023-10-12T00:38:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000111", 2023-10-12T00:38:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000048", 2023-10-12T00:38:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:7", 2023-10-12T00:38:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000020", 2023-10-12T00:38:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:1", 2023-10-12T00:38:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:0", 2023-10-12T00:38:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:8", 2023-10-12T00:38:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:0", 2023-10-12T00:38:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:3", 2023-10-12T00:38:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13", 2023-10-12T00:38:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000099", 2023-10-12T00:38:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000056", 2023-10-12T00:38:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000072", 2023-10-12T00:38:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000062", 2023-10-12T00:38:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000115", 2023-10-12T00:38:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.1", 2023-10-12T00:38:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000033", 2023-10-12T00:38:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:0", 2023-10-12T00:38:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:7", 2023-10-12T00:38:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8", 2023-10-12T00:38:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:5", 2023-10-12T00:38:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:7", 2023-10-12T00:38:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000025", 2023-10-12T00:38:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000123", 2023-10-12T00:38:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:5", 2023-10-12T00:38:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000108", 2023-10-12T00:38:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:8", 2023-10-12T00:38:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:4", 2023-10-12T00:38:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000094", 2023-10-12T00:38:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:6", 2023-10-12T00:38:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:2", 2023-10-12T00:38:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7", 2023-10-12T00:38:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000060", 2023-10-12T00:38:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000109", 2023-10-12T00:38:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:6", 2023-10-12T00:38:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:1", 2023-10-12T00:38:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000013", 2023-10-12T00:38:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:6", 2023-10-12T00:38:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:3", 2023-10-12T00:38:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:0", 2023-10-12T00:38:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.3.retry", 2023-10-12T00:38:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:7", 2023-10-12T00:38:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000051", 2023-10-12T00:38:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000106", 2023-10-12T00:38:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:1", 2023-10-12T00:38:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:9", 2023-10-12T00:38:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000017", 2023-10-12T00:38:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:5", 2023-10-12T00:38:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18", 2023-10-12T00:38:40.057 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000023", 2023-10-12T00:38:40.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:10", 2023-10-12T00:38:40.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:10", 2023-10-12T00:38:40.057 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:2", 2023-10-12T00:38:40.058 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000107", 2023-10-12T00:38:40.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14", 2023-10-12T00:38:40.058 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000046", 2023-10-12T00:38:40.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:10", 2023-10-12T00:38:40.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:2", 2023-10-12T00:38:40.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:8", 2023-10-12T00:38:40.059 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2", 2023-10-12T00:38:40.059 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:5", 2023-10-12T00:38:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:4", 2023-10-12T00:38:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000067", 2023-10-12T00:38:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:5", 2023-10-12T00:38:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.c93c7bc5-2183-452b-b2b7-6a812a26f746.0", 2023-10-12T00:38:40.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.9", 2023-10-12T00:38:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000086", 2023-10-12T00:38:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.5", 2023-10-12T00:38:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:9", 2023-10-12T00:38:40.061 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:8", 2023-10-12T00:38:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.0", 2023-10-12T00:38:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:6", 2023-10-12T00:38:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.3", 2023-10-12T00:38:40.062 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:7", 2023-10-12T00:38:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:8", 2023-10-12T00:38:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:1", 2023-10-12T00:38:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000050", 2023-10-12T00:38:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000102", 2023-10-12T00:38:40.063 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:4", 2023-10-12T00:38:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:4", 2023-10-12T00:38:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:6", 2023-10-12T00:38:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:9", 2023-10-12T00:38:40.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:3", 2023-10-12T00:38:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:8", 2023-10-12T00:38:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000124", 2023-10-12T00:38:40.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:1", 2023-10-12T00:38:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000001", 2023-10-12T00:38:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:9", 2023-10-12T00:38:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000126", 2023-10-12T00:38:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000055", 2023-10-12T00:38:40.066 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000061", 2023-10-12T00:38:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:10", 2023-10-12T00:38:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000097", 2023-10-12T00:38:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000042", 2023-10-12T00:38:40.067 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:5", 2023-10-12T00:38:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:6", 2023-10-12T00:38:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3.GHBGPfTPT61.0", 2023-10-12T00:38:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000075", 2023-10-12T00:38:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0.hR1G6ZD+ilt.0", 2023-10-12T00:38:40.068 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000120", 2023-10-12T00:38:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000030" 2023-10-12T00:38:40.069 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:38:40.069 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:38:40.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.071 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:40.119 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:40.119 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:40.160 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:38:40.160 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000078", 2023-10-12T00:38:40.161 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.history", 2023-10-12T00:38:40.161 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000070", 2023-10-12T00:38:40.161 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000104", 2023-10-12T00:38:40.161 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:6", 2023-10-12T00:38:40.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:8", 2023-10-12T00:38:40.162 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000026", 2023-10-12T00:38:40.162 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.8", 2023-10-12T00:38:40.162 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000028", 2023-10-12T00:38:40.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:3", 2023-10-12T00:38:40.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:1", 2023-10-12T00:38:40.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:10", 2023-10-12T00:38:40.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:3", 2023-10-12T00:38:40.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:7", 2023-10-12T00:38:40.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:2", 2023-10-12T00:38:40.164 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000040", 2023-10-12T00:38:40.164 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.6", 2023-10-12T00:38:40.164 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000015", 2023-10-12T00:38:40.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13", 2023-10-12T00:38:40.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000069", 2023-10-12T00:38:40.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000095", 2023-10-12T00:38:40.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000003", 2023-10-12T00:38:40.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000047", 2023-10-12T00:38:40.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:5", 2023-10-12T00:38:40.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:6", 2023-10-12T00:38:40.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:9", 2023-10-12T00:38:40.166 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000035", 2023-10-12T00:38:40.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:10", 2023-10-12T00:38:40.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000037", 2023-10-12T00:38:40.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:6", 2023-10-12T00:38:40.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16", 2023-10-12T00:38:40.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:7", 2023-10-12T00:38:40.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:2", 2023-10-12T00:38:40.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9", 2023-10-12T00:38:40.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000024", 2023-10-12T00:38:40.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000000", 2023-10-12T00:38:40.169 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000031", 2023-10-12T00:38:40.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:8", 2023-10-12T00:38:40.169 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3", 2023-10-12T00:38:40.170 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.2.retry", 2023-10-12T00:38:40.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8", 2023-10-12T00:38:40.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:8", 2023-10-12T00:38:40.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:6", 2023-10-12T00:38:40.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:1", 2023-10-12T00:38:40.171 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.1", 2023-10-12T00:38:40.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:2", 2023-10-12T00:38:40.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:7", 2023-10-12T00:38:40.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:0", 2023-10-12T00:38:40.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:3", 2023-10-12T00:38:40.172 INFO:teuthology.orchestra.run.smithi112.stdout: "data_loggenerations_metadata", 2023-10-12T00:38:40.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:9", 2023-10-12T00:38:40.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:3", 2023-10-12T00:38:40.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:10", 2023-10-12T00:38:40.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:3", 2023-10-12T00:38:40.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34", 2023-10-12T00:38:40.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000076", 2023-10-12T00:38:40.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000113", 2023-10-12T00:38:40.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:7", 2023-10-12T00:38:40.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000005", 2023-10-12T00:38:40.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000011", 2023-10-12T00:38:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000058", 2023-10-12T00:38:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000052", 2023-10-12T00:38:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:6", 2023-10-12T00:38:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:7", 2023-10-12T00:38:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2.mXzpl+b/9sU.0", 2023-10-12T00:38:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:6", 2023-10-12T00:38:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000088", 2023-10-12T00:38:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000080", 2023-10-12T00:38:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000090", 2023-10-12T00:38:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000110", 2023-10-12T00:38:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000096", 2023-10-12T00:38:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:8", 2023-10-12T00:38:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:4", 2023-10-12T00:38:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000087", 2023-10-12T00:38:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000008", 2023-10-12T00:38:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:9", 2023-10-12T00:38:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:10", 2023-10-12T00:38:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000006", 2023-10-12T00:38:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000029", 2023-10-12T00:38:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:10", 2023-10-12T00:38:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000089", 2023-10-12T00:38:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000022", 2023-10-12T00:38:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:1", 2023-10-12T00:38:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:8", 2023-10-12T00:38:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog.trim", 2023-10-12T00:38:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000016", 2023-10-12T00:38:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:7", 2023-10-12T00:38:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000041", 2023-10-12T00:38:40.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:3", 2023-10-12T00:38:40.182 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000018", 2023-10-12T00:38:40.182 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000092", 2023-10-12T00:38:40.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:4", 2023-10-12T00:38:40.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:1", 2023-10-12T00:38:40.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:3", 2023-10-12T00:38:40.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:0", 2023-10-12T00:38:40.183 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.4", 2023-10-12T00:38:40.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:2", 2023-10-12T00:38:40.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:4", 2023-10-12T00:38:40.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000014", 2023-10-12T00:38:40.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000112", 2023-10-12T00:38:40.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000007", 2023-10-12T00:38:40.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000021", 2023-10-12T00:38:40.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000064", 2023-10-12T00:38:40.185 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:0", 2023-10-12T00:38:40.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000071", 2023-10-12T00:38:40.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000074", 2023-10-12T00:38:40.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:1", 2023-10-12T00:38:40.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:8", 2023-10-12T00:38:40.186 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T00:38:40.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:7", 2023-10-12T00:38:40.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000081", 2023-10-12T00:38:40.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000009", 2023-10-12T00:38:40.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:9", 2023-10-12T00:38:40.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000121", 2023-10-12T00:38:40.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:4", 2023-10-12T00:38:40.188 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1", 2023-10-12T00:38:40.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:1", 2023-10-12T00:38:40.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:1", 2023-10-12T00:38:40.188 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000125", 2023-10-12T00:38:40.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000082", 2023-10-12T00:38:40.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:3", 2023-10-12T00:38:40.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:5", 2023-10-12T00:38:40.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:10", 2023-10-12T00:38:40.189 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.3.retry", 2023-10-12T00:38:40.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:2", 2023-10-12T00:38:40.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000105", 2023-10-12T00:38:40.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000059", 2023-10-12T00:38:40.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32", 2023-10-12T00:38:40.191 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.0.retry", 2023-10-12T00:38:40.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000077", 2023-10-12T00:38:40.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:1", 2023-10-12T00:38:40.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000032", 2023-10-12T00:38:40.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:5", 2023-10-12T00:38:40.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:9", 2023-10-12T00:38:40.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:0", 2023-10-12T00:38:40.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:0", 2023-10-12T00:38:40.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:9", 2023-10-12T00:38:40.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:4", 2023-10-12T00:38:40.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:0", 2023-10-12T00:38:40.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:2", 2023-10-12T00:38:40.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:0", 2023-10-12T00:38:40.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:9", 2023-10-12T00:38:40.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:8", 2023-10-12T00:38:40.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000053", 2023-10-12T00:38:40.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000091", 2023-10-12T00:38:40.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:6", 2023-10-12T00:38:40.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000065", 2023-10-12T00:38:40.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000083", 2023-10-12T00:38:40.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000010", 2023-10-12T00:38:40.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:8", 2023-10-12T00:38:40.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1", 2023-10-12T00:38:40.196 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.7", 2023-10-12T00:38:40.196 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.3", 2023-10-12T00:38:40.196 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:4", 2023-10-12T00:38:40.196 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:0", 2023-10-12T00:38:40.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:3", 2023-10-12T00:38:40.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:9", 2023-10-12T00:38:40.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:7", 2023-10-12T00:38:40.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:10", 2023-10-12T00:38:40.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:0", 2023-10-12T00:38:40.198 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.2", 2023-10-12T00:38:40.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17", 2023-10-12T00:38:40.198 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.2", 2023-10-12T00:38:40.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:1", 2023-10-12T00:38:40.199 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:5", 2023-10-12T00:38:40.199 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000045", 2023-10-12T00:38:40.199 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000002", 2023-10-12T00:38:40.199 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:3", 2023-10-12T00:38:40.199 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.0", 2023-10-12T00:38:40.200 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:10", 2023-10-12T00:38:40.200 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:9", 2023-10-12T00:38:40.200 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:5", 2023-10-12T00:38:40.200 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:8", 2023-10-12T00:38:40.200 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000116", 2023-10-12T00:38:40.201 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:0", 2023-10-12T00:38:40.201 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:10", 2023-10-12T00:38:40.201 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:9", 2023-10-12T00:38:40.201 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:10", 2023-10-12T00:38:40.202 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000034", 2023-10-12T00:38:40.202 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:10", 2023-10-12T00:38:40.202 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000101", 2023-10-12T00:38:40.202 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:7", 2023-10-12T00:38:40.202 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.1", 2023-10-12T00:38:40.203 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:0", 2023-10-12T00:38:40.203 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:8", 2023-10-12T00:38:40.203 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:6", 2023-10-12T00:38:40.203 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:3", 2023-10-12T00:38:40.204 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000079", 2023-10-12T00:38:40.204 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000049", 2023-10-12T00:38:40.204 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000117", 2023-10-12T00:38:40.204 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000044", 2023-10-12T00:38:40.204 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:5", 2023-10-12T00:38:40.205 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000066", 2023-10-12T00:38:40.205 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:9", 2023-10-12T00:38:40.205 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000068", 2023-10-12T00:38:40.205 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000085", 2023-10-12T00:38:40.206 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:4", 2023-10-12T00:38:40.206 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12", 2023-10-12T00:38:40.206 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:4", 2023-10-12T00:38:40.206 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.2", 2023-10-12T00:38:40.207 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000073", 2023-10-12T00:38:40.207 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000038", 2023-10-12T00:38:40.207 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.10", 2023-10-12T00:38:40.207 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000118", 2023-10-12T00:38:40.207 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33", 2023-10-12T00:38:40.208 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000036", 2023-10-12T00:38:40.208 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:1", 2023-10-12T00:38:40.208 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:1", 2023-10-12T00:38:40.208 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:2", 2023-10-12T00:38:40.209 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000103", 2023-10-12T00:38:40.209 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.3", 2023-10-12T00:38:40.209 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:9", 2023-10-12T00:38:40.209 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:10", 2023-10-12T00:38:40.210 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:6", 2023-10-12T00:38:40.210 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000119", 2023-10-12T00:38:40.210 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000098", 2023-10-12T00:38:40.210 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:10", 2023-10-12T00:38:40.211 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:9", 2023-10-12T00:38:40.211 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:3", 2023-10-12T00:38:40.211 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:2", 2023-10-12T00:38:40.211 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.0", 2023-10-12T00:38:40.211 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31", 2023-10-12T00:38:40.212 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000027", 2023-10-12T00:38:40.212 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status", 2023-10-12T00:38:40.212 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000019", 2023-10-12T00:38:40.212 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000039", 2023-10-12T00:38:40.212 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:3", 2023-10-12T00:38:40.213 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000100", 2023-10-12T00:38:40.213 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:8", 2023-10-12T00:38:40.213 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3.kWs9Jb+/V/a.0", 2023-10-12T00:38:40.213 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:6", 2023-10-12T00:38:40.213 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:0", 2023-10-12T00:38:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:10", 2023-10-12T00:38:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:2", 2023-10-12T00:38:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000093", 2023-10-12T00:38:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000004", 2023-10-12T00:38:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000063", 2023-10-12T00:38:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000122", 2023-10-12T00:38:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:5", 2023-10-12T00:38:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000057", 2023-10-12T00:38:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:1", 2023-10-12T00:38:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:3", 2023-10-12T00:38:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000054", 2023-10-12T00:38:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000114", 2023-10-12T00:38:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.2", 2023-10-12T00:38:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:0", 2023-10-12T00:38:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0", 2023-10-12T00:38:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:4", 2023-10-12T00:38:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:6", 2023-10-12T00:38:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:4", 2023-10-12T00:38:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000012", 2023-10-12T00:38:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:5", 2023-10-12T00:38:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:8", 2023-10-12T00:38:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:5", 2023-10-12T00:38:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000084", 2023-10-12T00:38:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000043", 2023-10-12T00:38:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:9", 2023-10-12T00:38:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:2", 2023-10-12T00:38:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.11", 2023-10-12T00:38:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:6", 2023-10-12T00:38:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:2", 2023-10-12T00:38:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:7", 2023-10-12T00:38:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000111", 2023-10-12T00:38:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000048", 2023-10-12T00:38:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000020", 2023-10-12T00:38:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:6", 2023-10-12T00:38:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:1", 2023-10-12T00:38:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000099", 2023-10-12T00:38:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000056", 2023-10-12T00:38:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:5", 2023-10-12T00:38:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:7", 2023-10-12T00:38:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:0", 2023-10-12T00:38:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7", 2023-10-12T00:38:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000072", 2023-10-12T00:38:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000062", 2023-10-12T00:38:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000115", 2023-10-12T00:38:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.1", 2023-10-12T00:38:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000033", 2023-10-12T00:38:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:2", 2023-10-12T00:38:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:3", 2023-10-12T00:38:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1.jIwIfY6zuH1.0", 2023-10-12T00:38:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:2", 2023-10-12T00:38:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000025", 2023-10-12T00:38:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000123", 2023-10-12T00:38:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:1", 2023-10-12T00:38:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:1", 2023-10-12T00:38:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000108", 2023-10-12T00:38:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000094", 2023-10-12T00:38:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:0", 2023-10-12T00:38:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-34:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.34:4", 2023-10-12T00:38:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:4", 2023-10-12T00:38:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:4", 2023-10-12T00:38:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:6", 2023-10-12T00:38:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000060", 2023-10-12T00:38:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:10", 2023-10-12T00:38:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15:0", 2023-10-12T00:38:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:1", 2023-10-12T00:38:40.228 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000109", 2023-10-12T00:38:40.228 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000013", 2023-10-12T00:38:40.228 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:7", 2023-10-12T00:38:40.228 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:7", 2023-10-12T00:38:40.229 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10", 2023-10-12T00:38:40.229 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:7", 2023-10-12T00:38:40.229 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-15:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.15", 2023-10-12T00:38:40.229 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:3", 2023-10-12T00:38:40.229 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:2", 2023-10-12T00:38:40.230 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000051", 2023-10-12T00:38:40.230 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000106", 2023-10-12T00:38:40.230 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:5", 2023-10-12T00:38:40.230 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:8", 2023-10-12T00:38:40.230 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000017", 2023-10-12T00:38:40.231 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-7:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.7:6", 2023-10-12T00:38:40.231 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:4", 2023-10-12T00:38:40.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000023", 2023-10-12T00:38:40.231 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11", 2023-10-12T00:38:40.232 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.1.retry", 2023-10-12T00:38:40.232 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:8", 2023-10-12T00:38:40.232 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-16:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.16:5", 2023-10-12T00:38:40.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000107", 2023-10-12T00:38:40.232 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:4", 2023-10-12T00:38:40.233 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:6", 2023-10-12T00:38:40.233 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000046", 2023-10-12T00:38:40.233 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:8", 2023-10-12T00:38:40.233 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-31:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.31:5", 2023-10-12T00:38:40.233 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14", 2023-10-12T00:38:40.234 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:3", 2023-10-12T00:38:40.234 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2", 2023-10-12T00:38:40.234 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000067", 2023-10-12T00:38:40.234 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:2", 2023-10-12T00:38:40.234 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-8:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.8:7", 2023-10-12T00:38:40.235 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:4", 2023-10-12T00:38:40.235 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:9", 2023-10-12T00:38:40.235 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.9", 2023-10-12T00:38:40.235 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000086", 2023-10-12T00:38:40.236 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.5", 2023-10-12T00:38:40.236 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:9", 2023-10-12T00:38:40.236 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.677f5a1d-3ab4-4b5f-af15-9890c2c79648.3", 2023-10-12T00:38:40.236 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-13:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.13:5", 2023-10-12T00:38:40.236 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.0", 2023-10-12T00:38:40.237 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:2", 2023-10-12T00:38:40.237 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.f8a42833-a8a4-4b21-a670-8605a7b80b21.3", 2023-10-12T00:38:40.237 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-14:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.14:0", 2023-10-12T00:38:40.237 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18", 2023-10-12T00:38:40.237 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000050", 2023-10-12T00:38:40.238 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:6", 2023-10-12T00:38:40.238 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000102", 2023-10-12T00:38:40.238 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-17:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.17:5", 2023-10-12T00:38:40.238 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000124", 2023-10-12T00:38:40.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000001", 2023-10-12T00:38:40.239 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-18:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.18:7", 2023-10-12T00:38:40.239 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:5", 2023-10-12T00:38:40.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000126", 2023-10-12T00:38:40.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000055", 2023-10-12T00:38:40.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000061", 2023-10-12T00:38:40.240 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:10", 2023-10-12T00:38:40.240 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-35:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5439.1:10", 2023-10-12T00:38:40.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000097", 2023-10-12T00:38:40.240 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-32:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.32:2", 2023-10-12T00:38:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:8", 2023-10-12T00:38:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000042", 2023-10-12T00:38:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:7", 2023-10-12T00:38:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-10:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.10:5", 2023-10-12T00:38:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000075", 2023-10-12T00:38:40.242 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-11:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.11:8", 2023-10-12T00:38:40.242 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000120", 2023-10-12T00:38:40.242 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000030", 2023-10-12T00:38:40.242 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-9:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.9:9", 2023-10-12T00:38:40.243 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-33:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.33:4", 2023-10-12T00:38:40.243 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0.6dgBKZGXfzM.0", 2023-10-12T00:38:40.243 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-12:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.12:1" 2023-10-12T00:38:40.243 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:38:40.243 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-10-12T00:38:40.245 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T00:38:40.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:40.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:40.340 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:38:40.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-10-12T00:38:40.340 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:39.879609Z" 2023-10-12T00:38:40.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-12T00:38:40.341 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:39.880468Z" 2023-10-12T00:38:40.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.342 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-10-12T00:38:40.342 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:39.881495Z" 2023-10-12T00:38:40.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.343 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-12T00:38:40.343 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:39.882531Z" 2023-10-12T00:38:40.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.343 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:38:40.345 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-10-12T00:38:40.345 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:38:40.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:40.347 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:40.397 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:40.398 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:40.439 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:40.440 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:40.440 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:40.440 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:40.441 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:40.441 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:38:40.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.441 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:40.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.442 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:40.442 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:40.442 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:40.442 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:38:40.443 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:40.443 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:40.443 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:40.443 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:23.283784Z" 2023-10-12T00:38:40.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.444 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:40.444 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:40.445 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:40.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-12T00:38:40.445 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:40.445 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:40.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:40.446 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:44.490949Z" 2023-10-12T00:38:40.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.447 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:40.447 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:40.447 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:40.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-12T00:38:40.447 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:40.448 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:40.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:40.448 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:34:13.466209Z" 2023-10-12T00:38:40.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:40.449 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:40.449 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:40.449 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:40.450 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:38:40.450 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:40.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:40.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:40.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:43.375485Z" 2023-10-12T00:38:40.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:40.451 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:40.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.451 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:40.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:38:40.452 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:38:40.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:40.453 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:40Z" 2023-10-12T00:38:40.453 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:40.454 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:23.283784Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:44.490949Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:34:13.466209Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:43.375485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:38:40Z" } 2023-10-12T00:38:40.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:23.283784Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:44.490949Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:34:13.466209Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:43.375485Z'}}] 2023-10-12T00:38:40.455 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-10-12T00:38:40.456 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000003324', 2: '00000000000000000000:00000000000000002625', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:38:40.457 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=1 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=3 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210 2023-10-12T00:38:45.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.459 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:45.565 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:45.565 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:38:45.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:45.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-10-12T00:38:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:39.879609Z" 2023-10-12T00:38:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-12T00:38:45.570 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:39.880468Z" 2023-10-12T00:38:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.572 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-10-12T00:38:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:39.881495Z" 2023-10-12T00:38:45.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:45.574 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-12T00:38:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.575 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:39.882531Z" 2023-10-12T00:38:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:38:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:38:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.577 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:45Z" 2023-10-12T00:38:45.578 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:45.579 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:39.879609Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004167", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:39.880468Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003467", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:39.881495Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:39.882531Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:38:45Z" } 2023-10-12T00:38:45.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:39.879609Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004167', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:39.880468Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:39.881495Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:39.882531Z'}}] 2023-10-12T00:38:45.580 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-10-12T00:38:45.580 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001211', 1: '00000000000000000000:00000000000000004167', 2: '00000000000000000000:00000000000000003467', 3: '00000000000000000000:00000000000000002055'} 2023-10-12T00:38:45.581 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:38:45.581 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T00:38:45.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.582 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:45.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:45.678 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:38:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:38:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:38:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-12T00:38:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:44.418283Z" 2023-10-12T00:38:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-12T00:38:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:34:13.888152Z" 2023-10-12T00:38:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:38:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:30:43.347442Z" 2023-10-12T00:38:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.682 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:38:45.683 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:38:45.684 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:38:45.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:45.685 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:45.737 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:45.738 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:45.778 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:45.778 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:45.779 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:45.779 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:45.779 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:45.779 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:38:45.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.780 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:45.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.780 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:45.780 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.781 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.781 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-12T00:38:45.781 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.781 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.782 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.782 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:21:03.300974Z" 2023-10-12T00:38:45.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.783 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:45.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.783 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-12T00:38:45.784 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.784 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:44.418283Z" 2023-10-12T00:38:45.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.785 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:45.785 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.786 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-12T00:38:45.786 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.786 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:34:13.888152Z" 2023-10-12T00:38:45.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:45.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:45.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:45.788 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:38:45.788 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-12T00:38:45.789 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:45.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:45.789 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:45.789 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:30:43.347442Z" 2023-10-12T00:38:45.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:45.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:45.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.791 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:45.791 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:38:45.792 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:38:45.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:45.792 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:45Z" 2023-10-12T00:38:45.792 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:45.793 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:21:03.300974Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:44.418283Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:34:13.888152Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:30:43.347442Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:38:45Z" } 2023-10-12T00:38:45.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:21:03.300974Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:44.418283Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:34:13.888152Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:30:43.347442Z'}}] 2023-10-12T00:38:45.794 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:38:45.795 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000651', 1: '00000000000000000000:00000000000000004029', 2: '00000000000000000000:00000000000000001640', 3: '00000000000000000000:00000000000000001210'} 2023-10-12T00:38:45.795 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:38:45.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-10-12T00:38:45.797 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-46 2023-10-12T00:38:45.798 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:38:45.798 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:38:45.798 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.799 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:38:45.799 DEBUG:boto:Data: 2023-10-12T00:38:45.800 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.800 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.801 DEBUG:boto:Params: {} 2023-10-12T00:38:45.802 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:38:45.802 DEBUG:boto:Token: None 2023-10-12T00:38:45.803 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/ 2023-10-12T00:38:45.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y8vhTRA4lWtyAOQ6PWvKRfiz3Ac= 2023-10-12T00:38:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y8vhTRA4lWtyAOQ6PWvKRfiz3Ac=', 'Content-Length': '0'} 2023-10-12T00:38:45.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4708440bc374aa8-0065274015-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.820 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-47 2023-10-12T00:38:45.820 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:38:45.820 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:38:45.821 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.821 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:38:45.822 DEBUG:boto:Data: 2023-10-12T00:38:45.822 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:45.823 DEBUG:boto:Port: 8001 2023-10-12T00:38:45.823 DEBUG:boto:Params: {} 2023-10-12T00:38:45.824 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:38:45.824 DEBUG:boto:Token: None 2023-10-12T00:38:45.825 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-47/ 2023-10-12T00:38:45.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YXTvna5c5FIhnmCnnatYy/Hxjw4= 2023-10-12T00:38:45.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YXTvna5c5FIhnmCnnatYy/Hxjw4=', 'Content-Length': '0'} 2023-10-12T00:38:45.855 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077703f23f6c36f29-0065274015-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.856 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:38:45.856 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:38:45.857 DEBUG:boto:Method: HEAD 2023-10-12T00:38:45.857 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:38:45.858 DEBUG:boto:Data: 2023-10-12T00:38:45.858 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.859 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.859 DEBUG:boto:Params: {} 2023-10-12T00:38:45.860 DEBUG:boto:Token: None 2023-10-12T00:38:45.860 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/ 2023-10-12T00:38:45.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs= 2023-10-12T00:38:45.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs=', 'Content-Length': '0'} 2023-10-12T00:38:45.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d98d19a616f583a-0065274015-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.865 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:38:45.865 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:38:45.866 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.866 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:38:45.866 DEBUG:boto:Data: 2023-10-12T00:38:45.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:45.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.868 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.868 DEBUG:boto:Params: {} 2023-10-12T00:38:45.869 DEBUG:boto:Token: None 2023-10-12T00:38:45.870 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/obj1 2023-10-12T00:38:45.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iiTET5rzUw8QArQkuI1o5jZkFC4= 2023-10-12T00:38:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iiTET5rzUw8QArQkuI1o5jZkFC4='} 2023-10-12T00:38:45.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0e862166d604e98-0065274015-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.878 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:38:45.878 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:38:45.878 DEBUG:boto:Method: HEAD 2023-10-12T00:38:45.880 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:38:45.880 DEBUG:boto:Data: 2023-10-12T00:38:45.880 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.881 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.882 DEBUG:boto:Params: {} 2023-10-12T00:38:45.882 DEBUG:boto:Token: None 2023-10-12T00:38:45.882 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/ 2023-10-12T00:38:45.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs= 2023-10-12T00:38:45.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs=', 'Content-Length': '0'} 2023-10-12T00:38:45.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0fdc3928472492d-0065274015-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.886 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:38:45.887 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:38:45.887 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.888 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:38:45.888 DEBUG:boto:Data: 2023-10-12T00:38:45.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:45.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.890 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.890 DEBUG:boto:Params: {} 2023-10-12T00:38:45.891 DEBUG:boto:Token: None 2023-10-12T00:38:45.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/obj2 2023-10-12T00:38:45.892 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oadkfisG5Jbagi3DuWM2YUCG+Hs= 2023-10-12T00:38:45.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oadkfisG5Jbagi3DuWM2YUCG+Hs='} 2023-10-12T00:38:45.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f0b77b49792fd49-0065274015-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.901 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:38:45.901 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:38:45.902 DEBUG:boto:Method: HEAD 2023-10-12T00:38:45.902 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:38:45.903 DEBUG:boto:Data: 2023-10-12T00:38:45.903 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.904 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.904 DEBUG:boto:Params: {} 2023-10-12T00:38:45.905 DEBUG:boto:Token: None 2023-10-12T00:38:45.905 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/ 2023-10-12T00:38:45.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs= 2023-10-12T00:38:45.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs=', 'Content-Length': '0'} 2023-10-12T00:38:45.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f94d3b3c95cbd43-0065274015-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.909 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:38:45.909 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:38:45.910 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.910 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:38:45.911 DEBUG:boto:Data: 2023-10-12T00:38:45.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:45.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.912 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.912 DEBUG:boto:Params: {} 2023-10-12T00:38:45.913 DEBUG:boto:Token: None 2023-10-12T00:38:45.913 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/obj3 2023-10-12T00:38:45.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+XXl2T4L3fTBC2S5aNbHPfHnr3Y= 2023-10-12T00:38:45.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+XXl2T4L3fTBC2S5aNbHPfHnr3Y='} 2023-10-12T00:38:45.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069417ff7716c5840-0065274015-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.920 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:38:45.921 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:38:45.921 DEBUG:boto:Method: HEAD 2023-10-12T00:38:45.922 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:38:45.922 DEBUG:boto:Data: 2023-10-12T00:38:45.923 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.924 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.924 DEBUG:boto:Params: {} 2023-10-12T00:38:45.925 DEBUG:boto:Token: None 2023-10-12T00:38:45.925 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/ 2023-10-12T00:38:45.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs= 2023-10-12T00:38:45.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:frK4BYItbX588Qt6HIMjrYNAmZs=', 'Content-Length': '0'} 2023-10-12T00:38:45.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083ab5bbbbf7bc94c-0065274015-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.929 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:38:45.930 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:38:45.930 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.930 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:38:45.931 DEBUG:boto:Data: 2023-10-12T00:38:45.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:45.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:38:45.932 DEBUG:boto:Port: 8000 2023-10-12T00:38:45.933 DEBUG:boto:Params: {} 2023-10-12T00:38:45.934 DEBUG:boto:Token: None 2023-10-12T00:38:45.934 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-46/obj4 2023-10-12T00:38:45.935 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d+eaXQ+cMN9vJy4hUTvb57drSvk= 2023-10-12T00:38:45.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d+eaXQ+cMN9vJy4hUTvb57drSvk='} 2023-10-12T00:38:45.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3a16c27a704d312-0065274015-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.979 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:38:45.980 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:38:45.980 DEBUG:boto:Method: HEAD 2023-10-12T00:38:45.981 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:38:45.981 DEBUG:boto:Data: 2023-10-12T00:38:45.981 DEBUG:boto:Headers: {} 2023-10-12T00:38:45.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:45.982 DEBUG:boto:Port: 8001 2023-10-12T00:38:45.983 DEBUG:boto:Params: {} 2023-10-12T00:38:45.983 DEBUG:boto:Token: None 2023-10-12T00:38:45.984 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-47/ 2023-10-12T00:38:45.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gx+YZltD6R/Mb8GivCLkBs4Qd7s= 2023-10-12T00:38:45.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gx+YZltD6R/Mb8GivCLkBs4Qd7s=', 'Content-Length': '0'} 2023-10-12T00:38:45.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9177dd566326171-0065274015-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.988 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:38:45.988 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:38:45.988 DEBUG:boto:Method: PUT 2023-10-12T00:38:45.989 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:38:45.989 DEBUG:boto:Data: 2023-10-12T00:38:45.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:45.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:45.991 DEBUG:boto:Port: 8001 2023-10-12T00:38:45.991 DEBUG:boto:Params: {} 2023-10-12T00:38:45.992 DEBUG:boto:Token: None 2023-10-12T00:38:45.992 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:45 GMT /wafmoa-47/obj1 2023-10-12T00:38:45.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g/l19DNXHngtTg3bswvHmWLQeXA= 2023-10-12T00:38:45.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g/l19DNXHngtTg3bswvHmWLQeXA='} 2023-10-12T00:38:45.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df258090b1a4d24b-0065274015-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:45.999 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:38:46.000 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:38:46.000 DEBUG:boto:Method: HEAD 2023-10-12T00:38:46.001 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:38:46.001 DEBUG:boto:Data: 2023-10-12T00:38:46.002 DEBUG:boto:Headers: {} 2023-10-12T00:38:46.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.002 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.003 DEBUG:boto:Params: {} 2023-10-12T00:38:46.003 DEBUG:boto:Token: None 2023-10-12T00:38:46.004 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/ 2023-10-12T00:38:46.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4= 2023-10-12T00:38:46.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4=', 'Content-Length': '0'} 2023-10-12T00:38:46.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000268cf760c7568e3b-0065274016-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.008 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:38:46.008 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:38:46.009 DEBUG:boto:Method: PUT 2023-10-12T00:38:46.009 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:38:46.010 DEBUG:boto:Data: 2023-10-12T00:38:46.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:46.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.011 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.011 DEBUG:boto:Params: {} 2023-10-12T00:38:46.012 DEBUG:boto:Token: None 2023-10-12T00:38:46.012 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/obj2 2023-10-12T00:38:46.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dVdi9jSB2UP5AHp8CduGxCCn9fk= 2023-10-12T00:38:46.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dVdi9jSB2UP5AHp8CduGxCCn9fk='} 2023-10-12T00:38:46.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e909f8db773d7fd-0065274016-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.059 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:38:46.060 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:38:46.060 DEBUG:boto:Method: HEAD 2023-10-12T00:38:46.061 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:38:46.061 DEBUG:boto:Data: 2023-10-12T00:38:46.061 DEBUG:boto:Headers: {} 2023-10-12T00:38:46.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.062 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.063 DEBUG:boto:Params: {} 2023-10-12T00:38:46.063 DEBUG:boto:Token: None 2023-10-12T00:38:46.064 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/ 2023-10-12T00:38:46.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4= 2023-10-12T00:38:46.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4=', 'Content-Length': '0'} 2023-10-12T00:38:46.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1615a41f7564e27-0065274016-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.067 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:38:46.068 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:38:46.068 DEBUG:boto:Method: PUT 2023-10-12T00:38:46.069 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:38:46.069 DEBUG:boto:Data: 2023-10-12T00:38:46.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:46.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.071 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.071 DEBUG:boto:Params: {} 2023-10-12T00:38:46.071 DEBUG:boto:Token: None 2023-10-12T00:38:46.072 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/obj3 2023-10-12T00:38:46.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wnZLFVqW2BtuQLCv6Ocufa6CNXw= 2023-10-12T00:38:46.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wnZLFVqW2BtuQLCv6Ocufa6CNXw='} 2023-10-12T00:38:46.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bda1d1a29f36491-0065274016-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.080 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:38:46.081 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:38:46.081 DEBUG:boto:Method: HEAD 2023-10-12T00:38:46.081 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:38:46.082 DEBUG:boto:Data: 2023-10-12T00:38:46.082 DEBUG:boto:Headers: {} 2023-10-12T00:38:46.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.083 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.084 DEBUG:boto:Params: {} 2023-10-12T00:38:46.084 DEBUG:boto:Token: None 2023-10-12T00:38:46.084 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/ 2023-10-12T00:38:46.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4= 2023-10-12T00:38:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AvSy2y5hIyBCKlaCZa90XdO/bs4=', 'Content-Length': '0'} 2023-10-12T00:38:46.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b01754efd7381f30-0065274016-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.088 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:38:46.089 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:38:46.089 DEBUG:boto:Method: PUT 2023-10-12T00:38:46.090 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:38:46.090 DEBUG:boto:Data: 2023-10-12T00:38:46.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:38:46.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:38:46.091 DEBUG:boto:Port: 8001 2023-10-12T00:38:46.092 DEBUG:boto:Params: {} 2023-10-12T00:38:46.092 DEBUG:boto:Token: None 2023-10-12T00:38:46.093 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:38:46 GMT /wafmoa-47/obj4 2023-10-12T00:38:46.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RhJkkR6MdLU23LT6F7zce5ssiPA= 2023-10-12T00:38:46.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:38:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RhJkkR6MdLU23LT6F7zce5ssiPA='} 2023-10-12T00:38:46.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab34d4c30628062b-0065274016-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:38:46.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:46.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:46.140 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:46.219 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:38:46.231 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:38:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:38:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:46.233 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:38:46.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071125.846963_478.1", 2023-10-12T00:38:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:45.846963Z" 2023-10-12T00:38:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:38:46.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.235 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:38:46.237 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071125.846963_478.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:46.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:38:46.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:46.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:46.239 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:46.288 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:46.289 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:46.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:46.316+0000 7f3647f3a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:46.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:46.317+0000 7f3647f3a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:46.328 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:46.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:46.329 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:46.329 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:46.329 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:46.333 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:46.333 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:46.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:46.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:46.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:46.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:46.335 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:46.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:46.335 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:46.336 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:46.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:46.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:46.337 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:46.337 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:46.337 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071119.768498_467.1", 2023-10-12T00:38:46.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:46.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:46.338 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:46.338 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:39.768498Z", 2023-10-12T00:38:46.338 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:46.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:46.339 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:46.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:46.340 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:46.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:46.340 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:46.340 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:46.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:46.341 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:46.341 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:46.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:46.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:46.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.342 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:46.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:46.343 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:46.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:46.343 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:46Z" 2023-10-12T00:38:46.343 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:46.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071119.768498_467.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:39.768498Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:46Z" } 2023-10-12T00:38:46.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071119.768498_467.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:39.768498Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:46.345 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071125.846963_478.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:46.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071119.768498_467.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:46.347 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071125.846963_478.1 target=1_1697071119.768498_467.1 2023-10-12T00:38:51.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:51.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:38:51.349 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:38:51.407 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:51.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:51.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:51.439+0000 7fcf63f8a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:51.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:51.440+0000 7fcf63f8a780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:38:51.453 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:38:51.454 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:38:51.454 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:38:51.454 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:38:51.455 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:38:51.455 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:38:51.455 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:51.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.456 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:38:51.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:51.456 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:38:51.456 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:51.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:51.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:38:51.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:51.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:38:51.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:51.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:38:51.458 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:51.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:51.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:51.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:38:51.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:51.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:51.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:38:51.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:51.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:51.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:51.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:38:51.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:51.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:51.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:51.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:38:51.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:51.463 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:51.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071125.846963_478.1", 2023-10-12T00:38:51.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:51.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:51.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:51.464 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:45.846963Z", 2023-10-12T00:38:51.464 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:51.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:51.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:38:51.465 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:38:51.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:38:51.466 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:38:51.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:38:51.466 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:38:51.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:38:51.467 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:38:51.467 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:38:51.467 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:38:51.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:51.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:38:51.468 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:38:51.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.469 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:38:51.469 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:38:51.469 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:38:51.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:38:51.469 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:38:51Z" 2023-10-12T00:38:51.470 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:38:51.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071125.846963_478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:45.846963Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:38:51Z" } 2023-10-12T00:38:51.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071125.846963_478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:45.846963Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:38:51.472 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071125.846963_478.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:51.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071125.846963_478.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:38:51.473 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:38:51.473 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-46 2023-10-12T00:38:51.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:38:51.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:38:51.476 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:38:51.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:38:51.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:38:51.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:51.568+0000 7f5f8ef1b780 1 waiting to reach incremental sync.. 2023-10-12T00:38:53.599 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:38:53.599+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T00:38:53.742 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.742+0000 7feb81a33700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:53.743 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.742+0000 7feb8b246700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:53.743 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.742+0000 7feb84a39700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:53.743 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.742+0000 7feb83a37700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:53.743 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.743+0000 7feb8da4b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:53.744 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:38:53.743+0000 7feb8623c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:38:56.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:38:56.569+0000 7f5f8ef1b780 1 waiting to reach incremental sync.. 2023-10-12T00:39:01.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:01.569+0000 7f5f8ef1b780 1 waiting to reach incremental sync.. 2023-10-12T00:39:06.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:06.570+0000 7f5f8ef1b780 1 bucket sync caught up with source: 2023-10-12T00:39:06.572 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.23.6, , 00000000001.22.6, , 00000000001.25.6, , , 00000000001.18.6, , , ] 2023-10-12T00:39:06.572 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.23.6, , 00000000001.22.6, , 00000000001.25.6, , , 00000000001.18.6, , , ] 2023-10-12T00:39:06.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:06.570+0000 7f5f8ef1b780 0 bucket checkpoint complete 2023-10-12T00:39:06.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-46 2023-10-12T00:39:06.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:06.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:06.590 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:39:06.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:06.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:06.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:06.686+0000 7fd51875c780 1 waiting to reach incremental sync.. 2023-10-12T00:39:11.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:11.687+0000 7fd51875c780 1 waiting to reach incremental sync.. 2023-10-12T00:39:16.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:16.689+0000 7fd51875c780 1 bucket sync caught up with source: 2023-10-12T00:39:16.690 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.164.6, , 00000000001.256.6, , 00000000001.235.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:16.691 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.164.6, , 00000000001.256.6, , 00000000001.235.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:16.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:16.689+0000 7fd51875c780 0 bucket checkpoint complete 2023-10-12T00:39:16.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-46 zones={a2, a1} 2023-10-12T00:39:16.707 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.707 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.708 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.708 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:16.709 DEBUG:boto:Data: 2023-10-12T00:39:16.709 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.710 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.710 DEBUG:boto:Params: {} 2023-10-12T00:39:16.711 DEBUG:boto:Token: None 2023-10-12T00:39:16.711 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RF30eV7azBusxGFXKB9JAnXOCnc= 2023-10-12T00:39:16.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RF30eV7azBusxGFXKB9JAnXOCnc=', 'Content-Length': '0'} 2023-10-12T00:39:16.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c2a47c826f67320-0065274034-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.716 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.716 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.717 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.717 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:16.718 DEBUG:boto:Data: 2023-10-12T00:39:16.718 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.719 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.720 DEBUG:boto:Params: {} 2023-10-12T00:39:16.720 DEBUG:boto:Token: None 2023-10-12T00:39:16.721 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RF30eV7azBusxGFXKB9JAnXOCnc= 2023-10-12T00:39:16.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RF30eV7azBusxGFXKB9JAnXOCnc=', 'Content-Length': '0'} 2023-10-12T00:39:16.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000155a86737efa5ca-0065274034-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:39:16.725 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.725 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.726 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.726 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.727 DEBUG:boto:Method: GET 2023-10-12T00:39:16.727 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.728 DEBUG:boto:Data: 2023-10-12T00:39:16.728 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.729 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.729 DEBUG:boto:Params: {} 2023-10-12T00:39:16.730 DEBUG:boto:Token: None 2023-10-12T00:39:16.730 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/?versions 2023-10-12T00:39:16.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8= 2023-10-12T00:39:16.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8=', 'Content-Length': '0'} 2023-10-12T00:39:16.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3959a203720986a-0065274034-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.735 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:16.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:16.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:16.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:16.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:39:16.738 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.738 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.738 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.739 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.739 DEBUG:boto:Method: GET 2023-10-12T00:39:16.740 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.740 DEBUG:boto:Data: 2023-10-12T00:39:16.740 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.741 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.742 DEBUG:boto:Params: {} 2023-10-12T00:39:16.742 DEBUG:boto:Token: None 2023-10-12T00:39:16.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/?versions 2023-10-12T00:39:16.743 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8= 2023-10-12T00:39:16.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8=', 'Content-Length': '0'} 2023-10-12T00:39:16.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c0be949b3d3b267-0065274034-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.747 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:16.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:16.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:16.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:16.749 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.750 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.750 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.750 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.751 DEBUG:boto:Method: GET 2023-10-12T00:39:16.751 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.752 DEBUG:boto:Data: 2023-10-12T00:39:16.752 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.753 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.753 DEBUG:boto:Params: {} 2023-10-12T00:39:16.754 DEBUG:boto:Token: None 2023-10-12T00:39:16.754 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/?versions 2023-10-12T00:39:16.755 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8= 2023-10-12T00:39:16.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8=', 'Content-Length': '0'} 2023-10-12T00:39:16.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1a95b2444d72da-0065274034-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.758 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.759 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.760 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.760 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.761 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.761 DEBUG:boto:Method: GET 2023-10-12T00:39:16.761 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:16.762 DEBUG:boto:Data: 2023-10-12T00:39:16.762 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.763 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.764 DEBUG:boto:Params: {} 2023-10-12T00:39:16.764 DEBUG:boto:Token: None 2023-10-12T00:39:16.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/?versions 2023-10-12T00:39:16.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8= 2023-10-12T00:39:16.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6UhOraVfam599fP7EZkVkGtzu8=', 'Content-Length': '0'} 2023-10-12T00:39:16.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e21fce2523b23fc6-0065274034-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.769 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:16.770 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.771 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.771 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.772 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.772 DEBUG:boto:Method: GET 2023-10-12T00:39:16.772 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.773 DEBUG:boto:Data: 2023-10-12T00:39:16.773 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.774 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.775 DEBUG:boto:Params: {} 2023-10-12T00:39:16.775 DEBUG:boto:Token: None 2023-10-12T00:39:16.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI= 2023-10-12T00:39:16.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI=', 'Content-Length': '0'} 2023-10-12T00:39:16.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3274de1863a4059-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.780 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.780 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.781 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.781 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.782 DEBUG:boto:Method: GET 2023-10-12T00:39:16.782 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.783 DEBUG:boto:Data: 2023-10-12T00:39:16.783 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.784 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.785 DEBUG:boto:Params: {} 2023-10-12T00:39:16.785 DEBUG:boto:Token: None 2023-10-12T00:39:16.786 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI= 2023-10-12T00:39:16.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI=', 'Content-Length': '0'} 2023-10-12T00:39:16.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f08594e9e1e53da-0065274034-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.789 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.790 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.790 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.790 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.791 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.791 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.792 DEBUG:boto:Data: 2023-10-12T00:39:16.792 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.793 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.793 DEBUG:boto:Params: {} 2023-10-12T00:39:16.794 DEBUG:boto:Token: None 2023-10-12T00:39:16.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WajE4teo3346Ar/K6kbzEgOu0LQ= 2023-10-12T00:39:16.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WajE4teo3346Ar/K6kbzEgOu0LQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c71dec4ddeb8f8c-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.798 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.798 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.799 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.799 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.800 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.800 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.800 DEBUG:boto:Data: 2023-10-12T00:39:16.801 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.802 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.802 DEBUG:boto:Params: {} 2023-10-12T00:39:16.803 DEBUG:boto:Token: None 2023-10-12T00:39:16.803 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WajE4teo3346Ar/K6kbzEgOu0LQ= 2023-10-12T00:39:16.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WajE4teo3346Ar/K6kbzEgOu0LQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008184f7cbfb3f555d-0065274034-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:16.807 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.807 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.808 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.808 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.809 DEBUG:boto:Method: GET 2023-10-12T00:39:16.809 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.810 DEBUG:boto:Data: 2023-10-12T00:39:16.810 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.811 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.811 DEBUG:boto:Params: {} 2023-10-12T00:39:16.812 DEBUG:boto:Token: None 2023-10-12T00:39:16.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI= 2023-10-12T00:39:16.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI=', 'Content-Length': '0'} 2023-10-12T00:39:16.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076dd6f852c154f42-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.858 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.859 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.860 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.860 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.860 DEBUG:boto:Method: GET 2023-10-12T00:39:16.861 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.861 DEBUG:boto:Data: 2023-10-12T00:39:16.862 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.863 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.863 DEBUG:boto:Params: {} 2023-10-12T00:39:16.864 DEBUG:boto:Token: None 2023-10-12T00:39:16.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:16.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI= 2023-10-12T00:39:16.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIr9jfEoPWV4l6D1eYULnNFYGPI=', 'Content-Length': '0'} 2023-10-12T00:39:16.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c74805d775d253b1-0065274034-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.868 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.869 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.869 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.869 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.870 DEBUG:boto:Method: GET 2023-10-12T00:39:16.870 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.871 DEBUG:boto:Data: 2023-10-12T00:39:16.871 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.872 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.872 DEBUG:boto:Params: {} 2023-10-12T00:39:16.873 DEBUG:boto:Token: None 2023-10-12T00:39:16.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ= 2023-10-12T00:39:16.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb369a8b0ee1a0fe-0065274034-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.878 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.878 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.879 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.879 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.880 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.880 DEBUG:boto:Method: GET 2023-10-12T00:39:16.881 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.882 DEBUG:boto:Data: 2023-10-12T00:39:16.883 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.884 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.885 DEBUG:boto:Params: {} 2023-10-12T00:39:16.885 DEBUG:boto:Token: None 2023-10-12T00:39:16.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.886 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ= 2023-10-12T00:39:16.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074352971cb951065-0065274034-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.890 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.891 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.891 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.892 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.892 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:39:16.892 DEBUG:boto:Data: 2023-10-12T00:39:16.893 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.894 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.895 DEBUG:boto:Params: {} 2023-10-12T00:39:16.895 DEBUG:boto:Token: None 2023-10-12T00:39:16.896 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1 2023-10-12T00:39:16.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9zKMOC1CQRYTPT2hanigcB6NGuY= 2023-10-12T00:39:16.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9zKMOC1CQRYTPT2hanigcB6NGuY=', 'Content-Length': '0'} 2023-10-12T00:39:16.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0e6db01a4cfc52a-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.899 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.900 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.900 DEBUG:boto:Method: HEAD 2023-10-12T00:39:16.900 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:39:16.901 DEBUG:boto:Data: 2023-10-12T00:39:16.901 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.902 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.903 DEBUG:boto:Params: {} 2023-10-12T00:39:16.903 DEBUG:boto:Token: None 2023-10-12T00:39:16.904 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1 2023-10-12T00:39:16.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9zKMOC1CQRYTPT2hanigcB6NGuY= 2023-10-12T00:39:16.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9zKMOC1CQRYTPT2hanigcB6NGuY=', 'Content-Length': '0'} 2023-10-12T00:39:16.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cedd69c11921b977-0065274034-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:16.908 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.908 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.909 DEBUG:boto:Method: GET 2023-10-12T00:39:16.909 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:39:16.910 DEBUG:boto:Data: 2023-10-12T00:39:16.910 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.911 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.912 DEBUG:boto:Params: {} 2023-10-12T00:39:16.912 DEBUG:boto:Token: None 2023-10-12T00:39:16.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1 2023-10-12T00:39:16.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YZcASIalC90GQQzJkHiKB2tk4u8= 2023-10-12T00:39:16.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YZcASIalC90GQQzJkHiKB2tk4u8=', 'Content-Length': '0'} 2023-10-12T00:39:16.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee4b807d84972357-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.916 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:16.917 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:16.917 DEBUG:boto:Method: GET 2023-10-12T00:39:16.918 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:39:16.918 DEBUG:boto:Data: 2023-10-12T00:39:16.918 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.919 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.920 DEBUG:boto:Params: {} 2023-10-12T00:39:16.920 DEBUG:boto:Token: None 2023-10-12T00:39:16.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj1 2023-10-12T00:39:16.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YZcASIalC90GQQzJkHiKB2tk4u8= 2023-10-12T00:39:16.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YZcASIalC90GQQzJkHiKB2tk4u8=', 'Content-Length': '0'} 2023-10-12T00:39:16.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009215d77c679760e0-0065274034-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.966 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.967 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.967 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.968 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.968 DEBUG:boto:Method: GET 2023-10-12T00:39:16.969 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.969 DEBUG:boto:Data: 2023-10-12T00:39:16.970 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.971 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.971 DEBUG:boto:Params: {} 2023-10-12T00:39:16.971 DEBUG:boto:Token: None 2023-10-12T00:39:16.972 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ= 2023-10-12T00:39:16.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f989f497ba272de-0065274034-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.977 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.977 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:16.978 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:16.978 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.981 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.982 DEBUG:boto:Method: GET 2023-10-12T00:39:16.982 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:39:16.983 DEBUG:boto:Data: 2023-10-12T00:39:16.983 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:16.984 DEBUG:boto:Port: 8000 2023-10-12T00:39:16.984 DEBUG:boto:Params: {} 2023-10-12T00:39:16.985 DEBUG:boto:Token: None 2023-10-12T00:39:16.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/ 2023-10-12T00:39:16.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ= 2023-10-12T00:39:16.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GctlRlxQeh2G0NWI853/sqpV7VQ=', 'Content-Length': '0'} 2023-10-12T00:39:16.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17ecf2612ab05d3-0065274034-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:16.990 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:16.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:16.991 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:16.992 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:16.992 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:16.992 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:16.993 DEBUG:boto:Method: GET 2023-10-12T00:39:16.993 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:16.994 DEBUG:boto:Data: 2023-10-12T00:39:16.995 DEBUG:boto:Headers: {} 2023-10-12T00:39:16.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:16.995 DEBUG:boto:Port: 8001 2023-10-12T00:39:16.996 DEBUG:boto:Params: {} 2023-10-12T00:39:16.996 DEBUG:boto:Token: None 2023-10-12T00:39:16.997 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:16 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:16.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pA+RDhvctsSt/VxdgsSdqdTosvI= 2023-10-12T00:39:16.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:16 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pA+RDhvctsSt/VxdgsSdqdTosvI=', 'Content-Length': '0'} 2023-10-12T00:39:17.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3a577b7fa1a783c-0065274034-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.042 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.043 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.043 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.044 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.044 DEBUG:boto:Method: GET 2023-10-12T00:39:17.045 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.045 DEBUG:boto:Data: 2023-10-12T00:39:17.046 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.046 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.047 DEBUG:boto:Params: {} 2023-10-12T00:39:17.047 DEBUG:boto:Token: None 2023-10-12T00:39:17.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c= 2023-10-12T00:39:17.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c=', 'Content-Length': '0'} 2023-10-12T00:39:17.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ebb5c0bc24320a4-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.052 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.053 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.053 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.053 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.054 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.054 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.055 DEBUG:boto:Data: 2023-10-12T00:39:17.055 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.056 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.056 DEBUG:boto:Params: {} 2023-10-12T00:39:17.057 DEBUG:boto:Token: None 2023-10-12T00:39:17.057 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c9GKVc97tFWnnNJEx1e2bS8b00M= 2023-10-12T00:39:17.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c9GKVc97tFWnnNJEx1e2bS8b00M=', 'Content-Length': '0'} 2023-10-12T00:39:17.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009af4579153732f0b-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.061 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.061 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.062 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.062 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.062 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.063 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.063 DEBUG:boto:Data: 2023-10-12T00:39:17.064 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.064 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.065 DEBUG:boto:Params: {} 2023-10-12T00:39:17.065 DEBUG:boto:Token: None 2023-10-12T00:39:17.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c9GKVc97tFWnnNJEx1e2bS8b00M= 2023-10-12T00:39:17.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c9GKVc97tFWnnNJEx1e2bS8b00M=', 'Content-Length': '0'} 2023-10-12T00:39:17.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000174cd8250ed1c33e-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:17.070 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.070 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.071 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.071 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.071 DEBUG:boto:Method: GET 2023-10-12T00:39:17.072 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.072 DEBUG:boto:Data: 2023-10-12T00:39:17.073 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.074 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.074 DEBUG:boto:Params: {} 2023-10-12T00:39:17.075 DEBUG:boto:Token: None 2023-10-12T00:39:17.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c= 2023-10-12T00:39:17.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c=', 'Content-Length': '0'} 2023-10-12T00:39:17.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f70f20e428383f1b-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.079 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.080 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.080 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.081 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.081 DEBUG:boto:Method: GET 2023-10-12T00:39:17.082 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.082 DEBUG:boto:Data: 2023-10-12T00:39:17.082 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.083 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.084 DEBUG:boto:Params: {} 2023-10-12T00:39:17.084 DEBUG:boto:Token: None 2023-10-12T00:39:17.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:39:17.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c= 2023-10-12T00:39:17.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wKrsYZdOgii2uDdw/cjNmBblw8c=', 'Content-Length': '0'} 2023-10-12T00:39:17.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d92433024a9fff9-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.088 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.089 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.089 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.090 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.090 DEBUG:boto:Method: GET 2023-10-12T00:39:17.091 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.091 DEBUG:boto:Data: 2023-10-12T00:39:17.092 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.092 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.093 DEBUG:boto:Params: {} 2023-10-12T00:39:17.093 DEBUG:boto:Token: None 2023-10-12T00:39:17.094 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf0d53b84bdd54e-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.098 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.099 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.099 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.099 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.100 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.100 DEBUG:boto:Method: GET 2023-10-12T00:39:17.101 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.101 DEBUG:boto:Data: 2023-10-12T00:39:17.102 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.102 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.103 DEBUG:boto:Params: {} 2023-10-12T00:39:17.103 DEBUG:boto:Token: None 2023-10-12T00:39:17.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9de2c4deae88afb-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.108 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.109 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.109 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.110 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.110 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:39:17.111 DEBUG:boto:Data: 2023-10-12T00:39:17.111 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.112 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.112 DEBUG:boto:Params: {} 2023-10-12T00:39:17.113 DEBUG:boto:Token: None 2023-10-12T00:39:17.113 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2 2023-10-12T00:39:17.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e00j7Uh3ocpYUkXCvFPnBSLxX7Y= 2023-10-12T00:39:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e00j7Uh3ocpYUkXCvFPnBSLxX7Y=', 'Content-Length': '0'} 2023-10-12T00:39:17.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0b80ac0e2b6f34f-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.117 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.118 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.118 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.119 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:39:17.119 DEBUG:boto:Data: 2023-10-12T00:39:17.120 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.120 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.121 DEBUG:boto:Params: {} 2023-10-12T00:39:17.121 DEBUG:boto:Token: None 2023-10-12T00:39:17.122 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2 2023-10-12T00:39:17.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e00j7Uh3ocpYUkXCvFPnBSLxX7Y= 2023-10-12T00:39:17.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e00j7Uh3ocpYUkXCvFPnBSLxX7Y=', 'Content-Length': '0'} 2023-10-12T00:39:17.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd72f61fd76aed6-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:17.126 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.126 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.127 DEBUG:boto:Method: GET 2023-10-12T00:39:17.127 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:39:17.127 DEBUG:boto:Data: 2023-10-12T00:39:17.128 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.129 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.129 DEBUG:boto:Params: {} 2023-10-12T00:39:17.130 DEBUG:boto:Token: None 2023-10-12T00:39:17.130 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2 2023-10-12T00:39:17.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NTq3k+JpCOKbsQn/L8tq/2kugoI= 2023-10-12T00:39:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NTq3k+JpCOKbsQn/L8tq/2kugoI=', 'Content-Length': '0'} 2023-10-12T00:39:17.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca0a026786230825-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.134 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:39:17.134 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:39:17.135 DEBUG:boto:Method: GET 2023-10-12T00:39:17.135 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:39:17.135 DEBUG:boto:Data: 2023-10-12T00:39:17.136 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.137 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.137 DEBUG:boto:Params: {} 2023-10-12T00:39:17.138 DEBUG:boto:Token: None 2023-10-12T00:39:17.138 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj2 2023-10-12T00:39:17.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NTq3k+JpCOKbsQn/L8tq/2kugoI= 2023-10-12T00:39:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NTq3k+JpCOKbsQn/L8tq/2kugoI=', 'Content-Length': '0'} 2023-10-12T00:39:17.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000968631822e2383c-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.182 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.185 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.186 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.187 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.187 DEBUG:boto:Method: GET 2023-10-12T00:39:17.188 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.188 DEBUG:boto:Data: 2023-10-12T00:39:17.189 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.189 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.190 DEBUG:boto:Params: {} 2023-10-12T00:39:17.190 DEBUG:boto:Token: None 2023-10-12T00:39:17.191 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.191 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007816fb35f9330308-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.195 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.197 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.198 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.198 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.198 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.199 DEBUG:boto:Method: GET 2023-10-12T00:39:17.199 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:39:17.200 DEBUG:boto:Data: 2023-10-12T00:39:17.200 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.202 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.202 DEBUG:boto:Params: {} 2023-10-12T00:39:17.203 DEBUG:boto:Token: None 2023-10-12T00:39:17.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5a37afb093ec21-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.208 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:17.209 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.209 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.210 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.210 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.211 DEBUG:boto:Method: GET 2023-10-12T00:39:17.211 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.212 DEBUG:boto:Data: 2023-10-12T00:39:17.212 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.213 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.213 DEBUG:boto:Params: {} 2023-10-12T00:39:17.214 DEBUG:boto:Token: None 2023-10-12T00:39:17.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY= 2023-10-12T00:39:17.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY=', 'Content-Length': '0'} 2023-10-12T00:39:17.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044851a206ad93a20-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.258 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.259 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.259 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.260 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.260 DEBUG:boto:Method: GET 2023-10-12T00:39:17.261 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.261 DEBUG:boto:Data: 2023-10-12T00:39:17.262 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.263 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.263 DEBUG:boto:Params: {} 2023-10-12T00:39:17.263 DEBUG:boto:Token: None 2023-10-12T00:39:17.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY= 2023-10-12T00:39:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY=', 'Content-Length': '0'} 2023-10-12T00:39:17.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000624708e84218f81c-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.268 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.268 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.269 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.269 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.270 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.270 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.271 DEBUG:boto:Data: 2023-10-12T00:39:17.271 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.272 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.272 DEBUG:boto:Params: {} 2023-10-12T00:39:17.273 DEBUG:boto:Token: None 2023-10-12T00:39:17.273 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.274 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VMt5bvozDvMHTnWJ2AFBGIxuUU0= 2023-10-12T00:39:17.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VMt5bvozDvMHTnWJ2AFBGIxuUU0=', 'Content-Length': '0'} 2023-10-12T00:39:17.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbfa61991654e2f1-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.277 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.277 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.278 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.278 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.279 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.279 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.280 DEBUG:boto:Data: 2023-10-12T00:39:17.280 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.281 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.281 DEBUG:boto:Params: {} 2023-10-12T00:39:17.282 DEBUG:boto:Token: None 2023-10-12T00:39:17.282 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VMt5bvozDvMHTnWJ2AFBGIxuUU0= 2023-10-12T00:39:17.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VMt5bvozDvMHTnWJ2AFBGIxuUU0=', 'Content-Length': '0'} 2023-10-12T00:39:17.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b16afc4508274dd-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:17.287 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.288 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.288 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.288 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.289 DEBUG:boto:Method: GET 2023-10-12T00:39:17.289 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.290 DEBUG:boto:Data: 2023-10-12T00:39:17.290 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.291 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.292 DEBUG:boto:Params: {} 2023-10-12T00:39:17.292 DEBUG:boto:Token: None 2023-10-12T00:39:17.293 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY= 2023-10-12T00:39:17.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY=', 'Content-Length': '0'} 2023-10-12T00:39:17.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243b3cb649315711-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.297 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.297 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.298 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.298 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.299 DEBUG:boto:Method: GET 2023-10-12T00:39:17.299 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.299 DEBUG:boto:Data: 2023-10-12T00:39:17.300 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.301 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.301 DEBUG:boto:Params: {} 2023-10-12T00:39:17.302 DEBUG:boto:Token: None 2023-10-12T00:39:17.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:39:17.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY= 2023-10-12T00:39:17.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qXWmi6k6kyywrS+xE8Kji8APtwY=', 'Content-Length': '0'} 2023-10-12T00:39:17.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007be0763bab64a8a7-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.306 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.306 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.307 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.307 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.308 DEBUG:boto:Method: GET 2023-10-12T00:39:17.308 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.309 DEBUG:boto:Data: 2023-10-12T00:39:17.309 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.310 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.311 DEBUG:boto:Params: {} 2023-10-12T00:39:17.311 DEBUG:boto:Token: None 2023-10-12T00:39:17.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000befc3d4b484a2b6a-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.316 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.316 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.317 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.317 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.318 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.318 DEBUG:boto:Method: GET 2023-10-12T00:39:17.319 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.319 DEBUG:boto:Data: 2023-10-12T00:39:17.320 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.320 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.321 DEBUG:boto:Params: {} 2023-10-12T00:39:17.321 DEBUG:boto:Token: None 2023-10-12T00:39:17.322 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058564f04b363c7a7-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.326 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.327 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.327 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.328 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.328 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:39:17.328 DEBUG:boto:Data: 2023-10-12T00:39:17.329 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.330 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.330 DEBUG:boto:Params: {} 2023-10-12T00:39:17.330 DEBUG:boto:Token: None 2023-10-12T00:39:17.331 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3 2023-10-12T00:39:17.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zhlsdBsUJpR6VTxYxvDlnnW7tNU= 2023-10-12T00:39:17.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zhlsdBsUJpR6VTxYxvDlnnW7tNU=', 'Content-Length': '0'} 2023-10-12T00:39:17.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097cfa48ca33c1f05-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.335 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.335 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.336 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.336 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:39:17.336 DEBUG:boto:Data: 2023-10-12T00:39:17.337 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.338 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.338 DEBUG:boto:Params: {} 2023-10-12T00:39:17.339 DEBUG:boto:Token: None 2023-10-12T00:39:17.339 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3 2023-10-12T00:39:17.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zhlsdBsUJpR6VTxYxvDlnnW7tNU= 2023-10-12T00:39:17.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zhlsdBsUJpR6VTxYxvDlnnW7tNU=', 'Content-Length': '0'} 2023-10-12T00:39:17.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6b825742c688bda-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:17.344 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.345 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.345 DEBUG:boto:Method: GET 2023-10-12T00:39:17.345 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:39:17.346 DEBUG:boto:Data: 2023-10-12T00:39:17.346 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.347 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.348 DEBUG:boto:Params: {} 2023-10-12T00:39:17.348 DEBUG:boto:Token: None 2023-10-12T00:39:17.349 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3 2023-10-12T00:39:17.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cnLPbrA6NG0FgVPXT2ozBju7CXE= 2023-10-12T00:39:17.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cnLPbrA6NG0FgVPXT2ozBju7CXE=', 'Content-Length': '0'} 2023-10-12T00:39:17.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000242acc02dac4e89d-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.352 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:39:17.353 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:39:17.353 DEBUG:boto:Method: GET 2023-10-12T00:39:17.354 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:39:17.354 DEBUG:boto:Data: 2023-10-12T00:39:17.355 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.356 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.356 DEBUG:boto:Params: {} 2023-10-12T00:39:17.357 DEBUG:boto:Token: None 2023-10-12T00:39:17.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj3 2023-10-12T00:39:17.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cnLPbrA6NG0FgVPXT2ozBju7CXE= 2023-10-12T00:39:17.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cnLPbrA6NG0FgVPXT2ozBju7CXE=', 'Content-Length': '0'} 2023-10-12T00:39:17.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ead107f6d796b10-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.406 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.407 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.407 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.408 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.408 DEBUG:boto:Method: GET 2023-10-12T00:39:17.409 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.409 DEBUG:boto:Data: 2023-10-12T00:39:17.410 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.411 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.411 DEBUG:boto:Params: {} 2023-10-12T00:39:17.412 DEBUG:boto:Token: None 2023-10-12T00:39:17.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a12657f53467f0-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.417 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.417 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.418 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.418 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.419 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.419 DEBUG:boto:Method: GET 2023-10-12T00:39:17.420 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:39:17.421 DEBUG:boto:Data: 2023-10-12T00:39:17.421 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.422 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.422 DEBUG:boto:Params: {} 2023-10-12T00:39:17.423 DEBUG:boto:Token: None 2023-10-12T00:39:17.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e841d374523f3be6-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.427 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:17.429 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.429 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.430 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.431 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.431 DEBUG:boto:Method: GET 2023-10-12T00:39:17.432 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.432 DEBUG:boto:Data: 2023-10-12T00:39:17.433 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.433 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.434 DEBUG:boto:Params: {} 2023-10-12T00:39:17.434 DEBUG:boto:Token: None 2023-10-12T00:39:17.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.435 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc= 2023-10-12T00:39:17.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc=', 'Content-Length': '0'} 2023-10-12T00:39:17.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6809dd291f42554-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.439 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.439 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.440 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.440 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.441 DEBUG:boto:Method: GET 2023-10-12T00:39:17.441 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.441 DEBUG:boto:Data: 2023-10-12T00:39:17.442 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.443 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.443 DEBUG:boto:Params: {} 2023-10-12T00:39:17.444 DEBUG:boto:Token: None 2023-10-12T00:39:17.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc= 2023-10-12T00:39:17.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc=', 'Content-Length': '0'} 2023-10-12T00:39:17.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000657393b377ff3cb1-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.448 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.448 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.449 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.449 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.450 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.450 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.451 DEBUG:boto:Data: 2023-10-12T00:39:17.451 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.452 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.452 DEBUG:boto:Params: {} 2023-10-12T00:39:17.453 DEBUG:boto:Token: None 2023-10-12T00:39:17.454 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.454 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADCrpbptdz6+PPsh6rakNjiXjgI= 2023-10-12T00:39:17.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADCrpbptdz6+PPsh6rakNjiXjgI=', 'Content-Length': '0'} 2023-10-12T00:39:17.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046c6f4a3c4b3b0f4-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.458 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.458 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.458 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.459 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.459 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.460 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.460 DEBUG:boto:Data: 2023-10-12T00:39:17.461 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.463 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.464 DEBUG:boto:Params: {} 2023-10-12T00:39:17.465 DEBUG:boto:Token: None 2023-10-12T00:39:17.465 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.466 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADCrpbptdz6+PPsh6rakNjiXjgI= 2023-10-12T00:39:17.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADCrpbptdz6+PPsh6rakNjiXjgI=', 'Content-Length': '0'} 2023-10-12T00:39:17.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000226851b4d3f7a243-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:17.470 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.471 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.471 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.471 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.472 DEBUG:boto:Method: GET 2023-10-12T00:39:17.472 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.473 DEBUG:boto:Data: 2023-10-12T00:39:17.473 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.474 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.474 DEBUG:boto:Params: {} 2023-10-12T00:39:17.475 DEBUG:boto:Token: None 2023-10-12T00:39:17.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc= 2023-10-12T00:39:17.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc=', 'Content-Length': '0'} 2023-10-12T00:39:17.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000711793b0c1660069-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.522 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.523 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.524 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.525 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.525 DEBUG:boto:Method: GET 2023-10-12T00:39:17.526 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.526 DEBUG:boto:Data: 2023-10-12T00:39:17.527 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.528 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.529 DEBUG:boto:Params: {} 2023-10-12T00:39:17.529 DEBUG:boto:Token: None 2023-10-12T00:39:17.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:39:17.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc= 2023-10-12T00:39:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NZ/Rz5JVprAuAOWq/fpq76N+ulc=', 'Content-Length': '0'} 2023-10-12T00:39:17.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017287a8fe86b7707-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.534 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.535 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.535 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.536 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.537 DEBUG:boto:Method: GET 2023-10-12T00:39:17.538 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.538 DEBUG:boto:Data: 2023-10-12T00:39:17.539 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.541 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.541 DEBUG:boto:Params: {} 2023-10-12T00:39:17.542 DEBUG:boto:Token: None 2023-10-12T00:39:17.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088269b8f9cc5bd2d-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.547 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.548 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.548 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.549 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.549 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.550 DEBUG:boto:Method: GET 2023-10-12T00:39:17.550 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.551 DEBUG:boto:Data: 2023-10-12T00:39:17.552 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.554 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.554 DEBUG:boto:Params: {} 2023-10-12T00:39:17.555 DEBUG:boto:Token: None 2023-10-12T00:39:17.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d52e14f834287dc-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.561 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.562 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.562 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.563 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.563 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:39:17.564 DEBUG:boto:Data: 2023-10-12T00:39:17.564 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.565 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.565 DEBUG:boto:Params: {} 2023-10-12T00:39:17.566 DEBUG:boto:Token: None 2023-10-12T00:39:17.566 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4 2023-10-12T00:39:17.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4iCf4zVNYwL1hdiyoe+YAVRUNb4= 2023-10-12T00:39:17.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4iCf4zVNYwL1hdiyoe+YAVRUNb4=', 'Content-Length': '0'} 2023-10-12T00:39:17.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f66bfb307ce6160-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.570 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.570 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.571 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.571 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:39:17.572 DEBUG:boto:Data: 2023-10-12T00:39:17.572 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.573 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.573 DEBUG:boto:Params: {} 2023-10-12T00:39:17.574 DEBUG:boto:Token: None 2023-10-12T00:39:17.574 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4 2023-10-12T00:39:17.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4iCf4zVNYwL1hdiyoe+YAVRUNb4= 2023-10-12T00:39:17.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4iCf4zVNYwL1hdiyoe+YAVRUNb4=', 'Content-Length': '0'} 2023-10-12T00:39:17.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092273fe1550b3d24-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:17.578 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.578 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.579 DEBUG:boto:Method: GET 2023-10-12T00:39:17.579 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:39:17.580 DEBUG:boto:Data: 2023-10-12T00:39:17.580 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.581 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.582 DEBUG:boto:Params: {} 2023-10-12T00:39:17.582 DEBUG:boto:Token: None 2023-10-12T00:39:17.583 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4 2023-10-12T00:39:17.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H7814/RinXXdgUYsdzPB4SFJEeY= 2023-10-12T00:39:17.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H7814/RinXXdgUYsdzPB4SFJEeY=', 'Content-Length': '0'} 2023-10-12T00:39:17.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000068679e73340ba1d-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.586 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:39:17.587 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:39:17.587 DEBUG:boto:Method: GET 2023-10-12T00:39:17.588 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:39:17.588 DEBUG:boto:Data: 2023-10-12T00:39:17.589 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.589 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.590 DEBUG:boto:Params: {} 2023-10-12T00:39:17.590 DEBUG:boto:Token: None 2023-10-12T00:39:17.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/obj4 2023-10-12T00:39:17.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H7814/RinXXdgUYsdzPB4SFJEeY= 2023-10-12T00:39:17.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H7814/RinXXdgUYsdzPB4SFJEeY=', 'Content-Length': '0'} 2023-10-12T00:39:17.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1154578f2dd71e-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.596 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.596 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.597 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.597 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.598 DEBUG:boto:Method: GET 2023-10-12T00:39:17.598 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.598 DEBUG:boto:Data: 2023-10-12T00:39:17.599 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.600 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.601 DEBUG:boto:Params: {} 2023-10-12T00:39:17.601 DEBUG:boto:Token: None 2023-10-12T00:39:17.602 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.602 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007342f85616718954-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.606 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.607 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:17.607 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:17.608 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.608 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.609 DEBUG:boto:Method: GET 2023-10-12T00:39:17.609 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:39:17.609 DEBUG:boto:Data: 2023-10-12T00:39:17.610 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.611 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.611 DEBUG:boto:Params: {} 2023-10-12T00:39:17.612 DEBUG:boto:Token: None 2023-10-12T00:39:17.612 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-46/ 2023-10-12T00:39:17.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY= 2023-10-12T00:39:17.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ewYTgayQnkdZOEDDpQgSujXbWmY=', 'Content-Length': '0'} 2023-10-12T00:39:17.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416c7fbbeebb2866-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.662 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-46 zones={a2, a1} 2023-10-12T00:39:17.664 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-47 2023-10-12T00:39:17.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:17.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:17.665 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:39:17.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:17.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:17.765+0000 7fa413aa8780 1 bucket sync caught up with source: 2023-10-12T00:39:17.767 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.14.6, , 00000000001.19.6, , 00000000001.23.6, , , 00000000001.20.6, , , ] 2023-10-12T00:39:17.767 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.14.6, , 00000000001.19.6, , 00000000001.23.6, , , 00000000001.20.6, , , ] 2023-10-12T00:39:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:17.765+0000 7fa413aa8780 0 bucket checkpoint complete 2023-10-12T00:39:17.779 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-47 2023-10-12T00:39:17.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:17.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:17.781 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:39:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:17.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:17.870+0000 7f3fb53a3780 1 bucket sync caught up with source: 2023-10-12T00:39:17.872 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.85.6, , 00000000001.188.6, , 00000000001.109.6, , , 00000000001.168.6, , , ] 2023-10-12T00:39:17.872 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.85.6, , 00000000001.188.6, , 00000000001.109.6, , , 00000000001.168.6, , , ] 2023-10-12T00:39:17.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:17.870+0000 7f3fb53a3780 0 bucket checkpoint complete 2023-10-12T00:39:17.884 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-47 zones={a2, a1} 2023-10-12T00:39:17.884 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.885 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.885 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.886 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:17.886 DEBUG:boto:Data: 2023-10-12T00:39:17.886 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.887 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.888 DEBUG:boto:Params: {} 2023-10-12T00:39:17.888 DEBUG:boto:Token: None 2023-10-12T00:39:17.889 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/ 2023-10-12T00:39:17.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Suot9tuWGWth+KEUJgB9u42sAyk= 2023-10-12T00:39:17.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Suot9tuWGWth+KEUJgB9u42sAyk=', 'Content-Length': '0'} 2023-10-12T00:39:17.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c3de013338cccb2-0065274035-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.893 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.893 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.893 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.894 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:17.894 DEBUG:boto:Data: 2023-10-12T00:39:17.895 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.895 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.896 DEBUG:boto:Params: {} 2023-10-12T00:39:17.896 DEBUG:boto:Token: None 2023-10-12T00:39:17.897 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/ 2023-10-12T00:39:17.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Suot9tuWGWth+KEUJgB9u42sAyk= 2023-10-12T00:39:17.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Suot9tuWGWth+KEUJgB9u42sAyk=', 'Content-Length': '0'} 2023-10-12T00:39:17.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0bab59660e7e798-0065274035-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.901 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:39:17.901 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.901 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.902 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.902 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.903 DEBUG:boto:Method: GET 2023-10-12T00:39:17.903 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.904 DEBUG:boto:Data: 2023-10-12T00:39:17.904 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.905 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.905 DEBUG:boto:Params: {} 2023-10-12T00:39:17.906 DEBUG:boto:Token: None 2023-10-12T00:39:17.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/?versions 2023-10-12T00:39:17.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU= 2023-10-12T00:39:17.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU=', 'Content-Length': '0'} 2023-10-12T00:39:17.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39bb319f7aac886-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.910 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:17.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:17.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:17.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:39:17.913 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.913 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.914 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.914 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.915 DEBUG:boto:Method: GET 2023-10-12T00:39:17.915 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.916 DEBUG:boto:Data: 2023-10-12T00:39:17.916 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.917 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.917 DEBUG:boto:Params: {} 2023-10-12T00:39:17.918 DEBUG:boto:Token: None 2023-10-12T00:39:17.918 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/?versions 2023-10-12T00:39:17.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU= 2023-10-12T00:39:17.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU=', 'Content-Length': '0'} 2023-10-12T00:39:17.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008221a986d77d2566-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.922 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:17.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:17.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:17.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:17.925 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.925 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.925 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.926 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.926 DEBUG:boto:Method: GET 2023-10-12T00:39:17.927 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.927 DEBUG:boto:Data: 2023-10-12T00:39:17.928 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.928 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.929 DEBUG:boto:Params: {} 2023-10-12T00:39:17.929 DEBUG:boto:Token: None 2023-10-12T00:39:17.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/?versions 2023-10-12T00:39:17.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU= 2023-10-12T00:39:17.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU=', 'Content-Length': '0'} 2023-10-12T00:39:17.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec8e7038c49d1e9c-0065274035-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.934 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.934 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:17.935 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:17.935 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.936 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.936 DEBUG:boto:Method: GET 2023-10-12T00:39:17.937 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:39:17.937 DEBUG:boto:Data: 2023-10-12T00:39:17.938 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.938 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.939 DEBUG:boto:Params: {} 2023-10-12T00:39:17.939 DEBUG:boto:Token: None 2023-10-12T00:39:17.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/?versions 2023-10-12T00:39:17.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU= 2023-10-12T00:39:17.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iyAM8pNifQoZydedFKNsgzFFvtU=', 'Content-Length': '0'} 2023-10-12T00:39:17.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e8a74e44de3fb5-0065274035-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.944 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:17.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:17.945 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.945 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.946 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.946 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.947 DEBUG:boto:Method: GET 2023-10-12T00:39:17.947 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.947 DEBUG:boto:Data: 2023-10-12T00:39:17.948 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.949 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.949 DEBUG:boto:Params: {} 2023-10-12T00:39:17.950 DEBUG:boto:Token: None 2023-10-12T00:39:17.950 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k= 2023-10-12T00:39:17.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k=', 'Content-Length': '0'} 2023-10-12T00:39:17.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000472b8efd5cc41bba-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.953 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.954 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.954 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.955 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.955 DEBUG:boto:Method: GET 2023-10-12T00:39:17.956 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.956 DEBUG:boto:Data: 2023-10-12T00:39:17.957 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.958 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.958 DEBUG:boto:Params: {} 2023-10-12T00:39:17.959 DEBUG:boto:Token: None 2023-10-12T00:39:17.959 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k= 2023-10-12T00:39:17.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k=', 'Content-Length': '0'} 2023-10-12T00:39:17.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007422fc12791c1437-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.962 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.963 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.963 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.964 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.964 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.965 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.965 DEBUG:boto:Data: 2023-10-12T00:39:17.965 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.966 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.967 DEBUG:boto:Params: {} 2023-10-12T00:39:17.967 DEBUG:boto:Token: None 2023-10-12T00:39:17.968 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vpvV5zme+87rinj6+Ph2lQyGTwA= 2023-10-12T00:39:17.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vpvV5zme+87rinj6+Ph2lQyGTwA=', 'Content-Length': '0'} 2023-10-12T00:39:17.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000393a18bcc5749ba1-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.971 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.972 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.972 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.973 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.974 DEBUG:boto:Method: HEAD 2023-10-12T00:39:17.975 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.975 DEBUG:boto:Data: 2023-10-12T00:39:17.976 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:17.977 DEBUG:boto:Port: 8000 2023-10-12T00:39:17.978 DEBUG:boto:Params: {} 2023-10-12T00:39:17.979 DEBUG:boto:Token: None 2023-10-12T00:39:17.979 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vpvV5zme+87rinj6+Ph2lQyGTwA= 2023-10-12T00:39:17.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vpvV5zme+87rinj6+Ph2lQyGTwA=', 'Content-Length': '0'} 2023-10-12T00:39:17.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bccaf8f270240ae-0065274035-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:17.984 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.984 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.985 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.985 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.985 DEBUG:boto:Method: GET 2023-10-12T00:39:17.986 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.987 DEBUG:boto:Data: 2023-10-12T00:39:17.987 DEBUG:boto:Headers: {} 2023-10-12T00:39:17.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:17.989 DEBUG:boto:Port: 8001 2023-10-12T00:39:17.989 DEBUG:boto:Params: {} 2023-10-12T00:39:17.990 DEBUG:boto:Token: None 2023-10-12T00:39:17.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:17 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.992 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k= 2023-10-12T00:39:17.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:17 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/b4JrXIIy9WHdp2NlqiOQ/dY0k=', 'Content-Length': '0'} 2023-10-12T00:39:17.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e935122b91abe21-0065274035-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:17.996 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:17.997 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:17.997 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.998 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.998 DEBUG:boto:Method: GET 2023-10-12T00:39:17.999 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:17.999 DEBUG:boto:Data: 2023-10-12T00:39:18.001 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.003 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.003 DEBUG:boto:Params: {} 2023-10-12T00:39:18.003 DEBUG:boto:Token: None 2023-10-12T00:39:18.004 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:39:18.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fom8+Uj1uUjLG4j95NDDblKa/6I= 2023-10-12T00:39:18.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fom8+Uj1uUjLG4j95NDDblKa/6I=', 'Content-Length': '0'} 2023-10-12T00:39:18.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb891705fc6c1aff-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.008 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.008 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.008 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.009 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.009 DEBUG:boto:Method: GET 2023-10-12T00:39:18.010 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.010 DEBUG:boto:Data: 2023-10-12T00:39:18.011 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.012 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.012 DEBUG:boto:Params: {} 2023-10-12T00:39:18.013 DEBUG:boto:Token: None 2023-10-12T00:39:18.013 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.014 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e531782e90e8e46-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.017 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.018 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.019 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.019 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.020 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.020 DEBUG:boto:Method: GET 2023-10-12T00:39:18.021 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.022 DEBUG:boto:Data: 2023-10-12T00:39:18.022 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.023 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.024 DEBUG:boto:Params: {} 2023-10-12T00:39:18.024 DEBUG:boto:Token: None 2023-10-12T00:39:18.025 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c856222d7c6687a2-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.070 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.072 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:18.072 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:18.073 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.073 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:39:18.073 DEBUG:boto:Data: 2023-10-12T00:39:18.074 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.075 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.075 DEBUG:boto:Params: {} 2023-10-12T00:39:18.076 DEBUG:boto:Token: None 2023-10-12T00:39:18.076 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj1 2023-10-12T00:39:18.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6PZenFxpj75RhO7LSUGRHQt3kVk= 2023-10-12T00:39:18.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6PZenFxpj75RhO7LSUGRHQt3kVk=', 'Content-Length': '0'} 2023-10-12T00:39:18.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf00de1ea7ece312-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.079 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:18.080 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:18.080 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.081 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:39:18.081 DEBUG:boto:Data: 2023-10-12T00:39:18.081 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.082 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.083 DEBUG:boto:Params: {} 2023-10-12T00:39:18.083 DEBUG:boto:Token: None 2023-10-12T00:39:18.084 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj1 2023-10-12T00:39:18.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6PZenFxpj75RhO7LSUGRHQt3kVk= 2023-10-12T00:39:18.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6PZenFxpj75RhO7LSUGRHQt3kVk=', 'Content-Length': '0'} 2023-10-12T00:39:18.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002577f74bb839c80d-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:18.088 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:18.088 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:18.089 DEBUG:boto:Method: GET 2023-10-12T00:39:18.090 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:39:18.090 DEBUG:boto:Data: 2023-10-12T00:39:18.090 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.091 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.092 DEBUG:boto:Params: {} 2023-10-12T00:39:18.092 DEBUG:boto:Token: None 2023-10-12T00:39:18.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj1 2023-10-12T00:39:18.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tuHkJ1YCVVDxinmLrCi4x5luZaU= 2023-10-12T00:39:18.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tuHkJ1YCVVDxinmLrCi4x5luZaU=', 'Content-Length': '0'} 2023-10-12T00:39:18.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da67ab54aea77616-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.138 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:39:18.139 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:39:18.140 DEBUG:boto:Method: GET 2023-10-12T00:39:18.140 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:39:18.141 DEBUG:boto:Data: 2023-10-12T00:39:18.141 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.142 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.143 DEBUG:boto:Params: {} 2023-10-12T00:39:18.143 DEBUG:boto:Token: None 2023-10-12T00:39:18.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj1 2023-10-12T00:39:18.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tuHkJ1YCVVDxinmLrCi4x5luZaU= 2023-10-12T00:39:18.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tuHkJ1YCVVDxinmLrCi4x5luZaU=', 'Content-Length': '0'} 2023-10-12T00:39:18.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f20f1405afa1fb9-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.147 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.148 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.148 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.149 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.149 DEBUG:boto:Method: GET 2023-10-12T00:39:18.150 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.150 DEBUG:boto:Data: 2023-10-12T00:39:18.151 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.151 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.152 DEBUG:boto:Params: {} 2023-10-12T00:39:18.152 DEBUG:boto:Token: None 2023-10-12T00:39:18.153 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b16f573043cfc8b6-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.157 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.158 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.159 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.159 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.160 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.160 DEBUG:boto:Method: GET 2023-10-12T00:39:18.161 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:39:18.161 DEBUG:boto:Data: 2023-10-12T00:39:18.162 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.163 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.163 DEBUG:boto:Params: {} 2023-10-12T00:39:18.164 DEBUG:boto:Token: None 2023-10-12T00:39:18.165 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e269c407943fd6-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.170 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:18.171 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.172 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.172 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.172 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.173 DEBUG:boto:Method: GET 2023-10-12T00:39:18.173 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.174 DEBUG:boto:Data: 2023-10-12T00:39:18.174 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.175 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.176 DEBUG:boto:Params: {} 2023-10-12T00:39:18.176 DEBUG:boto:Token: None 2023-10-12T00:39:18.177 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE= 2023-10-12T00:39:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE=', 'Content-Length': '0'} 2023-10-12T00:39:18.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e7b5ef751437124-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.181 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.181 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.182 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.182 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.183 DEBUG:boto:Method: GET 2023-10-12T00:39:18.183 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.184 DEBUG:boto:Data: 2023-10-12T00:39:18.184 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.185 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.186 DEBUG:boto:Params: {} 2023-10-12T00:39:18.186 DEBUG:boto:Token: None 2023-10-12T00:39:18.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE= 2023-10-12T00:39:18.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE=', 'Content-Length': '0'} 2023-10-12T00:39:18.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000063acffae56d95f2-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.190 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.191 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.191 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.191 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.192 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.192 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.193 DEBUG:boto:Data: 2023-10-12T00:39:18.193 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.194 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.194 DEBUG:boto:Params: {} 2023-10-12T00:39:18.195 DEBUG:boto:Token: None 2023-10-12T00:39:18.195 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5Ltq92v1ghxieucKeABFF6/OLB4= 2023-10-12T00:39:18.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5Ltq92v1ghxieucKeABFF6/OLB4=', 'Content-Length': '0'} 2023-10-12T00:39:18.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000318501fc60777e94-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.199 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.199 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.199 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.200 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.200 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.201 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.201 DEBUG:boto:Data: 2023-10-12T00:39:18.201 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.202 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.203 DEBUG:boto:Params: {} 2023-10-12T00:39:18.203 DEBUG:boto:Token: None 2023-10-12T00:39:18.204 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5Ltq92v1ghxieucKeABFF6/OLB4= 2023-10-12T00:39:18.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5Ltq92v1ghxieucKeABFF6/OLB4=', 'Content-Length': '0'} 2023-10-12T00:39:18.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081d1e1c36d345c11-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:18.208 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.208 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.208 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.209 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.209 DEBUG:boto:Method: GET 2023-10-12T00:39:18.210 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.210 DEBUG:boto:Data: 2023-10-12T00:39:18.211 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.212 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.212 DEBUG:boto:Params: {} 2023-10-12T00:39:18.212 DEBUG:boto:Token: None 2023-10-12T00:39:18.213 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE= 2023-10-12T00:39:18.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE=', 'Content-Length': '0'} 2023-10-12T00:39:18.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f608565962a33bb-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.216 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.217 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.217 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.218 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.218 DEBUG:boto:Method: GET 2023-10-12T00:39:18.218 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.219 DEBUG:boto:Data: 2023-10-12T00:39:18.219 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.220 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.221 DEBUG:boto:Params: {} 2023-10-12T00:39:18.221 DEBUG:boto:Token: None 2023-10-12T00:39:18.222 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:39:18.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE= 2023-10-12T00:39:18.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4PNMuNjgKjX2vlW3bqNkTJtYixE=', 'Content-Length': '0'} 2023-10-12T00:39:18.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f022ee1d49ff741-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.266 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.267 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.268 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.268 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.269 DEBUG:boto:Method: GET 2023-10-12T00:39:18.269 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.270 DEBUG:boto:Data: 2023-10-12T00:39:18.270 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.271 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.271 DEBUG:boto:Params: {} 2023-10-12T00:39:18.272 DEBUG:boto:Token: None 2023-10-12T00:39:18.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef5f633f0198e47-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.276 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.277 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.278 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.278 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.278 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.279 DEBUG:boto:Method: GET 2023-10-12T00:39:18.279 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.280 DEBUG:boto:Data: 2023-10-12T00:39:18.280 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.281 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.281 DEBUG:boto:Params: {} 2023-10-12T00:39:18.282 DEBUG:boto:Token: None 2023-10-12T00:39:18.282 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0aa98b53d77754f-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.287 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.287 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.288 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.288 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.289 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:39:18.289 DEBUG:boto:Data: 2023-10-12T00:39:18.290 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.290 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.291 DEBUG:boto:Params: {} 2023-10-12T00:39:18.291 DEBUG:boto:Token: None 2023-10-12T00:39:18.292 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2 2023-10-12T00:39:18.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:96Ojk9eeI0WtcqdS87Dcc+/K7OA= 2023-10-12T00:39:18.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:96Ojk9eeI0WtcqdS87Dcc+/K7OA=', 'Content-Length': '0'} 2023-10-12T00:39:18.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4af5c917711fd15-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.295 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.296 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.296 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.296 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:39:18.297 DEBUG:boto:Data: 2023-10-12T00:39:18.297 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.298 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.299 DEBUG:boto:Params: {} 2023-10-12T00:39:18.299 DEBUG:boto:Token: None 2023-10-12T00:39:18.300 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2 2023-10-12T00:39:18.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:96Ojk9eeI0WtcqdS87Dcc+/K7OA= 2023-10-12T00:39:18.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:96Ojk9eeI0WtcqdS87Dcc+/K7OA=', 'Content-Length': '0'} 2023-10-12T00:39:18.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae8968c5c45819ce-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:39:18.304 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.304 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.305 DEBUG:boto:Method: GET 2023-10-12T00:39:18.305 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:39:18.305 DEBUG:boto:Data: 2023-10-12T00:39:18.306 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.307 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.307 DEBUG:boto:Params: {} 2023-10-12T00:39:18.308 DEBUG:boto:Token: None 2023-10-12T00:39:18.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2 2023-10-12T00:39:18.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KkwR01NhSwSTuHCVE8bIcm1gvTQ= 2023-10-12T00:39:18.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KkwR01NhSwSTuHCVE8bIcm1gvTQ=', 'Content-Length': '0'} 2023-10-12T00:39:18.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ff9fe142b8660c-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.312 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:39:18.312 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:39:18.313 DEBUG:boto:Method: GET 2023-10-12T00:39:18.313 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:39:18.314 DEBUG:boto:Data: 2023-10-12T00:39:18.314 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.315 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.315 DEBUG:boto:Params: {} 2023-10-12T00:39:18.316 DEBUG:boto:Token: None 2023-10-12T00:39:18.316 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj2 2023-10-12T00:39:18.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KkwR01NhSwSTuHCVE8bIcm1gvTQ= 2023-10-12T00:39:18.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KkwR01NhSwSTuHCVE8bIcm1gvTQ=', 'Content-Length': '0'} 2023-10-12T00:39:18.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd2564090f033a93-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.320 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.321 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.321 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.322 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.322 DEBUG:boto:Method: GET 2023-10-12T00:39:18.323 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.323 DEBUG:boto:Data: 2023-10-12T00:39:18.324 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.324 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.325 DEBUG:boto:Params: {} 2023-10-12T00:39:18.325 DEBUG:boto:Token: None 2023-10-12T00:39:18.326 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd404a7f1ac1554-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.370 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.371 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.372 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.372 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.373 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.373 DEBUG:boto:Method: GET 2023-10-12T00:39:18.373 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:39:18.374 DEBUG:boto:Data: 2023-10-12T00:39:18.374 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.375 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.376 DEBUG:boto:Params: {} 2023-10-12T00:39:18.376 DEBUG:boto:Token: None 2023-10-12T00:39:18.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000498484741017fe2-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.381 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:18.382 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.382 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.383 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.383 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.384 DEBUG:boto:Method: GET 2023-10-12T00:39:18.385 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.385 DEBUG:boto:Data: 2023-10-12T00:39:18.385 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.386 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.387 DEBUG:boto:Params: {} 2023-10-12T00:39:18.387 DEBUG:boto:Token: None 2023-10-12T00:39:18.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0= 2023-10-12T00:39:18.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0=', 'Content-Length': '0'} 2023-10-12T00:39:18.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5d12163728c4889-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.391 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.392 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.392 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.393 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.393 DEBUG:boto:Method: GET 2023-10-12T00:39:18.393 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.394 DEBUG:boto:Data: 2023-10-12T00:39:18.394 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.395 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.396 DEBUG:boto:Params: {} 2023-10-12T00:39:18.396 DEBUG:boto:Token: None 2023-10-12T00:39:18.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0= 2023-10-12T00:39:18.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0=', 'Content-Length': '0'} 2023-10-12T00:39:18.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaf5b986a8fb3785-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.401 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.402 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.402 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.403 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.403 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.404 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.404 DEBUG:boto:Data: 2023-10-12T00:39:18.405 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.406 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.406 DEBUG:boto:Params: {} 2023-10-12T00:39:18.407 DEBUG:boto:Token: None 2023-10-12T00:39:18.407 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bwmdC/Ao4lKv0XoJefjZlBbW0CU= 2023-10-12T00:39:18.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bwmdC/Ao4lKv0XoJefjZlBbW0CU=', 'Content-Length': '0'} 2023-10-12T00:39:18.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c046e46ddb6565dd-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.412 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.413 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.414 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.415 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.415 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.415 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.416 DEBUG:boto:Data: 2023-10-12T00:39:18.416 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.417 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.418 DEBUG:boto:Params: {} 2023-10-12T00:39:18.418 DEBUG:boto:Token: None 2023-10-12T00:39:18.419 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bwmdC/Ao4lKv0XoJefjZlBbW0CU= 2023-10-12T00:39:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bwmdC/Ao4lKv0XoJefjZlBbW0CU=', 'Content-Length': '0'} 2023-10-12T00:39:18.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004855813929d8d644-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:18.423 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.424 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.424 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.425 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.425 DEBUG:boto:Method: GET 2023-10-12T00:39:18.426 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.426 DEBUG:boto:Data: 2023-10-12T00:39:18.427 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.428 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.428 DEBUG:boto:Params: {} 2023-10-12T00:39:18.429 DEBUG:boto:Token: None 2023-10-12T00:39:18.429 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.430 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0= 2023-10-12T00:39:18.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0=', 'Content-Length': '0'} 2023-10-12T00:39:18.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000565ae9997bd4c0e-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.433 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.434 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.434 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.435 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.435 DEBUG:boto:Method: GET 2023-10-12T00:39:18.436 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.436 DEBUG:boto:Data: 2023-10-12T00:39:18.437 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.437 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.438 DEBUG:boto:Params: {} 2023-10-12T00:39:18.438 DEBUG:boto:Token: None 2023-10-12T00:39:18.439 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:39:18.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0= 2023-10-12T00:39:18.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WNNppAKsu3W+z+Zwv8Au9TkCBt0=', 'Content-Length': '0'} 2023-10-12T00:39:18.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba840ef57dd6bff8-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.443 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.443 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.444 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.444 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.444 DEBUG:boto:Method: GET 2023-10-12T00:39:18.445 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.445 DEBUG:boto:Data: 2023-10-12T00:39:18.446 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.447 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.447 DEBUG:boto:Params: {} 2023-10-12T00:39:18.448 DEBUG:boto:Token: None 2023-10-12T00:39:18.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e37c934481625ac7-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.452 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.453 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.453 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.454 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.454 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.455 DEBUG:boto:Method: GET 2023-10-12T00:39:18.455 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.455 DEBUG:boto:Data: 2023-10-12T00:39:18.456 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.457 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.457 DEBUG:boto:Params: {} 2023-10-12T00:39:18.458 DEBUG:boto:Token: None 2023-10-12T00:39:18.458 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f719df6b2ad2672c-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.502 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.503 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.504 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.504 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.504 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:39:18.505 DEBUG:boto:Data: 2023-10-12T00:39:18.505 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.506 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.507 DEBUG:boto:Params: {} 2023-10-12T00:39:18.507 DEBUG:boto:Token: None 2023-10-12T00:39:18.508 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3 2023-10-12T00:39:18.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eHkS2E32dVFmKSGRxMxOG48iDOA= 2023-10-12T00:39:18.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eHkS2E32dVFmKSGRxMxOG48iDOA=', 'Content-Length': '0'} 2023-10-12T00:39:18.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bae7940e87d067c-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.511 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.512 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.512 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.513 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:39:18.513 DEBUG:boto:Data: 2023-10-12T00:39:18.514 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.515 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.515 DEBUG:boto:Params: {} 2023-10-12T00:39:18.515 DEBUG:boto:Token: None 2023-10-12T00:39:18.516 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3 2023-10-12T00:39:18.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eHkS2E32dVFmKSGRxMxOG48iDOA= 2023-10-12T00:39:18.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eHkS2E32dVFmKSGRxMxOG48iDOA=', 'Content-Length': '0'} 2023-10-12T00:39:18.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000412200f83a547917-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:39:18.520 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.520 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.521 DEBUG:boto:Method: GET 2023-10-12T00:39:18.521 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:39:18.522 DEBUG:boto:Data: 2023-10-12T00:39:18.522 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.523 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.523 DEBUG:boto:Params: {} 2023-10-12T00:39:18.524 DEBUG:boto:Token: None 2023-10-12T00:39:18.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3 2023-10-12T00:39:18.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PvpItUvt8Ypb0Ip9rEjI5e1uLOI= 2023-10-12T00:39:18.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PvpItUvt8Ypb0Ip9rEjI5e1uLOI=', 'Content-Length': '0'} 2023-10-12T00:39:18.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089166b55996a1f5f-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.570 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:39:18.571 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:39:18.572 DEBUG:boto:Method: GET 2023-10-12T00:39:18.572 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:39:18.573 DEBUG:boto:Data: 2023-10-12T00:39:18.573 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.576 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.576 DEBUG:boto:Params: {} 2023-10-12T00:39:18.577 DEBUG:boto:Token: None 2023-10-12T00:39:18.578 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj3 2023-10-12T00:39:18.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PvpItUvt8Ypb0Ip9rEjI5e1uLOI= 2023-10-12T00:39:18.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PvpItUvt8Ypb0Ip9rEjI5e1uLOI=', 'Content-Length': '0'} 2023-10-12T00:39:18.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ad16159c5594a4d-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.581 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.582 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.582 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.583 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.583 DEBUG:boto:Method: GET 2023-10-12T00:39:18.584 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.584 DEBUG:boto:Data: 2023-10-12T00:39:18.585 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.586 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.586 DEBUG:boto:Params: {} 2023-10-12T00:39:18.586 DEBUG:boto:Token: None 2023-10-12T00:39:18.587 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.588 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3cae1eaca6ff35-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.591 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.592 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.592 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.593 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.593 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.594 DEBUG:boto:Method: GET 2023-10-12T00:39:18.594 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:39:18.595 DEBUG:boto:Data: 2023-10-12T00:39:18.595 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.596 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.596 DEBUG:boto:Params: {} 2023-10-12T00:39:18.597 DEBUG:boto:Token: None 2023-10-12T00:39:18.597 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013151b5a51198ce2-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.601 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:18.602 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.603 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.603 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.604 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.604 DEBUG:boto:Method: GET 2023-10-12T00:39:18.605 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.605 DEBUG:boto:Data: 2023-10-12T00:39:18.605 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.606 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.607 DEBUG:boto:Params: {} 2023-10-12T00:39:18.607 DEBUG:boto:Token: None 2023-10-12T00:39:18.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM= 2023-10-12T00:39:18.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM=', 'Content-Length': '0'} 2023-10-12T00:39:18.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fd352bcf66e5c94-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.611 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.613 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.614 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.614 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.615 DEBUG:boto:Method: GET 2023-10-12T00:39:18.615 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.615 DEBUG:boto:Data: 2023-10-12T00:39:18.616 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.617 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.617 DEBUG:boto:Params: {} 2023-10-12T00:39:18.618 DEBUG:boto:Token: None 2023-10-12T00:39:18.618 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM= 2023-10-12T00:39:18.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM=', 'Content-Length': '0'} 2023-10-12T00:39:18.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039ab41a74e1a6f5e-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.622 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.623 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.623 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.623 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.624 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.624 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.625 DEBUG:boto:Data: 2023-10-12T00:39:18.625 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.626 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.626 DEBUG:boto:Params: {} 2023-10-12T00:39:18.627 DEBUG:boto:Token: None 2023-10-12T00:39:18.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7YYNM1oxRRXo2xjQ3QJYp22OVEY= 2023-10-12T00:39:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7YYNM1oxRRXo2xjQ3QJYp22OVEY=', 'Content-Length': '0'} 2023-10-12T00:39:18.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000843e25f30764a46e-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.630 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.631 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.631 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.631 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.632 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.632 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.633 DEBUG:boto:Data: 2023-10-12T00:39:18.633 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.634 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.635 DEBUG:boto:Params: {} 2023-10-12T00:39:18.635 DEBUG:boto:Token: None 2023-10-12T00:39:18.636 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7YYNM1oxRRXo2xjQ3QJYp22OVEY= 2023-10-12T00:39:18.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7YYNM1oxRRXo2xjQ3QJYp22OVEY=', 'Content-Length': '0'} 2023-10-12T00:39:18.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c84485baf44bff7-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:18.639 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.640 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.640 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.641 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.641 DEBUG:boto:Method: GET 2023-10-12T00:39:18.642 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.642 DEBUG:boto:Data: 2023-10-12T00:39:18.642 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.643 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.644 DEBUG:boto:Params: {} 2023-10-12T00:39:18.644 DEBUG:boto:Token: None 2023-10-12T00:39:18.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM= 2023-10-12T00:39:18.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM=', 'Content-Length': '0'} 2023-10-12T00:39:18.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048a40a24709ced28-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.648 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.648 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.649 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.649 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.650 DEBUG:boto:Method: GET 2023-10-12T00:39:18.650 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.651 DEBUG:boto:Data: 2023-10-12T00:39:18.651 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.652 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.652 DEBUG:boto:Params: {} 2023-10-12T00:39:18.653 DEBUG:boto:Token: None 2023-10-12T00:39:18.654 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:39:18.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM= 2023-10-12T00:39:18.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2mhwU/YvCE6qAWGeXrIW67aQbeM=', 'Content-Length': '0'} 2023-10-12T00:39:18.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ba71e7162e43581-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.657 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.658 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.658 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.659 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.659 DEBUG:boto:Method: GET 2023-10-12T00:39:18.659 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.660 DEBUG:boto:Data: 2023-10-12T00:39:18.660 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.661 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.662 DEBUG:boto:Params: {} 2023-10-12T00:39:18.662 DEBUG:boto:Token: None 2023-10-12T00:39:18.663 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000592f773511ba79b-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.667 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.667 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.668 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.668 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.669 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.669 DEBUG:boto:Method: GET 2023-10-12T00:39:18.670 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.670 DEBUG:boto:Data: 2023-10-12T00:39:18.671 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.671 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.672 DEBUG:boto:Params: {} 2023-10-12T00:39:18.672 DEBUG:boto:Token: None 2023-10-12T00:39:18.673 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.673 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b02dba09536d6526-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.677 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.678 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.678 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.679 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.679 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:39:18.679 DEBUG:boto:Data: 2023-10-12T00:39:18.680 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.681 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.681 DEBUG:boto:Params: {} 2023-10-12T00:39:18.681 DEBUG:boto:Token: None 2023-10-12T00:39:18.682 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4 2023-10-12T00:39:18.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:atiuURKi+KTnL/YdEtzfS/tgn10= 2023-10-12T00:39:18.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:atiuURKi+KTnL/YdEtzfS/tgn10=', 'Content-Length': '0'} 2023-10-12T00:39:18.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b574948f1bbd762-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.685 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.686 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.687 DEBUG:boto:Method: HEAD 2023-10-12T00:39:18.687 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:39:18.688 DEBUG:boto:Data: 2023-10-12T00:39:18.688 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.689 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.690 DEBUG:boto:Params: {} 2023-10-12T00:39:18.691 DEBUG:boto:Token: None 2023-10-12T00:39:18.692 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4 2023-10-12T00:39:18.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:atiuURKi+KTnL/YdEtzfS/tgn10= 2023-10-12T00:39:18.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:atiuURKi+KTnL/YdEtzfS/tgn10=', 'Content-Length': '0'} 2023-10-12T00:39:18.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000843c847c314c1306-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:39:18.696 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.697 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.697 DEBUG:boto:Method: GET 2023-10-12T00:39:18.698 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:39:18.698 DEBUG:boto:Data: 2023-10-12T00:39:18.699 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.700 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.700 DEBUG:boto:Params: {} 2023-10-12T00:39:18.701 DEBUG:boto:Token: None 2023-10-12T00:39:18.701 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4 2023-10-12T00:39:18.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NjjOWBzgY98HhxCt3+5qb6z+uz0= 2023-10-12T00:39:18.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NjjOWBzgY98HhxCt3+5qb6z+uz0=', 'Content-Length': '0'} 2023-10-12T00:39:18.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bbaf4e72f5facb3-0065274036-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.746 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:39:18.747 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:39:18.747 DEBUG:boto:Method: GET 2023-10-12T00:39:18.748 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:39:18.748 DEBUG:boto:Data: 2023-10-12T00:39:18.749 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.750 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.750 DEBUG:boto:Params: {} 2023-10-12T00:39:18.751 DEBUG:boto:Token: None 2023-10-12T00:39:18.751 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/obj4 2023-10-12T00:39:18.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NjjOWBzgY98HhxCt3+5qb6z+uz0= 2023-10-12T00:39:18.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NjjOWBzgY98HhxCt3+5qb6z+uz0=', 'Content-Length': '0'} 2023-10-12T00:39:18.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000964f54be0b51741-0065274036-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.755 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.755 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.756 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.756 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.757 DEBUG:boto:Method: GET 2023-10-12T00:39:18.757 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.758 DEBUG:boto:Data: 2023-10-12T00:39:18.758 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:18.759 DEBUG:boto:Port: 8001 2023-10-12T00:39:18.760 DEBUG:boto:Params: {} 2023-10-12T00:39:18.760 DEBUG:boto:Token: None 2023-10-12T00:39:18.761 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5993ca7487c1e6-0065274036-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.764 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.765 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:18.765 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:18.766 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.766 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.767 DEBUG:boto:Method: GET 2023-10-12T00:39:18.767 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:39:18.768 DEBUG:boto:Data: 2023-10-12T00:39:18.768 DEBUG:boto:Headers: {} 2023-10-12T00:39:18.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:18.769 DEBUG:boto:Port: 8000 2023-10-12T00:39:18.769 DEBUG:boto:Params: {} 2023-10-12T00:39:18.770 DEBUG:boto:Token: None 2023-10-12T00:39:18.770 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:18 GMT /wafmoa-47/ 2023-10-12T00:39:18.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q= 2023-10-12T00:39:18.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sueVv7b/dQPB5IdpEN02uaChz4Q=', 'Content-Length': '0'} 2023-10-12T00:39:18.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e58f6e14c0d2276-0065274036-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:18.814 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:18.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-47 zones={a2, a1} 2023-10-12T00:39:18.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:18.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:18.818 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:39:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:18.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'wafmoa-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:18.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'wafmoa-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:18.934 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket wafmoa-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:39:18.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:18.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:19.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:19.048 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:39:19.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:19.098 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:19.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'wafmoa-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:19.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'wafmoa-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:39:19.163 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket wafmoa-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:39:19.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:19.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:19.277 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:19.278 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:19.278 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.279 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:19.279 DEBUG:boto:Data: 2023-10-12T00:39:19.280 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.281 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.282 DEBUG:boto:Params: {} 2023-10-12T00:39:19.283 DEBUG:boto:Token: None 2023-10-12T00:39:19.283 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/ 2023-10-12T00:39:19.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4= 2023-10-12T00:39:19.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4=', 'Content-Length': '0'} 2023-10-12T00:39:19.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2596f5811d79fb0-0065274037-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.288 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:39:19.288 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:39:19.289 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.289 DEBUG:boto:Path: /wafmoa-46/obj5 2023-10-12T00:39:19.290 DEBUG:boto:Data: 2023-10-12T00:39:19.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.291 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.291 DEBUG:boto:Params: {} 2023-10-12T00:39:19.292 DEBUG:boto:Token: None 2023-10-12T00:39:19.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/obj5 2023-10-12T00:39:19.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i8AYWRffwy4nlH+23jh07zyPR8s= 2023-10-12T00:39:19.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i8AYWRffwy4nlH+23jh07zyPR8s='} 2023-10-12T00:39:19.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000225b5941ad14f7f8-0065274037-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.299 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:19.300 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:19.300 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.300 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:19.301 DEBUG:boto:Data: 2023-10-12T00:39:19.301 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.302 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.303 DEBUG:boto:Params: {} 2023-10-12T00:39:19.303 DEBUG:boto:Token: None 2023-10-12T00:39:19.303 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/ 2023-10-12T00:39:19.304 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4= 2023-10-12T00:39:19.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4=', 'Content-Length': '0'} 2023-10-12T00:39:19.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000046782303d8a780a-0065274037-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.307 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:39:19.308 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:39:19.308 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.309 DEBUG:boto:Path: /wafmoa-46/obj6 2023-10-12T00:39:19.309 DEBUG:boto:Data: 2023-10-12T00:39:19.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.311 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.312 DEBUG:boto:Params: {} 2023-10-12T00:39:19.313 DEBUG:boto:Token: None 2023-10-12T00:39:19.314 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/obj6 2023-10-12T00:39:19.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V2JwWZ9bp2eb8oviG5PHNR90jSY= 2023-10-12T00:39:19.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V2JwWZ9bp2eb8oviG5PHNR90jSY='} 2023-10-12T00:39:19.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e51e90215e9a49b0-0065274037-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.359 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:19.360 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:19.360 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.360 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:19.361 DEBUG:boto:Data: 2023-10-12T00:39:19.361 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.362 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.362 DEBUG:boto:Params: {} 2023-10-12T00:39:19.363 DEBUG:boto:Token: None 2023-10-12T00:39:19.364 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/ 2023-10-12T00:39:19.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4= 2023-10-12T00:39:19.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4=', 'Content-Length': '0'} 2023-10-12T00:39:19.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be905e64f620d00-0065274037-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.368 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:39:19.369 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:39:19.369 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.370 DEBUG:boto:Path: /wafmoa-46/obj7 2023-10-12T00:39:19.371 DEBUG:boto:Data: 2023-10-12T00:39:19.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.373 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.373 DEBUG:boto:Params: {} 2023-10-12T00:39:19.374 DEBUG:boto:Token: None 2023-10-12T00:39:19.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/obj7 2023-10-12T00:39:19.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jz3EMVxMycTcv2z5j91sEE4Ntjc= 2023-10-12T00:39:19.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jz3EMVxMycTcv2z5j91sEE4Ntjc='} 2023-10-12T00:39:19.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0481a89da6f0be6-0065274037-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.382 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:19.383 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:19.383 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.384 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:19.384 DEBUG:boto:Data: 2023-10-12T00:39:19.384 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.386 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.386 DEBUG:boto:Params: {} 2023-10-12T00:39:19.387 DEBUG:boto:Token: None 2023-10-12T00:39:19.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/ 2023-10-12T00:39:19.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4= 2023-10-12T00:39:19.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:soBiPk2noAWj55dDnI3J8gM96k4=', 'Content-Length': '0'} 2023-10-12T00:39:19.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bcaea787ea4208c-0065274037-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.391 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:39:19.392 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:39:19.392 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.393 DEBUG:boto:Path: /wafmoa-46/obj8 2023-10-12T00:39:19.393 DEBUG:boto:Data: 2023-10-12T00:39:19.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:19.395 DEBUG:boto:Port: 8000 2023-10-12T00:39:19.396 DEBUG:boto:Params: {} 2023-10-12T00:39:19.396 DEBUG:boto:Token: None 2023-10-12T00:39:19.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-46/obj8 2023-10-12T00:39:19.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fiwZN7vgzOj3xw7tjsmndCJYxow= 2023-10-12T00:39:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fiwZN7vgzOj3xw7tjsmndCJYxow='} 2023-10-12T00:39:19.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd6b5fe5531f8c09-0065274037-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.443 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:19.444 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:19.444 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.445 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:19.445 DEBUG:boto:Data: 2023-10-12T00:39:19.445 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.446 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.447 DEBUG:boto:Params: {} 2023-10-12T00:39:19.447 DEBUG:boto:Token: None 2023-10-12T00:39:19.448 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/ 2023-10-12T00:39:19.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs= 2023-10-12T00:39:19.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs=', 'Content-Length': '0'} 2023-10-12T00:39:19.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005627b2d2b7fd08f6-0065274037-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.452 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:39:19.452 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:39:19.452 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.453 DEBUG:boto:Path: /wafmoa-47/obj5 2023-10-12T00:39:19.453 DEBUG:boto:Data: 2023-10-12T00:39:19.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.455 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.455 DEBUG:boto:Params: {} 2023-10-12T00:39:19.455 DEBUG:boto:Token: None 2023-10-12T00:39:19.456 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/obj5 2023-10-12T00:39:19.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rKXQss3cvY0d9Q+W775h1hxf43s= 2023-10-12T00:39:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rKXQss3cvY0d9Q+W775h1hxf43s='} 2023-10-12T00:39:19.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ace4fc0227289d1c-0065274037-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.462 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:19.463 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:19.463 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.464 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:19.464 DEBUG:boto:Data: 2023-10-12T00:39:19.464 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.465 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.466 DEBUG:boto:Params: {} 2023-10-12T00:39:19.466 DEBUG:boto:Token: None 2023-10-12T00:39:19.467 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/ 2023-10-12T00:39:19.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs= 2023-10-12T00:39:19.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs=', 'Content-Length': '0'} 2023-10-12T00:39:19.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b1d90a9b79495ab-0065274037-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.471 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:39:19.472 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:39:19.472 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.473 DEBUG:boto:Path: /wafmoa-47/obj6 2023-10-12T00:39:19.473 DEBUG:boto:Data: 2023-10-12T00:39:19.474 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.474 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.475 DEBUG:boto:Params: {} 2023-10-12T00:39:19.475 DEBUG:boto:Token: None 2023-10-12T00:39:19.476 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/obj6 2023-10-12T00:39:19.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CNGDb6k+AVXlk/phzYrNJm9ljAk= 2023-10-12T00:39:19.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CNGDb6k+AVXlk/phzYrNJm9ljAk='} 2023-10-12T00:39:19.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8180e97066e46ab-0065274037-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.519 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:19.520 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:19.520 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.520 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:19.521 DEBUG:boto:Data: 2023-10-12T00:39:19.521 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.522 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.523 DEBUG:boto:Params: {} 2023-10-12T00:39:19.523 DEBUG:boto:Token: None 2023-10-12T00:39:19.524 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/ 2023-10-12T00:39:19.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs= 2023-10-12T00:39:19.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs=', 'Content-Length': '0'} 2023-10-12T00:39:19.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000289d8a245b3c914-0065274037-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.527 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:39:19.528 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:39:19.528 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.529 DEBUG:boto:Path: /wafmoa-47/obj7 2023-10-12T00:39:19.529 DEBUG:boto:Data: 2023-10-12T00:39:19.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.530 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.531 DEBUG:boto:Params: {} 2023-10-12T00:39:19.531 DEBUG:boto:Token: None 2023-10-12T00:39:19.532 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/obj7 2023-10-12T00:39:19.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eTMQPj9X+e9bgIzhxWWgrNixCVY= 2023-10-12T00:39:19.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eTMQPj9X+e9bgIzhxWWgrNixCVY='} 2023-10-12T00:39:19.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c4d204965d1b2e5-0065274037-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.538 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:39:19.538 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:39:19.539 DEBUG:boto:Method: HEAD 2023-10-12T00:39:19.539 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:39:19.539 DEBUG:boto:Data: 2023-10-12T00:39:19.540 DEBUG:boto:Headers: {} 2023-10-12T00:39:19.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.541 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.541 DEBUG:boto:Params: {} 2023-10-12T00:39:19.541 DEBUG:boto:Token: None 2023-10-12T00:39:19.542 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/ 2023-10-12T00:39:19.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs= 2023-10-12T00:39:19.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RVR0yLW4guUx3iQ2naHVMcpqwQs=', 'Content-Length': '0'} 2023-10-12T00:39:19.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3799ab37b534742-0065274037-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.546 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:39:19.546 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:39:19.546 DEBUG:boto:Method: PUT 2023-10-12T00:39:19.547 DEBUG:boto:Path: /wafmoa-47/obj8 2023-10-12T00:39:19.547 DEBUG:boto:Data: 2023-10-12T00:39:19.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:39:19.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:19.548 DEBUG:boto:Port: 8001 2023-10-12T00:39:19.549 DEBUG:boto:Params: {} 2023-10-12T00:39:19.549 DEBUG:boto:Token: None 2023-10-12T00:39:19.550 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:39:19 GMT /wafmoa-47/obj8 2023-10-12T00:39:19.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QoG8iorBOxd8wPcSbP86nVJhEzI= 2023-10-12T00:39:19.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:39:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QoG8iorBOxd8wPcSbP86nVJhEzI='} 2023-10-12T00:39:19.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042e5c9136c5223b4-0065274037-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:19.595 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-46 2023-10-12T00:39:19.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:19.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:19.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:39:19.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:19.690+0000 7f9fe5883780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:19.692 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.23.6, , 00000000001.22.6, , 00000000001.25.6, , , 00000000001.18.6, , , ] 2023-10-12T00:39:19.692 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.29.6, 00000000002.46.1, 00000000004.26.6, 00000000002.30.1, 00000000004.31.6, 00000000002.25.1, 00000000002.21.1, 00000000004.24.6, 00000000002.13.1, 00000000002.22.1, 00000000002.31.1] 2023-10-12T00:39:24.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:24.692+0000 7f9fe5883780 1 bucket sync caught up with source: 2023-10-12T00:39:24.693 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.29.6, 00000000002.46.1, 00000000004.26.6, 00000000002.30.1, 00000000004.31.6, 00000000002.25.1, 00000000002.21.1, 00000000004.24.6, 00000000002.13.1, 00000000002.22.1, 00000000002.31.1] 2023-10-12T00:39:24.693 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.29.6, 00000000002.46.1, 00000000004.26.6, 00000000002.30.1, 00000000004.31.6, 00000000002.25.1, 00000000002.21.1, 00000000004.24.6, 00000000002.13.1, 00000000002.22.1, 00000000002.31.1] 2023-10-12T00:39:24.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:24.692+0000 7f9fe5883780 0 bucket checkpoint complete 2023-10-12T00:39:24.710 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-46 2023-10-12T00:39:24.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:24.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:39:24.712 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:39:24.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:39:24.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:39:24.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:24.814+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:24.814 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.164.6, , 00000000001.256.6, , 00000000001.235.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:24.815 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:29.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:29.815+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:29.816 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:29.816 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:34.816+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:34.817 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:34.817 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:39.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:39.816+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:39.817 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:39.817 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:44.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:44.817+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:44.818 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:44.818 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:49.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:49.818+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:49.819 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:49.820 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:54.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:54.820+0000 7fb950f34780 1 waiting for incremental sync to catch up: 2023-10-12T00:39:54.821 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000001.256.6, , 00000000002.250.6, , , 00000000001.121.6, , , ] 2023-10-12T00:39:54.821 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:59.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:59.822+0000 7fb950f34780 1 bucket sync caught up with source: 2023-10-12T00:39:59.822 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:59.823 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.179.6, , 00000000002.264.6, , 00000000002.250.6, , , 00000000002.135.6, , , ] 2023-10-12T00:39:59.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:39:59.822+0000 7fb950f34780 0 bucket checkpoint complete 2023-10-12T00:39:59.838 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-46 zones={a2, a1} 2023-10-12T00:39:59.839 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.840 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.840 DEBUG:boto:Method: HEAD 2023-10-12T00:39:59.840 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:59.841 DEBUG:boto:Data: 2023-10-12T00:39:59.842 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.843 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.843 DEBUG:boto:Params: {} 2023-10-12T00:39:59.844 DEBUG:boto:Token: None 2023-10-12T00:39:59.844 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/ 2023-10-12T00:39:59.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MaRtC5irfFz2PPCSr1ovbMD94Y8= 2023-10-12T00:39:59.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MaRtC5irfFz2PPCSr1ovbMD94Y8=', 'Content-Length': '0'} 2023-10-12T00:39:59.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001151abd94c8c4cc9-006527405f-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.848 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.849 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.849 DEBUG:boto:Method: HEAD 2023-10-12T00:39:59.850 DEBUG:boto:Path: /wafmoa-46/ 2023-10-12T00:39:59.850 DEBUG:boto:Data: 2023-10-12T00:39:59.850 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.852 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.853 DEBUG:boto:Params: {} 2023-10-12T00:39:59.853 DEBUG:boto:Token: None 2023-10-12T00:39:59.854 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/ 2023-10-12T00:39:59.854 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MaRtC5irfFz2PPCSr1ovbMD94Y8= 2023-10-12T00:39:59.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MaRtC5irfFz2PPCSr1ovbMD94Y8=', 'Content-Length': '0'} 2023-10-12T00:39:59.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8514dbca24b9e02-006527405f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:39:59.858 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.859 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.859 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.860 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.860 DEBUG:boto:Method: GET 2023-10-12T00:39:59.860 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.861 DEBUG:boto:Data: 2023-10-12T00:39:59.861 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.862 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.863 DEBUG:boto:Params: {} 2023-10-12T00:39:59.864 DEBUG:boto:Token: None 2023-10-12T00:39:59.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/?versions 2023-10-12T00:39:59.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc= 2023-10-12T00:39:59.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc=', 'Content-Length': '0'} 2023-10-12T00:39:59.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c02cd3bc7d5c832-006527405f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.868 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:59.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:59.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:59.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:59.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:59.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:39:59.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:39:59.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:39:59.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:39:59.873 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:39:59.874 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.874 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.875 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.875 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.875 DEBUG:boto:Method: GET 2023-10-12T00:39:59.876 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.876 DEBUG:boto:Data: 2023-10-12T00:39:59.877 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.878 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.878 DEBUG:boto:Params: {} 2023-10-12T00:39:59.879 DEBUG:boto:Token: None 2023-10-12T00:39:59.879 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/?versions 2023-10-12T00:39:59.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc= 2023-10-12T00:39:59.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc=', 'Content-Length': '0'} 2023-10-12T00:39:59.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ad5f91ca053e480-006527405f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.884 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:59.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:39:59.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:39:59.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:39:59.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:39:59.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:39:59.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:39:59.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:39:59.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:39:59.888 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.888 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.888 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.889 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.889 DEBUG:boto:Method: GET 2023-10-12T00:39:59.890 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.890 DEBUG:boto:Data: 2023-10-12T00:39:59.890 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.891 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.892 DEBUG:boto:Params: {} 2023-10-12T00:39:59.892 DEBUG:boto:Token: None 2023-10-12T00:39:59.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/?versions 2023-10-12T00:39:59.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc= 2023-10-12T00:39:59.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc=', 'Content-Length': '0'} 2023-10-12T00:39:59.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35d00e8d05cf80e-006527405f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.897 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:59.898 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:39:59.899 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:39:59.899 DEBUG:boto:path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.900 DEBUG:boto:auth_path=/wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.900 DEBUG:boto:Method: GET 2023-10-12T00:39:59.900 DEBUG:boto:Path: /wafmoa-46/?versions&max-keys=999 2023-10-12T00:39:59.901 DEBUG:boto:Data: 2023-10-12T00:39:59.901 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.902 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.903 DEBUG:boto:Params: {} 2023-10-12T00:39:59.903 DEBUG:boto:Token: None 2023-10-12T00:39:59.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/?versions 2023-10-12T00:39:59.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc= 2023-10-12T00:39:59.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JsGClMqGCAutocJrkp8r4CDwpcc=', 'Content-Length': '0'} 2023-10-12T00:39:59.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c843d8049d7c48-006527405f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.910 DEBUG:boto:b'wafmoa-46999falseobj1nulltrue2023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:39:59.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:59.911 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.912 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.912 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.913 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.913 DEBUG:boto:Method: GET 2023-10-12T00:39:59.914 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.915 DEBUG:boto:Data: 2023-10-12T00:39:59.915 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.916 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.917 DEBUG:boto:Params: {} 2023-10-12T00:39:59.917 DEBUG:boto:Token: None 2023-10-12T00:39:59.918 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.918 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk= 2023-10-12T00:39:59.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk=', 'Content-Length': '0'} 2023-10-12T00:39:59.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af9c141da16baa58-006527405f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.922 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.923 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.923 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.924 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.924 DEBUG:boto:Method: GET 2023-10-12T00:39:59.925 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.925 DEBUG:boto:Data: 2023-10-12T00:39:59.926 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.927 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.927 DEBUG:boto:Params: {} 2023-10-12T00:39:59.928 DEBUG:boto:Token: None 2023-10-12T00:39:59.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk= 2023-10-12T00:39:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk=', 'Content-Length': '0'} 2023-10-12T00:39:59.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020f49800307c9519-006527405f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.932 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.933 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.933 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.934 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.934 DEBUG:boto:Method: HEAD 2023-10-12T00:39:59.935 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.936 DEBUG:boto:Data: 2023-10-12T00:39:59.936 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.937 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.937 DEBUG:boto:Params: {} 2023-10-12T00:39:59.938 DEBUG:boto:Token: None 2023-10-12T00:39:59.938 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H7ehofj3Ln6fGj3udrxPiOpv4h0= 2023-10-12T00:39:59.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H7ehofj3Ln6fGj3udrxPiOpv4h0=', 'Content-Length': '0'} 2023-10-12T00:39:59.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057916c54ba7a2ffd-006527405f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.942 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.943 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.943 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.943 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.944 DEBUG:boto:Method: HEAD 2023-10-12T00:39:59.944 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.945 DEBUG:boto:Data: 2023-10-12T00:39:59.946 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.946 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.947 DEBUG:boto:Params: {} 2023-10-12T00:39:59.947 DEBUG:boto:Token: None 2023-10-12T00:39:59.948 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H7ehofj3Ln6fGj3udrxPiOpv4h0= 2023-10-12T00:39:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H7ehofj3Ln6fGj3udrxPiOpv4h0=', 'Content-Length': '0'} 2023-10-12T00:39:59.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bae43f9f528bcf7f-006527405f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:39:59.952 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.952 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.953 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.953 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.954 DEBUG:boto:Method: GET 2023-10-12T00:39:59.955 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.955 DEBUG:boto:Data: 2023-10-12T00:39:59.956 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:39:59.957 DEBUG:boto:Port: 8001 2023-10-12T00:39:59.957 DEBUG:boto:Params: {} 2023-10-12T00:39:59.957 DEBUG:boto:Token: None 2023-10-12T00:39:59.958 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk= 2023-10-12T00:39:59.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk=', 'Content-Length': '0'} 2023-10-12T00:39:59.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a21f5cf3e9a42c5e-006527405f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:39:59.962 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:39:59.962 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:39:59.963 DEBUG:boto:path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.963 DEBUG:boto:auth_path=/wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.964 DEBUG:boto:Method: GET 2023-10-12T00:39:59.964 DEBUG:boto:Path: /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.965 DEBUG:boto:Data: 2023-10-12T00:39:59.965 DEBUG:boto:Headers: {} 2023-10-12T00:39:59.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:39:59.966 DEBUG:boto:Port: 8000 2023-10-12T00:39:59.967 DEBUG:boto:Params: {} 2023-10-12T00:39:59.967 DEBUG:boto:Token: None 2023-10-12T00:39:59.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:39:59 GMT /wafmoa-46/obj1?versionId=null 2023-10-12T00:39:59.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk= 2023-10-12T00:39:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:39:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU6ISQ1ITaPivmdD7eaNDEXnnQk=', 'Content-Length': '0'} 2023-10-12T00:39:59.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e054357fefc31f0-006527405f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:39:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.018 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.019 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.019 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.020 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.020 DEBUG:boto:Method: GET 2023-10-12T00:40:00.021 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.021 DEBUG:boto:Data: 2023-10-12T00:40:00.022 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.023 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.023 DEBUG:boto:Params: {} 2023-10-12T00:40:00.024 DEBUG:boto:Token: None 2023-10-12T00:40:00.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097defaec80ddf52f-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.028 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.029 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.030 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.030 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.030 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.031 DEBUG:boto:Method: GET 2023-10-12T00:40:00.031 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.032 DEBUG:boto:Data: 2023-10-12T00:40:00.032 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.033 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.033 DEBUG:boto:Params: {} 2023-10-12T00:40:00.034 DEBUG:boto:Token: None 2023-10-12T00:40:00.034 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.035 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf1e1ec0e8ad86c-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.038 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.039 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:40:00.040 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:40:00.040 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.040 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:40:00.041 DEBUG:boto:Data: 2023-10-12T00:40:00.041 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.042 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.042 DEBUG:boto:Params: {} 2023-10-12T00:40:00.043 DEBUG:boto:Token: None 2023-10-12T00:40:00.043 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj1 2023-10-12T00:40:00.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6c9tAsw+6zR8M1SQzmOXYOPl7kc= 2023-10-12T00:40:00.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6c9tAsw+6zR8M1SQzmOXYOPl7kc=', 'Content-Length': '0'} 2023-10-12T00:40:00.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b2dc7ff68943df1-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.046 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:40:00.047 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:40:00.047 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.048 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:40:00.048 DEBUG:boto:Data: 2023-10-12T00:40:00.049 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.049 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.050 DEBUG:boto:Params: {} 2023-10-12T00:40:00.050 DEBUG:boto:Token: None 2023-10-12T00:40:00.051 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj1 2023-10-12T00:40:00.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6c9tAsw+6zR8M1SQzmOXYOPl7kc= 2023-10-12T00:40:00.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6c9tAsw+6zR8M1SQzmOXYOPl7kc=', 'Content-Length': '0'} 2023-10-12T00:40:00.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c871dfd5b678524e-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:00.055 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:40:00.055 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:40:00.056 DEBUG:boto:Method: GET 2023-10-12T00:40:00.056 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:40:00.057 DEBUG:boto:Data: 2023-10-12T00:40:00.057 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.058 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.058 DEBUG:boto:Params: {} 2023-10-12T00:40:00.059 DEBUG:boto:Token: None 2023-10-12T00:40:00.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj1 2023-10-12T00:40:00.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J65OU0LzYfuo7le79dKC5I+ePls= 2023-10-12T00:40:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J65OU0LzYfuo7le79dKC5I+ePls=', 'Content-Length': '0'} 2023-10-12T00:40:00.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c69e91cb8e74a38b-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.102 DEBUG:boto:path=/wafmoa-46/obj1 2023-10-12T00:40:00.103 DEBUG:boto:auth_path=/wafmoa-46/obj1 2023-10-12T00:40:00.104 DEBUG:boto:Method: GET 2023-10-12T00:40:00.104 DEBUG:boto:Path: /wafmoa-46/obj1 2023-10-12T00:40:00.105 DEBUG:boto:Data: 2023-10-12T00:40:00.105 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.106 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.106 DEBUG:boto:Params: {} 2023-10-12T00:40:00.107 DEBUG:boto:Token: None 2023-10-12T00:40:00.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj1 2023-10-12T00:40:00.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J65OU0LzYfuo7le79dKC5I+ePls= 2023-10-12T00:40:00.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J65OU0LzYfuo7le79dKC5I+ePls=', 'Content-Length': '0'} 2023-10-12T00:40:00.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060933d9a431747d1-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.113 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.114 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.114 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.115 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.115 DEBUG:boto:Method: GET 2023-10-12T00:40:00.115 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.116 DEBUG:boto:Data: 2023-10-12T00:40:00.116 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.118 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.118 DEBUG:boto:Params: {} 2023-10-12T00:40:00.119 DEBUG:boto:Token: None 2023-10-12T00:40:00.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe29d35e5f444ba8-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.123 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.124 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.124 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.125 DEBUG:boto:path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.125 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.125 DEBUG:boto:Method: GET 2023-10-12T00:40:00.126 DEBUG:boto:Path: /wafmoa-46/?prefix=obj1 2023-10-12T00:40:00.126 DEBUG:boto:Data: 2023-10-12T00:40:00.127 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.128 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.128 DEBUG:boto:Params: {} 2023-10-12T00:40:00.128 DEBUG:boto:Token: None 2023-10-12T00:40:00.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa29158ed5ec3b12-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.133 DEBUG:boto:b'wafmoa-46obj11000falseobj12023-10-12T00:38:45.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:00.134 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.135 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.135 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.136 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.136 DEBUG:boto:Method: GET 2023-10-12T00:40:00.137 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.137 DEBUG:boto:Data: 2023-10-12T00:40:00.138 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.138 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.139 DEBUG:boto:Params: {} 2023-10-12T00:40:00.139 DEBUG:boto:Token: None 2023-10-12T00:40:00.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0= 2023-10-12T00:40:00.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0=', 'Content-Length': '0'} 2023-10-12T00:40:00.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055371404d12bb673-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.143 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.144 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.144 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.145 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.145 DEBUG:boto:Method: GET 2023-10-12T00:40:00.145 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.146 DEBUG:boto:Data: 2023-10-12T00:40:00.146 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.147 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.148 DEBUG:boto:Params: {} 2023-10-12T00:40:00.148 DEBUG:boto:Token: None 2023-10-12T00:40:00.149 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0= 2023-10-12T00:40:00.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0=', 'Content-Length': '0'} 2023-10-12T00:40:00.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e0034d54719ed3b-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.152 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.153 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.153 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.153 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.154 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.154 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.155 DEBUG:boto:Data: 2023-10-12T00:40:00.155 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.156 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.156 DEBUG:boto:Params: {} 2023-10-12T00:40:00.157 DEBUG:boto:Token: None 2023-10-12T00:40:00.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L8ej7aaFsI1YLl9AO2HdJBYVNNU= 2023-10-12T00:40:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L8ej7aaFsI1YLl9AO2HdJBYVNNU=', 'Content-Length': '0'} 2023-10-12T00:40:00.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000313e04a5784ebe25-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.160 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.161 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.161 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.162 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.162 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.162 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.163 DEBUG:boto:Data: 2023-10-12T00:40:00.163 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.164 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.164 DEBUG:boto:Params: {} 2023-10-12T00:40:00.165 DEBUG:boto:Token: None 2023-10-12T00:40:00.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L8ej7aaFsI1YLl9AO2HdJBYVNNU= 2023-10-12T00:40:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L8ej7aaFsI1YLl9AO2HdJBYVNNU=', 'Content-Length': '0'} 2023-10-12T00:40:00.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf74a9ba8a9692a1-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:00.169 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.170 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.170 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.171 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.171 DEBUG:boto:Method: GET 2023-10-12T00:40:00.171 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.172 DEBUG:boto:Data: 2023-10-12T00:40:00.172 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.173 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.174 DEBUG:boto:Params: {} 2023-10-12T00:40:00.174 DEBUG:boto:Token: None 2023-10-12T00:40:00.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0= 2023-10-12T00:40:00.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0=', 'Content-Length': '0'} 2023-10-12T00:40:00.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b2746f0f2c218a-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.218 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.219 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.219 DEBUG:boto:path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.220 DEBUG:boto:auth_path=/wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.220 DEBUG:boto:Method: GET 2023-10-12T00:40:00.221 DEBUG:boto:Path: /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.221 DEBUG:boto:Data: 2023-10-12T00:40:00.222 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.223 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.223 DEBUG:boto:Params: {} 2023-10-12T00:40:00.224 DEBUG:boto:Token: None 2023-10-12T00:40:00.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2?versionId=null 2023-10-12T00:40:00.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0= 2023-10-12T00:40:00.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:45PImkrYM1cXDOCD5ZzVre9JQK0=', 'Content-Length': '0'} 2023-10-12T00:40:00.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000356c92de008b4b9-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.228 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.229 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.229 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.230 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.230 DEBUG:boto:Method: GET 2023-10-12T00:40:00.231 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.231 DEBUG:boto:Data: 2023-10-12T00:40:00.231 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.232 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.233 DEBUG:boto:Params: {} 2023-10-12T00:40:00.233 DEBUG:boto:Token: None 2023-10-12T00:40:00.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.234 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a887b77c2a2d806-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.239 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.239 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.240 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.241 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.241 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.242 DEBUG:boto:Method: GET 2023-10-12T00:40:00.242 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.243 DEBUG:boto:Data: 2023-10-12T00:40:00.244 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.245 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.245 DEBUG:boto:Params: {} 2023-10-12T00:40:00.246 DEBUG:boto:Token: None 2023-10-12T00:40:00.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d3d1e49ba6ad97-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.252 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.252 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.253 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.253 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.254 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:40:00.254 DEBUG:boto:Data: 2023-10-12T00:40:00.254 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.255 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.256 DEBUG:boto:Params: {} 2023-10-12T00:40:00.256 DEBUG:boto:Token: None 2023-10-12T00:40:00.257 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2 2023-10-12T00:40:00.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GVbMmiY3s9DwzyOOAy3wEIl/hM= 2023-10-12T00:40:00.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GVbMmiY3s9DwzyOOAy3wEIl/hM=', 'Content-Length': '0'} 2023-10-12T00:40:00.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae799a67a92c5892-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.260 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.261 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.261 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.262 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:40:00.262 DEBUG:boto:Data: 2023-10-12T00:40:00.262 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.263 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.264 DEBUG:boto:Params: {} 2023-10-12T00:40:00.264 DEBUG:boto:Token: None 2023-10-12T00:40:00.265 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2 2023-10-12T00:40:00.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GVbMmiY3s9DwzyOOAy3wEIl/hM= 2023-10-12T00:40:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GVbMmiY3s9DwzyOOAy3wEIl/hM=', 'Content-Length': '0'} 2023-10-12T00:40:00.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cee2bf39d569dcc-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:00.269 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.269 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.270 DEBUG:boto:Method: GET 2023-10-12T00:40:00.270 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:40:00.271 DEBUG:boto:Data: 2023-10-12T00:40:00.271 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.272 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.273 DEBUG:boto:Params: {} 2023-10-12T00:40:00.273 DEBUG:boto:Token: None 2023-10-12T00:40:00.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2 2023-10-12T00:40:00.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rs2TLgGAW9YHR4CmX6mDGIgp6dA= 2023-10-12T00:40:00.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rs2TLgGAW9YHR4CmX6mDGIgp6dA=', 'Content-Length': '0'} 2023-10-12T00:40:00.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd06a0688803ad61-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.278 DEBUG:boto:path=/wafmoa-46/obj2 2023-10-12T00:40:00.279 DEBUG:boto:auth_path=/wafmoa-46/obj2 2023-10-12T00:40:00.279 DEBUG:boto:Method: GET 2023-10-12T00:40:00.280 DEBUG:boto:Path: /wafmoa-46/obj2 2023-10-12T00:40:00.280 DEBUG:boto:Data: 2023-10-12T00:40:00.280 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.282 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.282 DEBUG:boto:Params: {} 2023-10-12T00:40:00.283 DEBUG:boto:Token: None 2023-10-12T00:40:00.284 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj2 2023-10-12T00:40:00.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rs2TLgGAW9YHR4CmX6mDGIgp6dA= 2023-10-12T00:40:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rs2TLgGAW9YHR4CmX6mDGIgp6dA=', 'Content-Length': '0'} 2023-10-12T00:40:00.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000312c63f5edfc7cd8-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.330 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.331 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.332 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.332 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.332 DEBUG:boto:Method: GET 2023-10-12T00:40:00.333 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.333 DEBUG:boto:Data: 2023-10-12T00:40:00.334 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.335 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.335 DEBUG:boto:Params: {} 2023-10-12T00:40:00.336 DEBUG:boto:Token: None 2023-10-12T00:40:00.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e87cfeeb2e7029-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.340 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.341 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.342 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.342 DEBUG:boto:path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.343 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.343 DEBUG:boto:Method: GET 2023-10-12T00:40:00.344 DEBUG:boto:Path: /wafmoa-46/?prefix=obj2 2023-10-12T00:40:00.344 DEBUG:boto:Data: 2023-10-12T00:40:00.345 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.346 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.346 DEBUG:boto:Params: {} 2023-10-12T00:40:00.347 DEBUG:boto:Token: None 2023-10-12T00:40:00.347 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80c3ca5f6964a8d-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.352 DEBUG:boto:b'wafmoa-46obj21000falseobj22023-10-12T00:38:45.895Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:00.353 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.353 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.354 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.354 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.355 DEBUG:boto:Method: GET 2023-10-12T00:40:00.356 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.356 DEBUG:boto:Data: 2023-10-12T00:40:00.357 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.357 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.358 DEBUG:boto:Params: {} 2023-10-12T00:40:00.359 DEBUG:boto:Token: None 2023-10-12T00:40:00.359 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM= 2023-10-12T00:40:00.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM=', 'Content-Length': '0'} 2023-10-12T00:40:00.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000592533dc4e4c3546-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.363 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.363 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.364 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.364 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.365 DEBUG:boto:Method: GET 2023-10-12T00:40:00.365 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.365 DEBUG:boto:Data: 2023-10-12T00:40:00.366 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.367 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.367 DEBUG:boto:Params: {} 2023-10-12T00:40:00.368 DEBUG:boto:Token: None 2023-10-12T00:40:00.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM= 2023-10-12T00:40:00.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM=', 'Content-Length': '0'} 2023-10-12T00:40:00.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e7c7ed764e5414d-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.372 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.372 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.373 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.373 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.374 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.374 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.374 DEBUG:boto:Data: 2023-10-12T00:40:00.375 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.376 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.376 DEBUG:boto:Params: {} 2023-10-12T00:40:00.377 DEBUG:boto:Token: None 2023-10-12T00:40:00.377 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuSxZSLT5m7MttzPYYneJwEaz6A= 2023-10-12T00:40:00.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuSxZSLT5m7MttzPYYneJwEaz6A=', 'Content-Length': '0'} 2023-10-12T00:40:00.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000865ca4fabf363545-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.382 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.382 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.382 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.383 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.383 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.384 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.384 DEBUG:boto:Data: 2023-10-12T00:40:00.385 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.386 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.386 DEBUG:boto:Params: {} 2023-10-12T00:40:00.387 DEBUG:boto:Token: None 2023-10-12T00:40:00.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuSxZSLT5m7MttzPYYneJwEaz6A= 2023-10-12T00:40:00.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuSxZSLT5m7MttzPYYneJwEaz6A=', 'Content-Length': '0'} 2023-10-12T00:40:00.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0ed8fc15570ea4-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:00.391 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.392 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.392 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.392 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.393 DEBUG:boto:Method: GET 2023-10-12T00:40:00.393 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.394 DEBUG:boto:Data: 2023-10-12T00:40:00.394 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.395 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.396 DEBUG:boto:Params: {} 2023-10-12T00:40:00.396 DEBUG:boto:Token: None 2023-10-12T00:40:00.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM= 2023-10-12T00:40:00.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM=', 'Content-Length': '0'} 2023-10-12T00:40:00.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002620a2d1b8c0ef76-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.442 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.443 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.443 DEBUG:boto:path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.444 DEBUG:boto:auth_path=/wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.444 DEBUG:boto:Method: GET 2023-10-12T00:40:00.445 DEBUG:boto:Path: /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.445 DEBUG:boto:Data: 2023-10-12T00:40:00.446 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.447 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.447 DEBUG:boto:Params: {} 2023-10-12T00:40:00.448 DEBUG:boto:Token: None 2023-10-12T00:40:00.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3?versionId=null 2023-10-12T00:40:00.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM= 2023-10-12T00:40:00.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fh7magbEdYnLn2sll69EDWBMdBM=', 'Content-Length': '0'} 2023-10-12T00:40:00.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004be2257b288e1afc-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.452 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.453 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.453 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.454 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.454 DEBUG:boto:Method: GET 2023-10-12T00:40:00.455 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.455 DEBUG:boto:Data: 2023-10-12T00:40:00.456 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.456 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.457 DEBUG:boto:Params: {} 2023-10-12T00:40:00.458 DEBUG:boto:Token: None 2023-10-12T00:40:00.458 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d097b369ecbd6de-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.462 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.463 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.463 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.464 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.464 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.465 DEBUG:boto:Method: GET 2023-10-12T00:40:00.465 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.466 DEBUG:boto:Data: 2023-10-12T00:40:00.466 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.467 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.467 DEBUG:boto:Params: {} 2023-10-12T00:40:00.468 DEBUG:boto:Token: None 2023-10-12T00:40:00.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964e35ce78269ee8-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.473 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.473 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.474 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.474 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.475 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:40:00.475 DEBUG:boto:Data: 2023-10-12T00:40:00.475 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.476 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.477 DEBUG:boto:Params: {} 2023-10-12T00:40:00.477 DEBUG:boto:Token: None 2023-10-12T00:40:00.478 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3 2023-10-12T00:40:00.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m+g8bljrcKCGcJs/7YbU3w1hVxM= 2023-10-12T00:40:00.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m+g8bljrcKCGcJs/7YbU3w1hVxM=', 'Content-Length': '0'} 2023-10-12T00:40:00.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085cb4db5b58aabbe-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.481 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.481 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.482 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.482 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:40:00.483 DEBUG:boto:Data: 2023-10-12T00:40:00.483 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.484 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.484 DEBUG:boto:Params: {} 2023-10-12T00:40:00.485 DEBUG:boto:Token: None 2023-10-12T00:40:00.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3 2023-10-12T00:40:00.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m+g8bljrcKCGcJs/7YbU3w1hVxM= 2023-10-12T00:40:00.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m+g8bljrcKCGcJs/7YbU3w1hVxM=', 'Content-Length': '0'} 2023-10-12T00:40:00.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f543286e4337adf-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:00.490 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.490 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.491 DEBUG:boto:Method: GET 2023-10-12T00:40:00.491 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:40:00.491 DEBUG:boto:Data: 2023-10-12T00:40:00.492 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.493 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.493 DEBUG:boto:Params: {} 2023-10-12T00:40:00.494 DEBUG:boto:Token: None 2023-10-12T00:40:00.494 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3 2023-10-12T00:40:00.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0jdK40+znK0365drP+rmMxFJxYg= 2023-10-12T00:40:00.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0jdK40+znK0365drP+rmMxFJxYg=', 'Content-Length': '0'} 2023-10-12T00:40:00.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074fccd2e2e13db2f-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.498 DEBUG:boto:path=/wafmoa-46/obj3 2023-10-12T00:40:00.498 DEBUG:boto:auth_path=/wafmoa-46/obj3 2023-10-12T00:40:00.499 DEBUG:boto:Method: GET 2023-10-12T00:40:00.499 DEBUG:boto:Path: /wafmoa-46/obj3 2023-10-12T00:40:00.500 DEBUG:boto:Data: 2023-10-12T00:40:00.500 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.502 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.503 DEBUG:boto:Params: {} 2023-10-12T00:40:00.503 DEBUG:boto:Token: None 2023-10-12T00:40:00.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj3 2023-10-12T00:40:00.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0jdK40+znK0365drP+rmMxFJxYg= 2023-10-12T00:40:00.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0jdK40+znK0365drP+rmMxFJxYg=', 'Content-Length': '0'} 2023-10-12T00:40:00.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e99c6ad69209477-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.508 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.509 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.509 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.509 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.510 DEBUG:boto:Method: GET 2023-10-12T00:40:00.511 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.511 DEBUG:boto:Data: 2023-10-12T00:40:00.511 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.512 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.513 DEBUG:boto:Params: {} 2023-10-12T00:40:00.513 DEBUG:boto:Token: None 2023-10-12T00:40:00.514 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491a331d09bbeef6-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.518 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.522 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.523 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.523 DEBUG:boto:path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.524 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.524 DEBUG:boto:Method: GET 2023-10-12T00:40:00.525 DEBUG:boto:Path: /wafmoa-46/?prefix=obj3 2023-10-12T00:40:00.525 DEBUG:boto:Data: 2023-10-12T00:40:00.525 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.526 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.527 DEBUG:boto:Params: {} 2023-10-12T00:40:00.527 DEBUG:boto:Token: None 2023-10-12T00:40:00.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a8aa01b7c00a6e9-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.532 DEBUG:boto:b'wafmoa-46obj31000falseobj32023-10-12T00:38:45.917Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:00.533 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.534 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.534 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.534 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.535 DEBUG:boto:Method: GET 2023-10-12T00:40:00.535 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.536 DEBUG:boto:Data: 2023-10-12T00:40:00.536 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.537 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.537 DEBUG:boto:Params: {} 2023-10-12T00:40:00.538 DEBUG:boto:Token: None 2023-10-12T00:40:00.538 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4= 2023-10-12T00:40:00.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4=', 'Content-Length': '0'} 2023-10-12T00:40:00.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef9b7432cf6977a7-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.542 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.543 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.543 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.543 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.544 DEBUG:boto:Method: GET 2023-10-12T00:40:00.544 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.545 DEBUG:boto:Data: 2023-10-12T00:40:00.545 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.546 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.547 DEBUG:boto:Params: {} 2023-10-12T00:40:00.547 DEBUG:boto:Token: None 2023-10-12T00:40:00.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4= 2023-10-12T00:40:00.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4=', 'Content-Length': '0'} 2023-10-12T00:40:00.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000985554fb3329aaff-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.551 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.552 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.552 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.553 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.553 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.554 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.554 DEBUG:boto:Data: 2023-10-12T00:40:00.554 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.555 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.556 DEBUG:boto:Params: {} 2023-10-12T00:40:00.556 DEBUG:boto:Token: None 2023-10-12T00:40:00.557 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zpPGjYWuFZzBUkbK0ekVja+aY/0= 2023-10-12T00:40:00.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zpPGjYWuFZzBUkbK0ekVja+aY/0=', 'Content-Length': '0'} 2023-10-12T00:40:00.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e34d88b2629c85-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.560 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.561 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.561 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.562 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.562 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.562 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.563 DEBUG:boto:Data: 2023-10-12T00:40:00.563 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.564 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.564 DEBUG:boto:Params: {} 2023-10-12T00:40:00.565 DEBUG:boto:Token: None 2023-10-12T00:40:00.566 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zpPGjYWuFZzBUkbK0ekVja+aY/0= 2023-10-12T00:40:00.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zpPGjYWuFZzBUkbK0ekVja+aY/0=', 'Content-Length': '0'} 2023-10-12T00:40:00.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001293a8bf6e091a8f-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:00.569 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.570 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.570 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.571 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.571 DEBUG:boto:Method: GET 2023-10-12T00:40:00.572 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.572 DEBUG:boto:Data: 2023-10-12T00:40:00.572 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.573 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.574 DEBUG:boto:Params: {} 2023-10-12T00:40:00.574 DEBUG:boto:Token: None 2023-10-12T00:40:00.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4= 2023-10-12T00:40:00.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4=', 'Content-Length': '0'} 2023-10-12T00:40:00.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c8ad2fc4a1a9e9a-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.578 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.579 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.579 DEBUG:boto:path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.580 DEBUG:boto:auth_path=/wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.580 DEBUG:boto:Method: GET 2023-10-12T00:40:00.581 DEBUG:boto:Path: /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.581 DEBUG:boto:Data: 2023-10-12T00:40:00.581 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.582 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.583 DEBUG:boto:Params: {} 2023-10-12T00:40:00.583 DEBUG:boto:Token: None 2023-10-12T00:40:00.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4?versionId=null 2023-10-12T00:40:00.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4= 2023-10-12T00:40:00.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4x2Bj5ZN3jHzG5x4wTB5Pv3tL4=', 'Content-Length': '0'} 2023-10-12T00:40:00.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f2f4450539fa477-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.630 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.631 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.632 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.632 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.632 DEBUG:boto:Method: GET 2023-10-12T00:40:00.633 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.633 DEBUG:boto:Data: 2023-10-12T00:40:00.634 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.635 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.635 DEBUG:boto:Params: {} 2023-10-12T00:40:00.636 DEBUG:boto:Token: None 2023-10-12T00:40:00.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d5917ee730252f8-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.641 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.642 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.642 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.643 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.643 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.644 DEBUG:boto:Method: GET 2023-10-12T00:40:00.644 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.645 DEBUG:boto:Data: 2023-10-12T00:40:00.645 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.646 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.646 DEBUG:boto:Params: {} 2023-10-12T00:40:00.647 DEBUG:boto:Token: None 2023-10-12T00:40:00.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d59ecd2a04d20a-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.652 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.652 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.653 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.653 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.654 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:40:00.654 DEBUG:boto:Data: 2023-10-12T00:40:00.655 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.655 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.656 DEBUG:boto:Params: {} 2023-10-12T00:40:00.656 DEBUG:boto:Token: None 2023-10-12T00:40:00.657 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4 2023-10-12T00:40:00.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/yrXqEoF1LuCAlPUexZ/FUjwq6E= 2023-10-12T00:40:00.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/yrXqEoF1LuCAlPUexZ/FUjwq6E=', 'Content-Length': '0'} 2023-10-12T00:40:00.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004474ed0f9f1cc1cb-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.661 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.661 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.662 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.662 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:40:00.663 DEBUG:boto:Data: 2023-10-12T00:40:00.663 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.664 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.664 DEBUG:boto:Params: {} 2023-10-12T00:40:00.665 DEBUG:boto:Token: None 2023-10-12T00:40:00.665 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4 2023-10-12T00:40:00.666 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/yrXqEoF1LuCAlPUexZ/FUjwq6E= 2023-10-12T00:40:00.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/yrXqEoF1LuCAlPUexZ/FUjwq6E=', 'Content-Length': '0'} 2023-10-12T00:40:00.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e8e0c33e54f7a34-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:00.669 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.670 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.670 DEBUG:boto:Method: GET 2023-10-12T00:40:00.671 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:40:00.671 DEBUG:boto:Data: 2023-10-12T00:40:00.672 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.673 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.673 DEBUG:boto:Params: {} 2023-10-12T00:40:00.674 DEBUG:boto:Token: None 2023-10-12T00:40:00.674 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4 2023-10-12T00:40:00.675 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MGjTAJHmgQTeluqoz5ne91mGjM4= 2023-10-12T00:40:00.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MGjTAJHmgQTeluqoz5ne91mGjM4=', 'Content-Length': '0'} 2023-10-12T00:40:00.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058b999f91f72903a-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.678 DEBUG:boto:path=/wafmoa-46/obj4 2023-10-12T00:40:00.679 DEBUG:boto:auth_path=/wafmoa-46/obj4 2023-10-12T00:40:00.679 DEBUG:boto:Method: GET 2023-10-12T00:40:00.679 DEBUG:boto:Path: /wafmoa-46/obj4 2023-10-12T00:40:00.680 DEBUG:boto:Data: 2023-10-12T00:40:00.680 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.681 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.682 DEBUG:boto:Params: {} 2023-10-12T00:40:00.682 DEBUG:boto:Token: None 2023-10-12T00:40:00.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj4 2023-10-12T00:40:00.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MGjTAJHmgQTeluqoz5ne91mGjM4= 2023-10-12T00:40:00.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MGjTAJHmgQTeluqoz5ne91mGjM4=', 'Content-Length': '0'} 2023-10-12T00:40:00.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df32cc9b837c7c4-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.687 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.687 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.688 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.688 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.689 DEBUG:boto:Method: GET 2023-10-12T00:40:00.689 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.689 DEBUG:boto:Data: 2023-10-12T00:40:00.690 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.691 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.691 DEBUG:boto:Params: {} 2023-10-12T00:40:00.692 DEBUG:boto:Token: None 2023-10-12T00:40:00.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e217feeda32b61-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.697 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.697 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.698 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.698 DEBUG:boto:path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.699 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.699 DEBUG:boto:Method: GET 2023-10-12T00:40:00.700 DEBUG:boto:Path: /wafmoa-46/?prefix=obj4 2023-10-12T00:40:00.700 DEBUG:boto:Data: 2023-10-12T00:40:00.700 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.701 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.702 DEBUG:boto:Params: {} 2023-10-12T00:40:00.702 DEBUG:boto:Token: None 2023-10-12T00:40:00.703 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081dd0853252931bd-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.750 DEBUG:boto:b'wafmoa-46obj41000falseobj42023-10-12T00:38:45.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:00.754 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.754 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.755 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.755 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.756 DEBUG:boto:Method: GET 2023-10-12T00:40:00.756 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.757 DEBUG:boto:Data: 2023-10-12T00:40:00.757 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.758 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.758 DEBUG:boto:Params: {} 2023-10-12T00:40:00.759 DEBUG:boto:Token: None 2023-10-12T00:40:00.759 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q= 2023-10-12T00:40:00.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q=', 'Content-Length': '0'} 2023-10-12T00:40:00.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050acabc0adce807d-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.764 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.764 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.765 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.765 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.765 DEBUG:boto:Method: GET 2023-10-12T00:40:00.766 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.766 DEBUG:boto:Data: 2023-10-12T00:40:00.767 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.768 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.768 DEBUG:boto:Params: {} 2023-10-12T00:40:00.768 DEBUG:boto:Token: None 2023-10-12T00:40:00.769 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q= 2023-10-12T00:40:00.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q=', 'Content-Length': '0'} 2023-10-12T00:40:00.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9da345361ddad0e-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.773 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.773 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.774 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.774 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.775 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.775 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.775 DEBUG:boto:Data: 2023-10-12T00:40:00.776 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.777 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.777 DEBUG:boto:Params: {} 2023-10-12T00:40:00.777 DEBUG:boto:Token: None 2023-10-12T00:40:00.778 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.778 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eR+3RpoqFq8Jags2nQ3220QGiWk= 2023-10-12T00:40:00.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eR+3RpoqFq8Jags2nQ3220QGiWk=', 'Content-Length': '0'} 2023-10-12T00:40:00.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8d7e15f06be20f6-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.782 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.782 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.782 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.783 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.783 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.784 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.784 DEBUG:boto:Data: 2023-10-12T00:40:00.785 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.785 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.786 DEBUG:boto:Params: {} 2023-10-12T00:40:00.786 DEBUG:boto:Token: None 2023-10-12T00:40:00.787 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eR+3RpoqFq8Jags2nQ3220QGiWk= 2023-10-12T00:40:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eR+3RpoqFq8Jags2nQ3220QGiWk=', 'Content-Length': '0'} 2023-10-12T00:40:00.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013702a1183f37325-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:00.791 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.791 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.792 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.792 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.793 DEBUG:boto:Method: GET 2023-10-12T00:40:00.793 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.793 DEBUG:boto:Data: 2023-10-12T00:40:00.794 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.795 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.795 DEBUG:boto:Params: {} 2023-10-12T00:40:00.796 DEBUG:boto:Token: None 2023-10-12T00:40:00.796 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q= 2023-10-12T00:40:00.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q=', 'Content-Length': '0'} 2023-10-12T00:40:00.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001742200d8f8485e8-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.800 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.801 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.801 DEBUG:boto:path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.802 DEBUG:boto:auth_path=/wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.802 DEBUG:boto:Method: GET 2023-10-12T00:40:00.803 DEBUG:boto:Path: /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.803 DEBUG:boto:Data: 2023-10-12T00:40:00.804 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.805 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.805 DEBUG:boto:Params: {} 2023-10-12T00:40:00.806 DEBUG:boto:Token: None 2023-10-12T00:40:00.806 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5?versionId=null 2023-10-12T00:40:00.807 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q= 2023-10-12T00:40:00.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AEM124MIZjyX9bLVZ8m92bjW84Q=', 'Content-Length': '0'} 2023-10-12T00:40:00.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047f1856fe31a32c4-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.810 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.811 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.811 DEBUG:boto:path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.812 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.812 DEBUG:boto:Method: GET 2023-10-12T00:40:00.813 DEBUG:boto:Path: /wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.813 DEBUG:boto:Data: 2023-10-12T00:40:00.813 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.814 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.815 DEBUG:boto:Params: {} 2023-10-12T00:40:00.815 DEBUG:boto:Token: None 2023-10-12T00:40:00.816 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.816 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006daf2cee58a2c131-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.862 DEBUG:boto:b'wafmoa-46obj51000falseobj52023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.863 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.864 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.864 DEBUG:boto:path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.865 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.865 DEBUG:boto:Method: GET 2023-10-12T00:40:00.866 DEBUG:boto:Path: /wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.866 DEBUG:boto:Data: 2023-10-12T00:40:00.867 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.868 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.868 DEBUG:boto:Params: {} 2023-10-12T00:40:00.869 DEBUG:boto:Token: None 2023-10-12T00:40:00.869 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0af1665cb15bff8-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.874 DEBUG:boto:b'wafmoa-46obj51000falseobj52023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.874 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.875 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.875 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.876 DEBUG:boto:Path: /wafmoa-46/obj5 2023-10-12T00:40:00.876 DEBUG:boto:Data: 2023-10-12T00:40:00.877 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.877 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.878 DEBUG:boto:Params: {} 2023-10-12T00:40:00.878 DEBUG:boto:Token: None 2023-10-12T00:40:00.879 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5 2023-10-12T00:40:00.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CcS0fqvcKSI2IDFDZgoM9olaTlo= 2023-10-12T00:40:00.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CcS0fqvcKSI2IDFDZgoM9olaTlo=', 'Content-Length': '0'} 2023-10-12T00:40:00.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e16cb45149b54bc6-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.882 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.883 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.883 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.884 DEBUG:boto:Path: /wafmoa-46/obj5 2023-10-12T00:40:00.884 DEBUG:boto:Data: 2023-10-12T00:40:00.885 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.885 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.886 DEBUG:boto:Params: {} 2023-10-12T00:40:00.886 DEBUG:boto:Token: None 2023-10-12T00:40:00.887 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5 2023-10-12T00:40:00.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CcS0fqvcKSI2IDFDZgoM9olaTlo= 2023-10-12T00:40:00.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CcS0fqvcKSI2IDFDZgoM9olaTlo=', 'Content-Length': '0'} 2023-10-12T00:40:00.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088bab07e8e4a17db-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:00.891 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.891 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.891 DEBUG:boto:Method: GET 2023-10-12T00:40:00.892 DEBUG:boto:Path: /wafmoa-46/obj5 2023-10-12T00:40:00.892 DEBUG:boto:Data: 2023-10-12T00:40:00.893 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.894 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.894 DEBUG:boto:Params: {} 2023-10-12T00:40:00.895 DEBUG:boto:Token: None 2023-10-12T00:40:00.895 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5 2023-10-12T00:40:00.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FD5tquB41/ZjzHRCmPvFMbaL5dM= 2023-10-12T00:40:00.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FD5tquB41/ZjzHRCmPvFMbaL5dM=', 'Content-Length': '0'} 2023-10-12T00:40:00.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cb72764261a4011-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.899 DEBUG:boto:path=/wafmoa-46/obj5 2023-10-12T00:40:00.899 DEBUG:boto:auth_path=/wafmoa-46/obj5 2023-10-12T00:40:00.900 DEBUG:boto:Method: GET 2023-10-12T00:40:00.900 DEBUG:boto:Path: /wafmoa-46/obj5 2023-10-12T00:40:00.901 DEBUG:boto:Data: 2023-10-12T00:40:00.901 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.902 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.903 DEBUG:boto:Params: {} 2023-10-12T00:40:00.903 DEBUG:boto:Token: None 2023-10-12T00:40:00.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj5 2023-10-12T00:40:00.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FD5tquB41/ZjzHRCmPvFMbaL5dM= 2023-10-12T00:40:00.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FD5tquB41/ZjzHRCmPvFMbaL5dM=', 'Content-Length': '0'} 2023-10-12T00:40:00.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecf4dd2dcbf9fcd7-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.908 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.908 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.909 DEBUG:boto:path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.909 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.909 DEBUG:boto:Method: GET 2023-10-12T00:40:00.910 DEBUG:boto:Path: /wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.910 DEBUG:boto:Data: 2023-10-12T00:40:00.911 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.912 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.912 DEBUG:boto:Params: {} 2023-10-12T00:40:00.913 DEBUG:boto:Token: None 2023-10-12T00:40:00.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a48fcb95287750-0065274060-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.917 DEBUG:boto:b'wafmoa-46obj51000falseobj52023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.918 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:00.919 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:00.919 DEBUG:boto:path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.919 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.920 DEBUG:boto:Method: GET 2023-10-12T00:40:00.920 DEBUG:boto:Path: /wafmoa-46/?prefix=obj5 2023-10-12T00:40:00.921 DEBUG:boto:Data: 2023-10-12T00:40:00.921 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.922 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.923 DEBUG:boto:Params: {} 2023-10-12T00:40:00.923 DEBUG:boto:Token: None 2023-10-12T00:40:00.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/ 2023-10-12T00:40:00.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk= 2023-10-12T00:40:00.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k/oYfD0aEJRREFiZHd6P2B4JNnk=', 'Content-Length': '0'} 2023-10-12T00:40:00.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351e2b5a4ae89171-0065274060-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.930 DEBUG:boto:b'wafmoa-46obj51000falseobj52023-10-12T00:39:19.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:00.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:00.931 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:00.931 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:00.932 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.932 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.933 DEBUG:boto:Method: GET 2023-10-12T00:40:00.933 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.934 DEBUG:boto:Data: 2023-10-12T00:40:00.934 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.935 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.935 DEBUG:boto:Params: {} 2023-10-12T00:40:00.936 DEBUG:boto:Token: None 2023-10-12T00:40:00.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xTrds449MIwWN60xj2EhPA383as= 2023-10-12T00:40:00.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xTrds449MIwWN60xj2EhPA383as=', 'Content-Length': '0'} 2023-10-12T00:40:00.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc4143cd909375f6-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.940 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:00.941 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:00.941 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.942 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.942 DEBUG:boto:Method: GET 2023-10-12T00:40:00.943 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.943 DEBUG:boto:Data: 2023-10-12T00:40:00.944 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:00.945 DEBUG:boto:Port: 8000 2023-10-12T00:40:00.945 DEBUG:boto:Params: {} 2023-10-12T00:40:00.945 DEBUG:boto:Token: None 2023-10-12T00:40:00.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xTrds449MIwWN60xj2EhPA383as= 2023-10-12T00:40:00.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xTrds449MIwWN60xj2EhPA383as=', 'Content-Length': '0'} 2023-10-12T00:40:00.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e1f2c19c894c01f-0065274060-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.990 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:00.991 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:00.991 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.992 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.992 DEBUG:boto:Method: HEAD 2023-10-12T00:40:00.993 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.993 DEBUG:boto:Data: 2023-10-12T00:40:00.994 DEBUG:boto:Headers: {} 2023-10-12T00:40:00.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:00.994 DEBUG:boto:Port: 8001 2023-10-12T00:40:00.995 DEBUG:boto:Params: {} 2023-10-12T00:40:00.995 DEBUG:boto:Token: None 2023-10-12T00:40:00.996 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:00 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:00.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1I9iaz3VVDkYUj23B7qvzhPHtE8= 2023-10-12T00:40:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1I9iaz3VVDkYUj23B7qvzhPHtE8=', 'Content-Length': '0'} 2023-10-12T00:40:00.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004bb267da2cb9f47-0065274060-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:00.999 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.000 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.000 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.001 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.001 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.002 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.002 DEBUG:boto:Data: 2023-10-12T00:40:01.003 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.003 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.004 DEBUG:boto:Params: {} 2023-10-12T00:40:01.004 DEBUG:boto:Token: None 2023-10-12T00:40:01.005 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sc1t6cay7n20QXyunb9dNxtNFhQ= 2023-10-12T00:40:01.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sc1t6cay7n20QXyunb9dNxtNFhQ=', 'Content-Length': '0'} 2023-10-12T00:40:01.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b5ea7d1b93e0af6-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:01.009 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.009 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.010 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.010 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.011 DEBUG:boto:Method: GET 2023-10-12T00:40:01.011 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.011 DEBUG:boto:Data: 2023-10-12T00:40:01.012 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.013 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.013 DEBUG:boto:Params: {} 2023-10-12T00:40:01.014 DEBUG:boto:Token: None 2023-10-12T00:40:01.014 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nSGRbMl8Djm8zPHcpILaQWnBxfY= 2023-10-12T00:40:01.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nSGRbMl8Djm8zPHcpILaQWnBxfY=', 'Content-Length': '0'} 2023-10-12T00:40:01.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f679a24d7f13c4-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.058 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.059 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.059 DEBUG:boto:path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.060 DEBUG:boto:auth_path=/wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.060 DEBUG:boto:Method: GET 2023-10-12T00:40:01.061 DEBUG:boto:Path: /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.061 DEBUG:boto:Data: 2023-10-12T00:40:01.062 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.063 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.063 DEBUG:boto:Params: {} 2023-10-12T00:40:01.064 DEBUG:boto:Token: None 2023-10-12T00:40:01.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6?versionId=null 2023-10-12T00:40:01.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nSGRbMl8Djm8zPHcpILaQWnBxfY= 2023-10-12T00:40:01.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nSGRbMl8Djm8zPHcpILaQWnBxfY=', 'Content-Length': '0'} 2023-10-12T00:40:01.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000baee1a7560609365-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.068 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.068 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.069 DEBUG:boto:path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.069 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.070 DEBUG:boto:Method: GET 2023-10-12T00:40:01.070 DEBUG:boto:Path: /wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.071 DEBUG:boto:Data: 2023-10-12T00:40:01.071 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.072 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.072 DEBUG:boto:Params: {} 2023-10-12T00:40:01.073 DEBUG:boto:Token: None 2023-10-12T00:40:01.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526b1cb3a2a78185-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.078 DEBUG:boto:b'wafmoa-46obj61000falseobj62023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.079 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.080 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.080 DEBUG:boto:path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.080 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.081 DEBUG:boto:Method: GET 2023-10-12T00:40:01.081 DEBUG:boto:Path: /wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.082 DEBUG:boto:Data: 2023-10-12T00:40:01.082 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.083 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.083 DEBUG:boto:Params: {} 2023-10-12T00:40:01.084 DEBUG:boto:Token: None 2023-10-12T00:40:01.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7101e3a8a2364f6-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.088 DEBUG:boto:b'wafmoa-46obj61000falseobj62023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.089 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.090 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.090 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.090 DEBUG:boto:Path: /wafmoa-46/obj6 2023-10-12T00:40:01.091 DEBUG:boto:Data: 2023-10-12T00:40:01.091 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.092 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.093 DEBUG:boto:Params: {} 2023-10-12T00:40:01.093 DEBUG:boto:Token: None 2023-10-12T00:40:01.093 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6 2023-10-12T00:40:01.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GF+UW6iqyx4pDFzZCjqJlk/MzU4= 2023-10-12T00:40:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GF+UW6iqyx4pDFzZCjqJlk/MzU4=', 'Content-Length': '0'} 2023-10-12T00:40:01.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fd3c2718fb7f3d8-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.097 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.097 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.098 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.098 DEBUG:boto:Path: /wafmoa-46/obj6 2023-10-12T00:40:01.099 DEBUG:boto:Data: 2023-10-12T00:40:01.099 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.100 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.100 DEBUG:boto:Params: {} 2023-10-12T00:40:01.101 DEBUG:boto:Token: None 2023-10-12T00:40:01.101 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6 2023-10-12T00:40:01.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GF+UW6iqyx4pDFzZCjqJlk/MzU4= 2023-10-12T00:40:01.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GF+UW6iqyx4pDFzZCjqJlk/MzU4=', 'Content-Length': '0'} 2023-10-12T00:40:01.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df0ba79370226069-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:01.106 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.106 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.107 DEBUG:boto:Method: GET 2023-10-12T00:40:01.107 DEBUG:boto:Path: /wafmoa-46/obj6 2023-10-12T00:40:01.108 DEBUG:boto:Data: 2023-10-12T00:40:01.108 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.109 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.109 DEBUG:boto:Params: {} 2023-10-12T00:40:01.110 DEBUG:boto:Token: None 2023-10-12T00:40:01.110 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6 2023-10-12T00:40:01.111 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WLWUr0lazOpg9XoToljcRp6AH84= 2023-10-12T00:40:01.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WLWUr0lazOpg9XoToljcRp6AH84=', 'Content-Length': '0'} 2023-10-12T00:40:01.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008334d25ff4ee41e0-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.115 DEBUG:boto:path=/wafmoa-46/obj6 2023-10-12T00:40:01.116 DEBUG:boto:auth_path=/wafmoa-46/obj6 2023-10-12T00:40:01.116 DEBUG:boto:Method: GET 2023-10-12T00:40:01.117 DEBUG:boto:Path: /wafmoa-46/obj6 2023-10-12T00:40:01.117 DEBUG:boto:Data: 2023-10-12T00:40:01.117 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.118 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.119 DEBUG:boto:Params: {} 2023-10-12T00:40:01.119 DEBUG:boto:Token: None 2023-10-12T00:40:01.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj6 2023-10-12T00:40:01.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WLWUr0lazOpg9XoToljcRp6AH84= 2023-10-12T00:40:01.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WLWUr0lazOpg9XoToljcRp6AH84=', 'Content-Length': '0'} 2023-10-12T00:40:01.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d672e13ac69a8b8-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.124 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.124 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.125 DEBUG:boto:path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.125 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.125 DEBUG:boto:Method: GET 2023-10-12T00:40:01.126 DEBUG:boto:Path: /wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.126 DEBUG:boto:Data: 2023-10-12T00:40:01.127 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.128 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.128 DEBUG:boto:Params: {} 2023-10-12T00:40:01.129 DEBUG:boto:Token: None 2023-10-12T00:40:01.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.130 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f9bb566f10d780-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.133 DEBUG:boto:b'wafmoa-46obj61000falseobj62023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.134 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.134 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.135 DEBUG:boto:path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.135 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.136 DEBUG:boto:Method: GET 2023-10-12T00:40:01.136 DEBUG:boto:Path: /wafmoa-46/?prefix=obj6 2023-10-12T00:40:01.137 DEBUG:boto:Data: 2023-10-12T00:40:01.137 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.138 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.139 DEBUG:boto:Params: {} 2023-10-12T00:40:01.139 DEBUG:boto:Token: None 2023-10-12T00:40:01.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0e16baa7b4b0de-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.186 DEBUG:boto:b'wafmoa-46obj61000falseobj62023-10-12T00:39:19.318Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:01.188 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.188 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.189 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.189 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.189 DEBUG:boto:Method: GET 2023-10-12T00:40:01.190 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.190 DEBUG:boto:Data: 2023-10-12T00:40:01.191 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.192 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.192 DEBUG:boto:Params: {} 2023-10-12T00:40:01.193 DEBUG:boto:Token: None 2023-10-12T00:40:01.193 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.194 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I= 2023-10-12T00:40:01.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I=', 'Content-Length': '0'} 2023-10-12T00:40:01.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003459dde19e8540ad-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.197 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.198 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.198 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.199 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.199 DEBUG:boto:Method: GET 2023-10-12T00:40:01.200 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.200 DEBUG:boto:Data: 2023-10-12T00:40:01.201 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.202 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.202 DEBUG:boto:Params: {} 2023-10-12T00:40:01.202 DEBUG:boto:Token: None 2023-10-12T00:40:01.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I= 2023-10-12T00:40:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I=', 'Content-Length': '0'} 2023-10-12T00:40:01.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ba8a2e16a1a305b-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.207 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.208 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.208 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.208 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.209 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.209 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.210 DEBUG:boto:Data: 2023-10-12T00:40:01.210 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.211 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.211 DEBUG:boto:Params: {} 2023-10-12T00:40:01.212 DEBUG:boto:Token: None 2023-10-12T00:40:01.212 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ovyL5NBHFTXbLYJy0W3XNQFc2VQ= 2023-10-12T00:40:01.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ovyL5NBHFTXbLYJy0W3XNQFc2VQ=', 'Content-Length': '0'} 2023-10-12T00:40:01.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae9cf90eccd56ba2-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.216 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.216 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.217 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.217 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.217 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.218 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.218 DEBUG:boto:Data: 2023-10-12T00:40:01.219 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.220 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.220 DEBUG:boto:Params: {} 2023-10-12T00:40:01.220 DEBUG:boto:Token: None 2023-10-12T00:40:01.221 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ovyL5NBHFTXbLYJy0W3XNQFc2VQ= 2023-10-12T00:40:01.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ovyL5NBHFTXbLYJy0W3XNQFc2VQ=', 'Content-Length': '0'} 2023-10-12T00:40:01.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca332f1923e8c81e-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:01.225 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.225 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.226 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.226 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.226 DEBUG:boto:Method: GET 2023-10-12T00:40:01.227 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.227 DEBUG:boto:Data: 2023-10-12T00:40:01.228 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.229 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.229 DEBUG:boto:Params: {} 2023-10-12T00:40:01.230 DEBUG:boto:Token: None 2023-10-12T00:40:01.230 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.231 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I= 2023-10-12T00:40:01.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I=', 'Content-Length': '0'} 2023-10-12T00:40:01.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005255aa1ba2d0997f-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.234 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.234 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.235 DEBUG:boto:path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.235 DEBUG:boto:auth_path=/wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.235 DEBUG:boto:Method: GET 2023-10-12T00:40:01.236 DEBUG:boto:Path: /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.236 DEBUG:boto:Data: 2023-10-12T00:40:01.237 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.238 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.238 DEBUG:boto:Params: {} 2023-10-12T00:40:01.239 DEBUG:boto:Token: None 2023-10-12T00:40:01.239 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7?versionId=null 2023-10-12T00:40:01.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I= 2023-10-12T00:40:01.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tVPZytQ6JVzfLyBOpkBMGvSvO3I=', 'Content-Length': '0'} 2023-10-12T00:40:01.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b454dd630dd46dc7-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.243 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.244 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.244 DEBUG:boto:path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.244 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.245 DEBUG:boto:Method: GET 2023-10-12T00:40:01.245 DEBUG:boto:Path: /wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.246 DEBUG:boto:Data: 2023-10-12T00:40:01.246 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.247 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.247 DEBUG:boto:Params: {} 2023-10-12T00:40:01.248 DEBUG:boto:Token: None 2023-10-12T00:40:01.248 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835fee4190a8b5bb-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.253 DEBUG:boto:b'wafmoa-46obj71000falseobj72023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.253 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.254 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.254 DEBUG:boto:path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.255 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.255 DEBUG:boto:Method: GET 2023-10-12T00:40:01.256 DEBUG:boto:Path: /wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.256 DEBUG:boto:Data: 2023-10-12T00:40:01.256 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.257 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.258 DEBUG:boto:Params: {} 2023-10-12T00:40:01.258 DEBUG:boto:Token: None 2023-10-12T00:40:01.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f2f140edae14409-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.310 DEBUG:boto:b'wafmoa-46obj71000falseobj72023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.312 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.312 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.312 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.313 DEBUG:boto:Path: /wafmoa-46/obj7 2023-10-12T00:40:01.313 DEBUG:boto:Data: 2023-10-12T00:40:01.314 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.315 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.315 DEBUG:boto:Params: {} 2023-10-12T00:40:01.316 DEBUG:boto:Token: None 2023-10-12T00:40:01.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7 2023-10-12T00:40:01.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zPoL3Mx4dUI3liF4sszG05/Yon4= 2023-10-12T00:40:01.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zPoL3Mx4dUI3liF4sszG05/Yon4=', 'Content-Length': '0'} 2023-10-12T00:40:01.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beac23ae34076cb6-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.321 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.321 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.322 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.323 DEBUG:boto:Path: /wafmoa-46/obj7 2023-10-12T00:40:01.323 DEBUG:boto:Data: 2023-10-12T00:40:01.324 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.325 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.325 DEBUG:boto:Params: {} 2023-10-12T00:40:01.326 DEBUG:boto:Token: None 2023-10-12T00:40:01.326 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7 2023-10-12T00:40:01.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zPoL3Mx4dUI3liF4sszG05/Yon4= 2023-10-12T00:40:01.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zPoL3Mx4dUI3liF4sszG05/Yon4=', 'Content-Length': '0'} 2023-10-12T00:40:01.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018b525e656c2431f-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:01.330 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.331 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.331 DEBUG:boto:Method: GET 2023-10-12T00:40:01.332 DEBUG:boto:Path: /wafmoa-46/obj7 2023-10-12T00:40:01.332 DEBUG:boto:Data: 2023-10-12T00:40:01.332 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.333 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.334 DEBUG:boto:Params: {} 2023-10-12T00:40:01.334 DEBUG:boto:Token: None 2023-10-12T00:40:01.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7 2023-10-12T00:40:01.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WC4fF33kH5LWb+lqAc9gbNfTkuo= 2023-10-12T00:40:01.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WC4fF33kH5LWb+lqAc9gbNfTkuo=', 'Content-Length': '0'} 2023-10-12T00:40:01.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdfbb2ccb7436652-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.342 DEBUG:boto:path=/wafmoa-46/obj7 2023-10-12T00:40:01.343 DEBUG:boto:auth_path=/wafmoa-46/obj7 2023-10-12T00:40:01.343 DEBUG:boto:Method: GET 2023-10-12T00:40:01.344 DEBUG:boto:Path: /wafmoa-46/obj7 2023-10-12T00:40:01.344 DEBUG:boto:Data: 2023-10-12T00:40:01.345 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.345 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.346 DEBUG:boto:Params: {} 2023-10-12T00:40:01.346 DEBUG:boto:Token: None 2023-10-12T00:40:01.347 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj7 2023-10-12T00:40:01.347 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WC4fF33kH5LWb+lqAc9gbNfTkuo= 2023-10-12T00:40:01.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WC4fF33kH5LWb+lqAc9gbNfTkuo=', 'Content-Length': '0'} 2023-10-12T00:40:01.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001177dd90630750d6-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.352 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.353 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.353 DEBUG:boto:path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.354 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.354 DEBUG:boto:Method: GET 2023-10-12T00:40:01.355 DEBUG:boto:Path: /wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.355 DEBUG:boto:Data: 2023-10-12T00:40:01.356 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.357 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.357 DEBUG:boto:Params: {} 2023-10-12T00:40:01.358 DEBUG:boto:Token: None 2023-10-12T00:40:01.359 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095dd4f45118b9e35-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.363 DEBUG:boto:b'wafmoa-46obj71000falseobj72023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.364 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.365 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.365 DEBUG:boto:path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.365 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.366 DEBUG:boto:Method: GET 2023-10-12T00:40:01.367 DEBUG:boto:Path: /wafmoa-46/?prefix=obj7 2023-10-12T00:40:01.367 DEBUG:boto:Data: 2023-10-12T00:40:01.368 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.369 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.369 DEBUG:boto:Params: {} 2023-10-12T00:40:01.370 DEBUG:boto:Token: None 2023-10-12T00:40:01.370 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806a6a74133e70cb-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.375 DEBUG:boto:b'wafmoa-46obj71000falseobj72023-10-12T00:39:19.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:01.376 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.377 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.378 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.378 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.379 DEBUG:boto:Method: GET 2023-10-12T00:40:01.379 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.380 DEBUG:boto:Data: 2023-10-12T00:40:01.380 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.382 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.382 DEBUG:boto:Params: {} 2023-10-12T00:40:01.383 DEBUG:boto:Token: None 2023-10-12T00:40:01.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY= 2023-10-12T00:40:01.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY=', 'Content-Length': '0'} 2023-10-12T00:40:01.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da6ab1d1bbd94d3b-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.390 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.391 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.391 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.393 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.394 DEBUG:boto:Method: GET 2023-10-12T00:40:01.394 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.395 DEBUG:boto:Data: 2023-10-12T00:40:01.396 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.397 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.398 DEBUG:boto:Params: {} 2023-10-12T00:40:01.399 DEBUG:boto:Token: None 2023-10-12T00:40:01.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY= 2023-10-12T00:40:01.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY=', 'Content-Length': '0'} 2023-10-12T00:40:01.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000874357180f0acb39-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.405 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.405 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.406 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.406 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.407 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.407 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.408 DEBUG:boto:Data: 2023-10-12T00:40:01.408 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.409 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.410 DEBUG:boto:Params: {} 2023-10-12T00:40:01.410 DEBUG:boto:Token: None 2023-10-12T00:40:01.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xsr4Vy+0vk1/FZin0z653R63nB0= 2023-10-12T00:40:01.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xsr4Vy+0vk1/FZin0z653R63nB0=', 'Content-Length': '0'} 2023-10-12T00:40:01.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee8944beb1df2160-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.415 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.415 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.416 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.416 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.417 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.417 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.418 DEBUG:boto:Data: 2023-10-12T00:40:01.418 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.419 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.420 DEBUG:boto:Params: {} 2023-10-12T00:40:01.420 DEBUG:boto:Token: None 2023-10-12T00:40:01.421 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xsr4Vy+0vk1/FZin0z653R63nB0= 2023-10-12T00:40:01.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xsr4Vy+0vk1/FZin0z653R63nB0=', 'Content-Length': '0'} 2023-10-12T00:40:01.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fdb6d3bbe5512bc-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:01.425 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.426 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.427 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.427 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.427 DEBUG:boto:Method: GET 2023-10-12T00:40:01.428 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.429 DEBUG:boto:Data: 2023-10-12T00:40:01.429 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.430 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.431 DEBUG:boto:Params: {} 2023-10-12T00:40:01.431 DEBUG:boto:Token: None 2023-10-12T00:40:01.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY= 2023-10-12T00:40:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY=', 'Content-Length': '0'} 2023-10-12T00:40:01.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a18589f48634b10b-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.436 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.437 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.437 DEBUG:boto:path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.438 DEBUG:boto:auth_path=/wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.438 DEBUG:boto:Method: GET 2023-10-12T00:40:01.439 DEBUG:boto:Path: /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.439 DEBUG:boto:Data: 2023-10-12T00:40:01.440 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.441 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.441 DEBUG:boto:Params: {} 2023-10-12T00:40:01.442 DEBUG:boto:Token: None 2023-10-12T00:40:01.442 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8?versionId=null 2023-10-12T00:40:01.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY= 2023-10-12T00:40:01.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z7sjaQ8Ne1CHbSkUzz5M3FJfGxY=', 'Content-Length': '0'} 2023-10-12T00:40:01.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1c15018ade16a48-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.446 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.446 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.447 DEBUG:boto:path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.447 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.448 DEBUG:boto:Method: GET 2023-10-12T00:40:01.448 DEBUG:boto:Path: /wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.449 DEBUG:boto:Data: 2023-10-12T00:40:01.449 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.450 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.450 DEBUG:boto:Params: {} 2023-10-12T00:40:01.451 DEBUG:boto:Token: None 2023-10-12T00:40:01.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f396324b452af851-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.455 DEBUG:boto:b'wafmoa-46obj81000falseobj82023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.456 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.457 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.457 DEBUG:boto:path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.458 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.458 DEBUG:boto:Method: GET 2023-10-12T00:40:01.458 DEBUG:boto:Path: /wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.459 DEBUG:boto:Data: 2023-10-12T00:40:01.459 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.460 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.461 DEBUG:boto:Params: {} 2023-10-12T00:40:01.461 DEBUG:boto:Token: None 2023-10-12T00:40:01.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.462 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9fdc0321cb63211-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.466 DEBUG:boto:b'wafmoa-46obj81000falseobj82023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.466 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.467 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.467 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.468 DEBUG:boto:Path: /wafmoa-46/obj8 2023-10-12T00:40:01.468 DEBUG:boto:Data: 2023-10-12T00:40:01.469 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.469 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.470 DEBUG:boto:Params: {} 2023-10-12T00:40:01.470 DEBUG:boto:Token: None 2023-10-12T00:40:01.471 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8 2023-10-12T00:40:01.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgZoqN1e90pHyfXHmw5K171GRFE= 2023-10-12T00:40:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgZoqN1e90pHyfXHmw5K171GRFE=', 'Content-Length': '0'} 2023-10-12T00:40:01.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021a27fe8e06d9880-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.474 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.475 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.475 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.476 DEBUG:boto:Path: /wafmoa-46/obj8 2023-10-12T00:40:01.476 DEBUG:boto:Data: 2023-10-12T00:40:01.477 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.477 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.478 DEBUG:boto:Params: {} 2023-10-12T00:40:01.478 DEBUG:boto:Token: None 2023-10-12T00:40:01.479 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8 2023-10-12T00:40:01.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MgZoqN1e90pHyfXHmw5K171GRFE= 2023-10-12T00:40:01.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MgZoqN1e90pHyfXHmw5K171GRFE=', 'Content-Length': '0'} 2023-10-12T00:40:01.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064bdf82754e75fd9-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:01.483 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.483 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.484 DEBUG:boto:Method: GET 2023-10-12T00:40:01.484 DEBUG:boto:Path: /wafmoa-46/obj8 2023-10-12T00:40:01.484 DEBUG:boto:Data: 2023-10-12T00:40:01.485 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.486 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.486 DEBUG:boto:Params: {} 2023-10-12T00:40:01.487 DEBUG:boto:Token: None 2023-10-12T00:40:01.487 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8 2023-10-12T00:40:01.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jlk3YJArl10h2Syx6q3rQSX/r7s= 2023-10-12T00:40:01.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jlk3YJArl10h2Syx6q3rQSX/r7s=', 'Content-Length': '0'} 2023-10-12T00:40:01.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-46:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009199d92ad09c2852-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.538 DEBUG:boto:path=/wafmoa-46/obj8 2023-10-12T00:40:01.539 DEBUG:boto:auth_path=/wafmoa-46/obj8 2023-10-12T00:40:01.539 DEBUG:boto:Method: GET 2023-10-12T00:40:01.540 DEBUG:boto:Path: /wafmoa-46/obj8 2023-10-12T00:40:01.540 DEBUG:boto:Data: 2023-10-12T00:40:01.541 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.542 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.542 DEBUG:boto:Params: {} 2023-10-12T00:40:01.545 DEBUG:boto:Token: None 2023-10-12T00:40:01.546 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/obj8 2023-10-12T00:40:01.546 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jlk3YJArl10h2Syx6q3rQSX/r7s= 2023-10-12T00:40:01.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jlk3YJArl10h2Syx6q3rQSX/r7s=', 'Content-Length': '0'} 2023-10-12T00:40:01.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043bb2fbedd8b94a2-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.550 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.550 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.550 DEBUG:boto:path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.551 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.551 DEBUG:boto:Method: GET 2023-10-12T00:40:01.552 DEBUG:boto:Path: /wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.552 DEBUG:boto:Data: 2023-10-12T00:40:01.553 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.554 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.554 DEBUG:boto:Params: {} 2023-10-12T00:40:01.554 DEBUG:boto:Token: None 2023-10-12T00:40:01.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065946466cc472990-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.559 DEBUG:boto:b'wafmoa-46obj81000falseobj82023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.560 DEBUG:boto:path=/wafmoa-46/ 2023-10-12T00:40:01.560 DEBUG:boto:auth_path=/wafmoa-46/ 2023-10-12T00:40:01.561 DEBUG:boto:path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.561 DEBUG:boto:auth_path=/wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.561 DEBUG:boto:Method: GET 2023-10-12T00:40:01.562 DEBUG:boto:Path: /wafmoa-46/?prefix=obj8 2023-10-12T00:40:01.562 DEBUG:boto:Data: 2023-10-12T00:40:01.563 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.564 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.564 DEBUG:boto:Params: {} 2023-10-12T00:40:01.564 DEBUG:boto:Token: None 2023-10-12T00:40:01.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-46/ 2023-10-12T00:40:01.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk= 2023-10-12T00:40:01.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyENiOcwZqW2EU3Pgs8vJXOPvkk=', 'Content-Length': '0'} 2023-10-12T00:40:01.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02561e0abe781f9-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.569 DEBUG:boto:b'wafmoa-46obj81000falseobj82023-10-12T00:39:19.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-46 zones={a2, a1} 2023-10-12T00:40:01.571 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-47 2023-10-12T00:40:01.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:01.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:01.573 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:01.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:01.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:01.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:01.669+0000 7fbb3c0c9780 1 bucket sync caught up with source: 2023-10-12T00:40:01.669 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.18.6, 00000000002.27.1, 00000000004.27.6, 00000000002.20.1, 00000000004.29.6, 00000000002.29.1, 00000000002.31.1, 00000000004.24.6, 00000000002.29.1, 00000000002.19.1, 00000000002.25.1] 2023-10-12T00:40:01.669 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.18.6, 00000000002.27.1, 00000000004.27.6, 00000000002.20.1, 00000000004.29.6, 00000000002.29.1, 00000000002.31.1, 00000000004.24.6, 00000000002.29.1, 00000000002.19.1, 00000000002.25.1] 2023-10-12T00:40:01.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:01.669+0000 7fbb3c0c9780 0 bucket checkpoint complete 2023-10-12T00:40:01.682 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-47 2023-10-12T00:40:01.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:01.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:01.684 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:01.774+0000 7f455dbff780 1 bucket sync caught up with source: 2023-10-12T00:40:01.774 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.88.6, , 00000000002.196.6, , 00000000002.115.6, , , 00000000002.173.6, , , ] 2023-10-12T00:40:01.775 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.88.6, , 00000000002.196.6, , 00000000002.115.6, , , 00000000002.173.6, , , ] 2023-10-12T00:40:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:01.774+0000 7f455dbff780 0 bucket checkpoint complete 2023-10-12T00:40:01.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-47 zones={a2, a1} 2023-10-12T00:40:01.787 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.787 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.788 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.788 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:40:01.788 DEBUG:boto:Data: 2023-10-12T00:40:01.789 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.790 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.790 DEBUG:boto:Params: {} 2023-10-12T00:40:01.791 DEBUG:boto:Token: None 2023-10-12T00:40:01.791 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/ 2023-10-12T00:40:01.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XofkSNwgCTkGAMiXHVd7zGfKoms= 2023-10-12T00:40:01.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XofkSNwgCTkGAMiXHVd7zGfKoms=', 'Content-Length': '0'} 2023-10-12T00:40:01.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044bb45afb2cc4053-0065274061-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.795 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.795 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.796 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.796 DEBUG:boto:Path: /wafmoa-47/ 2023-10-12T00:40:01.797 DEBUG:boto:Data: 2023-10-12T00:40:01.797 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.798 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.798 DEBUG:boto:Params: {} 2023-10-12T00:40:01.799 DEBUG:boto:Token: None 2023-10-12T00:40:01.799 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/ 2023-10-12T00:40:01.800 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XofkSNwgCTkGAMiXHVd7zGfKoms= 2023-10-12T00:40:01.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XofkSNwgCTkGAMiXHVd7zGfKoms=', 'Content-Length': '0'} 2023-10-12T00:40:01.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009adb3e8cf464a624-0065274061-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:40:01.803 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.804 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.804 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.804 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.805 DEBUG:boto:Method: GET 2023-10-12T00:40:01.805 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.806 DEBUG:boto:Data: 2023-10-12T00:40:01.806 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.807 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.807 DEBUG:boto:Params: {} 2023-10-12T00:40:01.808 DEBUG:boto:Token: None 2023-10-12T00:40:01.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/?versions 2023-10-12T00:40:01.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus= 2023-10-12T00:40:01.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus=', 'Content-Length': '0'} 2023-10-12T00:40:01.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0de491a7fad077b-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.812 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:01.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:01.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:01.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:01.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:40:01.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:40:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:40:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:40:01.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:40:01.817 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.817 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.818 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.818 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.819 DEBUG:boto:Method: GET 2023-10-12T00:40:01.819 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.819 DEBUG:boto:Data: 2023-10-12T00:40:01.820 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.821 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.821 DEBUG:boto:Params: {} 2023-10-12T00:40:01.822 DEBUG:boto:Token: None 2023-10-12T00:40:01.822 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/?versions 2023-10-12T00:40:01.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus= 2023-10-12T00:40:01.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus=', 'Content-Length': '0'} 2023-10-12T00:40:01.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea6bf2ff00cab37-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.826 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:01.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:01.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:40:01.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:40:01.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:40:01.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:40:01.830 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.830 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.831 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.831 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.831 DEBUG:boto:Method: GET 2023-10-12T00:40:01.832 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.832 DEBUG:boto:Data: 2023-10-12T00:40:01.833 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.833 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.834 DEBUG:boto:Params: {} 2023-10-12T00:40:01.834 DEBUG:boto:Token: None 2023-10-12T00:40:01.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/?versions 2023-10-12T00:40:01.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus= 2023-10-12T00:40:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus=', 'Content-Length': '0'} 2023-10-12T00:40:01.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d051c8921150c13-0065274061-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.838 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.839 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.840 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.840 DEBUG:boto:path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.841 DEBUG:boto:auth_path=/wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.841 DEBUG:boto:Method: GET 2023-10-12T00:40:01.842 DEBUG:boto:Path: /wafmoa-47/?versions&max-keys=999 2023-10-12T00:40:01.842 DEBUG:boto:Data: 2023-10-12T00:40:01.842 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.843 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.844 DEBUG:boto:Params: {} 2023-10-12T00:40:01.844 DEBUG:boto:Token: None 2023-10-12T00:40:01.845 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/?versions 2023-10-12T00:40:01.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus= 2023-10-12T00:40:01.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S9NEBVBb+fb3PPa4HV0KrR14Cus=', 'Content-Length': '0'} 2023-10-12T00:40:01.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1e7d470eec902f-0065274061-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.849 DEBUG:boto:b'wafmoa-47999falseobj1nulltrue2023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:01.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:01.850 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.850 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.851 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.851 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.852 DEBUG:boto:Method: GET 2023-10-12T00:40:01.852 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.852 DEBUG:boto:Data: 2023-10-12T00:40:01.853 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.854 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.854 DEBUG:boto:Params: {} 2023-10-12T00:40:01.855 DEBUG:boto:Token: None 2023-10-12T00:40:01.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow= 2023-10-12T00:40:01.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow=', 'Content-Length': '0'} 2023-10-12T00:40:01.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e17d097529ff543f-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.859 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.859 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.860 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.860 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.861 DEBUG:boto:Method: GET 2023-10-12T00:40:01.861 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.862 DEBUG:boto:Data: 2023-10-12T00:40:01.862 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.863 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.863 DEBUG:boto:Params: {} 2023-10-12T00:40:01.864 DEBUG:boto:Token: None 2023-10-12T00:40:01.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow= 2023-10-12T00:40:01.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow=', 'Content-Length': '0'} 2023-10-12T00:40:01.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000420a3e2dd4c88bfc-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.910 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.911 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.911 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.912 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.912 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.913 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.913 DEBUG:boto:Data: 2023-10-12T00:40:01.914 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.914 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.915 DEBUG:boto:Params: {} 2023-10-12T00:40:01.915 DEBUG:boto:Token: None 2023-10-12T00:40:01.916 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q43fKZ1JJLqUEsL8FL64NYuatbI= 2023-10-12T00:40:01.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q43fKZ1JJLqUEsL8FL64NYuatbI=', 'Content-Length': '0'} 2023-10-12T00:40:01.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c16bb52e09798e-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.919 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.920 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.920 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.921 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.921 DEBUG:boto:Method: HEAD 2023-10-12T00:40:01.922 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.922 DEBUG:boto:Data: 2023-10-12T00:40:01.923 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.924 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.925 DEBUG:boto:Params: {} 2023-10-12T00:40:01.925 DEBUG:boto:Token: None 2023-10-12T00:40:01.926 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q43fKZ1JJLqUEsL8FL64NYuatbI= 2023-10-12T00:40:01.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q43fKZ1JJLqUEsL8FL64NYuatbI=', 'Content-Length': '0'} 2023-10-12T00:40:01.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe2954051df4f0ee-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:01.931 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.932 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.932 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.932 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.933 DEBUG:boto:Method: GET 2023-10-12T00:40:01.933 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.934 DEBUG:boto:Data: 2023-10-12T00:40:01.934 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.935 DEBUG:boto:Port: 8001 2023-10-12T00:40:01.936 DEBUG:boto:Params: {} 2023-10-12T00:40:01.936 DEBUG:boto:Token: None 2023-10-12T00:40:01.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow= 2023-10-12T00:40:01.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow=', 'Content-Length': '0'} 2023-10-12T00:40:01.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000413c9453db6ae425-0065274061-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.986 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:01.987 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:01.988 DEBUG:boto:path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.988 DEBUG:boto:auth_path=/wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.988 DEBUG:boto:Method: GET 2023-10-12T00:40:01.989 DEBUG:boto:Path: /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.989 DEBUG:boto:Data: 2023-10-12T00:40:01.990 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:01.991 DEBUG:boto:Port: 8000 2023-10-12T00:40:01.991 DEBUG:boto:Params: {} 2023-10-12T00:40:01.992 DEBUG:boto:Token: None 2023-10-12T00:40:01.992 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:01 GMT /wafmoa-47/obj1?versionId=null 2023-10-12T00:40:01.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow= 2023-10-12T00:40:01.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tp7WMS97gwBqT2k1oggXXAOG9Ow=', 'Content-Length': '0'} 2023-10-12T00:40:01.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000042ac25b15d1053a-0065274061-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:01.995 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:01.996 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:01.996 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:01.997 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:01.997 DEBUG:boto:Method: GET 2023-10-12T00:40:01.998 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:40:01.998 DEBUG:boto:Data: 2023-10-12T00:40:01.999 DEBUG:boto:Headers: {} 2023-10-12T00:40:01.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:01.999 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.000 DEBUG:boto:Params: {} 2023-10-12T00:40:02.000 DEBUG:boto:Token: None 2023-10-12T00:40:02.001 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d122ec97532d514d-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.005 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.005 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.006 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.006 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.007 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.007 DEBUG:boto:Method: GET 2023-10-12T00:40:02.008 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.008 DEBUG:boto:Data: 2023-10-12T00:40:02.008 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.009 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.010 DEBUG:boto:Params: {} 2023-10-12T00:40:02.011 DEBUG:boto:Token: None 2023-10-12T00:40:02.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ec5b0b78e95ab97-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.015 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.015 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:02.016 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:02.016 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.017 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:40:02.017 DEBUG:boto:Data: 2023-10-12T00:40:02.017 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.018 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.019 DEBUG:boto:Params: {} 2023-10-12T00:40:02.019 DEBUG:boto:Token: None 2023-10-12T00:40:02.020 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj1 2023-10-12T00:40:02.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWmINECUs3PPjIrVDwL0y3dwJms= 2023-10-12T00:40:02.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWmINECUs3PPjIrVDwL0y3dwJms=', 'Content-Length': '0'} 2023-10-12T00:40:02.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000700ee74084a69554-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.023 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:02.023 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:02.024 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.024 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:40:02.024 DEBUG:boto:Data: 2023-10-12T00:40:02.025 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.026 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.026 DEBUG:boto:Params: {} 2023-10-12T00:40:02.027 DEBUG:boto:Token: None 2023-10-12T00:40:02.027 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj1 2023-10-12T00:40:02.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWmINECUs3PPjIrVDwL0y3dwJms= 2023-10-12T00:40:02.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWmINECUs3PPjIrVDwL0y3dwJms=', 'Content-Length': '0'} 2023-10-12T00:40:02.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4a7673a423a4170-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:02.031 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:02.031 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:02.032 DEBUG:boto:Method: GET 2023-10-12T00:40:02.032 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:40:02.033 DEBUG:boto:Data: 2023-10-12T00:40:02.033 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.034 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.034 DEBUG:boto:Params: {} 2023-10-12T00:40:02.035 DEBUG:boto:Token: None 2023-10-12T00:40:02.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj1 2023-10-12T00:40:02.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V8EYNjfUs0N83czqaaLWmCcjqLU= 2023-10-12T00:40:02.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V8EYNjfUs0N83czqaaLWmCcjqLU=', 'Content-Length': '0'} 2023-10-12T00:40:02.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8c4ca6bfc7b881d-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.040 DEBUG:boto:path=/wafmoa-47/obj1 2023-10-12T00:40:02.040 DEBUG:boto:auth_path=/wafmoa-47/obj1 2023-10-12T00:40:02.041 DEBUG:boto:Method: GET 2023-10-12T00:40:02.041 DEBUG:boto:Path: /wafmoa-47/obj1 2023-10-12T00:40:02.041 DEBUG:boto:Data: 2023-10-12T00:40:02.042 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.043 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.043 DEBUG:boto:Params: {} 2023-10-12T00:40:02.044 DEBUG:boto:Token: None 2023-10-12T00:40:02.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj1 2023-10-12T00:40:02.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V8EYNjfUs0N83czqaaLWmCcjqLU= 2023-10-12T00:40:02.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V8EYNjfUs0N83czqaaLWmCcjqLU=', 'Content-Length': '0'} 2023-10-12T00:40:02.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001006ddfda59535dd-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.094 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.095 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.095 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.096 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.096 DEBUG:boto:Method: GET 2023-10-12T00:40:02.097 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.097 DEBUG:boto:Data: 2023-10-12T00:40:02.098 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.098 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.099 DEBUG:boto:Params: {} 2023-10-12T00:40:02.099 DEBUG:boto:Token: None 2023-10-12T00:40:02.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4003528a3639c3-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.104 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.105 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.106 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.106 DEBUG:boto:path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.107 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.108 DEBUG:boto:Method: GET 2023-10-12T00:40:02.109 DEBUG:boto:Path: /wafmoa-47/?prefix=obj1 2023-10-12T00:40:02.109 DEBUG:boto:Data: 2023-10-12T00:40:02.110 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.111 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.111 DEBUG:boto:Params: {} 2023-10-12T00:40:02.112 DEBUG:boto:Token: None 2023-10-12T00:40:02.113 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a957e850ef4edf3-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.117 DEBUG:boto:b'wafmoa-47obj11000falseobj12023-10-12T00:38:45.996Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:02.118 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.119 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.119 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.119 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.120 DEBUG:boto:Method: GET 2023-10-12T00:40:02.120 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.121 DEBUG:boto:Data: 2023-10-12T00:40:02.121 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.122 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.123 DEBUG:boto:Params: {} 2023-10-12T00:40:02.123 DEBUG:boto:Token: None 2023-10-12T00:40:02.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY= 2023-10-12T00:40:02.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY=', 'Content-Length': '0'} 2023-10-12T00:40:02.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092048135036dd9a3-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.127 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.128 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.128 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.129 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.129 DEBUG:boto:Method: GET 2023-10-12T00:40:02.129 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.130 DEBUG:boto:Data: 2023-10-12T00:40:02.130 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.131 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.132 DEBUG:boto:Params: {} 2023-10-12T00:40:02.133 DEBUG:boto:Token: None 2023-10-12T00:40:02.133 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.134 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY= 2023-10-12T00:40:02.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY=', 'Content-Length': '0'} 2023-10-12T00:40:02.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fb3eb903b8e7208-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.137 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.138 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.138 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.138 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.139 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.140 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.140 DEBUG:boto:Data: 2023-10-12T00:40:02.141 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.142 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.142 DEBUG:boto:Params: {} 2023-10-12T00:40:02.143 DEBUG:boto:Token: None 2023-10-12T00:40:02.143 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jvKpvHE+UPnlo/bqWpO+w4CWl3c= 2023-10-12T00:40:02.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jvKpvHE+UPnlo/bqWpO+w4CWl3c=', 'Content-Length': '0'} 2023-10-12T00:40:02.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006897bd5e61c5cc58-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.148 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.148 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.149 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.149 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.150 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.150 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.151 DEBUG:boto:Data: 2023-10-12T00:40:02.151 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.152 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.153 DEBUG:boto:Params: {} 2023-10-12T00:40:02.153 DEBUG:boto:Token: None 2023-10-12T00:40:02.154 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.154 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jvKpvHE+UPnlo/bqWpO+w4CWl3c= 2023-10-12T00:40:02.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jvKpvHE+UPnlo/bqWpO+w4CWl3c=', 'Content-Length': '0'} 2023-10-12T00:40:02.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003009d924ecf76844-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:02.160 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.161 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.162 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.162 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.163 DEBUG:boto:Method: GET 2023-10-12T00:40:02.163 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.164 DEBUG:boto:Data: 2023-10-12T00:40:02.164 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.165 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.165 DEBUG:boto:Params: {} 2023-10-12T00:40:02.166 DEBUG:boto:Token: None 2023-10-12T00:40:02.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY= 2023-10-12T00:40:02.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY=', 'Content-Length': '0'} 2023-10-12T00:40:02.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0bbc4d90c05aa61-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.210 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.211 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.211 DEBUG:boto:path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.212 DEBUG:boto:auth_path=/wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.212 DEBUG:boto:Method: GET 2023-10-12T00:40:02.213 DEBUG:boto:Path: /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.213 DEBUG:boto:Data: 2023-10-12T00:40:02.214 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.215 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.215 DEBUG:boto:Params: {} 2023-10-12T00:40:02.216 DEBUG:boto:Token: None 2023-10-12T00:40:02.216 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2?versionId=null 2023-10-12T00:40:02.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY= 2023-10-12T00:40:02.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pPdlVU/LDEx/60TV0c7WygN7jkY=', 'Content-Length': '0'} 2023-10-12T00:40:02.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ee562bacfd6a995-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.220 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.220 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.221 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.221 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.222 DEBUG:boto:Method: GET 2023-10-12T00:40:02.222 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.223 DEBUG:boto:Data: 2023-10-12T00:40:02.223 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.224 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.225 DEBUG:boto:Params: {} 2023-10-12T00:40:02.225 DEBUG:boto:Token: None 2023-10-12T00:40:02.226 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b9232ad68321ea4-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.229 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.230 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.231 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.231 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.232 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.232 DEBUG:boto:Method: GET 2023-10-12T00:40:02.233 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.233 DEBUG:boto:Data: 2023-10-12T00:40:02.233 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.234 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.235 DEBUG:boto:Params: {} 2023-10-12T00:40:02.235 DEBUG:boto:Token: None 2023-10-12T00:40:02.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0fa03759841b6c-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.240 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.241 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.241 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.242 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.242 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:40:02.243 DEBUG:boto:Data: 2023-10-12T00:40:02.243 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.244 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.244 DEBUG:boto:Params: {} 2023-10-12T00:40:02.245 DEBUG:boto:Token: None 2023-10-12T00:40:02.245 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2 2023-10-12T00:40:02.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nA0bQFmCMz+MIQyoKRpeIigwUEk= 2023-10-12T00:40:02.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nA0bQFmCMz+MIQyoKRpeIigwUEk=', 'Content-Length': '0'} 2023-10-12T00:40:02.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c6e6769b6d78ea4-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.248 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.249 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.249 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.250 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:40:02.250 DEBUG:boto:Data: 2023-10-12T00:40:02.251 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.251 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.252 DEBUG:boto:Params: {} 2023-10-12T00:40:02.252 DEBUG:boto:Token: None 2023-10-12T00:40:02.253 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2 2023-10-12T00:40:02.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nA0bQFmCMz+MIQyoKRpeIigwUEk= 2023-10-12T00:40:02.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nA0bQFmCMz+MIQyoKRpeIigwUEk=', 'Content-Length': '0'} 2023-10-12T00:40:02.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2616edb6be828b4-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:02.256 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.257 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.257 DEBUG:boto:Method: GET 2023-10-12T00:40:02.258 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:40:02.258 DEBUG:boto:Data: 2023-10-12T00:40:02.259 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.259 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.260 DEBUG:boto:Params: {} 2023-10-12T00:40:02.260 DEBUG:boto:Token: None 2023-10-12T00:40:02.261 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2 2023-10-12T00:40:02.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXuukYE+l+EqSeFsugoBdVuwgTY= 2023-10-12T00:40:02.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXuukYE+l+EqSeFsugoBdVuwgTY=', 'Content-Length': '0'} 2023-10-12T00:40:02.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1fb38c640af64c1-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.264 DEBUG:boto:path=/wafmoa-47/obj2 2023-10-12T00:40:02.264 DEBUG:boto:auth_path=/wafmoa-47/obj2 2023-10-12T00:40:02.265 DEBUG:boto:Method: GET 2023-10-12T00:40:02.265 DEBUG:boto:Path: /wafmoa-47/obj2 2023-10-12T00:40:02.266 DEBUG:boto:Data: 2023-10-12T00:40:02.266 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.267 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.268 DEBUG:boto:Params: {} 2023-10-12T00:40:02.268 DEBUG:boto:Token: None 2023-10-12T00:40:02.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj2 2023-10-12T00:40:02.269 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXuukYE+l+EqSeFsugoBdVuwgTY= 2023-10-12T00:40:02.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXuukYE+l+EqSeFsugoBdVuwgTY=', 'Content-Length': '0'} 2023-10-12T00:40:02.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059745b17e640ff4a-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.272 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.272 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.273 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.274 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.274 DEBUG:boto:Method: GET 2023-10-12T00:40:02.275 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.275 DEBUG:boto:Data: 2023-10-12T00:40:02.275 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.276 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.277 DEBUG:boto:Params: {} 2023-10-12T00:40:02.277 DEBUG:boto:Token: None 2023-10-12T00:40:02.278 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebd7b8650fbac03e-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.281 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.282 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.283 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.283 DEBUG:boto:path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.283 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.284 DEBUG:boto:Method: GET 2023-10-12T00:40:02.284 DEBUG:boto:Path: /wafmoa-47/?prefix=obj2 2023-10-12T00:40:02.285 DEBUG:boto:Data: 2023-10-12T00:40:02.285 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.286 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.286 DEBUG:boto:Params: {} 2023-10-12T00:40:02.287 DEBUG:boto:Token: None 2023-10-12T00:40:02.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d47180117df70c-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.291 DEBUG:boto:b'wafmoa-47obj21000falseobj22023-10-12T00:38:46.016Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:02.292 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.293 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.293 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.293 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.294 DEBUG:boto:Method: GET 2023-10-12T00:40:02.294 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.295 DEBUG:boto:Data: 2023-10-12T00:40:02.295 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.296 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.296 DEBUG:boto:Params: {} 2023-10-12T00:40:02.297 DEBUG:boto:Token: None 2023-10-12T00:40:02.297 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.298 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4= 2023-10-12T00:40:02.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4=', 'Content-Length': '0'} 2023-10-12T00:40:02.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000639c7f63606e2a08-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.301 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.301 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.302 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.302 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.302 DEBUG:boto:Method: GET 2023-10-12T00:40:02.303 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.303 DEBUG:boto:Data: 2023-10-12T00:40:02.304 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.305 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.305 DEBUG:boto:Params: {} 2023-10-12T00:40:02.305 DEBUG:boto:Token: None 2023-10-12T00:40:02.306 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4= 2023-10-12T00:40:02.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4=', 'Content-Length': '0'} 2023-10-12T00:40:02.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098f7065d1fac17f9-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.350 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.351 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.351 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.352 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.352 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.353 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.353 DEBUG:boto:Data: 2023-10-12T00:40:02.354 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.355 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.355 DEBUG:boto:Params: {} 2023-10-12T00:40:02.356 DEBUG:boto:Token: None 2023-10-12T00:40:02.356 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FG8y+l44AKte+7bDxZ8tIuNd38= 2023-10-12T00:40:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FG8y+l44AKte+7bDxZ8tIuNd38=', 'Content-Length': '0'} 2023-10-12T00:40:02.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8813661d43056b6-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.361 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.362 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.362 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.363 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.363 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.364 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.364 DEBUG:boto:Data: 2023-10-12T00:40:02.364 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.365 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.366 DEBUG:boto:Params: {} 2023-10-12T00:40:02.366 DEBUG:boto:Token: None 2023-10-12T00:40:02.367 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FG8y+l44AKte+7bDxZ8tIuNd38= 2023-10-12T00:40:02.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FG8y+l44AKte+7bDxZ8tIuNd38=', 'Content-Length': '0'} 2023-10-12T00:40:02.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe1d6e3383ea0c2d-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:02.370 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.371 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.371 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.372 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.372 DEBUG:boto:Method: GET 2023-10-12T00:40:02.373 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.373 DEBUG:boto:Data: 2023-10-12T00:40:02.373 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.374 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.375 DEBUG:boto:Params: {} 2023-10-12T00:40:02.375 DEBUG:boto:Token: None 2023-10-12T00:40:02.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4= 2023-10-12T00:40:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4=', 'Content-Length': '0'} 2023-10-12T00:40:02.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d0addf014b6a8df-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.379 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.379 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.380 DEBUG:boto:path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.380 DEBUG:boto:auth_path=/wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.380 DEBUG:boto:Method: GET 2023-10-12T00:40:02.381 DEBUG:boto:Path: /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.381 DEBUG:boto:Data: 2023-10-12T00:40:02.382 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.383 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.383 DEBUG:boto:Params: {} 2023-10-12T00:40:02.384 DEBUG:boto:Token: None 2023-10-12T00:40:02.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3?versionId=null 2023-10-12T00:40:02.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4= 2023-10-12T00:40:02.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:o8TnJN+2x/FgSNAQ50fLnPBJTi4=', 'Content-Length': '0'} 2023-10-12T00:40:02.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd4d7f1de0806b9-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.387 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.388 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.388 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.388 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.389 DEBUG:boto:Method: GET 2023-10-12T00:40:02.389 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.390 DEBUG:boto:Data: 2023-10-12T00:40:02.390 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.391 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.391 DEBUG:boto:Params: {} 2023-10-12T00:40:02.392 DEBUG:boto:Token: None 2023-10-12T00:40:02.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4b72cf4990c3bf-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.396 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.397 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.397 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.397 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.398 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.398 DEBUG:boto:Method: GET 2023-10-12T00:40:02.399 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.399 DEBUG:boto:Data: 2023-10-12T00:40:02.400 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.400 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.401 DEBUG:boto:Params: {} 2023-10-12T00:40:02.401 DEBUG:boto:Token: None 2023-10-12T00:40:02.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba5837690210521-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.405 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.406 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.406 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.407 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.407 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:40:02.408 DEBUG:boto:Data: 2023-10-12T00:40:02.408 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.409 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.409 DEBUG:boto:Params: {} 2023-10-12T00:40:02.410 DEBUG:boto:Token: None 2023-10-12T00:40:02.410 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3 2023-10-12T00:40:02.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oIGgnfd0jDmCCOj1KXsiOn2iNv0= 2023-10-12T00:40:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oIGgnfd0jDmCCOj1KXsiOn2iNv0=', 'Content-Length': '0'} 2023-10-12T00:40:02.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aceb82bc8008056-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.413 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.413 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.414 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.414 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:40:02.415 DEBUG:boto:Data: 2023-10-12T00:40:02.415 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.416 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.416 DEBUG:boto:Params: {} 2023-10-12T00:40:02.417 DEBUG:boto:Token: None 2023-10-12T00:40:02.417 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3 2023-10-12T00:40:02.418 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oIGgnfd0jDmCCOj1KXsiOn2iNv0= 2023-10-12T00:40:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oIGgnfd0jDmCCOj1KXsiOn2iNv0=', 'Content-Length': '0'} 2023-10-12T00:40:02.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eeaec4d86d43770b-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:02.421 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.422 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.422 DEBUG:boto:Method: GET 2023-10-12T00:40:02.422 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:40:02.423 DEBUG:boto:Data: 2023-10-12T00:40:02.423 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.424 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.425 DEBUG:boto:Params: {} 2023-10-12T00:40:02.425 DEBUG:boto:Token: None 2023-10-12T00:40:02.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3 2023-10-12T00:40:02.426 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KXg4HwvUyzMqOeUnI9Vcq6H2vpI= 2023-10-12T00:40:02.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KXg4HwvUyzMqOeUnI9Vcq6H2vpI=', 'Content-Length': '0'} 2023-10-12T00:40:02.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058927fde91c93343-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.429 DEBUG:boto:path=/wafmoa-47/obj3 2023-10-12T00:40:02.430 DEBUG:boto:auth_path=/wafmoa-47/obj3 2023-10-12T00:40:02.430 DEBUG:boto:Method: GET 2023-10-12T00:40:02.431 DEBUG:boto:Path: /wafmoa-47/obj3 2023-10-12T00:40:02.431 DEBUG:boto:Data: 2023-10-12T00:40:02.432 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.432 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.433 DEBUG:boto:Params: {} 2023-10-12T00:40:02.433 DEBUG:boto:Token: None 2023-10-12T00:40:02.434 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj3 2023-10-12T00:40:02.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KXg4HwvUyzMqOeUnI9Vcq6H2vpI= 2023-10-12T00:40:02.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KXg4HwvUyzMqOeUnI9Vcq6H2vpI=', 'Content-Length': '0'} 2023-10-12T00:40:02.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005098dec3062f86a2-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.437 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.437 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.438 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.438 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.439 DEBUG:boto:Method: GET 2023-10-12T00:40:02.439 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.440 DEBUG:boto:Data: 2023-10-12T00:40:02.440 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.441 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.441 DEBUG:boto:Params: {} 2023-10-12T00:40:02.442 DEBUG:boto:Token: None 2023-10-12T00:40:02.442 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f79ab098bb192757-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.446 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.446 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.447 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.447 DEBUG:boto:path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.448 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.448 DEBUG:boto:Method: GET 2023-10-12T00:40:02.448 DEBUG:boto:Path: /wafmoa-47/?prefix=obj3 2023-10-12T00:40:02.449 DEBUG:boto:Data: 2023-10-12T00:40:02.449 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.450 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.451 DEBUG:boto:Params: {} 2023-10-12T00:40:02.451 DEBUG:boto:Token: None 2023-10-12T00:40:02.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7841bfb668536c0-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.455 DEBUG:boto:b'wafmoa-47obj31000falseobj32023-10-12T00:38:46.075Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:02.457 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.457 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.458 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.458 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.459 DEBUG:boto:Method: GET 2023-10-12T00:40:02.459 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.460 DEBUG:boto:Data: 2023-10-12T00:40:02.460 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.461 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.461 DEBUG:boto:Params: {} 2023-10-12T00:40:02.462 DEBUG:boto:Token: None 2023-10-12T00:40:02.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U= 2023-10-12T00:40:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U=', 'Content-Length': '0'} 2023-10-12T00:40:02.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033dd7e2bd597b537-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.466 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.467 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.467 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.467 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.468 DEBUG:boto:Method: GET 2023-10-12T00:40:02.468 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.469 DEBUG:boto:Data: 2023-10-12T00:40:02.469 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.470 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.471 DEBUG:boto:Params: {} 2023-10-12T00:40:02.471 DEBUG:boto:Token: None 2023-10-12T00:40:02.472 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U= 2023-10-12T00:40:02.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U=', 'Content-Length': '0'} 2023-10-12T00:40:02.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d6166ab35b28465-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.522 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.523 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.523 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.524 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.524 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.524 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.525 DEBUG:boto:Data: 2023-10-12T00:40:02.525 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.526 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.527 DEBUG:boto:Params: {} 2023-10-12T00:40:02.529 DEBUG:boto:Token: None 2023-10-12T00:40:02.529 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TDomS1EjJA2c6OU4yg5r+YGnCWs= 2023-10-12T00:40:02.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TDomS1EjJA2c6OU4yg5r+YGnCWs=', 'Content-Length': '0'} 2023-10-12T00:40:02.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083d5d6858898a5dc-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.533 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.533 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.534 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.534 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.535 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.535 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.535 DEBUG:boto:Data: 2023-10-12T00:40:02.536 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.537 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.537 DEBUG:boto:Params: {} 2023-10-12T00:40:02.538 DEBUG:boto:Token: None 2023-10-12T00:40:02.538 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TDomS1EjJA2c6OU4yg5r+YGnCWs= 2023-10-12T00:40:02.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TDomS1EjJA2c6OU4yg5r+YGnCWs=', 'Content-Length': '0'} 2023-10-12T00:40:02.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099b46fdbc36852cc-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:02.542 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.542 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.543 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.543 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.543 DEBUG:boto:Method: GET 2023-10-12T00:40:02.544 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.544 DEBUG:boto:Data: 2023-10-12T00:40:02.545 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.546 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.546 DEBUG:boto:Params: {} 2023-10-12T00:40:02.547 DEBUG:boto:Token: None 2023-10-12T00:40:02.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U= 2023-10-12T00:40:02.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U=', 'Content-Length': '0'} 2023-10-12T00:40:02.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e593a106e819b0c-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.550 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.551 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.551 DEBUG:boto:path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.552 DEBUG:boto:auth_path=/wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.552 DEBUG:boto:Method: GET 2023-10-12T00:40:02.553 DEBUG:boto:Path: /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.553 DEBUG:boto:Data: 2023-10-12T00:40:02.553 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.554 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.555 DEBUG:boto:Params: {} 2023-10-12T00:40:02.555 DEBUG:boto:Token: None 2023-10-12T00:40:02.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4?versionId=null 2023-10-12T00:40:02.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U= 2023-10-12T00:40:02.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AoJ8iz8d+LGlRcWqk5o12mkYc/U=', 'Content-Length': '0'} 2023-10-12T00:40:02.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba4fc8b12ec5a41b-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.559 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.560 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.560 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.560 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.561 DEBUG:boto:Method: GET 2023-10-12T00:40:02.561 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.562 DEBUG:boto:Data: 2023-10-12T00:40:02.562 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.565 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.566 DEBUG:boto:Params: {} 2023-10-12T00:40:02.566 DEBUG:boto:Token: None 2023-10-12T00:40:02.567 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc42d5c65eb9d14c-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.571 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.572 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.572 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.573 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.573 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.574 DEBUG:boto:Method: GET 2023-10-12T00:40:02.574 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.574 DEBUG:boto:Data: 2023-10-12T00:40:02.575 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.576 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.576 DEBUG:boto:Params: {} 2023-10-12T00:40:02.577 DEBUG:boto:Token: None 2023-10-12T00:40:02.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566f585bf667fdd9-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.581 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.582 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.582 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.582 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.583 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:40:02.583 DEBUG:boto:Data: 2023-10-12T00:40:02.584 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.585 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.585 DEBUG:boto:Params: {} 2023-10-12T00:40:02.585 DEBUG:boto:Token: None 2023-10-12T00:40:02.586 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4 2023-10-12T00:40:02.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9WzH0NJwqUMMJn96vTg5QSsXy4= 2023-10-12T00:40:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9WzH0NJwqUMMJn96vTg5QSsXy4=', 'Content-Length': '0'} 2023-10-12T00:40:02.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e92dad27c6e5ff3-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.589 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.589 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.590 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.590 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:40:02.591 DEBUG:boto:Data: 2023-10-12T00:40:02.591 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.592 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.592 DEBUG:boto:Params: {} 2023-10-12T00:40:02.593 DEBUG:boto:Token: None 2023-10-12T00:40:02.593 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4 2023-10-12T00:40:02.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9WzH0NJwqUMMJn96vTg5QSsXy4= 2023-10-12T00:40:02.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9WzH0NJwqUMMJn96vTg5QSsXy4=', 'Content-Length': '0'} 2023-10-12T00:40:02.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bca2459b90950a4-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:02.597 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.597 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.598 DEBUG:boto:Method: GET 2023-10-12T00:40:02.598 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:40:02.599 DEBUG:boto:Data: 2023-10-12T00:40:02.599 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.600 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.600 DEBUG:boto:Params: {} 2023-10-12T00:40:02.601 DEBUG:boto:Token: None 2023-10-12T00:40:02.601 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4 2023-10-12T00:40:02.602 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kMvvDV21RYAL1bi73QkOdZBApRw= 2023-10-12T00:40:02.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kMvvDV21RYAL1bi73QkOdZBApRw=', 'Content-Length': '0'} 2023-10-12T00:40:02.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c30a172c0ac98f44-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.650 DEBUG:boto:path=/wafmoa-47/obj4 2023-10-12T00:40:02.651 DEBUG:boto:auth_path=/wafmoa-47/obj4 2023-10-12T00:40:02.652 DEBUG:boto:Method: GET 2023-10-12T00:40:02.652 DEBUG:boto:Path: /wafmoa-47/obj4 2023-10-12T00:40:02.652 DEBUG:boto:Data: 2023-10-12T00:40:02.653 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.654 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.654 DEBUG:boto:Params: {} 2023-10-12T00:40:02.655 DEBUG:boto:Token: None 2023-10-12T00:40:02.655 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj4 2023-10-12T00:40:02.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kMvvDV21RYAL1bi73QkOdZBApRw= 2023-10-12T00:40:02.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kMvvDV21RYAL1bi73QkOdZBApRw=', 'Content-Length': '0'} 2023-10-12T00:40:02.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:38:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb4d7bc64ac02ec9-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.659 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.659 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.660 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.660 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.661 DEBUG:boto:Method: GET 2023-10-12T00:40:02.661 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.661 DEBUG:boto:Data: 2023-10-12T00:40:02.662 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.663 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.663 DEBUG:boto:Params: {} 2023-10-12T00:40:02.664 DEBUG:boto:Token: None 2023-10-12T00:40:02.664 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373a933f7048cacd-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.668 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.668 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.669 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.669 DEBUG:boto:path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.670 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.670 DEBUG:boto:Method: GET 2023-10-12T00:40:02.671 DEBUG:boto:Path: /wafmoa-47/?prefix=obj4 2023-10-12T00:40:02.671 DEBUG:boto:Data: 2023-10-12T00:40:02.672 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.672 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.673 DEBUG:boto:Params: {} 2023-10-12T00:40:02.673 DEBUG:boto:Token: None 2023-10-12T00:40:02.674 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081888eacd27dd3c5-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.718 DEBUG:boto:b'wafmoa-47obj41000falseobj42023-10-12T00:38:46.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:02.720 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.720 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.721 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.721 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.721 DEBUG:boto:Method: GET 2023-10-12T00:40:02.722 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.722 DEBUG:boto:Data: 2023-10-12T00:40:02.723 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.724 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.724 DEBUG:boto:Params: {} 2023-10-12T00:40:02.725 DEBUG:boto:Token: None 2023-10-12T00:40:02.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0= 2023-10-12T00:40:02.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0=', 'Content-Length': '0'} 2023-10-12T00:40:02.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8e7f04943042eaa-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.730 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.731 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.731 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.732 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.733 DEBUG:boto:Method: GET 2023-10-12T00:40:02.734 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.735 DEBUG:boto:Data: 2023-10-12T00:40:02.735 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.737 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.738 DEBUG:boto:Params: {} 2023-10-12T00:40:02.739 DEBUG:boto:Token: None 2023-10-12T00:40:02.740 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0= 2023-10-12T00:40:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0=', 'Content-Length': '0'} 2023-10-12T00:40:02.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0ef2c2e68c20827-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.745 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.746 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.746 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.747 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.748 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.748 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.749 DEBUG:boto:Data: 2023-10-12T00:40:02.750 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.751 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.752 DEBUG:boto:Params: {} 2023-10-12T00:40:02.752 DEBUG:boto:Token: None 2023-10-12T00:40:02.753 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XT/WP8PBt5d7H1obQoUVsQOYEwQ= 2023-10-12T00:40:02.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XT/WP8PBt5d7H1obQoUVsQOYEwQ=', 'Content-Length': '0'} 2023-10-12T00:40:02.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b301e057b5d734-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.757 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.758 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.758 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.759 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.759 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.759 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.760 DEBUG:boto:Data: 2023-10-12T00:40:02.761 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.762 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.762 DEBUG:boto:Params: {} 2023-10-12T00:40:02.763 DEBUG:boto:Token: None 2023-10-12T00:40:02.764 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XT/WP8PBt5d7H1obQoUVsQOYEwQ= 2023-10-12T00:40:02.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XT/WP8PBt5d7H1obQoUVsQOYEwQ=', 'Content-Length': '0'} 2023-10-12T00:40:02.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a03c7742c2843245-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:02.770 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.770 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.771 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.771 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.772 DEBUG:boto:Method: GET 2023-10-12T00:40:02.772 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.773 DEBUG:boto:Data: 2023-10-12T00:40:02.773 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.774 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.775 DEBUG:boto:Params: {} 2023-10-12T00:40:02.775 DEBUG:boto:Token: None 2023-10-12T00:40:02.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0= 2023-10-12T00:40:02.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0=', 'Content-Length': '0'} 2023-10-12T00:40:02.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7861df22e1b8fa1-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.780 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.780 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.781 DEBUG:boto:path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.781 DEBUG:boto:auth_path=/wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.781 DEBUG:boto:Method: GET 2023-10-12T00:40:02.782 DEBUG:boto:Path: /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.782 DEBUG:boto:Data: 2023-10-12T00:40:02.783 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.784 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.784 DEBUG:boto:Params: {} 2023-10-12T00:40:02.785 DEBUG:boto:Token: None 2023-10-12T00:40:02.785 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5?versionId=null 2023-10-12T00:40:02.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0= 2023-10-12T00:40:02.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jsshsoqtU9/722weQjLuYDwG5t0=', 'Content-Length': '0'} 2023-10-12T00:40:02.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d94199aede9e68ec-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.789 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.789 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.790 DEBUG:boto:path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.791 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.791 DEBUG:boto:Method: GET 2023-10-12T00:40:02.791 DEBUG:boto:Path: /wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.792 DEBUG:boto:Data: 2023-10-12T00:40:02.792 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.793 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.794 DEBUG:boto:Params: {} 2023-10-12T00:40:02.794 DEBUG:boto:Token: None 2023-10-12T00:40:02.795 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c011ab448fc8476-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.799 DEBUG:boto:b'wafmoa-47obj51000falseobj52023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.799 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.800 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.800 DEBUG:boto:path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.801 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.801 DEBUG:boto:Method: GET 2023-10-12T00:40:02.802 DEBUG:boto:Path: /wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.802 DEBUG:boto:Data: 2023-10-12T00:40:02.803 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.804 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.804 DEBUG:boto:Params: {} 2023-10-12T00:40:02.805 DEBUG:boto:Token: None 2023-10-12T00:40:02.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062fa7f72cba21c5d-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.809 DEBUG:boto:b'wafmoa-47obj51000falseobj52023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.810 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.810 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.811 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.811 DEBUG:boto:Path: /wafmoa-47/obj5 2023-10-12T00:40:02.812 DEBUG:boto:Data: 2023-10-12T00:40:02.812 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.813 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.814 DEBUG:boto:Params: {} 2023-10-12T00:40:02.814 DEBUG:boto:Token: None 2023-10-12T00:40:02.815 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5 2023-10-12T00:40:02.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I31GaXoBHK0oGkzdHervx22hzWs= 2023-10-12T00:40:02.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I31GaXoBHK0oGkzdHervx22hzWs=', 'Content-Length': '0'} 2023-10-12T00:40:02.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd623bc88300b77f-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.818 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.818 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.819 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.819 DEBUG:boto:Path: /wafmoa-47/obj5 2023-10-12T00:40:02.819 DEBUG:boto:Data: 2023-10-12T00:40:02.820 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.821 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.821 DEBUG:boto:Params: {} 2023-10-12T00:40:02.822 DEBUG:boto:Token: None 2023-10-12T00:40:02.822 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5 2023-10-12T00:40:02.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I31GaXoBHK0oGkzdHervx22hzWs= 2023-10-12T00:40:02.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I31GaXoBHK0oGkzdHervx22hzWs=', 'Content-Length': '0'} 2023-10-12T00:40:02.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06952dfa3ddb95e-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:40:02.826 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.826 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.827 DEBUG:boto:Method: GET 2023-10-12T00:40:02.827 DEBUG:boto:Path: /wafmoa-47/obj5 2023-10-12T00:40:02.828 DEBUG:boto:Data: 2023-10-12T00:40:02.828 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.829 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.829 DEBUG:boto:Params: {} 2023-10-12T00:40:02.830 DEBUG:boto:Token: None 2023-10-12T00:40:02.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5 2023-10-12T00:40:02.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:21WzhwxypWPYVjqyNFImpmo6c2w= 2023-10-12T00:40:02.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:21WzhwxypWPYVjqyNFImpmo6c2w=', 'Content-Length': '0'} 2023-10-12T00:40:02.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c214b351b1c1e19f-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.834 DEBUG:boto:path=/wafmoa-47/obj5 2023-10-12T00:40:02.834 DEBUG:boto:auth_path=/wafmoa-47/obj5 2023-10-12T00:40:02.835 DEBUG:boto:Method: GET 2023-10-12T00:40:02.835 DEBUG:boto:Path: /wafmoa-47/obj5 2023-10-12T00:40:02.836 DEBUG:boto:Data: 2023-10-12T00:40:02.836 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.837 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.837 DEBUG:boto:Params: {} 2023-10-12T00:40:02.838 DEBUG:boto:Token: None 2023-10-12T00:40:02.838 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj5 2023-10-12T00:40:02.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:21WzhwxypWPYVjqyNFImpmo6c2w= 2023-10-12T00:40:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:21WzhwxypWPYVjqyNFImpmo6c2w=', 'Content-Length': '0'} 2023-10-12T00:40:02.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e30011fda73da6c1-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.842 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.842 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.843 DEBUG:boto:path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.843 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.844 DEBUG:boto:Method: GET 2023-10-12T00:40:02.844 DEBUG:boto:Path: /wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.845 DEBUG:boto:Data: 2023-10-12T00:40:02.845 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.846 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.846 DEBUG:boto:Params: {} 2023-10-12T00:40:02.847 DEBUG:boto:Token: None 2023-10-12T00:40:02.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c957eaa6b08066a4-0065274062-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.851 DEBUG:boto:b'wafmoa-47obj51000falseobj52023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.852 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:02.852 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:02.853 DEBUG:boto:path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.853 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.854 DEBUG:boto:Method: GET 2023-10-12T00:40:02.854 DEBUG:boto:Path: /wafmoa-47/?prefix=obj5 2023-10-12T00:40:02.855 DEBUG:boto:Data: 2023-10-12T00:40:02.855 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.856 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.856 DEBUG:boto:Params: {} 2023-10-12T00:40:02.857 DEBUG:boto:Token: None 2023-10-12T00:40:02.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/ 2023-10-12T00:40:02.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY= 2023-10-12T00:40:02.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MnNipf2gjdWt6Aal8jC525R3BpY=', 'Content-Length': '0'} 2023-10-12T00:40:02.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c5af86bbfa524d-0065274062-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.861 DEBUG:boto:b'wafmoa-47obj51000falseobj52023-10-12T00:39:19.460Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:02.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:02.862 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.863 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.863 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.864 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.864 DEBUG:boto:Method: GET 2023-10-12T00:40:02.864 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.865 DEBUG:boto:Data: 2023-10-12T00:40:02.866 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.867 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.868 DEBUG:boto:Params: {} 2023-10-12T00:40:02.868 DEBUG:boto:Token: None 2023-10-12T00:40:02.869 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668= 2023-10-12T00:40:02.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668=', 'Content-Length': '0'} 2023-10-12T00:40:02.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d757e9683bff7a5-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.914 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.915 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.915 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.916 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.916 DEBUG:boto:Method: GET 2023-10-12T00:40:02.917 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.917 DEBUG:boto:Data: 2023-10-12T00:40:02.918 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.919 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.919 DEBUG:boto:Params: {} 2023-10-12T00:40:02.920 DEBUG:boto:Token: None 2023-10-12T00:40:02.920 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668= 2023-10-12T00:40:02.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668=', 'Content-Length': '0'} 2023-10-12T00:40:02.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9e672f3a973e900-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.924 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.924 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.925 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.925 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.926 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.926 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.926 DEBUG:boto:Data: 2023-10-12T00:40:02.927 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.928 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.928 DEBUG:boto:Params: {} 2023-10-12T00:40:02.929 DEBUG:boto:Token: None 2023-10-12T00:40:02.929 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ylDjO1F1KCr//679rvB3tHW9Rks= 2023-10-12T00:40:02.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ylDjO1F1KCr//679rvB3tHW9Rks=', 'Content-Length': '0'} 2023-10-12T00:40:02.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000767b94057c795bdd-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.933 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.933 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.933 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.934 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.934 DEBUG:boto:Method: HEAD 2023-10-12T00:40:02.935 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.935 DEBUG:boto:Data: 2023-10-12T00:40:02.936 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.936 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.937 DEBUG:boto:Params: {} 2023-10-12T00:40:02.937 DEBUG:boto:Token: None 2023-10-12T00:40:02.938 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ylDjO1F1KCr//679rvB3tHW9Rks= 2023-10-12T00:40:02.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ylDjO1F1KCr//679rvB3tHW9Rks=', 'Content-Length': '0'} 2023-10-12T00:40:02.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000993225adf1c956fa-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:02.941 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.942 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.942 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.942 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.943 DEBUG:boto:Method: GET 2023-10-12T00:40:02.943 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.944 DEBUG:boto:Data: 2023-10-12T00:40:02.944 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:02.945 DEBUG:boto:Port: 8001 2023-10-12T00:40:02.946 DEBUG:boto:Params: {} 2023-10-12T00:40:02.946 DEBUG:boto:Token: None 2023-10-12T00:40:02.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668= 2023-10-12T00:40:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668=', 'Content-Length': '0'} 2023-10-12T00:40:02.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b70aa3735ac08c0-0065274062-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:02.950 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:02.950 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:02.951 DEBUG:boto:path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.951 DEBUG:boto:auth_path=/wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.952 DEBUG:boto:Method: GET 2023-10-12T00:40:02.952 DEBUG:boto:Path: /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.952 DEBUG:boto:Data: 2023-10-12T00:40:02.953 DEBUG:boto:Headers: {} 2023-10-12T00:40:02.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:02.954 DEBUG:boto:Port: 8000 2023-10-12T00:40:02.954 DEBUG:boto:Params: {} 2023-10-12T00:40:02.955 DEBUG:boto:Token: None 2023-10-12T00:40:02.955 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:02 GMT /wafmoa-47/obj6?versionId=null 2023-10-12T00:40:02.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668= 2023-10-12T00:40:02.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhAZyI0v7TDw3lLlLittpvqp668=', 'Content-Length': '0'} 2023-10-12T00:40:02.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000543516ce02055e84-0065274062-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.002 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.003 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.004 DEBUG:boto:path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.004 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.004 DEBUG:boto:Method: GET 2023-10-12T00:40:03.005 DEBUG:boto:Path: /wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.005 DEBUG:boto:Data: 2023-10-12T00:40:03.006 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.007 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.008 DEBUG:boto:Params: {} 2023-10-12T00:40:03.008 DEBUG:boto:Token: None 2023-10-12T00:40:03.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6bf7a74af216d6-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.013 DEBUG:boto:b'wafmoa-47obj61000falseobj62023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.013 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.014 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.014 DEBUG:boto:path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.015 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.015 DEBUG:boto:Method: GET 2023-10-12T00:40:03.016 DEBUG:boto:Path: /wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.016 DEBUG:boto:Data: 2023-10-12T00:40:03.017 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.018 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.018 DEBUG:boto:Params: {} 2023-10-12T00:40:03.019 DEBUG:boto:Token: None 2023-10-12T00:40:03.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcaee561c9f2e72e-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.023 DEBUG:boto:b'wafmoa-47obj61000falseobj62023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.024 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:03.025 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:03.025 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.026 DEBUG:boto:Path: /wafmoa-47/obj6 2023-10-12T00:40:03.026 DEBUG:boto:Data: 2023-10-12T00:40:03.026 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.027 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.028 DEBUG:boto:Params: {} 2023-10-12T00:40:03.028 DEBUG:boto:Token: None 2023-10-12T00:40:03.029 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj6 2023-10-12T00:40:03.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ljtdoBl38K7vkMJ1fMvIaYnbm8o= 2023-10-12T00:40:03.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ljtdoBl38K7vkMJ1fMvIaYnbm8o=', 'Content-Length': '0'} 2023-10-12T00:40:03.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000975878bf055f0afe-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.032 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:03.033 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:03.033 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.034 DEBUG:boto:Path: /wafmoa-47/obj6 2023-10-12T00:40:03.034 DEBUG:boto:Data: 2023-10-12T00:40:03.035 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.036 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.036 DEBUG:boto:Params: {} 2023-10-12T00:40:03.037 DEBUG:boto:Token: None 2023-10-12T00:40:03.037 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj6 2023-10-12T00:40:03.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ljtdoBl38K7vkMJ1fMvIaYnbm8o= 2023-10-12T00:40:03.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ljtdoBl38K7vkMJ1fMvIaYnbm8o=', 'Content-Length': '0'} 2023-10-12T00:40:03.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005616ecd39dd20e89-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:40:03.041 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:03.041 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:03.042 DEBUG:boto:Method: GET 2023-10-12T00:40:03.042 DEBUG:boto:Path: /wafmoa-47/obj6 2023-10-12T00:40:03.043 DEBUG:boto:Data: 2023-10-12T00:40:03.043 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.044 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.045 DEBUG:boto:Params: {} 2023-10-12T00:40:03.045 DEBUG:boto:Token: None 2023-10-12T00:40:03.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj6 2023-10-12T00:40:03.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PHsmIbYFJuqD499I3m/12kE7gbM= 2023-10-12T00:40:03.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PHsmIbYFJuqD499I3m/12kE7gbM=', 'Content-Length': '0'} 2023-10-12T00:40:03.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b40329312700c0ff-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.049 DEBUG:boto:path=/wafmoa-47/obj6 2023-10-12T00:40:03.050 DEBUG:boto:auth_path=/wafmoa-47/obj6 2023-10-12T00:40:03.050 DEBUG:boto:Method: GET 2023-10-12T00:40:03.051 DEBUG:boto:Path: /wafmoa-47/obj6 2023-10-12T00:40:03.051 DEBUG:boto:Data: 2023-10-12T00:40:03.052 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.053 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.053 DEBUG:boto:Params: {} 2023-10-12T00:40:03.054 DEBUG:boto:Token: None 2023-10-12T00:40:03.054 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj6 2023-10-12T00:40:03.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PHsmIbYFJuqD499I3m/12kE7gbM= 2023-10-12T00:40:03.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PHsmIbYFJuqD499I3m/12kE7gbM=', 'Content-Length': '0'} 2023-10-12T00:40:03.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001468941f6a07463-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.058 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.058 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.059 DEBUG:boto:path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.059 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.059 DEBUG:boto:Method: GET 2023-10-12T00:40:03.060 DEBUG:boto:Path: /wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.060 DEBUG:boto:Data: 2023-10-12T00:40:03.061 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.062 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.062 DEBUG:boto:Params: {} 2023-10-12T00:40:03.063 DEBUG:boto:Token: None 2023-10-12T00:40:03.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3fac860fb0ae88d-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.067 DEBUG:boto:b'wafmoa-47obj61000falseobj62023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.068 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.068 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.069 DEBUG:boto:path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.069 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.070 DEBUG:boto:Method: GET 2023-10-12T00:40:03.070 DEBUG:boto:Path: /wafmoa-47/?prefix=obj6 2023-10-12T00:40:03.071 DEBUG:boto:Data: 2023-10-12T00:40:03.071 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.072 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.072 DEBUG:boto:Params: {} 2023-10-12T00:40:03.073 DEBUG:boto:Token: None 2023-10-12T00:40:03.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076406296a111448a-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.077 DEBUG:boto:b'wafmoa-47obj61000falseobj62023-10-12T00:39:19.479Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:03.078 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.079 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.079 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.080 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.080 DEBUG:boto:Method: GET 2023-10-12T00:40:03.081 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.081 DEBUG:boto:Data: 2023-10-12T00:40:03.082 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.082 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.083 DEBUG:boto:Params: {} 2023-10-12T00:40:03.083 DEBUG:boto:Token: None 2023-10-12T00:40:03.084 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk= 2023-10-12T00:40:03.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk=', 'Content-Length': '0'} 2023-10-12T00:40:03.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bb9fa6caee98f31-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.088 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.088 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.089 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.089 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.090 DEBUG:boto:Method: GET 2023-10-12T00:40:03.090 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.090 DEBUG:boto:Data: 2023-10-12T00:40:03.091 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.092 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.092 DEBUG:boto:Params: {} 2023-10-12T00:40:03.093 DEBUG:boto:Token: None 2023-10-12T00:40:03.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk= 2023-10-12T00:40:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk=', 'Content-Length': '0'} 2023-10-12T00:40:03.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b576ddb647edab5-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.097 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.098 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.098 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.099 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.099 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.100 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.100 DEBUG:boto:Data: 2023-10-12T00:40:03.101 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.101 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.102 DEBUG:boto:Params: {} 2023-10-12T00:40:03.102 DEBUG:boto:Token: None 2023-10-12T00:40:03.103 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.103 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cY4WSx3keL/AzRl+0DedXNN+irc= 2023-10-12T00:40:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cY4WSx3keL/AzRl+0DedXNN+irc=', 'Content-Length': '0'} 2023-10-12T00:40:03.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000018dfe889c4bb033-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.106 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.107 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.107 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.108 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.108 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.109 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.109 DEBUG:boto:Data: 2023-10-12T00:40:03.109 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.110 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.111 DEBUG:boto:Params: {} 2023-10-12T00:40:03.111 DEBUG:boto:Token: None 2023-10-12T00:40:03.112 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cY4WSx3keL/AzRl+0DedXNN+irc= 2023-10-12T00:40:03.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cY4WSx3keL/AzRl+0DedXNN+irc=', 'Content-Length': '0'} 2023-10-12T00:40:03.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d19e03347354322d-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:03.116 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.116 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.117 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.117 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.118 DEBUG:boto:Method: GET 2023-10-12T00:40:03.118 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.119 DEBUG:boto:Data: 2023-10-12T00:40:03.119 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.120 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.120 DEBUG:boto:Params: {} 2023-10-12T00:40:03.121 DEBUG:boto:Token: None 2023-10-12T00:40:03.121 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk= 2023-10-12T00:40:03.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk=', 'Content-Length': '0'} 2023-10-12T00:40:03.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ba355b3521260db-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.166 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.167 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.168 DEBUG:boto:path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.168 DEBUG:boto:auth_path=/wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.169 DEBUG:boto:Method: GET 2023-10-12T00:40:03.169 DEBUG:boto:Path: /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.169 DEBUG:boto:Data: 2023-10-12T00:40:03.170 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.171 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.172 DEBUG:boto:Params: {} 2023-10-12T00:40:03.173 DEBUG:boto:Token: None 2023-10-12T00:40:03.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7?versionId=null 2023-10-12T00:40:03.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk= 2023-10-12T00:40:03.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qCfx/E5U43aQcy5FEis/vi/TTIk=', 'Content-Length': '0'} 2023-10-12T00:40:03.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011b839e8e4e34c2d-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.177 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.177 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.178 DEBUG:boto:path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.178 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.179 DEBUG:boto:Method: GET 2023-10-12T00:40:03.179 DEBUG:boto:Path: /wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.180 DEBUG:boto:Data: 2023-10-12T00:40:03.181 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.182 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.183 DEBUG:boto:Params: {} 2023-10-12T00:40:03.183 DEBUG:boto:Token: None 2023-10-12T00:40:03.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccda038f46da96f2-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.187 DEBUG:boto:b'wafmoa-47obj71000falseobj72023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.188 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.189 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.189 DEBUG:boto:path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.189 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.190 DEBUG:boto:Method: GET 2023-10-12T00:40:03.190 DEBUG:boto:Path: /wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.191 DEBUG:boto:Data: 2023-10-12T00:40:03.191 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.192 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.193 DEBUG:boto:Params: {} 2023-10-12T00:40:03.193 DEBUG:boto:Token: None 2023-10-12T00:40:03.194 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.194 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c98ac48ec4b51be8-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.197 DEBUG:boto:b'wafmoa-47obj71000falseobj72023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.198 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.198 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.199 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.199 DEBUG:boto:Path: /wafmoa-47/obj7 2023-10-12T00:40:03.200 DEBUG:boto:Data: 2023-10-12T00:40:03.200 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.201 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.201 DEBUG:boto:Params: {} 2023-10-12T00:40:03.202 DEBUG:boto:Token: None 2023-10-12T00:40:03.202 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7 2023-10-12T00:40:03.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lGO803pZs9gbAmQwebtbvKnh4Ys= 2023-10-12T00:40:03.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lGO803pZs9gbAmQwebtbvKnh4Ys=', 'Content-Length': '0'} 2023-10-12T00:40:03.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000430e75554839b4f2-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.206 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.206 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.207 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.207 DEBUG:boto:Path: /wafmoa-47/obj7 2023-10-12T00:40:03.208 DEBUG:boto:Data: 2023-10-12T00:40:03.208 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.209 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.209 DEBUG:boto:Params: {} 2023-10-12T00:40:03.210 DEBUG:boto:Token: None 2023-10-12T00:40:03.210 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7 2023-10-12T00:40:03.211 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lGO803pZs9gbAmQwebtbvKnh4Ys= 2023-10-12T00:40:03.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lGO803pZs9gbAmQwebtbvKnh4Ys=', 'Content-Length': '0'} 2023-10-12T00:40:03.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000946ca8f781e1e1d4-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:40:03.214 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.215 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.215 DEBUG:boto:Method: GET 2023-10-12T00:40:03.216 DEBUG:boto:Path: /wafmoa-47/obj7 2023-10-12T00:40:03.216 DEBUG:boto:Data: 2023-10-12T00:40:03.217 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.218 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.218 DEBUG:boto:Params: {} 2023-10-12T00:40:03.219 DEBUG:boto:Token: None 2023-10-12T00:40:03.219 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7 2023-10-12T00:40:03.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+KXLZfzsy3roLcWqSbdPbIGC3Y= 2023-10-12T00:40:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+KXLZfzsy3roLcWqSbdPbIGC3Y=', 'Content-Length': '0'} 2023-10-12T00:40:03.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d17e665a3cf2afa-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.223 DEBUG:boto:path=/wafmoa-47/obj7 2023-10-12T00:40:03.223 DEBUG:boto:auth_path=/wafmoa-47/obj7 2023-10-12T00:40:03.224 DEBUG:boto:Method: GET 2023-10-12T00:40:03.224 DEBUG:boto:Path: /wafmoa-47/obj7 2023-10-12T00:40:03.225 DEBUG:boto:Data: 2023-10-12T00:40:03.225 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.226 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.227 DEBUG:boto:Params: {} 2023-10-12T00:40:03.227 DEBUG:boto:Token: None 2023-10-12T00:40:03.227 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj7 2023-10-12T00:40:03.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x+KXLZfzsy3roLcWqSbdPbIGC3Y= 2023-10-12T00:40:03.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x+KXLZfzsy3roLcWqSbdPbIGC3Y=', 'Content-Length': '0'} 2023-10-12T00:40:03.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d944b30cf5a2bee-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.231 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.231 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.232 DEBUG:boto:path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.232 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.233 DEBUG:boto:Method: GET 2023-10-12T00:40:03.233 DEBUG:boto:Path: /wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.233 DEBUG:boto:Data: 2023-10-12T00:40:03.234 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.235 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.235 DEBUG:boto:Params: {} 2023-10-12T00:40:03.236 DEBUG:boto:Token: None 2023-10-12T00:40:03.237 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682ede1b9935f98e-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.240 DEBUG:boto:b'wafmoa-47obj71000falseobj72023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.241 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.242 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.242 DEBUG:boto:path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.242 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.243 DEBUG:boto:Method: GET 2023-10-12T00:40:03.243 DEBUG:boto:Path: /wafmoa-47/?prefix=obj7 2023-10-12T00:40:03.244 DEBUG:boto:Data: 2023-10-12T00:40:03.244 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.245 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.245 DEBUG:boto:Params: {} 2023-10-12T00:40:03.246 DEBUG:boto:Token: None 2023-10-12T00:40:03.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cefc1ea8d8bbb73-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.290 DEBUG:boto:b'wafmoa-47obj71000falseobj72023-10-12T00:39:19.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:03.292 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.292 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.292 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.293 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.293 DEBUG:boto:Method: GET 2023-10-12T00:40:03.294 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.294 DEBUG:boto:Data: 2023-10-12T00:40:03.295 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.296 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.296 DEBUG:boto:Params: {} 2023-10-12T00:40:03.297 DEBUG:boto:Token: None 2023-10-12T00:40:03.297 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.298 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM= 2023-10-12T00:40:03.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM=', 'Content-Length': '0'} 2023-10-12T00:40:03.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c59405d85bca9396-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.301 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.301 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.302 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.302 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.303 DEBUG:boto:Method: GET 2023-10-12T00:40:03.303 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.304 DEBUG:boto:Data: 2023-10-12T00:40:03.304 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.305 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.305 DEBUG:boto:Params: {} 2023-10-12T00:40:03.306 DEBUG:boto:Token: None 2023-10-12T00:40:03.306 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM= 2023-10-12T00:40:03.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM=', 'Content-Length': '0'} 2023-10-12T00:40:03.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbd434ef82713d4b-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.311 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.312 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.312 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.313 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.313 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.314 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.314 DEBUG:boto:Data: 2023-10-12T00:40:03.314 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.315 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.316 DEBUG:boto:Params: {} 2023-10-12T00:40:03.316 DEBUG:boto:Token: None 2023-10-12T00:40:03.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BrXSn5brqYAuHeNEJNoAVfw6w8U= 2023-10-12T00:40:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BrXSn5brqYAuHeNEJNoAVfw6w8U=', 'Content-Length': '0'} 2023-10-12T00:40:03.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013cd29f5b5c75912-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.320 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.320 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.321 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.321 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.321 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.322 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.322 DEBUG:boto:Data: 2023-10-12T00:40:03.323 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.323 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.324 DEBUG:boto:Params: {} 2023-10-12T00:40:03.324 DEBUG:boto:Token: None 2023-10-12T00:40:03.325 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BrXSn5brqYAuHeNEJNoAVfw6w8U= 2023-10-12T00:40:03.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BrXSn5brqYAuHeNEJNoAVfw6w8U=', 'Content-Length': '0'} 2023-10-12T00:40:03.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000194dbae2171cd91-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:03.330 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.330 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.331 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.331 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.331 DEBUG:boto:Method: GET 2023-10-12T00:40:03.332 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.332 DEBUG:boto:Data: 2023-10-12T00:40:03.333 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.333 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.334 DEBUG:boto:Params: {} 2023-10-12T00:40:03.334 DEBUG:boto:Token: None 2023-10-12T00:40:03.335 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM= 2023-10-12T00:40:03.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM=', 'Content-Length': '0'} 2023-10-12T00:40:03.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000730125ff54c87890-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.338 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.339 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.339 DEBUG:boto:path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.340 DEBUG:boto:auth_path=/wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.340 DEBUG:boto:Method: GET 2023-10-12T00:40:03.340 DEBUG:boto:Path: /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.341 DEBUG:boto:Data: 2023-10-12T00:40:03.341 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.342 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.342 DEBUG:boto:Params: {} 2023-10-12T00:40:03.343 DEBUG:boto:Token: None 2023-10-12T00:40:03.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8?versionId=null 2023-10-12T00:40:03.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM= 2023-10-12T00:40:03.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uR10CgJmD0Jd9U0nhYB3e2XXLrM=', 'Content-Length': '0'} 2023-10-12T00:40:03.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017d56171148b57c2-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.347 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.347 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.348 DEBUG:boto:path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.348 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.349 DEBUG:boto:Method: GET 2023-10-12T00:40:03.349 DEBUG:boto:Path: /wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.350 DEBUG:boto:Data: 2023-10-12T00:40:03.350 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.351 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.351 DEBUG:boto:Params: {} 2023-10-12T00:40:03.352 DEBUG:boto:Token: None 2023-10-12T00:40:03.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b93bde192e0eddc-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.356 DEBUG:boto:b'wafmoa-47obj81000falseobj82023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.357 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.357 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.358 DEBUG:boto:path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.358 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.358 DEBUG:boto:Method: GET 2023-10-12T00:40:03.359 DEBUG:boto:Path: /wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.359 DEBUG:boto:Data: 2023-10-12T00:40:03.360 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.361 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.361 DEBUG:boto:Params: {} 2023-10-12T00:40:03.362 DEBUG:boto:Token: None 2023-10-12T00:40:03.362 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed21d9f9de57b5a3-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.410 DEBUG:boto:b'wafmoa-47obj81000falseobj82023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.411 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.412 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.412 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.413 DEBUG:boto:Path: /wafmoa-47/obj8 2023-10-12T00:40:03.413 DEBUG:boto:Data: 2023-10-12T00:40:03.414 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.414 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.415 DEBUG:boto:Params: {} 2023-10-12T00:40:03.415 DEBUG:boto:Token: None 2023-10-12T00:40:03.416 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8 2023-10-12T00:40:03.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LG5prGcKto456kak8IkgN4pFA8c= 2023-10-12T00:40:03.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LG5prGcKto456kak8IkgN4pFA8c=', 'Content-Length': '0'} 2023-10-12T00:40:03.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038b65265b0dec0f4-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.420 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.420 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.420 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.421 DEBUG:boto:Path: /wafmoa-47/obj8 2023-10-12T00:40:03.421 DEBUG:boto:Data: 2023-10-12T00:40:03.422 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.422 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.423 DEBUG:boto:Params: {} 2023-10-12T00:40:03.424 DEBUG:boto:Token: None 2023-10-12T00:40:03.424 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8 2023-10-12T00:40:03.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LG5prGcKto456kak8IkgN4pFA8c= 2023-10-12T00:40:03.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LG5prGcKto456kak8IkgN4pFA8c=', 'Content-Length': '0'} 2023-10-12T00:40:03.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f83c913abfd6fcd1-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:40:03.428 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.428 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.429 DEBUG:boto:Method: GET 2023-10-12T00:40:03.430 DEBUG:boto:Path: /wafmoa-47/obj8 2023-10-12T00:40:03.430 DEBUG:boto:Data: 2023-10-12T00:40:03.431 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.431 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.432 DEBUG:boto:Params: {} 2023-10-12T00:40:03.432 DEBUG:boto:Token: None 2023-10-12T00:40:03.433 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8 2023-10-12T00:40:03.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xB4pwpASPnB54aLWJADtfsD5nas= 2023-10-12T00:40:03.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xB4pwpASPnB54aLWJADtfsD5nas=', 'Content-Length': '0'} 2023-10-12T00:40:03.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fcd0241d96c3869-0065274063-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.479 DEBUG:boto:path=/wafmoa-47/obj8 2023-10-12T00:40:03.480 DEBUG:boto:auth_path=/wafmoa-47/obj8 2023-10-12T00:40:03.480 DEBUG:boto:Method: GET 2023-10-12T00:40:03.481 DEBUG:boto:Path: /wafmoa-47/obj8 2023-10-12T00:40:03.481 DEBUG:boto:Data: 2023-10-12T00:40:03.482 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.483 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.483 DEBUG:boto:Params: {} 2023-10-12T00:40:03.484 DEBUG:boto:Token: None 2023-10-12T00:40:03.484 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/obj8 2023-10-12T00:40:03.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xB4pwpASPnB54aLWJADtfsD5nas= 2023-10-12T00:40:03.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xB4pwpASPnB54aLWJADtfsD5nas=', 'Content-Length': '0'} 2023-10-12T00:40:03.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-47:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d5bd366f9b93089-0065274063-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.488 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.489 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.489 DEBUG:boto:path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.490 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.490 DEBUG:boto:Method: GET 2023-10-12T00:40:03.490 DEBUG:boto:Path: /wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.491 DEBUG:boto:Data: 2023-10-12T00:40:03.491 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.493 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.493 DEBUG:boto:Params: {} 2023-10-12T00:40:03.494 DEBUG:boto:Token: None 2023-10-12T00:40:03.495 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c17405684ac277-0065274063-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.498 DEBUG:boto:b'wafmoa-47obj81000falseobj82023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.499 DEBUG:boto:path=/wafmoa-47/ 2023-10-12T00:40:03.500 DEBUG:boto:auth_path=/wafmoa-47/ 2023-10-12T00:40:03.500 DEBUG:boto:path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.500 DEBUG:boto:auth_path=/wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.501 DEBUG:boto:Method: GET 2023-10-12T00:40:03.501 DEBUG:boto:Path: /wafmoa-47/?prefix=obj8 2023-10-12T00:40:03.502 DEBUG:boto:Data: 2023-10-12T00:40:03.502 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.503 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.503 DEBUG:boto:Params: {} 2023-10-12T00:40:03.504 DEBUG:boto:Token: None 2023-10-12T00:40:03.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-47/ 2023-10-12T00:40:03.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI= 2023-10-12T00:40:03.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BYENDXIr9eegxGVXJIwyzqrEQAI=', 'Content-Length': '0'} 2023-10-12T00:40:03.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a151f72b934012e-0065274063-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.508 DEBUG:boto:b'wafmoa-47obj81000falseobj82023-10-12T00:39:19.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:03.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-47 zones={a2, a1} 2023-10-12T00:40:03.509 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T00:40:03.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.511 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:03.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:03.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:03.607 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:40:03.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-12T00:40:03.608 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:35.837822Z" 2023-10-12T00:40:03.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-12T00:40:03.609 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:26.980072Z" 2023-10-12T00:40:03.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-10-12T00:40:03.610 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:19.261193Z" 2023-10-12T00:40:03.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-12T00:40:03.610 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:37.018420Z" 2023-10-12T00:40:03.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.611 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:40:03.612 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-10-12T00:40:03.612 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:40:03.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.614 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:03.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:03.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:03.709 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:40:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-12T00:40:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:35.837822Z" 2023-10-12T00:40:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-12T00:40:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:26.980072Z" 2023-10-12T00:40:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-10-12T00:40:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:19.261193Z" 2023-10-12T00:40:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-12T00:40:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:37.018420Z" 2023-10-12T00:40:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:40:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:40:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:03Z" 2023-10-12T00:40:03.722 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:03.723 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:35.837822Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006987", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:26.980072Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005159", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:19.261193Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:37.018420Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:40:03Z" } 2023-10-12T00:40:03.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:35.837822Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006987', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:26.980072Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:19.261193Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:37.018420Z'}}] 2023-10-12T00:40:03.724 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-10-12T00:40:03.724 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002903', 1: '00000000000000000000:00000000000000006987', 2: '00000000000000000000:00000000000000005159', 3: '00000000000000000000:00000000000000004316'} 2023-10-12T00:40:03.725 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:40:03.725 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T00:40:03.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.726 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:03.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:03.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:03.822 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:40:03.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.823 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-10-12T00:40:03.823 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:40.448046Z" 2023-10-12T00:40:03.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006000", 2023-10-12T00:40:03.824 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:40.449578Z" 2023-10-12T00:40:03.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-12T00:40:03.825 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:38:46.923941Z" 2023-10-12T00:40:03.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.826 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-12T00:40:03.826 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:39:40.450947Z" 2023-10-12T00:40:03.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.826 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:40:03.827 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002482', 3: '00000000000000000000:00000000000000002620'} 2023-10-12T00:40:03.828 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:40:03.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:03.829 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:03.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:03.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:03.922 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:03.923 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:40:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:03.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-10-12T00:40:03.925 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:40.448046Z" 2023-10-12T00:40:03.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.927 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006000", 2023-10-12T00:40:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.928 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:40.449578Z" 2023-10-12T00:40:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.929 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-12T00:40:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.930 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:38:46.923941Z" 2023-10-12T00:40:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:40:03.932 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-12T00:40:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:39:40.450947Z" 2023-10-12T00:40:03.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:03.934 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:40:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:40:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:03.935 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:03Z" 2023-10-12T00:40:03.935 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:03.936 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:40.448046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:40.449578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:38:46.923941Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:39:40.450947Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:40:03Z" } 2023-10-12T00:40:03.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:40.448046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:40.449578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:38:46.923941Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:39:40.450947Z'}}] 2023-10-12T00:40:03.937 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002482', 3: '00000000000000000000:00000000000000002620'} 2023-10-12T00:40:03.938 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000002482', 3: '00000000000000000000:00000000000000002620'} 2023-10-12T00:40:03.938 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:40:03.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-10-12T00:40:03.940 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-48 2023-10-12T00:40:03.941 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:03.941 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:03.942 DEBUG:boto:Method: PUT 2023-10-12T00:40:03.942 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:03.942 DEBUG:boto:Data: 2023-10-12T00:40:03.943 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.944 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.944 DEBUG:boto:Params: {} 2023-10-12T00:40:03.944 DEBUG:boto:Token: None 2023-10-12T00:40:03.945 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-48/ 2023-10-12T00:40:03.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WC3CeGkM9Xfd8aA7KCwJcOQm7l0= 2023-10-12T00:40:03.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WC3CeGkM9Xfd8aA7KCwJcOQm7l0=', 'Content-Length': '0'} 2023-10-12T00:40:03.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa308fa38b8ceb85-0065274063-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.961 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-49 2023-10-12T00:40:03.961 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:03.961 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:03.962 DEBUG:boto:Method: PUT 2023-10-12T00:40:03.962 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:03.963 DEBUG:boto:Data: 2023-10-12T00:40:03.963 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:03.964 DEBUG:boto:Port: 8001 2023-10-12T00:40:03.964 DEBUG:boto:Params: {} 2023-10-12T00:40:03.965 DEBUG:boto:Token: None 2023-10-12T00:40:03.965 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-49/ 2023-10-12T00:40:03.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V2OfMZ3I0mQX7vYnP3JW8otiByI= 2023-10-12T00:40:03.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V2OfMZ3I0mQX7vYnP3JW8otiByI=', 'Content-Length': '0'} 2023-10-12T00:40:03.989 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2e31a0a6bfbab37-0065274063-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.989 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:03.990 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:03.990 DEBUG:boto:Method: HEAD 2023-10-12T00:40:03.991 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:03.991 DEBUG:boto:Data: 2023-10-12T00:40:03.992 DEBUG:boto:Headers: {} 2023-10-12T00:40:03.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:03.992 DEBUG:boto:Port: 8000 2023-10-12T00:40:03.993 DEBUG:boto:Params: {} 2023-10-12T00:40:03.993 DEBUG:boto:Token: None 2023-10-12T00:40:03.994 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:03 GMT /wafmoa-48/ 2023-10-12T00:40:03.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vM/tC5hL3MX5vQ1z3iJwiPZutHQ= 2023-10-12T00:40:03.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vM/tC5hL3MX5vQ1z3iJwiPZutHQ=', 'Content-Length': '0'} 2023-10-12T00:40:03.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092fe099dfc5d03fd-0065274063-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:03.998 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:04.001 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:04.002 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.002 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:40:04.003 DEBUG:boto:Data: 2023-10-12T00:40:04.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.004 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.004 DEBUG:boto:Params: {} 2023-10-12T00:40:04.005 DEBUG:boto:Token: None 2023-10-12T00:40:04.005 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/obj1 2023-10-12T00:40:04.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VspOQXEeA4guaZ5FUbLl5tAb1jY= 2023-10-12T00:40:04.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VspOQXEeA4guaZ5FUbLl5tAb1jY='} 2023-10-12T00:40:04.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000061fd4338ec9f951-0065274064-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.012 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:04.013 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:04.013 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.013 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:04.014 DEBUG:boto:Data: 2023-10-12T00:40:04.014 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.015 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.015 DEBUG:boto:Params: {} 2023-10-12T00:40:04.016 DEBUG:boto:Token: None 2023-10-12T00:40:04.016 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/ 2023-10-12T00:40:04.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU= 2023-10-12T00:40:04.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU=', 'Content-Length': '0'} 2023-10-12T00:40:04.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a8c3047493b9c95-0065274064-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.020 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:04.020 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:04.021 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.021 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:40:04.022 DEBUG:boto:Data: 2023-10-12T00:40:04.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.023 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.024 DEBUG:boto:Params: {} 2023-10-12T00:40:04.024 DEBUG:boto:Token: None 2023-10-12T00:40:04.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/obj2 2023-10-12T00:40:04.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QlImJI7ZmoUpOUaWOZqt43aIaDk= 2023-10-12T00:40:04.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QlImJI7ZmoUpOUaWOZqt43aIaDk='} 2023-10-12T00:40:04.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9ec9012ac1dd855-0065274064-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.071 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:04.072 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:04.072 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.072 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:04.073 DEBUG:boto:Data: 2023-10-12T00:40:04.073 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.074 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.075 DEBUG:boto:Params: {} 2023-10-12T00:40:04.075 DEBUG:boto:Token: None 2023-10-12T00:40:04.076 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/ 2023-10-12T00:40:04.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU= 2023-10-12T00:40:04.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU=', 'Content-Length': '0'} 2023-10-12T00:40:04.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6917f6c1a45d143-0065274064-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.080 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:04.080 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:04.081 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.081 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:40:04.082 DEBUG:boto:Data: 2023-10-12T00:40:04.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.083 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.083 DEBUG:boto:Params: {} 2023-10-12T00:40:04.084 DEBUG:boto:Token: None 2023-10-12T00:40:04.084 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/obj3 2023-10-12T00:40:04.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T9JE12LnGAZTctZcksbLWdSgw1o= 2023-10-12T00:40:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T9JE12LnGAZTctZcksbLWdSgw1o='} 2023-10-12T00:40:04.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fb9733df19668e3-0065274064-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.091 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:04.091 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:04.091 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.092 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:04.093 DEBUG:boto:Data: 2023-10-12T00:40:04.093 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.094 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.094 DEBUG:boto:Params: {} 2023-10-12T00:40:04.095 DEBUG:boto:Token: None 2023-10-12T00:40:04.095 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/ 2023-10-12T00:40:04.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU= 2023-10-12T00:40:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cMBXxOwFh8UCZ+PK/9X/M/26yaU=', 'Content-Length': '0'} 2023-10-12T00:40:04.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040b355f2f8d3388e-0065274064-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.099 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:04.099 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:04.100 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.100 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:40:04.100 DEBUG:boto:Data: 2023-10-12T00:40:04.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:04.102 DEBUG:boto:Port: 8000 2023-10-12T00:40:04.102 DEBUG:boto:Params: {} 2023-10-12T00:40:04.103 DEBUG:boto:Token: None 2023-10-12T00:40:04.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-48/obj4 2023-10-12T00:40:04.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ntiQGnYlscSgA7UwCC04vj8LoD0= 2023-10-12T00:40:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ntiQGnYlscSgA7UwCC04vj8LoD0='} 2023-10-12T00:40:04.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cc9e9ee651472af-0065274064-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.147 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:04.148 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:04.148 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.148 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:04.149 DEBUG:boto:Data: 2023-10-12T00:40:04.149 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.150 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.150 DEBUG:boto:Params: {} 2023-10-12T00:40:04.151 DEBUG:boto:Token: None 2023-10-12T00:40:04.152 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/ 2023-10-12T00:40:04.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0= 2023-10-12T00:40:04.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0=', 'Content-Length': '0'} 2023-10-12T00:40:04.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008415fc516e7ab570-0065274064-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.156 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:04.157 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:04.157 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.158 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:40:04.158 DEBUG:boto:Data: 2023-10-12T00:40:04.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.159 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.160 DEBUG:boto:Params: {} 2023-10-12T00:40:04.160 DEBUG:boto:Token: None 2023-10-12T00:40:04.161 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/obj1 2023-10-12T00:40:04.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O58Hppe8tIwhC/b3YM9I3xXRpDs= 2023-10-12T00:40:04.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O58Hppe8tIwhC/b3YM9I3xXRpDs='} 2023-10-12T00:40:04.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0b4c86076cdbdc7-0065274064-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.168 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:04.168 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:04.169 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.169 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:04.170 DEBUG:boto:Data: 2023-10-12T00:40:04.170 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.171 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.171 DEBUG:boto:Params: {} 2023-10-12T00:40:04.172 DEBUG:boto:Token: None 2023-10-12T00:40:04.172 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/ 2023-10-12T00:40:04.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0= 2023-10-12T00:40:04.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0=', 'Content-Length': '0'} 2023-10-12T00:40:04.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003ccdd9f6ea0b281-0065274064-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.177 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:04.177 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:04.178 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.178 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:40:04.178 DEBUG:boto:Data: 2023-10-12T00:40:04.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.180 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.180 DEBUG:boto:Params: {} 2023-10-12T00:40:04.181 DEBUG:boto:Token: None 2023-10-12T00:40:04.181 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/obj2 2023-10-12T00:40:04.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NU3xAvHB4xkZ7qypZi40GhHSRhE= 2023-10-12T00:40:04.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NU3xAvHB4xkZ7qypZi40GhHSRhE='} 2023-10-12T00:40:04.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c170a1e70d6b58ae-0065274064-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.188 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:04.188 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:04.189 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.189 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:04.190 DEBUG:boto:Data: 2023-10-12T00:40:04.191 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.191 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.192 DEBUG:boto:Params: {} 2023-10-12T00:40:04.192 DEBUG:boto:Token: None 2023-10-12T00:40:04.193 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/ 2023-10-12T00:40:04.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0= 2023-10-12T00:40:04.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0=', 'Content-Length': '0'} 2023-10-12T00:40:04.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a480a95863dbb15c-0065274064-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.201 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:04.201 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:04.202 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.202 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:40:04.203 DEBUG:boto:Data: 2023-10-12T00:40:04.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.207 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.208 DEBUG:boto:Params: {} 2023-10-12T00:40:04.208 DEBUG:boto:Token: None 2023-10-12T00:40:04.209 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/obj3 2023-10-12T00:40:04.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0QwC2o71yNVviuas45O0dxAJIRE= 2023-10-12T00:40:04.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0QwC2o71yNVviuas45O0dxAJIRE='} 2023-10-12T00:40:04.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd3dd66e989e3c83-0065274064-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.259 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:04.260 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:04.260 DEBUG:boto:Method: HEAD 2023-10-12T00:40:04.260 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:04.261 DEBUG:boto:Data: 2023-10-12T00:40:04.261 DEBUG:boto:Headers: {} 2023-10-12T00:40:04.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.262 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.262 DEBUG:boto:Params: {} 2023-10-12T00:40:04.263 DEBUG:boto:Token: None 2023-10-12T00:40:04.263 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/ 2023-10-12T00:40:04.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0= 2023-10-12T00:40:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mhQ5b5gameFcNIKBVF89VVPAas0=', 'Content-Length': '0'} 2023-10-12T00:40:04.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065855d5c1754c0f8-0065274064-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.267 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:04.267 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:04.268 DEBUG:boto:Method: PUT 2023-10-12T00:40:04.268 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:40:04.269 DEBUG:boto:Data: 2023-10-12T00:40:04.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:40:04.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:04.270 DEBUG:boto:Port: 8001 2023-10-12T00:40:04.270 DEBUG:boto:Params: {} 2023-10-12T00:40:04.271 DEBUG:boto:Token: None 2023-10-12T00:40:04.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:40:04 GMT /wafmoa-49/obj4 2023-10-12T00:40:04.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:81BAUpABFe5trAL//awxPQNxmjQ= 2023-10-12T00:40:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:40:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:81BAUpABFe5trAL//awxPQNxmjQ='} 2023-10-12T00:40:04.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000511d4382ec02e6aa-0065274064-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:04.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:04.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:04.279 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:04.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:04.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:04.357 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:40:04.369 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:40:04.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.370 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:04.370 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:40:04.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:04.371 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:40:04.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.372 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071203.980933_494.1", 2023-10-12T00:40:04.372 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:40:03.980933Z" 2023-10-12T00:40:04.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.373 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:04.373 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:40:04.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.373 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:40:04.374 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:04.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:40:04.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:04.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:04.376 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:04.426 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:04.427 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:04.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:04.455+0000 7fa6ec31d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:04.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:04.456+0000 7fa6ec31d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:04.467 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:04.468 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:04.468 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:04.468 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:40:04.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:04.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.469 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:04.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.469 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:04.469 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:04.470 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:04.470 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:04.470 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:04.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:40:04.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:40:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:04.472 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:40:04.473 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.474 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071203.954221_487.1", 2023-10-12T00:40:04.475 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:40:03.954221Z", 2023-10-12T00:40:04.476 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:04.477 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:04.478 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:40:04.479 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:40:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:40:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:04Z" 2023-10-12T00:40:04.482 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:04.483 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071203.954221_487.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:40:03.954221Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:40:04Z" } 2023-10-12T00:40:04.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071203.954221_487.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:40:03.954221Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:40:04.484 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:04.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.954221_487.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:04.485 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071203.980933_494.1 target=1_1697071203.954221_487.1 2023-10-12T00:40:09.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:09.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:09.487 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:09.543 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:09.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:09.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:09.575+0000 7f9067586780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:09.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:09.576+0000 7f9067586780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:09.589 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:09.589 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:09.589 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:09.589 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:09.590 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:09.590 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:40:09.590 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:09.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.590 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:09.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:09.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:09.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:09.591 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:09.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:09.592 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:09.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:40:09.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:09.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:40:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:40:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071203.980933_494.1", 2023-10-12T00:40:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:40:03.980933Z", 2023-10-12T00:40:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:40:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:40:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:40:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:09Z" 2023-10-12T00:40:09.603 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:09.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071203.980933_494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:40:03.980933Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:40:09Z" } 2023-10-12T00:40:09.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071203.980933_494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:40:03.980933Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:40:09.605 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:09.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:09.606 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:40:09.607 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-48 2023-10-12T00:40:09.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:09.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:09.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:09.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:09.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:09.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:09.699+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:14.699+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:19.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:19.700+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:24.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:24.700+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:29.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:29.701+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:34.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:34.702+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:39.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:39.704+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:44.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:44.704+0000 7fe2245d8780 1 waiting to reach incremental sync.. 2023-10-12T00:40:49.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:49.706+0000 7fe2245d8780 1 bucket sync caught up with source: 2023-10-12T00:40:49.707 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.27.6, , 00000000001.32.6, , 00000000001.22.6, , , 00000000001.33.6, , , ] 2023-10-12T00:40:49.707 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.27.6, , 00000000001.32.6, , 00000000001.22.6, , , 00000000001.33.6, , , ] 2023-10-12T00:40:49.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:49.706+0000 7fe2245d8780 0 bucket checkpoint complete 2023-10-12T00:40:49.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-48 2023-10-12T00:40:49.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:49.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:49.725 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:49.786 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:49.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:49.823+0000 7f16c5fc0780 1 bucket sync caught up with source: 2023-10-12T00:40:49.823 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.148.6, , 00000000001.96.6, , 00000000001.228.6, , , 00000000001.138.6, , , ] 2023-10-12T00:40:49.824 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.148.6, , 00000000001.96.6, , 00000000001.228.6, , , 00000000001.138.6, , , ] 2023-10-12T00:40:49.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:49.823+0000 7f16c5fc0780 0 bucket checkpoint complete 2023-10-12T00:40:49.836 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-48 zones={a2, a1} 2023-10-12T00:40:49.837 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.837 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.838 DEBUG:boto:Method: HEAD 2023-10-12T00:40:49.839 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:49.839 DEBUG:boto:Data: 2023-10-12T00:40:49.840 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:49.841 DEBUG:boto:Port: 8001 2023-10-12T00:40:49.842 DEBUG:boto:Params: {} 2023-10-12T00:40:49.842 DEBUG:boto:Token: None 2023-10-12T00:40:49.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/ 2023-10-12T00:40:49.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TNyj+0p9vErUSjgnVOSqn8oNY5c= 2023-10-12T00:40:49.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TNyj+0p9vErUSjgnVOSqn8oNY5c=', 'Content-Length': '0'} 2023-10-12T00:40:49.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1a36076f3998acc-0065274091-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.850 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.851 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.851 DEBUG:boto:Method: HEAD 2023-10-12T00:40:49.852 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:40:49.853 DEBUG:boto:Data: 2023-10-12T00:40:49.853 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:49.855 DEBUG:boto:Port: 8000 2023-10-12T00:40:49.855 DEBUG:boto:Params: {} 2023-10-12T00:40:49.856 DEBUG:boto:Token: None 2023-10-12T00:40:49.857 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/ 2023-10-12T00:40:49.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TNyj+0p9vErUSjgnVOSqn8oNY5c= 2023-10-12T00:40:49.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TNyj+0p9vErUSjgnVOSqn8oNY5c=', 'Content-Length': '0'} 2023-10-12T00:40:49.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8b0aeec0c98c2f9-0065274091-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.861 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:40:49.862 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.863 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.863 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.864 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.865 DEBUG:boto:Method: GET 2023-10-12T00:40:49.865 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.866 DEBUG:boto:Data: 2023-10-12T00:40:49.867 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:49.868 DEBUG:boto:Port: 8001 2023-10-12T00:40:49.869 DEBUG:boto:Params: {} 2023-10-12T00:40:49.869 DEBUG:boto:Token: None 2023-10-12T00:40:49.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/?versions 2023-10-12T00:40:49.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ= 2023-10-12T00:40:49.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ=', 'Content-Length': '0'} 2023-10-12T00:40:49.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066e5ddafdee49504-0065274091-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.874 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:49.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:49.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:49.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:49.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:49.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:40:49.879 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.879 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.880 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.881 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.881 DEBUG:boto:Method: GET 2023-10-12T00:40:49.882 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.883 DEBUG:boto:Data: 2023-10-12T00:40:49.883 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:49.884 DEBUG:boto:Port: 8000 2023-10-12T00:40:49.885 DEBUG:boto:Params: {} 2023-10-12T00:40:49.886 DEBUG:boto:Token: None 2023-10-12T00:40:49.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/?versions 2023-10-12T00:40:49.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ= 2023-10-12T00:40:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ=', 'Content-Length': '0'} 2023-10-12T00:40:49.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d83302d61935c21c-0065274091-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.891 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:49.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:49.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:49.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:49.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:49.894 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.894 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.895 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.895 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.896 DEBUG:boto:Method: GET 2023-10-12T00:40:49.896 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.897 DEBUG:boto:Data: 2023-10-12T00:40:49.897 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:49.898 DEBUG:boto:Port: 8001 2023-10-12T00:40:49.898 DEBUG:boto:Params: {} 2023-10-12T00:40:49.899 DEBUG:boto:Token: None 2023-10-12T00:40:49.899 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/?versions 2023-10-12T00:40:49.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ= 2023-10-12T00:40:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ=', 'Content-Length': '0'} 2023-10-12T00:40:49.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0d92ae21fbabd81-0065274091-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.904 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:49.905 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:49.906 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:49.906 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.907 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.908 DEBUG:boto:Method: GET 2023-10-12T00:40:49.908 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:40:49.909 DEBUG:boto:Data: 2023-10-12T00:40:49.909 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:49.910 DEBUG:boto:Port: 8000 2023-10-12T00:40:49.911 DEBUG:boto:Params: {} 2023-10-12T00:40:49.912 DEBUG:boto:Token: None 2023-10-12T00:40:49.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/?versions 2023-10-12T00:40:49.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ= 2023-10-12T00:40:49.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZlcBWJvetgTQ5k2r04VB58SeLHQ=', 'Content-Length': '0'} 2023-10-12T00:40:49.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0fdb6f03e2ae53-0065274091-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.917 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:49.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:49.919 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:49.919 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:49.920 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.920 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.921 DEBUG:boto:Method: GET 2023-10-12T00:40:49.921 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.922 DEBUG:boto:Data: 2023-10-12T00:40:49.922 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:49.924 DEBUG:boto:Port: 8001 2023-10-12T00:40:49.924 DEBUG:boto:Params: {} 2023-10-12T00:40:49.925 DEBUG:boto:Token: None 2023-10-12T00:40:49.925 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s3IM3sSmpiw/1rkXx/z6NKpPNgg= 2023-10-12T00:40:49.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s3IM3sSmpiw/1rkXx/z6NKpPNgg=', 'Content-Length': '0'} 2023-10-12T00:40:49.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce8997f61c8c3373-0065274091-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.970 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:49.971 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:49.972 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.973 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.973 DEBUG:boto:Method: GET 2023-10-12T00:40:49.974 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.974 DEBUG:boto:Data: 2023-10-12T00:40:49.975 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:49.976 DEBUG:boto:Port: 8000 2023-10-12T00:40:49.976 DEBUG:boto:Params: {} 2023-10-12T00:40:49.977 DEBUG:boto:Token: None 2023-10-12T00:40:49.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s3IM3sSmpiw/1rkXx/z6NKpPNgg= 2023-10-12T00:40:49.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s3IM3sSmpiw/1rkXx/z6NKpPNgg=', 'Content-Length': '0'} 2023-10-12T00:40:49.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007954d5537bb12955-0065274091-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.982 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:49.983 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:49.983 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.984 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.984 DEBUG:boto:Method: HEAD 2023-10-12T00:40:49.985 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.985 DEBUG:boto:Data: 2023-10-12T00:40:49.986 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:49.987 DEBUG:boto:Port: 8001 2023-10-12T00:40:49.988 DEBUG:boto:Params: {} 2023-10-12T00:40:49.988 DEBUG:boto:Token: None 2023-10-12T00:40:49.990 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.990 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cglFrEyLL3kj67xgPmHK3npatkc= 2023-10-12T00:40:49.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cglFrEyLL3kj67xgPmHK3npatkc=', 'Content-Length': '0'} 2023-10-12T00:40:49.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4a7e5d503282b3f-0065274091-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:49.993 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:49.993 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:49.994 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.994 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.995 DEBUG:boto:Method: HEAD 2023-10-12T00:40:49.995 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.995 DEBUG:boto:Data: 2023-10-12T00:40:49.996 DEBUG:boto:Headers: {} 2023-10-12T00:40:49.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:49.997 DEBUG:boto:Port: 8000 2023-10-12T00:40:49.997 DEBUG:boto:Params: {} 2023-10-12T00:40:49.998 DEBUG:boto:Token: None 2023-10-12T00:40:49.998 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:49 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:49.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cglFrEyLL3kj67xgPmHK3npatkc= 2023-10-12T00:40:49.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cglFrEyLL3kj67xgPmHK3npatkc=', 'Content-Length': '0'} 2023-10-12T00:40:50.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000299810b2d680d90e-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:50.002 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.002 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.003 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.003 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.004 DEBUG:boto:Method: GET 2023-10-12T00:40:50.004 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.004 DEBUG:boto:Data: 2023-10-12T00:40:50.005 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.006 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.006 DEBUG:boto:Params: {} 2023-10-12T00:40:50.007 DEBUG:boto:Token: None 2023-10-12T00:40:50.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ifa/ROpGU8jE9MB4VtHrRxpM7S4= 2023-10-12T00:40:50.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ifa/ROpGU8jE9MB4VtHrRxpM7S4=', 'Content-Length': '0'} 2023-10-12T00:40:50.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a06b9e7f713d168-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.010 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.011 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.013 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.013 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.014 DEBUG:boto:Method: GET 2023-10-12T00:40:50.014 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.015 DEBUG:boto:Data: 2023-10-12T00:40:50.015 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.016 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.017 DEBUG:boto:Params: {} 2023-10-12T00:40:50.017 DEBUG:boto:Token: None 2023-10-12T00:40:50.018 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:40:50.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ifa/ROpGU8jE9MB4VtHrRxpM7S4= 2023-10-12T00:40:50.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ifa/ROpGU8jE9MB4VtHrRxpM7S4=', 'Content-Length': '0'} 2023-10-12T00:40:50.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b8cb76838883e16-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.021 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.022 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.022 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.023 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.023 DEBUG:boto:Method: GET 2023-10-12T00:40:50.023 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.024 DEBUG:boto:Data: 2023-10-12T00:40:50.024 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.025 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.026 DEBUG:boto:Params: {} 2023-10-12T00:40:50.026 DEBUG:boto:Token: None 2023-10-12T00:40:50.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c3b70fe742e13b-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.030 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.031 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.032 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.032 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.032 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.033 DEBUG:boto:Method: GET 2023-10-12T00:40:50.033 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.034 DEBUG:boto:Data: 2023-10-12T00:40:50.034 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.035 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.036 DEBUG:boto:Params: {} 2023-10-12T00:40:50.036 DEBUG:boto:Token: None 2023-10-12T00:40:50.037 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3711e7c215419a-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.041 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.041 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.042 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.042 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.043 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:40:50.043 DEBUG:boto:Data: 2023-10-12T00:40:50.043 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.044 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.045 DEBUG:boto:Params: {} 2023-10-12T00:40:50.045 DEBUG:boto:Token: None 2023-10-12T00:40:50.046 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1 2023-10-12T00:40:50.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZRVbq1ywbsHVSvXkUNLaHp7ykr8= 2023-10-12T00:40:50.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZRVbq1ywbsHVSvXkUNLaHp7ykr8=', 'Content-Length': '0'} 2023-10-12T00:40:50.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e2ea88e59c3057-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.049 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.049 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.050 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.050 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:40:50.051 DEBUG:boto:Data: 2023-10-12T00:40:50.051 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.052 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.052 DEBUG:boto:Params: {} 2023-10-12T00:40:50.053 DEBUG:boto:Token: None 2023-10-12T00:40:50.053 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1 2023-10-12T00:40:50.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZRVbq1ywbsHVSvXkUNLaHp7ykr8= 2023-10-12T00:40:50.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZRVbq1ywbsHVSvXkUNLaHp7ykr8=', 'Content-Length': '0'} 2023-10-12T00:40:50.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c34eade437bd48ba-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:50.057 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.057 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.058 DEBUG:boto:Method: GET 2023-10-12T00:40:50.058 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:40:50.058 DEBUG:boto:Data: 2023-10-12T00:40:50.059 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.060 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.060 DEBUG:boto:Params: {} 2023-10-12T00:40:50.061 DEBUG:boto:Token: None 2023-10-12T00:40:50.061 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1 2023-10-12T00:40:50.062 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ky9QSyJXLYTt7M/pI3q5XM9k4xg= 2023-10-12T00:40:50.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ky9QSyJXLYTt7M/pI3q5XM9k4xg=', 'Content-Length': '0'} 2023-10-12T00:40:50.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eae8ad12f7d68395-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.064 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:40:50.065 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:40:50.065 DEBUG:boto:Method: GET 2023-10-12T00:40:50.066 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:40:50.066 DEBUG:boto:Data: 2023-10-12T00:40:50.067 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.067 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.068 DEBUG:boto:Params: {} 2023-10-12T00:40:50.068 DEBUG:boto:Token: None 2023-10-12T00:40:50.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj1 2023-10-12T00:40:50.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ky9QSyJXLYTt7M/pI3q5XM9k4xg= 2023-10-12T00:40:50.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ky9QSyJXLYTt7M/pI3q5XM9k4xg=', 'Content-Length': '0'} 2023-10-12T00:40:50.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039843be3cadfba33-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.072 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.072 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.073 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.073 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.074 DEBUG:boto:Method: GET 2023-10-12T00:40:50.074 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.075 DEBUG:boto:Data: 2023-10-12T00:40:50.075 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.076 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.076 DEBUG:boto:Params: {} 2023-10-12T00:40:50.077 DEBUG:boto:Token: None 2023-10-12T00:40:50.077 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.078 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc37e9be04c8992c-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.081 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.082 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.082 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.083 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.083 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.084 DEBUG:boto:Method: GET 2023-10-12T00:40:50.084 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:40:50.084 DEBUG:boto:Data: 2023-10-12T00:40:50.085 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.086 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.086 DEBUG:boto:Params: {} 2023-10-12T00:40:50.087 DEBUG:boto:Token: None 2023-10-12T00:40:50.087 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d07128d02a0b5e-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.090 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:50.091 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.092 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.092 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.093 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.093 DEBUG:boto:Method: GET 2023-10-12T00:40:50.094 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.094 DEBUG:boto:Data: 2023-10-12T00:40:50.094 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.095 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.096 DEBUG:boto:Params: {} 2023-10-12T00:40:50.096 DEBUG:boto:Token: None 2023-10-12T00:40:50.097 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8= 2023-10-12T00:40:50.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8=', 'Content-Length': '0'} 2023-10-12T00:40:50.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8d7f9f3d3589543-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.101 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.101 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.102 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.102 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.103 DEBUG:boto:Method: GET 2023-10-12T00:40:50.104 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.104 DEBUG:boto:Data: 2023-10-12T00:40:50.105 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.106 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.107 DEBUG:boto:Params: {} 2023-10-12T00:40:50.108 DEBUG:boto:Token: None 2023-10-12T00:40:50.109 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8= 2023-10-12T00:40:50.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8=', 'Content-Length': '0'} 2023-10-12T00:40:50.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b50f24fd16765fb-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.114 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.114 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.115 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.115 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.115 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.116 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.116 DEBUG:boto:Data: 2023-10-12T00:40:50.117 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.118 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.118 DEBUG:boto:Params: {} 2023-10-12T00:40:50.119 DEBUG:boto:Token: None 2023-10-12T00:40:50.119 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QLsGBQGUgAUgWXHXahT6fHUECOQ= 2023-10-12T00:40:50.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QLsGBQGUgAUgWXHXahT6fHUECOQ=', 'Content-Length': '0'} 2023-10-12T00:40:50.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f37410b0c4273f7-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.123 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.124 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.124 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.124 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.125 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.125 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.126 DEBUG:boto:Data: 2023-10-12T00:40:50.126 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.127 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.127 DEBUG:boto:Params: {} 2023-10-12T00:40:50.128 DEBUG:boto:Token: None 2023-10-12T00:40:50.129 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QLsGBQGUgAUgWXHXahT6fHUECOQ= 2023-10-12T00:40:50.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QLsGBQGUgAUgWXHXahT6fHUECOQ=', 'Content-Length': '0'} 2023-10-12T00:40:50.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053c1ddffeaa58882-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:50.133 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.133 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.134 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.134 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.135 DEBUG:boto:Method: GET 2023-10-12T00:40:50.135 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.136 DEBUG:boto:Data: 2023-10-12T00:40:50.136 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.137 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.137 DEBUG:boto:Params: {} 2023-10-12T00:40:50.138 DEBUG:boto:Token: None 2023-10-12T00:40:50.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8= 2023-10-12T00:40:50.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8=', 'Content-Length': '0'} 2023-10-12T00:40:50.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e615275ed5467af4-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.143 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.143 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.144 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.144 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.145 DEBUG:boto:Method: GET 2023-10-12T00:40:50.145 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.146 DEBUG:boto:Data: 2023-10-12T00:40:50.146 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.147 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.147 DEBUG:boto:Params: {} 2023-10-12T00:40:50.148 DEBUG:boto:Token: None 2023-10-12T00:40:50.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:40:50.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8= 2023-10-12T00:40:50.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlgyCwxF5BVMiot26Ei2Z1z2dO8=', 'Content-Length': '0'} 2023-10-12T00:40:50.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081f14f250b432086-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.152 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.152 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.153 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.153 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.154 DEBUG:boto:Method: GET 2023-10-12T00:40:50.154 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.155 DEBUG:boto:Data: 2023-10-12T00:40:50.155 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.156 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.156 DEBUG:boto:Params: {} 2023-10-12T00:40:50.157 DEBUG:boto:Token: None 2023-10-12T00:40:50.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059cd950a28b2d73f-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.161 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.162 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.163 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.164 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.164 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.165 DEBUG:boto:Method: GET 2023-10-12T00:40:50.166 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.166 DEBUG:boto:Data: 2023-10-12T00:40:50.167 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.168 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.169 DEBUG:boto:Params: {} 2023-10-12T00:40:50.169 DEBUG:boto:Token: None 2023-10-12T00:40:50.170 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.171 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de60e58edd7dfced-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.174 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.175 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.175 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.176 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.176 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:40:50.177 DEBUG:boto:Data: 2023-10-12T00:40:50.177 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.178 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.178 DEBUG:boto:Params: {} 2023-10-12T00:40:50.179 DEBUG:boto:Token: None 2023-10-12T00:40:50.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2 2023-10-12T00:40:50.180 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KFYo/1qmBD5AOJIkdrg5kEenl34= 2023-10-12T00:40:50.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KFYo/1qmBD5AOJIkdrg5kEenl34=', 'Content-Length': '0'} 2023-10-12T00:40:50.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b44ec4a06fa5f6-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.183 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.183 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.184 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.184 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:40:50.185 DEBUG:boto:Data: 2023-10-12T00:40:50.185 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.186 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.186 DEBUG:boto:Params: {} 2023-10-12T00:40:50.187 DEBUG:boto:Token: None 2023-10-12T00:40:50.187 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2 2023-10-12T00:40:50.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KFYo/1qmBD5AOJIkdrg5kEenl34= 2023-10-12T00:40:50.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KFYo/1qmBD5AOJIkdrg5kEenl34=', 'Content-Length': '0'} 2023-10-12T00:40:50.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bf4d38add07b2b1-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:50.191 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.192 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.193 DEBUG:boto:Method: GET 2023-10-12T00:40:50.193 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:40:50.193 DEBUG:boto:Data: 2023-10-12T00:40:50.194 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.195 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.196 DEBUG:boto:Params: {} 2023-10-12T00:40:50.196 DEBUG:boto:Token: None 2023-10-12T00:40:50.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2 2023-10-12T00:40:50.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGoJ6Wr3Tf8ui449LW0sF+nBT6c= 2023-10-12T00:40:50.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGoJ6Wr3Tf8ui449LW0sF+nBT6c=', 'Content-Length': '0'} 2023-10-12T00:40:50.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d2a62829eaae907-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.200 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:40:50.201 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:40:50.201 DEBUG:boto:Method: GET 2023-10-12T00:40:50.202 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:40:50.202 DEBUG:boto:Data: 2023-10-12T00:40:50.203 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.203 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.204 DEBUG:boto:Params: {} 2023-10-12T00:40:50.204 DEBUG:boto:Token: None 2023-10-12T00:40:50.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj2 2023-10-12T00:40:50.205 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uGoJ6Wr3Tf8ui449LW0sF+nBT6c= 2023-10-12T00:40:50.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uGoJ6Wr3Tf8ui449LW0sF+nBT6c=', 'Content-Length': '0'} 2023-10-12T00:40:50.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdbf4d885dfca67d-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.208 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.209 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.209 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.209 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.210 DEBUG:boto:Method: GET 2023-10-12T00:40:50.211 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.211 DEBUG:boto:Data: 2023-10-12T00:40:50.212 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.212 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.213 DEBUG:boto:Params: {} 2023-10-12T00:40:50.213 DEBUG:boto:Token: None 2023-10-12T00:40:50.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf7718aaca0f7403-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.218 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.218 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.219 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.219 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.220 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.220 DEBUG:boto:Method: GET 2023-10-12T00:40:50.221 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:40:50.221 DEBUG:boto:Data: 2023-10-12T00:40:50.221 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.222 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.223 DEBUG:boto:Params: {} 2023-10-12T00:40:50.224 DEBUG:boto:Token: None 2023-10-12T00:40:50.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0a28f8c1ea6529-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.228 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:50.229 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.230 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.230 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.231 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.231 DEBUG:boto:Method: GET 2023-10-12T00:40:50.232 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.232 DEBUG:boto:Data: 2023-10-12T00:40:50.233 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.233 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.234 DEBUG:boto:Params: {} 2023-10-12T00:40:50.234 DEBUG:boto:Token: None 2023-10-12T00:40:50.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0= 2023-10-12T00:40:50.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0=', 'Content-Length': '0'} 2023-10-12T00:40:50.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8e8dc359c02d96a-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.238 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.239 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.239 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.240 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.240 DEBUG:boto:Method: GET 2023-10-12T00:40:50.241 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.241 DEBUG:boto:Data: 2023-10-12T00:40:50.242 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.243 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.243 DEBUG:boto:Params: {} 2023-10-12T00:40:50.243 DEBUG:boto:Token: None 2023-10-12T00:40:50.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0= 2023-10-12T00:40:50.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0=', 'Content-Length': '0'} 2023-10-12T00:40:50.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a541f3a687b618-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.247 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.248 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.248 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.249 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.249 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.249 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.250 DEBUG:boto:Data: 2023-10-12T00:40:50.250 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.251 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.251 DEBUG:boto:Params: {} 2023-10-12T00:40:50.252 DEBUG:boto:Token: None 2023-10-12T00:40:50.252 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFxx8iez/d5kFtA1YFn7tOc90/Q= 2023-10-12T00:40:50.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFxx8iez/d5kFtA1YFn7tOc90/Q=', 'Content-Length': '0'} 2023-10-12T00:40:50.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006da2057b99a580e5-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.255 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.256 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.256 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.257 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.257 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.257 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.258 DEBUG:boto:Data: 2023-10-12T00:40:50.258 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.259 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.259 DEBUG:boto:Params: {} 2023-10-12T00:40:50.260 DEBUG:boto:Token: None 2023-10-12T00:40:50.260 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFxx8iez/d5kFtA1YFn7tOc90/Q= 2023-10-12T00:40:50.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFxx8iez/d5kFtA1YFn7tOc90/Q=', 'Content-Length': '0'} 2023-10-12T00:40:50.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084016747e043c41c-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:50.267 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.268 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.268 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.269 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.269 DEBUG:boto:Method: GET 2023-10-12T00:40:50.270 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.270 DEBUG:boto:Data: 2023-10-12T00:40:50.271 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.271 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.272 DEBUG:boto:Params: {} 2023-10-12T00:40:50.272 DEBUG:boto:Token: None 2023-10-12T00:40:50.273 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0= 2023-10-12T00:40:50.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0=', 'Content-Length': '0'} 2023-10-12T00:40:50.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000467ac683b313fb66-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.318 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.319 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.319 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.320 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.320 DEBUG:boto:Method: GET 2023-10-12T00:40:50.321 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.321 DEBUG:boto:Data: 2023-10-12T00:40:50.322 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.322 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.323 DEBUG:boto:Params: {} 2023-10-12T00:40:50.323 DEBUG:boto:Token: None 2023-10-12T00:40:50.324 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:40:50.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0= 2023-10-12T00:40:50.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sFNL3VBGtb+c8gn6auv8bbvL0V0=', 'Content-Length': '0'} 2023-10-12T00:40:50.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fb4c4750acb07c4-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.328 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.328 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.329 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.329 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.330 DEBUG:boto:Method: GET 2023-10-12T00:40:50.330 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.331 DEBUG:boto:Data: 2023-10-12T00:40:50.331 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.332 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.332 DEBUG:boto:Params: {} 2023-10-12T00:40:50.333 DEBUG:boto:Token: None 2023-10-12T00:40:50.333 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4848c708231f6f7-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.337 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.338 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.339 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.339 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.339 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.340 DEBUG:boto:Method: GET 2023-10-12T00:40:50.340 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.341 DEBUG:boto:Data: 2023-10-12T00:40:50.341 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.342 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.343 DEBUG:boto:Params: {} 2023-10-12T00:40:50.343 DEBUG:boto:Token: None 2023-10-12T00:40:50.344 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16f486d9c3eb186-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.390 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.391 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.392 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.392 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.393 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:40:50.393 DEBUG:boto:Data: 2023-10-12T00:40:50.393 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.394 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.395 DEBUG:boto:Params: {} 2023-10-12T00:40:50.395 DEBUG:boto:Token: None 2023-10-12T00:40:50.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3 2023-10-12T00:40:50.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vFvvGmoym63BDgzGs8bwRjksO18= 2023-10-12T00:40:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vFvvGmoym63BDgzGs8bwRjksO18=', 'Content-Length': '0'} 2023-10-12T00:40:50.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092f3371def84aaf2-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.399 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.400 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.400 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.401 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:40:50.401 DEBUG:boto:Data: 2023-10-12T00:40:50.401 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.402 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.403 DEBUG:boto:Params: {} 2023-10-12T00:40:50.403 DEBUG:boto:Token: None 2023-10-12T00:40:50.404 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3 2023-10-12T00:40:50.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vFvvGmoym63BDgzGs8bwRjksO18= 2023-10-12T00:40:50.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vFvvGmoym63BDgzGs8bwRjksO18=', 'Content-Length': '0'} 2023-10-12T00:40:50.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052deae3167830a68-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:50.407 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.408 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.408 DEBUG:boto:Method: GET 2023-10-12T00:40:50.409 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:40:50.409 DEBUG:boto:Data: 2023-10-12T00:40:50.410 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.411 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.411 DEBUG:boto:Params: {} 2023-10-12T00:40:50.411 DEBUG:boto:Token: None 2023-10-12T00:40:50.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3 2023-10-12T00:40:50.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PTD+Mb/6KK0UANlQ8ekjseLNqyo= 2023-10-12T00:40:50.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PTD+Mb/6KK0UANlQ8ekjseLNqyo=', 'Content-Length': '0'} 2023-10-12T00:40:50.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1771fc041b278f6-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.415 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:40:50.416 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:40:50.416 DEBUG:boto:Method: GET 2023-10-12T00:40:50.417 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:40:50.417 DEBUG:boto:Data: 2023-10-12T00:40:50.418 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.418 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.419 DEBUG:boto:Params: {} 2023-10-12T00:40:50.419 DEBUG:boto:Token: None 2023-10-12T00:40:50.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj3 2023-10-12T00:40:50.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PTD+Mb/6KK0UANlQ8ekjseLNqyo= 2023-10-12T00:40:50.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PTD+Mb/6KK0UANlQ8ekjseLNqyo=', 'Content-Length': '0'} 2023-10-12T00:40:50.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9e29b75b1122490-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.424 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.424 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.424 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.425 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.425 DEBUG:boto:Method: GET 2023-10-12T00:40:50.426 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.426 DEBUG:boto:Data: 2023-10-12T00:40:50.427 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.428 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.428 DEBUG:boto:Params: {} 2023-10-12T00:40:50.428 DEBUG:boto:Token: None 2023-10-12T00:40:50.429 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf925fefe772a23-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.478 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.479 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.480 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.480 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.481 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.481 DEBUG:boto:Method: GET 2023-10-12T00:40:50.482 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:40:50.482 DEBUG:boto:Data: 2023-10-12T00:40:50.483 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.484 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.484 DEBUG:boto:Params: {} 2023-10-12T00:40:50.485 DEBUG:boto:Token: None 2023-10-12T00:40:50.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000824a8f31d0d793e-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.489 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:50.490 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.490 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.491 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.491 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.492 DEBUG:boto:Method: GET 2023-10-12T00:40:50.492 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.493 DEBUG:boto:Data: 2023-10-12T00:40:50.493 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.494 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.494 DEBUG:boto:Params: {} 2023-10-12T00:40:50.495 DEBUG:boto:Token: None 2023-10-12T00:40:50.495 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk= 2023-10-12T00:40:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk=', 'Content-Length': '0'} 2023-10-12T00:40:50.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aacbfe11c213f4a-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.499 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.499 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.500 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.500 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.500 DEBUG:boto:Method: GET 2023-10-12T00:40:50.501 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.501 DEBUG:boto:Data: 2023-10-12T00:40:50.502 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.503 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.503 DEBUG:boto:Params: {} 2023-10-12T00:40:50.504 DEBUG:boto:Token: None 2023-10-12T00:40:50.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk= 2023-10-12T00:40:50.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk=', 'Content-Length': '0'} 2023-10-12T00:40:50.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f3adaf0650d1f2c-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.507 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.508 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.508 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.509 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.509 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.509 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.510 DEBUG:boto:Data: 2023-10-12T00:40:50.510 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.511 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.512 DEBUG:boto:Params: {} 2023-10-12T00:40:50.512 DEBUG:boto:Token: None 2023-10-12T00:40:50.512 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bsobs9KqCyyb5GLBgIPBXjBdLpI= 2023-10-12T00:40:50.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bsobs9KqCyyb5GLBgIPBXjBdLpI=', 'Content-Length': '0'} 2023-10-12T00:40:50.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038947c465684bd36-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.516 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.516 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.517 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.517 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.518 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.518 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.518 DEBUG:boto:Data: 2023-10-12T00:40:50.519 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.520 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.520 DEBUG:boto:Params: {} 2023-10-12T00:40:50.521 DEBUG:boto:Token: None 2023-10-12T00:40:50.521 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bsobs9KqCyyb5GLBgIPBXjBdLpI= 2023-10-12T00:40:50.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bsobs9KqCyyb5GLBgIPBXjBdLpI=', 'Content-Length': '0'} 2023-10-12T00:40:50.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c91d144e03e327e-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:50.525 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.525 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.526 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.526 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.527 DEBUG:boto:Method: GET 2023-10-12T00:40:50.527 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.527 DEBUG:boto:Data: 2023-10-12T00:40:50.528 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.529 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.529 DEBUG:boto:Params: {} 2023-10-12T00:40:50.530 DEBUG:boto:Token: None 2023-10-12T00:40:50.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk= 2023-10-12T00:40:50.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk=', 'Content-Length': '0'} 2023-10-12T00:40:50.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b69c4e74c5621a8-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.534 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.534 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.535 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.535 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.535 DEBUG:boto:Method: GET 2023-10-12T00:40:50.536 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.536 DEBUG:boto:Data: 2023-10-12T00:40:50.537 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.538 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.538 DEBUG:boto:Params: {} 2023-10-12T00:40:50.539 DEBUG:boto:Token: None 2023-10-12T00:40:50.539 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:40:50.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk= 2023-10-12T00:40:50.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krLdJWOplyFREJ+J4aQ71H8zdXk=', 'Content-Length': '0'} 2023-10-12T00:40:50.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d710abb232b16630-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.542 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.543 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.543 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.544 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.544 DEBUG:boto:Method: GET 2023-10-12T00:40:50.545 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.545 DEBUG:boto:Data: 2023-10-12T00:40:50.545 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.546 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.547 DEBUG:boto:Params: {} 2023-10-12T00:40:50.547 DEBUG:boto:Token: None 2023-10-12T00:40:50.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0255b5f6890b7ea-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.552 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.552 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.553 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.553 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.554 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.554 DEBUG:boto:Method: GET 2023-10-12T00:40:50.555 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.555 DEBUG:boto:Data: 2023-10-12T00:40:50.556 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.557 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.557 DEBUG:boto:Params: {} 2023-10-12T00:40:50.557 DEBUG:boto:Token: None 2023-10-12T00:40:50.558 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d9a377ef9169d0-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.562 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.562 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.563 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.563 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.564 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:40:50.564 DEBUG:boto:Data: 2023-10-12T00:40:50.564 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.565 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.566 DEBUG:boto:Params: {} 2023-10-12T00:40:50.566 DEBUG:boto:Token: None 2023-10-12T00:40:50.567 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4 2023-10-12T00:40:50.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9y+6dUf7Ds24utTJcQRE6U0gX3g= 2023-10-12T00:40:50.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9y+6dUf7Ds24utTJcQRE6U0gX3g=', 'Content-Length': '0'} 2023-10-12T00:40:50.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cbd1b3ab3997f98-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.570 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.571 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.571 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.571 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:40:50.572 DEBUG:boto:Data: 2023-10-12T00:40:50.572 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.573 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.573 DEBUG:boto:Params: {} 2023-10-12T00:40:50.574 DEBUG:boto:Token: None 2023-10-12T00:40:50.574 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4 2023-10-12T00:40:50.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9y+6dUf7Ds24utTJcQRE6U0gX3g= 2023-10-12T00:40:50.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9y+6dUf7Ds24utTJcQRE6U0gX3g=', 'Content-Length': '0'} 2023-10-12T00:40:50.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a883a6cf146cdee0-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:50.578 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.579 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.579 DEBUG:boto:Method: GET 2023-10-12T00:40:50.580 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:40:50.580 DEBUG:boto:Data: 2023-10-12T00:40:50.580 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.581 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.582 DEBUG:boto:Params: {} 2023-10-12T00:40:50.582 DEBUG:boto:Token: None 2023-10-12T00:40:50.583 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4 2023-10-12T00:40:50.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JlNnRUjC92shHnpX+zM9INs0kCw= 2023-10-12T00:40:50.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JlNnRUjC92shHnpX+zM9INs0kCw=', 'Content-Length': '0'} 2023-10-12T00:40:50.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde7e3ccae9ce24e-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.586 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:40:50.587 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:40:50.587 DEBUG:boto:Method: GET 2023-10-12T00:40:50.587 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:40:50.588 DEBUG:boto:Data: 2023-10-12T00:40:50.588 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.589 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.590 DEBUG:boto:Params: {} 2023-10-12T00:40:50.590 DEBUG:boto:Token: None 2023-10-12T00:40:50.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/obj4 2023-10-12T00:40:50.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JlNnRUjC92shHnpX+zM9INs0kCw= 2023-10-12T00:40:50.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JlNnRUjC92shHnpX+zM9INs0kCw=', 'Content-Length': '0'} 2023-10-12T00:40:50.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a82919f9a446c19-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.594 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.595 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.595 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.595 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.596 DEBUG:boto:Method: GET 2023-10-12T00:40:50.596 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.597 DEBUG:boto:Data: 2023-10-12T00:40:50.597 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.598 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.598 DEBUG:boto:Params: {} 2023-10-12T00:40:50.599 DEBUG:boto:Token: None 2023-10-12T00:40:50.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052cdd9c845f3e8fd-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.604 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.605 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:40:50.605 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:40:50.606 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.607 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.607 DEBUG:boto:Method: GET 2023-10-12T00:40:50.608 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:40:50.608 DEBUG:boto:Data: 2023-10-12T00:40:50.609 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.610 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.610 DEBUG:boto:Params: {} 2023-10-12T00:40:50.611 DEBUG:boto:Token: None 2023-10-12T00:40:50.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-48/ 2023-10-12T00:40:50.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA= 2023-10-12T00:40:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Z6rrVCWexT5KSco4Ra+1JiiXGA=', 'Content-Length': '0'} 2023-10-12T00:40:50.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c360cedc03d9b87d-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.615 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.616 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-48 zones={a2, a1} 2023-10-12T00:40:50.616 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-49 2023-10-12T00:40:50.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:50.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:50.618 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:50.674 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:50.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:50.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:50.708+0000 7f976d9d4780 1 bucket sync caught up with source: 2023-10-12T00:40:50.709 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.14.6, , 00000000001.26.6, , 00000000001.14.6, , , 00000000001.21.6, , , ] 2023-10-12T00:40:50.709 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.14.6, , 00000000001.26.6, , 00000000001.14.6, , , 00000000001.21.6, , , ] 2023-10-12T00:40:50.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:50.708+0000 7f976d9d4780 0 bucket checkpoint complete 2023-10-12T00:40:50.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-49 2023-10-12T00:40:50.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:50.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:40:50.723 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:40:50.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:50.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:50.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:50.812+0000 7f50d14f5780 1 bucket sync caught up with source: 2023-10-12T00:40:50.813 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.178.6, , 00000000001.298.6, , 00000000001.65.6, , , 00000000001.161.6, , , ] 2023-10-12T00:40:50.813 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.178.6, , 00000000001.298.6, , 00000000001.65.6, , , 00000000001.161.6, , , ] 2023-10-12T00:40:50.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:50.812+0000 7f50d14f5780 0 bucket checkpoint complete 2023-10-12T00:40:50.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-49 zones={a2, a1} 2023-10-12T00:40:50.826 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.826 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.827 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.827 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:50.828 DEBUG:boto:Data: 2023-10-12T00:40:50.828 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.829 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.829 DEBUG:boto:Params: {} 2023-10-12T00:40:50.830 DEBUG:boto:Token: None 2023-10-12T00:40:50.830 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/ 2023-10-12T00:40:50.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yc37wthh/PrnA14AoO12ImsNEBQ= 2023-10-12T00:40:50.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yc37wthh/PrnA14AoO12ImsNEBQ=', 'Content-Length': '0'} 2023-10-12T00:40:50.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b05113f474a8629-0065274092-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.834 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.834 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.835 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.835 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:40:50.836 DEBUG:boto:Data: 2023-10-12T00:40:50.836 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.837 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.837 DEBUG:boto:Params: {} 2023-10-12T00:40:50.838 DEBUG:boto:Token: None 2023-10-12T00:40:50.838 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/ 2023-10-12T00:40:50.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yc37wthh/PrnA14AoO12ImsNEBQ= 2023-10-12T00:40:50.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yc37wthh/PrnA14AoO12ImsNEBQ=', 'Content-Length': '0'} 2023-10-12T00:40:50.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c7cedcc09ce7c9d-0065274092-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:40:50.842 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.843 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.844 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.844 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.844 DEBUG:boto:Method: GET 2023-10-12T00:40:50.845 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.845 DEBUG:boto:Data: 2023-10-12T00:40:50.846 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.847 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.847 DEBUG:boto:Params: {} 2023-10-12T00:40:50.847 DEBUG:boto:Token: None 2023-10-12T00:40:50.848 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/?versions 2023-10-12T00:40:50.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE= 2023-10-12T00:40:50.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE=', 'Content-Length': '0'} 2023-10-12T00:40:50.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009c5a81e782a7ffd-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.852 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:50.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:50.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:50.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:50.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:40:50.855 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.855 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.856 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.856 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.857 DEBUG:boto:Method: GET 2023-10-12T00:40:50.857 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.858 DEBUG:boto:Data: 2023-10-12T00:40:50.858 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.859 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.859 DEBUG:boto:Params: {} 2023-10-12T00:40:50.860 DEBUG:boto:Token: None 2023-10-12T00:40:50.860 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/?versions 2023-10-12T00:40:50.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE= 2023-10-12T00:40:50.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE=', 'Content-Length': '0'} 2023-10-12T00:40:50.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef6eb8d6dec400f-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.865 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:40:50.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:40:50.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:40:50.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:40:50.867 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.868 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.868 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.869 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.869 DEBUG:boto:Method: GET 2023-10-12T00:40:50.869 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.870 DEBUG:boto:Data: 2023-10-12T00:40:50.870 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.871 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.872 DEBUG:boto:Params: {} 2023-10-12T00:40:50.872 DEBUG:boto:Token: None 2023-10-12T00:40:50.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/?versions 2023-10-12T00:40:50.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE= 2023-10-12T00:40:50.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE=', 'Content-Length': '0'} 2023-10-12T00:40:50.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6291003ab147bca-0065274092-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.879 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.881 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.882 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.882 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.882 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.883 DEBUG:boto:Method: GET 2023-10-12T00:40:50.883 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:40:50.884 DEBUG:boto:Data: 2023-10-12T00:40:50.884 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.885 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.885 DEBUG:boto:Params: {} 2023-10-12T00:40:50.886 DEBUG:boto:Token: None 2023-10-12T00:40:50.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/?versions 2023-10-12T00:40:50.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE= 2023-10-12T00:40:50.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qxYzJGlZxKI197ZAyi3YQdAazZE=', 'Content-Length': '0'} 2023-10-12T00:40:50.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003abc892474ae1198-0065274092-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.891 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:50.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:50.893 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.893 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.894 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.894 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.894 DEBUG:boto:Method: GET 2023-10-12T00:40:50.895 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.896 DEBUG:boto:Data: 2023-10-12T00:40:50.896 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.898 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.898 DEBUG:boto:Params: {} 2023-10-12T00:40:50.899 DEBUG:boto:Token: None 2023-10-12T00:40:50.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.901 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY= 2023-10-12T00:40:50.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY=', 'Content-Length': '0'} 2023-10-12T00:40:50.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d298360ac3c0332-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.946 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.947 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.947 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.948 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.949 DEBUG:boto:Method: GET 2023-10-12T00:40:50.949 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.949 DEBUG:boto:Data: 2023-10-12T00:40:50.950 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.951 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.952 DEBUG:boto:Params: {} 2023-10-12T00:40:50.952 DEBUG:boto:Token: None 2023-10-12T00:40:50.953 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY= 2023-10-12T00:40:50.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY=', 'Content-Length': '0'} 2023-10-12T00:40:50.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031e3b8a4f350852a-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.956 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.957 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.957 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.958 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.958 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.959 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.959 DEBUG:boto:Data: 2023-10-12T00:40:50.960 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.961 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.961 DEBUG:boto:Params: {} 2023-10-12T00:40:50.962 DEBUG:boto:Token: None 2023-10-12T00:40:50.962 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wSCugjyt5TkON8ARPhrvYONIC7I= 2023-10-12T00:40:50.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wSCugjyt5TkON8ARPhrvYONIC7I=', 'Content-Length': '0'} 2023-10-12T00:40:50.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed1d3495fc5dbab6-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.966 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.966 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.967 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.967 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.967 DEBUG:boto:Method: HEAD 2023-10-12T00:40:50.968 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.968 DEBUG:boto:Data: 2023-10-12T00:40:50.969 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.969 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.970 DEBUG:boto:Params: {} 2023-10-12T00:40:50.970 DEBUG:boto:Token: None 2023-10-12T00:40:50.971 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wSCugjyt5TkON8ARPhrvYONIC7I= 2023-10-12T00:40:50.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wSCugjyt5TkON8ARPhrvYONIC7I=', 'Content-Length': '0'} 2023-10-12T00:40:50.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000505b1935d83d44a-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:50.975 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.975 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.975 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.976 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.976 DEBUG:boto:Method: GET 2023-10-12T00:40:50.977 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.977 DEBUG:boto:Data: 2023-10-12T00:40:50.978 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.978 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.979 DEBUG:boto:Params: {} 2023-10-12T00:40:50.979 DEBUG:boto:Token: None 2023-10-12T00:40:50.980 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY= 2023-10-12T00:40:50.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY=', 'Content-Length': '0'} 2023-10-12T00:40:50.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a7accedbe5eccac-0065274092-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.984 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:50.984 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:50.985 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.985 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.986 DEBUG:boto:Method: GET 2023-10-12T00:40:50.986 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.987 DEBUG:boto:Data: 2023-10-12T00:40:50.987 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:50.988 DEBUG:boto:Port: 8000 2023-10-12T00:40:50.988 DEBUG:boto:Params: {} 2023-10-12T00:40:50.989 DEBUG:boto:Token: None 2023-10-12T00:40:50.989 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:40:50.990 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY= 2023-10-12T00:40:50.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v29bH3aRRQFX8T2+d6ezQJw1sGY=', 'Content-Length': '0'} 2023-10-12T00:40:50.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b081e7c9ff2c20a1-0065274092-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:50.993 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:50.994 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:50.994 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:50.995 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:50.995 DEBUG:boto:Method: GET 2023-10-12T00:40:50.996 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:40:50.996 DEBUG:boto:Data: 2023-10-12T00:40:50.997 DEBUG:boto:Headers: {} 2023-10-12T00:40:50.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:50.997 DEBUG:boto:Port: 8001 2023-10-12T00:40:50.998 DEBUG:boto:Params: {} 2023-10-12T00:40:50.998 DEBUG:boto:Token: None 2023-10-12T00:40:50.999 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:50 GMT /wafmoa-49/ 2023-10-12T00:40:50.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vRlyEOxQI2+buYe6AOG9FjxBD7Y= 2023-10-12T00:40:51.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vRlyEOxQI2+buYe6AOG9FjxBD7Y=', 'Content-Length': '0'} 2023-10-12T00:40:51.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002525eb602468d326-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.003 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.003 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.004 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.004 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.005 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.005 DEBUG:boto:Method: GET 2023-10-12T00:40:51.006 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.006 DEBUG:boto:Data: 2023-10-12T00:40:51.006 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.007 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.008 DEBUG:boto:Params: {} 2023-10-12T00:40:51.008 DEBUG:boto:Token: None 2023-10-12T00:40:51.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006404d410e1eb6d1a-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.013 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.013 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:51.014 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:51.014 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.015 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:40:51.015 DEBUG:boto:Data: 2023-10-12T00:40:51.016 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.016 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.017 DEBUG:boto:Params: {} 2023-10-12T00:40:51.017 DEBUG:boto:Token: None 2023-10-12T00:40:51.018 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj1 2023-10-12T00:40:51.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8nKQuFSiSr0oR0t4BnoVUH4+/Zs= 2023-10-12T00:40:51.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8nKQuFSiSr0oR0t4BnoVUH4+/Zs=', 'Content-Length': '0'} 2023-10-12T00:40:51.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073372a1c88808029-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.021 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:51.021 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:51.022 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.022 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:40:51.023 DEBUG:boto:Data: 2023-10-12T00:40:51.023 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.024 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.024 DEBUG:boto:Params: {} 2023-10-12T00:40:51.025 DEBUG:boto:Token: None 2023-10-12T00:40:51.025 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj1 2023-10-12T00:40:51.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8nKQuFSiSr0oR0t4BnoVUH4+/Zs= 2023-10-12T00:40:51.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8nKQuFSiSr0oR0t4BnoVUH4+/Zs=', 'Content-Length': '0'} 2023-10-12T00:40:51.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013d4033338ec9d60-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:40:51.029 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:51.029 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:51.030 DEBUG:boto:Method: GET 2023-10-12T00:40:51.030 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:40:51.031 DEBUG:boto:Data: 2023-10-12T00:40:51.031 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.032 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.032 DEBUG:boto:Params: {} 2023-10-12T00:40:51.033 DEBUG:boto:Token: None 2023-10-12T00:40:51.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj1 2023-10-12T00:40:51.034 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MqkWOsvqu2au6TtE360NcfsGb70= 2023-10-12T00:40:51.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MqkWOsvqu2au6TtE360NcfsGb70=', 'Content-Length': '0'} 2023-10-12T00:40:51.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f12722a8bca2b6-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.078 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:40:51.079 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:40:51.082 DEBUG:boto:Method: GET 2023-10-12T00:40:51.082 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:40:51.083 DEBUG:boto:Data: 2023-10-12T00:40:51.084 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.085 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.086 DEBUG:boto:Params: {} 2023-10-12T00:40:51.087 DEBUG:boto:Token: None 2023-10-12T00:40:51.088 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj1 2023-10-12T00:40:51.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MqkWOsvqu2au6TtE360NcfsGb70= 2023-10-12T00:40:51.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MqkWOsvqu2au6TtE360NcfsGb70=', 'Content-Length': '0'} 2023-10-12T00:40:51.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6b0076b42b09e18-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.092 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.092 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.093 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.093 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.094 DEBUG:boto:Method: GET 2023-10-12T00:40:51.094 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.095 DEBUG:boto:Data: 2023-10-12T00:40:51.096 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.096 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.097 DEBUG:boto:Params: {} 2023-10-12T00:40:51.097 DEBUG:boto:Token: None 2023-10-12T00:40:51.098 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.098 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000229ab1aca9a34827-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.103 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.104 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.104 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.105 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.105 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.106 DEBUG:boto:Method: GET 2023-10-12T00:40:51.107 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:40:51.107 DEBUG:boto:Data: 2023-10-12T00:40:51.108 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.109 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.110 DEBUG:boto:Params: {} 2023-10-12T00:40:51.111 DEBUG:boto:Token: None 2023-10-12T00:40:51.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070db5807dd754651-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.117 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:51.118 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.119 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.119 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.119 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.120 DEBUG:boto:Method: GET 2023-10-12T00:40:51.121 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.121 DEBUG:boto:Data: 2023-10-12T00:40:51.121 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.122 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.123 DEBUG:boto:Params: {} 2023-10-12T00:40:51.123 DEBUG:boto:Token: None 2023-10-12T00:40:51.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0= 2023-10-12T00:40:51.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0=', 'Content-Length': '0'} 2023-10-12T00:40:51.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de584a2af37ce5fc-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.127 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.128 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.128 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.128 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.129 DEBUG:boto:Method: GET 2023-10-12T00:40:51.129 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.130 DEBUG:boto:Data: 2023-10-12T00:40:51.130 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.131 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.131 DEBUG:boto:Params: {} 2023-10-12T00:40:51.132 DEBUG:boto:Token: None 2023-10-12T00:40:51.132 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.133 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0= 2023-10-12T00:40:51.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0=', 'Content-Length': '0'} 2023-10-12T00:40:51.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8162c608f220600-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.136 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.136 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.137 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.137 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.138 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.138 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.139 DEBUG:boto:Data: 2023-10-12T00:40:51.139 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.140 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.140 DEBUG:boto:Params: {} 2023-10-12T00:40:51.141 DEBUG:boto:Token: None 2023-10-12T00:40:51.141 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.142 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BC91s6g919JpaWe8aX7YtBvS8IY= 2023-10-12T00:40:51.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BC91s6g919JpaWe8aX7YtBvS8IY=', 'Content-Length': '0'} 2023-10-12T00:40:51.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4e34954783c263f-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.144 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.145 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.145 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.146 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.146 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.146 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.147 DEBUG:boto:Data: 2023-10-12T00:40:51.147 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.148 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.149 DEBUG:boto:Params: {} 2023-10-12T00:40:51.149 DEBUG:boto:Token: None 2023-10-12T00:40:51.150 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BC91s6g919JpaWe8aX7YtBvS8IY= 2023-10-12T00:40:51.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BC91s6g919JpaWe8aX7YtBvS8IY=', 'Content-Length': '0'} 2023-10-12T00:40:51.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000949bd61eb3a2f81-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:51.153 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.154 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.154 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.155 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.155 DEBUG:boto:Method: GET 2023-10-12T00:40:51.156 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.156 DEBUG:boto:Data: 2023-10-12T00:40:51.157 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.157 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.158 DEBUG:boto:Params: {} 2023-10-12T00:40:51.159 DEBUG:boto:Token: None 2023-10-12T00:40:51.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0= 2023-10-12T00:40:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0=', 'Content-Length': '0'} 2023-10-12T00:40:51.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf6594606e4daa9e-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.202 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.203 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.203 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.204 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.204 DEBUG:boto:Method: GET 2023-10-12T00:40:51.205 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.205 DEBUG:boto:Data: 2023-10-12T00:40:51.206 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.206 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.207 DEBUG:boto:Params: {} 2023-10-12T00:40:51.207 DEBUG:boto:Token: None 2023-10-12T00:40:51.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:40:51.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0= 2023-10-12T00:40:51.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0J9iZVXZFxIhPmDtGnGNONDdSm0=', 'Content-Length': '0'} 2023-10-12T00:40:51.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022ccfcc9749c9417-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.212 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.212 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.213 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.213 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.214 DEBUG:boto:Method: GET 2023-10-12T00:40:51.214 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.214 DEBUG:boto:Data: 2023-10-12T00:40:51.215 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.216 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.216 DEBUG:boto:Params: {} 2023-10-12T00:40:51.217 DEBUG:boto:Token: None 2023-10-12T00:40:51.217 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c225ea3a85d4fc0b-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.221 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.222 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.223 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.223 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.224 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.224 DEBUG:boto:Method: GET 2023-10-12T00:40:51.224 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.225 DEBUG:boto:Data: 2023-10-12T00:40:51.225 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.226 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.227 DEBUG:boto:Params: {} 2023-10-12T00:40:51.227 DEBUG:boto:Token: None 2023-10-12T00:40:51.228 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000431c41dadbe0efa-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.274 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.275 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.276 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.276 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.276 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:40:51.277 DEBUG:boto:Data: 2023-10-12T00:40:51.277 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.278 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.279 DEBUG:boto:Params: {} 2023-10-12T00:40:51.279 DEBUG:boto:Token: None 2023-10-12T00:40:51.279 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2 2023-10-12T00:40:51.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b/ePHNUsK8rLJ1MZwCqutZY69OY= 2023-10-12T00:40:51.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b/ePHNUsK8rLJ1MZwCqutZY69OY=', 'Content-Length': '0'} 2023-10-12T00:40:51.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d803404f9adcb38a-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.283 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.283 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.284 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.284 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:40:51.285 DEBUG:boto:Data: 2023-10-12T00:40:51.285 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.286 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.286 DEBUG:boto:Params: {} 2023-10-12T00:40:51.288 DEBUG:boto:Token: None 2023-10-12T00:40:51.291 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2 2023-10-12T00:40:51.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b/ePHNUsK8rLJ1MZwCqutZY69OY= 2023-10-12T00:40:51.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b/ePHNUsK8rLJ1MZwCqutZY69OY=', 'Content-Length': '0'} 2023-10-12T00:40:51.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a051d592381b1e6-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:40:51.295 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.296 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.296 DEBUG:boto:Method: GET 2023-10-12T00:40:51.297 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:40:51.297 DEBUG:boto:Data: 2023-10-12T00:40:51.298 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.299 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.300 DEBUG:boto:Params: {} 2023-10-12T00:40:51.301 DEBUG:boto:Token: None 2023-10-12T00:40:51.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2 2023-10-12T00:40:51.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cMqTJOgRX3T8w7SFPDIfMa6y/9w= 2023-10-12T00:40:51.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cMqTJOgRX3T8w7SFPDIfMa6y/9w=', 'Content-Length': '0'} 2023-10-12T00:40:51.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd4e0e932a098719-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.305 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:40:51.305 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:40:51.306 DEBUG:boto:Method: GET 2023-10-12T00:40:51.306 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:40:51.307 DEBUG:boto:Data: 2023-10-12T00:40:51.307 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.308 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.308 DEBUG:boto:Params: {} 2023-10-12T00:40:51.309 DEBUG:boto:Token: None 2023-10-12T00:40:51.309 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj2 2023-10-12T00:40:51.310 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cMqTJOgRX3T8w7SFPDIfMa6y/9w= 2023-10-12T00:40:51.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cMqTJOgRX3T8w7SFPDIfMa6y/9w=', 'Content-Length': '0'} 2023-10-12T00:40:51.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023260ecaa2928d62-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.313 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.314 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.314 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.314 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.315 DEBUG:boto:Method: GET 2023-10-12T00:40:51.315 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.316 DEBUG:boto:Data: 2023-10-12T00:40:51.316 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.317 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.317 DEBUG:boto:Params: {} 2023-10-12T00:40:51.318 DEBUG:boto:Token: None 2023-10-12T00:40:51.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be0f2cfb346a8357-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.322 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.323 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.324 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.324 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.324 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.325 DEBUG:boto:Method: GET 2023-10-12T00:40:51.325 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:40:51.326 DEBUG:boto:Data: 2023-10-12T00:40:51.326 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.327 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.327 DEBUG:boto:Params: {} 2023-10-12T00:40:51.328 DEBUG:boto:Token: None 2023-10-12T00:40:51.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fbdf5b401d79cb-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.332 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:51.334 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.334 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.334 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.335 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.335 DEBUG:boto:Method: GET 2023-10-12T00:40:51.336 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.336 DEBUG:boto:Data: 2023-10-12T00:40:51.337 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.337 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.338 DEBUG:boto:Params: {} 2023-10-12T00:40:51.338 DEBUG:boto:Token: None 2023-10-12T00:40:51.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q= 2023-10-12T00:40:51.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q=', 'Content-Length': '0'} 2023-10-12T00:40:51.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e02236d9cbf8df68-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.342 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.343 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.343 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.344 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.344 DEBUG:boto:Method: GET 2023-10-12T00:40:51.345 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.345 DEBUG:boto:Data: 2023-10-12T00:40:51.345 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.346 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.347 DEBUG:boto:Params: {} 2023-10-12T00:40:51.347 DEBUG:boto:Token: None 2023-10-12T00:40:51.348 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q= 2023-10-12T00:40:51.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q=', 'Content-Length': '0'} 2023-10-12T00:40:51.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323c0c1abf8e07a1-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.351 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.352 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.352 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.352 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.353 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.353 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.354 DEBUG:boto:Data: 2023-10-12T00:40:51.354 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.355 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.355 DEBUG:boto:Params: {} 2023-10-12T00:40:51.356 DEBUG:boto:Token: None 2023-10-12T00:40:51.356 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mojFONoy0+4dcao040NPBo90zsk= 2023-10-12T00:40:51.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mojFONoy0+4dcao040NPBo90zsk=', 'Content-Length': '0'} 2023-10-12T00:40:51.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b5e33c4fdba19d9-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.360 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.360 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.360 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.361 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.361 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.362 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.362 DEBUG:boto:Data: 2023-10-12T00:40:51.362 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.363 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.364 DEBUG:boto:Params: {} 2023-10-12T00:40:51.364 DEBUG:boto:Token: None 2023-10-12T00:40:51.365 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mojFONoy0+4dcao040NPBo90zsk= 2023-10-12T00:40:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mojFONoy0+4dcao040NPBo90zsk=', 'Content-Length': '0'} 2023-10-12T00:40:51.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1da72f0b1fa153b-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:51.368 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.369 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.369 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.370 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.370 DEBUG:boto:Method: GET 2023-10-12T00:40:51.370 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.371 DEBUG:boto:Data: 2023-10-12T00:40:51.371 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.372 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.373 DEBUG:boto:Params: {} 2023-10-12T00:40:51.373 DEBUG:boto:Token: None 2023-10-12T00:40:51.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q= 2023-10-12T00:40:51.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q=', 'Content-Length': '0'} 2023-10-12T00:40:51.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc8d04722a0db5c9-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.377 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.377 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.378 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.378 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.379 DEBUG:boto:Method: GET 2023-10-12T00:40:51.379 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.380 DEBUG:boto:Data: 2023-10-12T00:40:51.380 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.381 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.381 DEBUG:boto:Params: {} 2023-10-12T00:40:51.382 DEBUG:boto:Token: None 2023-10-12T00:40:51.382 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:40:51.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q= 2023-10-12T00:40:51.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q7a2nbdViKN8nsG71DCD+Dk+n+Q=', 'Content-Length': '0'} 2023-10-12T00:40:51.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e7dcaf62f3a3646-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.386 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.386 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.386 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.387 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.387 DEBUG:boto:Method: GET 2023-10-12T00:40:51.388 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.388 DEBUG:boto:Data: 2023-10-12T00:40:51.389 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.389 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.390 DEBUG:boto:Params: {} 2023-10-12T00:40:51.390 DEBUG:boto:Token: None 2023-10-12T00:40:51.391 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ab5aaae0edb71e-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.395 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.396 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.396 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.397 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.397 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.397 DEBUG:boto:Method: GET 2023-10-12T00:40:51.398 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.398 DEBUG:boto:Data: 2023-10-12T00:40:51.399 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.400 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.400 DEBUG:boto:Params: {} 2023-10-12T00:40:51.400 DEBUG:boto:Token: None 2023-10-12T00:40:51.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b400d01d72be06da-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.405 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.405 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.406 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.406 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.406 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:40:51.407 DEBUG:boto:Data: 2023-10-12T00:40:51.407 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.408 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.408 DEBUG:boto:Params: {} 2023-10-12T00:40:51.409 DEBUG:boto:Token: None 2023-10-12T00:40:51.409 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3 2023-10-12T00:40:51.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3fVMNPBTMjjDR/TEgsmy+QGd9dQ= 2023-10-12T00:40:51.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3fVMNPBTMjjDR/TEgsmy+QGd9dQ=', 'Content-Length': '0'} 2023-10-12T00:40:51.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000719e2cc748a4c4f4-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.413 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.413 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.413 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.414 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:40:51.414 DEBUG:boto:Data: 2023-10-12T00:40:51.415 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.415 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.416 DEBUG:boto:Params: {} 2023-10-12T00:40:51.416 DEBUG:boto:Token: None 2023-10-12T00:40:51.417 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3 2023-10-12T00:40:51.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3fVMNPBTMjjDR/TEgsmy+QGd9dQ= 2023-10-12T00:40:51.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3fVMNPBTMjjDR/TEgsmy+QGd9dQ=', 'Content-Length': '0'} 2023-10-12T00:40:51.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d640a5c68387eb22-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:40:51.421 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.421 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.421 DEBUG:boto:Method: GET 2023-10-12T00:40:51.422 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:40:51.423 DEBUG:boto:Data: 2023-10-12T00:40:51.423 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.424 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.425 DEBUG:boto:Params: {} 2023-10-12T00:40:51.425 DEBUG:boto:Token: None 2023-10-12T00:40:51.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3 2023-10-12T00:40:51.426 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PFcT6YQHUh8CwtJAoYWjJGQSSPg= 2023-10-12T00:40:51.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PFcT6YQHUh8CwtJAoYWjJGQSSPg=', 'Content-Length': '0'} 2023-10-12T00:40:51.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5f17d37aae4a201-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.470 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:40:51.471 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:40:51.471 DEBUG:boto:Method: GET 2023-10-12T00:40:51.472 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:40:51.472 DEBUG:boto:Data: 2023-10-12T00:40:51.473 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.474 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.474 DEBUG:boto:Params: {} 2023-10-12T00:40:51.475 DEBUG:boto:Token: None 2023-10-12T00:40:51.475 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj3 2023-10-12T00:40:51.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PFcT6YQHUh8CwtJAoYWjJGQSSPg= 2023-10-12T00:40:51.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PFcT6YQHUh8CwtJAoYWjJGQSSPg=', 'Content-Length': '0'} 2023-10-12T00:40:51.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004207b68d08a6b96-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.479 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.480 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.480 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.480 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.481 DEBUG:boto:Method: GET 2023-10-12T00:40:51.481 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.482 DEBUG:boto:Data: 2023-10-12T00:40:51.482 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.483 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.484 DEBUG:boto:Params: {} 2023-10-12T00:40:51.484 DEBUG:boto:Token: None 2023-10-12T00:40:51.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000704ca02fac236b7-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.489 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.489 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.490 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.491 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.491 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.491 DEBUG:boto:Method: GET 2023-10-12T00:40:51.492 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:40:51.495 DEBUG:boto:Data: 2023-10-12T00:40:51.495 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.497 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.497 DEBUG:boto:Params: {} 2023-10-12T00:40:51.498 DEBUG:boto:Token: None 2023-10-12T00:40:51.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000906fdd7afeaf1b3-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.550 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:51.552 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.553 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.554 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.555 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.555 DEBUG:boto:Method: GET 2023-10-12T00:40:51.556 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.558 DEBUG:boto:Data: 2023-10-12T00:40:51.558 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.560 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.561 DEBUG:boto:Params: {} 2023-10-12T00:40:51.562 DEBUG:boto:Token: None 2023-10-12T00:40:51.562 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.563 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE= 2023-10-12T00:40:51.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE=', 'Content-Length': '0'} 2023-10-12T00:40:51.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4bb9a3a7e29ef1b-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.568 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.569 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.569 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.570 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.570 DEBUG:boto:Method: GET 2023-10-12T00:40:51.571 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.572 DEBUG:boto:Data: 2023-10-12T00:40:51.572 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.574 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.575 DEBUG:boto:Params: {} 2023-10-12T00:40:51.576 DEBUG:boto:Token: None 2023-10-12T00:40:51.576 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE= 2023-10-12T00:40:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE=', 'Content-Length': '0'} 2023-10-12T00:40:51.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017120ea8e8b9112a-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.582 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.582 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.583 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.584 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.584 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.585 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.586 DEBUG:boto:Data: 2023-10-12T00:40:51.586 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.588 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.588 DEBUG:boto:Params: {} 2023-10-12T00:40:51.589 DEBUG:boto:Token: None 2023-10-12T00:40:51.590 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KVzSuwcm0gC4i/BJiDvyeF+yy+Y= 2023-10-12T00:40:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KVzSuwcm0gC4i/BJiDvyeF+yy+Y=', 'Content-Length': '0'} 2023-10-12T00:40:51.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0b85228c78e5ad-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.595 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.595 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.596 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.597 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.597 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.598 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.599 DEBUG:boto:Data: 2023-10-12T00:40:51.600 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.601 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.601 DEBUG:boto:Params: {} 2023-10-12T00:40:51.602 DEBUG:boto:Token: None 2023-10-12T00:40:51.603 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KVzSuwcm0gC4i/BJiDvyeF+yy+Y= 2023-10-12T00:40:51.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KVzSuwcm0gC4i/BJiDvyeF+yy+Y=', 'Content-Length': '0'} 2023-10-12T00:40:51.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010d8111b6f365760-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:51.607 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.608 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.608 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.609 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.609 DEBUG:boto:Method: GET 2023-10-12T00:40:51.610 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.610 DEBUG:boto:Data: 2023-10-12T00:40:51.610 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.611 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.612 DEBUG:boto:Params: {} 2023-10-12T00:40:51.612 DEBUG:boto:Token: None 2023-10-12T00:40:51.613 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE= 2023-10-12T00:40:51.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE=', 'Content-Length': '0'} 2023-10-12T00:40:51.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072e8c4bbd3f6d074-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.616 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.617 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.617 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.618 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.619 DEBUG:boto:Method: GET 2023-10-12T00:40:51.619 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.620 DEBUG:boto:Data: 2023-10-12T00:40:51.620 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.622 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.622 DEBUG:boto:Params: {} 2023-10-12T00:40:51.623 DEBUG:boto:Token: None 2023-10-12T00:40:51.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:40:51.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE= 2023-10-12T00:40:51.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZX1BJySYAns5DJaZUdSvjK54GsE=', 'Content-Length': '0'} 2023-10-12T00:40:51.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b09562ff7cda1eac-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.628 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.629 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.629 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.629 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.630 DEBUG:boto:Method: GET 2023-10-12T00:40:51.630 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.631 DEBUG:boto:Data: 2023-10-12T00:40:51.631 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.632 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.632 DEBUG:boto:Params: {} 2023-10-12T00:40:51.633 DEBUG:boto:Token: None 2023-10-12T00:40:51.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000598c893f4cdc8d4e-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.637 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.638 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.638 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.639 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.639 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.640 DEBUG:boto:Method: GET 2023-10-12T00:40:51.640 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.641 DEBUG:boto:Data: 2023-10-12T00:40:51.641 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.642 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.642 DEBUG:boto:Params: {} 2023-10-12T00:40:51.643 DEBUG:boto:Token: None 2023-10-12T00:40:51.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c244b1f94ef134bb-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.647 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.648 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.648 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.649 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.649 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:40:51.649 DEBUG:boto:Data: 2023-10-12T00:40:51.650 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.651 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.651 DEBUG:boto:Params: {} 2023-10-12T00:40:51.652 DEBUG:boto:Token: None 2023-10-12T00:40:51.652 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4 2023-10-12T00:40:51.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Vj/mKTqlypX++PK07X9y7GcpdTM= 2023-10-12T00:40:51.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Vj/mKTqlypX++PK07X9y7GcpdTM=', 'Content-Length': '0'} 2023-10-12T00:40:51.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046b056872b416720-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.655 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.656 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.656 DEBUG:boto:Method: HEAD 2023-10-12T00:40:51.656 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:40:51.657 DEBUG:boto:Data: 2023-10-12T00:40:51.657 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.658 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.658 DEBUG:boto:Params: {} 2023-10-12T00:40:51.659 DEBUG:boto:Token: None 2023-10-12T00:40:51.659 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4 2023-10-12T00:40:51.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Vj/mKTqlypX++PK07X9y7GcpdTM= 2023-10-12T00:40:51.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Vj/mKTqlypX++PK07X9y7GcpdTM=', 'Content-Length': '0'} 2023-10-12T00:40:51.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0e4a6a983639bb8-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:40:51.663 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.663 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.664 DEBUG:boto:Method: GET 2023-10-12T00:40:51.664 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:40:51.665 DEBUG:boto:Data: 2023-10-12T00:40:51.665 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.666 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.667 DEBUG:boto:Params: {} 2023-10-12T00:40:51.667 DEBUG:boto:Token: None 2023-10-12T00:40:51.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4 2023-10-12T00:40:51.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TSnJq0yQtc64hToItdKvThkpZRQ= 2023-10-12T00:40:51.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TSnJq0yQtc64hToItdKvThkpZRQ=', 'Content-Length': '0'} 2023-10-12T00:40:51.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005666dbab9c25a14b-0065274093-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.714 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:40:51.715 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:40:51.716 DEBUG:boto:Method: GET 2023-10-12T00:40:51.716 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:40:51.717 DEBUG:boto:Data: 2023-10-12T00:40:51.717 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.718 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.719 DEBUG:boto:Params: {} 2023-10-12T00:40:51.719 DEBUG:boto:Token: None 2023-10-12T00:40:51.720 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/obj4 2023-10-12T00:40:51.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TSnJq0yQtc64hToItdKvThkpZRQ= 2023-10-12T00:40:51.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TSnJq0yQtc64hToItdKvThkpZRQ=', 'Content-Length': '0'} 2023-10-12T00:40:51.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e852af8cf7bfa632-0065274093-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.723 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.724 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.724 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.724 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.725 DEBUG:boto:Method: GET 2023-10-12T00:40:51.725 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.726 DEBUG:boto:Data: 2023-10-12T00:40:51.726 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:40:51.727 DEBUG:boto:Port: 8001 2023-10-12T00:40:51.727 DEBUG:boto:Params: {} 2023-10-12T00:40:51.728 DEBUG:boto:Token: None 2023-10-12T00:40:51.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d2f06533fd65749-0065274093-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.732 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.733 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:40:51.734 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:40:51.734 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.734 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.735 DEBUG:boto:Method: GET 2023-10-12T00:40:51.735 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:40:51.736 DEBUG:boto:Data: 2023-10-12T00:40:51.736 DEBUG:boto:Headers: {} 2023-10-12T00:40:51.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:40:51.737 DEBUG:boto:Port: 8000 2023-10-12T00:40:51.737 DEBUG:boto:Params: {} 2023-10-12T00:40:51.738 DEBUG:boto:Token: None 2023-10-12T00:40:51.738 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:40:51 GMT /wafmoa-49/ 2023-10-12T00:40:51.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g= 2023-10-12T00:40:51.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:40:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IozriMqK+1HGCQp27nd8KhBiy/g=', 'Content-Length': '0'} 2023-10-12T00:40:51.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7d9294d2516ea7-0065274093-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:40:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:40:51.782 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:40:51.783 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-49 zones={a2, a1} 2023-10-12T00:40:51.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:40:51.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:40:51.785 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:40:51.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:51.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:51.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wafmoa-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:40:51.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wafmoa-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:40:51.901 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wafmoa-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:40:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:52.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:52.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:52.016 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:52.063 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:52.064 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:52.092 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:40:52.104 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:40:52.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:52.105 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:40:52.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.106 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:52.106 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:40:52.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.107 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071252.002073_499.1", 2023-10-12T00:40:52.107 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:40:52.002073Z" 2023-10-12T00:40:52.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.108 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:52.108 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:40:52.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.108 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:40:52.109 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071252.002073_499.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:52.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:40:52.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:52.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:52.111 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:52.161 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:52.162 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:52.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:52.189+0000 7fe15105d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:52.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:52.190+0000 7fe15105d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:52.202 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:52.203 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:52.203 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:52.203 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:52.203 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:52.203 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:40:52.204 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:52.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.204 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:52.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.205 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:52.205 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:52.205 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:52.205 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:52.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:52.206 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:40:52.206 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:52.206 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:40:52.206 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:52.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.207 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:52.207 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:52.207 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:52.208 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:52.208 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:52.208 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:52.209 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:52.209 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:40:52.209 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:52.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.210 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:52.211 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:52.211 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:52.211 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071203.980933_494.1", 2023-10-12T00:40:52.211 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:52.211 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:52.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:52.212 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:40:03.980933Z", 2023-10-12T00:40:52.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:52.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:52.213 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:52.213 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:52.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:52.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:52.214 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:52.214 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:52.214 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:52.214 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:40:52.215 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:52.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:52.215 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:52.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.216 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:52.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:40:52.216 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:40:52.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:52.216 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:52Z" 2023-10-12T00:40:52.217 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:52.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071203.980933_494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:40:03.980933Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:40:52Z" } 2023-10-12T00:40:52.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071203.980933_494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:40:03.980933Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:40:52.219 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071252.002073_499.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:52.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:52.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071252.002073_499.1 target=1_1697071203.980933_494.1 2023-10-12T00:40:53.746 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.745+0000 7feb8ca49700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:53.746 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.745+0000 7feb83236700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:53.746 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.745+0000 7feb7ea2d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:53.748 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.746+0000 7feb7e22c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:53.748 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.746+0000 7feb84a39700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:53.749 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T00:40:53.746+0000 7feb86a3d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-12T00:40:57.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:57.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:40:57.222 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:40:57.280 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:40:57.280 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:40:57.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:57.313+0000 7f863b62e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:57.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:40:57.314+0000 7f863b62e780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:40:57.330 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:40:57.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:40:57.331 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:40:57.331 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:40:57.332 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:40:57.332 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:40:57.332 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:57.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.333 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:40:57.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:57.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:40:57.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:57.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:57.335 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:40:57.335 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:57.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:40:57.335 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:57.336 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:40:57.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:57.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:57.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:57.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:40:57.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:57.338 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:57.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:40:57.339 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:57.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:57.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:57.339 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:40:57.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:57.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:57.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:57.341 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:40:57.341 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:57.341 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:57.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071203.980933_494.1", 2023-10-12T00:40:57.342 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:57.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:57.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:57.342 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:40:03.980933Z", 2023-10-12T00:40:57.343 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:57.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:57.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:40:57.344 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:40:57.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:40:57.344 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:40:57.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:40:57.345 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:40:57.345 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:40:57.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:40:57.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:40:57.346 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:40:57.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:57.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:40:57.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:40:57.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.347 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:40:57.347 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:40:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:40:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:40:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:40:57Z" 2023-10-12T00:40:57.349 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:40:57.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071203.980933_494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:40:03.980933Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:40:57Z" } 2023-10-12T00:40:57.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071203.980933_494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:40:03.980933Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:40:57.351 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071252.002073_499.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:57.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071203.980933_494.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:40:57.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071252.002073_499.1 target=1_1697071203.980933_494.1 2023-10-12T00:41:02.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:02.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:02.356 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:02.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:02.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:02.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:02.444+0000 7f9027a0c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:02.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:02.445+0000 7f9027a0c780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:02.459 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:41:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:41:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:41:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:41:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:41:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:41:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:41:02.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:02.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:41:02.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:02.464 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:02.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:41:02.465 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:02.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:41:02.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:02.467 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:41:02.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:02.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:02.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:02.470 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:41:02.470 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:02.470 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:02.471 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:41:02.471 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:02.471 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:02.472 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:02.472 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:41:02.472 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:02.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:02.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:02.474 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:41:02.474 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:02.475 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:02.475 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071252.002073_499.1", 2023-10-12T00:41:02.475 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:02.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:02.476 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:02.476 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:40:52.002073Z", 2023-10-12T00:41:02.477 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:02.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:02.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:02.479 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:41:02.479 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:02.480 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:02.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:41:02.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:02.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:02.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:02.481 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:41:02.482 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:02.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:02.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:02.483 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:41:02.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.483 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:41:02.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:41:02.484 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:41:02.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:02.484 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:41:02Z" 2023-10-12T00:41:02.484 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:41:02.485 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071252.002073_499.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:40:52.002073Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:41:02Z" } 2023-10-12T00:41:02.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071252.002073_499.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:40:52.002073Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:41:02.486 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071252.002073_499.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:02.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071252.002073_499.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:02.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:41:02.488 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:02.488 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:02.488 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.489 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:02.489 DEBUG:boto:Data: 2023-10-12T00:41:02.490 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.491 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.491 DEBUG:boto:Params: {} 2023-10-12T00:41:02.491 DEBUG:boto:Token: None 2023-10-12T00:41:02.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/ 2023-10-12T00:41:02.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc= 2023-10-12T00:41:02.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc=', 'Content-Length': '0'} 2023-10-12T00:41:02.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc267f3c164091c4-006527409e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.496 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:02.497 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:02.497 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.497 DEBUG:boto:Path: /wafmoa-48/obj5 2023-10-12T00:41:02.498 DEBUG:boto:Data: 2023-10-12T00:41:02.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.499 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.499 DEBUG:boto:Params: {} 2023-10-12T00:41:02.500 DEBUG:boto:Token: None 2023-10-12T00:41:02.500 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/obj5 2023-10-12T00:41:02.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h1D5e9PTDL50Hv1Qn/xWzzcyoD0= 2023-10-12T00:41:02.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h1D5e9PTDL50Hv1Qn/xWzzcyoD0='} 2023-10-12T00:41:02.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000185123a58cb258ee-006527409e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.508 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:02.508 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:02.508 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.509 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:02.509 DEBUG:boto:Data: 2023-10-12T00:41:02.510 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.510 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.511 DEBUG:boto:Params: {} 2023-10-12T00:41:02.511 DEBUG:boto:Token: None 2023-10-12T00:41:02.512 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/ 2023-10-12T00:41:02.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc= 2023-10-12T00:41:02.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc=', 'Content-Length': '0'} 2023-10-12T00:41:02.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ac556d5987bad81-006527409e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.515 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:02.516 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:02.516 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.517 DEBUG:boto:Path: /wafmoa-48/obj6 2023-10-12T00:41:02.517 DEBUG:boto:Data: 2023-10-12T00:41:02.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.518 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.519 DEBUG:boto:Params: {} 2023-10-12T00:41:02.519 DEBUG:boto:Token: None 2023-10-12T00:41:02.520 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/obj6 2023-10-12T00:41:02.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yD3KNn8U/UbvdMgI3vx5Q38q3N8= 2023-10-12T00:41:02.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yD3KNn8U/UbvdMgI3vx5Q38q3N8='} 2023-10-12T00:41:02.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c89ea1bb2ac1a25-006527409e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.563 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:02.563 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:02.564 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.565 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:02.565 DEBUG:boto:Data: 2023-10-12T00:41:02.566 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.567 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.567 DEBUG:boto:Params: {} 2023-10-12T00:41:02.568 DEBUG:boto:Token: None 2023-10-12T00:41:02.568 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/ 2023-10-12T00:41:02.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc= 2023-10-12T00:41:02.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc=', 'Content-Length': '0'} 2023-10-12T00:41:02.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcf859a2f5abc543-006527409e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.572 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:02.572 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:02.573 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.573 DEBUG:boto:Path: /wafmoa-48/obj7 2023-10-12T00:41:02.574 DEBUG:boto:Data: 2023-10-12T00:41:02.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.575 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.575 DEBUG:boto:Params: {} 2023-10-12T00:41:02.576 DEBUG:boto:Token: None 2023-10-12T00:41:02.576 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/obj7 2023-10-12T00:41:02.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jcZaltMfhjNHCfNvPUfyUIqGOd8= 2023-10-12T00:41:02.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jcZaltMfhjNHCfNvPUfyUIqGOd8='} 2023-10-12T00:41:02.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000350d4c26a3d07057-006527409e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.585 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:02.585 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:02.586 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.586 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:02.587 DEBUG:boto:Data: 2023-10-12T00:41:02.587 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.588 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.588 DEBUG:boto:Params: {} 2023-10-12T00:41:02.589 DEBUG:boto:Token: None 2023-10-12T00:41:02.589 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/ 2023-10-12T00:41:02.590 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc= 2023-10-12T00:41:02.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ja5lk7bqbV9SAENQ1REe/CoFaYc=', 'Content-Length': '0'} 2023-10-12T00:41:02.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5ba5a3c240e350f-006527409e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.593 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:02.593 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:02.594 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.594 DEBUG:boto:Path: /wafmoa-48/obj8 2023-10-12T00:41:02.595 DEBUG:boto:Data: 2023-10-12T00:41:02.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:02.596 DEBUG:boto:Port: 8000 2023-10-12T00:41:02.596 DEBUG:boto:Params: {} 2023-10-12T00:41:02.597 DEBUG:boto:Token: None 2023-10-12T00:41:02.597 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-48/obj8 2023-10-12T00:41:02.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NHKPbw3eXKAT7DcLffFZpLFRq+I= 2023-10-12T00:41:02.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NHKPbw3eXKAT7DcLffFZpLFRq+I='} 2023-10-12T00:41:02.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2eb00823934560f-006527409e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.643 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:02.644 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:02.644 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.645 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:02.645 DEBUG:boto:Data: 2023-10-12T00:41:02.645 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.646 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.647 DEBUG:boto:Params: {} 2023-10-12T00:41:02.647 DEBUG:boto:Token: None 2023-10-12T00:41:02.648 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/ 2023-10-12T00:41:02.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk= 2023-10-12T00:41:02.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk=', 'Content-Length': '0'} 2023-10-12T00:41:02.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000235ea3fba7478c5-006527409e-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.652 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:02.652 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:02.653 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.653 DEBUG:boto:Path: /wafmoa-49/obj5 2023-10-12T00:41:02.653 DEBUG:boto:Data: 2023-10-12T00:41:02.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.655 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.655 DEBUG:boto:Params: {} 2023-10-12T00:41:02.656 DEBUG:boto:Token: None 2023-10-12T00:41:02.656 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/obj5 2023-10-12T00:41:02.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GDx2pA0dq47NO+haR8FuoZippyw= 2023-10-12T00:41:02.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GDx2pA0dq47NO+haR8FuoZippyw='} 2023-10-12T00:41:02.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b168125b897d25c-006527409e-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.662 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:02.662 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:02.663 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.663 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:02.663 DEBUG:boto:Data: 2023-10-12T00:41:02.664 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.665 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.665 DEBUG:boto:Params: {} 2023-10-12T00:41:02.666 DEBUG:boto:Token: None 2023-10-12T00:41:02.666 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/ 2023-10-12T00:41:02.667 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk= 2023-10-12T00:41:02.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk=', 'Content-Length': '0'} 2023-10-12T00:41:02.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050b49b6cbfbf0de3-006527409e-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.670 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:02.671 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:02.671 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.672 DEBUG:boto:Path: /wafmoa-49/obj6 2023-10-12T00:41:02.672 DEBUG:boto:Data: 2023-10-12T00:41:02.672 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.673 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.674 DEBUG:boto:Params: {} 2023-10-12T00:41:02.674 DEBUG:boto:Token: None 2023-10-12T00:41:02.674 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/obj6 2023-10-12T00:41:02.675 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YJVxVdy0/06XRG7tNfiI+Um7n3M= 2023-10-12T00:41:02.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YJVxVdy0/06XRG7tNfiI+Um7n3M='} 2023-10-12T00:41:02.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068bcba40d9de8bd1-006527409e-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.719 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:02.720 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:02.720 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.720 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:02.721 DEBUG:boto:Data: 2023-10-12T00:41:02.721 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.722 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.723 DEBUG:boto:Params: {} 2023-10-12T00:41:02.723 DEBUG:boto:Token: None 2023-10-12T00:41:02.724 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/ 2023-10-12T00:41:02.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk= 2023-10-12T00:41:02.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk=', 'Content-Length': '0'} 2023-10-12T00:41:02.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c0c34333a3e8457-006527409e-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.728 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:02.728 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:02.729 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.729 DEBUG:boto:Path: /wafmoa-49/obj7 2023-10-12T00:41:02.729 DEBUG:boto:Data: 2023-10-12T00:41:02.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.731 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.731 DEBUG:boto:Params: {} 2023-10-12T00:41:02.731 DEBUG:boto:Token: None 2023-10-12T00:41:02.732 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/obj7 2023-10-12T00:41:02.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/5c7Eeqt9a7DCWdguT8xUjfxH2c= 2023-10-12T00:41:02.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/5c7Eeqt9a7DCWdguT8xUjfxH2c='} 2023-10-12T00:41:02.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c9266a88cdb3b58-006527409e-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.737 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:02.737 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:02.738 DEBUG:boto:Method: HEAD 2023-10-12T00:41:02.738 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:02.739 DEBUG:boto:Data: 2023-10-12T00:41:02.739 DEBUG:boto:Headers: {} 2023-10-12T00:41:02.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.740 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.740 DEBUG:boto:Params: {} 2023-10-12T00:41:02.741 DEBUG:boto:Token: None 2023-10-12T00:41:02.741 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/ 2023-10-12T00:41:02.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk= 2023-10-12T00:41:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnvky0DCm0wmeRqYyKvivhCs5dk=', 'Content-Length': '0'} 2023-10-12T00:41:02.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b199df3a93b839c5-006527409e-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.745 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:02.745 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:02.746 DEBUG:boto:Method: PUT 2023-10-12T00:41:02.746 DEBUG:boto:Path: /wafmoa-49/obj8 2023-10-12T00:41:02.747 DEBUG:boto:Data: 2023-10-12T00:41:02.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T00:41:02.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:02.748 DEBUG:boto:Port: 8001 2023-10-12T00:41:02.749 DEBUG:boto:Params: {} 2023-10-12T00:41:02.749 DEBUG:boto:Token: None 2023-10-12T00:41:02.750 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 00:41:02 GMT /wafmoa-49/obj8 2023-10-12T00:41:02.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T40qDtvKSbwAklA2YmSuQjKAdTk= 2023-10-12T00:41:02.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T40qDtvKSbwAklA2YmSuQjKAdTk='} 2023-10-12T00:41:02.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006185a87e95dd1c57-006527409e-1689-a2'), ('Date', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:02.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'wafmoa-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:41:02.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'wafmoa-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:41:02.796 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket wafmoa-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:41:02.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:02.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:02.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'wafmoa-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:41:02.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'wafmoa-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:41:02.913 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket wafmoa-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:41:02.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:02.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:03.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-48 2023-10-12T00:41:03.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:03.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:03.030 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:03.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:03.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:03.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:03.121+0000 7fe93d036780 1 waiting for incremental sync to catch up: 2023-10-12T00:41:03.122 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.27.6, , 00000000001.32.6, , 00000000001.22.6, , , 00000000001.33.6, , , ] 2023-10-12T00:41:03.122 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.31.1, 00000000002.29.1, 00000000004.36.1, 00000000002.31.1, 00000000004.27.1, 00000000002.23.1, 00000000002.34.1, 00000000004.38.1, 00000000002.25.1, 00000000002.30.1, 00000000002.20.1] 2023-10-12T00:41:08.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:08.122+0000 7fe93d036780 1 bucket sync caught up with source: 2023-10-12T00:41:08.123 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.31.1, 00000000002.29.1, 00000000004.36.1, 00000000002.31.1, 00000000004.27.1, 00000000002.23.1, 00000000002.34.1, 00000000004.38.1, 00000000002.25.1, 00000000002.30.1, 00000000002.20.1] 2023-10-12T00:41:08.123 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.31.1, 00000000002.29.1, 00000000004.36.1, 00000000002.31.1, 00000000004.27.1, 00000000002.23.1, 00000000002.34.1, 00000000004.38.1, 00000000002.25.1, 00000000002.30.1, 00000000002.20.1] 2023-10-12T00:41:08.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:08.122+0000 7fe93d036780 0 bucket checkpoint complete 2023-10-12T00:41:08.142 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-48 2023-10-12T00:41:08.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:08.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:08.144 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:08.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:08.247+0000 7f281ffbd780 1 waiting for incremental sync to catch up: 2023-10-12T00:41:08.249 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.148.6, , 00000000001.96.6, , 00000000001.228.6, , , 00000000001.138.6, , , ] 2023-10-12T00:41:08.249 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.154.1, 00000000002.193.1, 00000000004.102.1, 00000000002.280.1, 00000000004.238.1, 00000000002.200.1, 00000000002.265.1, 00000000004.148.1, 00000000002.175.1, 00000000002.196.1, 00000000002.159.1] 2023-10-12T00:41:13.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:13.249+0000 7f281ffbd780 1 bucket sync caught up with source: 2023-10-12T00:41:13.250 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.154.1, 00000000002.193.1, 00000000004.102.1, 00000000002.280.1, 00000000004.238.1, 00000000002.200.1, 00000000002.265.1, 00000000004.148.1, 00000000002.175.1, 00000000002.196.1, 00000000002.159.1] 2023-10-12T00:41:13.250 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.154.1, 00000000002.193.1, 00000000004.102.1, 00000000002.280.1, 00000000004.238.1, 00000000002.200.1, 00000000002.265.1, 00000000004.148.1, 00000000002.175.1, 00000000002.196.1, 00000000002.159.1] 2023-10-12T00:41:13.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:13.249+0000 7f281ffbd780 0 bucket checkpoint complete 2023-10-12T00:41:13.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-48 zones={a2, a1} 2023-10-12T00:41:13.267 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.267 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.268 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.268 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:13.268 DEBUG:boto:Data: 2023-10-12T00:41:13.269 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.270 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.270 DEBUG:boto:Params: {} 2023-10-12T00:41:13.271 DEBUG:boto:Token: None 2023-10-12T00:41:13.271 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:peiQU6jcxQYeIEfsjWpNQerHJ9Y= 2023-10-12T00:41:13.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:peiQU6jcxQYeIEfsjWpNQerHJ9Y=', 'Content-Length': '0'} 2023-10-12T00:41:13.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e46992640c713955-00652740a9-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.275 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.275 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.276 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.276 DEBUG:boto:Path: /wafmoa-48/ 2023-10-12T00:41:13.277 DEBUG:boto:Data: 2023-10-12T00:41:13.277 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.278 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.278 DEBUG:boto:Params: {} 2023-10-12T00:41:13.279 DEBUG:boto:Token: None 2023-10-12T00:41:13.279 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:peiQU6jcxQYeIEfsjWpNQerHJ9Y= 2023-10-12T00:41:13.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:peiQU6jcxQYeIEfsjWpNQerHJ9Y=', 'Content-Length': '0'} 2023-10-12T00:41:13.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b3abf756ef5dca3-00652740a9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:41:13.283 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.284 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.284 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.285 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.285 DEBUG:boto:Method: GET 2023-10-12T00:41:13.285 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.286 DEBUG:boto:Data: 2023-10-12T00:41:13.286 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.287 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.288 DEBUG:boto:Params: {} 2023-10-12T00:41:13.288 DEBUG:boto:Token: None 2023-10-12T00:41:13.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/?versions 2023-10-12T00:41:13.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY= 2023-10-12T00:41:13.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY=', 'Content-Length': '0'} 2023-10-12T00:41:13.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae16cf4e7eafd7f3-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.292 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:41:13.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:41:13.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:41:13.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:41:13.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:41:13.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:41:13.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:41:13.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:41:13.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:41:13.297 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.297 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.298 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.298 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.299 DEBUG:boto:Method: GET 2023-10-12T00:41:13.299 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.299 DEBUG:boto:Data: 2023-10-12T00:41:13.300 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.301 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.301 DEBUG:boto:Params: {} 2023-10-12T00:41:13.301 DEBUG:boto:Token: None 2023-10-12T00:41:13.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/?versions 2023-10-12T00:41:13.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY= 2023-10-12T00:41:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY=', 'Content-Length': '0'} 2023-10-12T00:41:13.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b557889bb638184b-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.306 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:41:13.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:41:13.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:41:13.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:41:13.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:41:13.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:41:13.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:41:13.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:41:13.310 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.310 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.310 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.311 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.311 DEBUG:boto:Method: GET 2023-10-12T00:41:13.312 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.312 DEBUG:boto:Data: 2023-10-12T00:41:13.313 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.314 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.314 DEBUG:boto:Params: {} 2023-10-12T00:41:13.314 DEBUG:boto:Token: None 2023-10-12T00:41:13.315 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/?versions 2023-10-12T00:41:13.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY= 2023-10-12T00:41:13.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY=', 'Content-Length': '0'} 2023-10-12T00:41:13.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e56be5c687e2612-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.319 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.320 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.321 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.321 DEBUG:boto:path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.321 DEBUG:boto:auth_path=/wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.322 DEBUG:boto:Method: GET 2023-10-12T00:41:13.322 DEBUG:boto:Path: /wafmoa-48/?versions&max-keys=999 2023-10-12T00:41:13.323 DEBUG:boto:Data: 2023-10-12T00:41:13.323 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.324 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.324 DEBUG:boto:Params: {} 2023-10-12T00:41:13.325 DEBUG:boto:Token: None 2023-10-12T00:41:13.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/?versions 2023-10-12T00:41:13.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY= 2023-10-12T00:41:13.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ftZEqThWDwMKVIQKAruwM4CNyCY=', 'Content-Length': '0'} 2023-10-12T00:41:13.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da2e168aa6ea5df-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.329 DEBUG:boto:b'wafmoa-48999falseobj1nulltrue2023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:13.331 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.331 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.332 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.332 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.333 DEBUG:boto:Method: GET 2023-10-12T00:41:13.333 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.333 DEBUG:boto:Data: 2023-10-12T00:41:13.334 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.335 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.335 DEBUG:boto:Params: {} 2023-10-12T00:41:13.336 DEBUG:boto:Token: None 2023-10-12T00:41:13.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs= 2023-10-12T00:41:13.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs=', 'Content-Length': '0'} 2023-10-12T00:41:13.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d8392ff9c424065-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.340 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.340 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.341 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.341 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.342 DEBUG:boto:Method: GET 2023-10-12T00:41:13.342 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.342 DEBUG:boto:Data: 2023-10-12T00:41:13.343 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.344 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.344 DEBUG:boto:Params: {} 2023-10-12T00:41:13.345 DEBUG:boto:Token: None 2023-10-12T00:41:13.345 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs= 2023-10-12T00:41:13.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs=', 'Content-Length': '0'} 2023-10-12T00:41:13.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029538a7a320322f3-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.349 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.349 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.349 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.350 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.350 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.351 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.351 DEBUG:boto:Data: 2023-10-12T00:41:13.351 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.352 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.353 DEBUG:boto:Params: {} 2023-10-12T00:41:13.353 DEBUG:boto:Token: None 2023-10-12T00:41:13.353 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oS+DCrALnUyRmDUGTTEpkaXu6Jk= 2023-10-12T00:41:13.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oS+DCrALnUyRmDUGTTEpkaXu6Jk=', 'Content-Length': '0'} 2023-10-12T00:41:13.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086be539857cd47b0-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.356 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.357 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.357 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.358 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.358 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.359 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.359 DEBUG:boto:Data: 2023-10-12T00:41:13.359 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.360 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.361 DEBUG:boto:Params: {} 2023-10-12T00:41:13.361 DEBUG:boto:Token: None 2023-10-12T00:41:13.362 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.362 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oS+DCrALnUyRmDUGTTEpkaXu6Jk= 2023-10-12T00:41:13.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oS+DCrALnUyRmDUGTTEpkaXu6Jk=', 'Content-Length': '0'} 2023-10-12T00:41:13.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daa0415133c306cd-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:13.365 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.366 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.366 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.366 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.367 DEBUG:boto:Method: GET 2023-10-12T00:41:13.367 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.368 DEBUG:boto:Data: 2023-10-12T00:41:13.368 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.369 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.369 DEBUG:boto:Params: {} 2023-10-12T00:41:13.370 DEBUG:boto:Token: None 2023-10-12T00:41:13.371 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs= 2023-10-12T00:41:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs=', 'Content-Length': '0'} 2023-10-12T00:41:13.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34de791b2083042-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.414 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.415 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.415 DEBUG:boto:path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.416 DEBUG:boto:auth_path=/wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.416 DEBUG:boto:Method: GET 2023-10-12T00:41:13.417 DEBUG:boto:Path: /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.417 DEBUG:boto:Data: 2023-10-12T00:41:13.418 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.419 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.419 DEBUG:boto:Params: {} 2023-10-12T00:41:13.420 DEBUG:boto:Token: None 2023-10-12T00:41:13.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1?versionId=null 2023-10-12T00:41:13.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs= 2023-10-12T00:41:13.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U5GL/TgV4YrNl/8MHLjWks/bmCs=', 'Content-Length': '0'} 2023-10-12T00:41:13.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009736d56fc5146906-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.424 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.424 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.424 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.425 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.425 DEBUG:boto:Method: GET 2023-10-12T00:41:13.426 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.427 DEBUG:boto:Data: 2023-10-12T00:41:13.427 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.429 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.429 DEBUG:boto:Params: {} 2023-10-12T00:41:13.430 DEBUG:boto:Token: None 2023-10-12T00:41:13.431 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb38141af08301be-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.436 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.437 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.438 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.438 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.439 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.439 DEBUG:boto:Method: GET 2023-10-12T00:41:13.440 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.441 DEBUG:boto:Data: 2023-10-12T00:41:13.441 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.442 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.443 DEBUG:boto:Params: {} 2023-10-12T00:41:13.443 DEBUG:boto:Token: None 2023-10-12T00:41:13.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031b2ac13cbb4b00e-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.449 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.450 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.451 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.452 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.452 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:41:13.453 DEBUG:boto:Data: 2023-10-12T00:41:13.453 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.455 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.455 DEBUG:boto:Params: {} 2023-10-12T00:41:13.458 DEBUG:boto:Token: None 2023-10-12T00:41:13.458 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1 2023-10-12T00:41:13.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5quEgaQX/w1KRpGnGXdhwVlQveI= 2023-10-12T00:41:13.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5quEgaQX/w1KRpGnGXdhwVlQveI=', 'Content-Length': '0'} 2023-10-12T00:41:13.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d2006fa2ed44160-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.463 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.463 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.464 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.464 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:41:13.465 DEBUG:boto:Data: 2023-10-12T00:41:13.466 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.467 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.467 DEBUG:boto:Params: {} 2023-10-12T00:41:13.468 DEBUG:boto:Token: None 2023-10-12T00:41:13.469 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1 2023-10-12T00:41:13.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5quEgaQX/w1KRpGnGXdhwVlQveI= 2023-10-12T00:41:13.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5quEgaQX/w1KRpGnGXdhwVlQveI=', 'Content-Length': '0'} 2023-10-12T00:41:13.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c28d5cc6935c0bc-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:13.473 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.473 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.474 DEBUG:boto:Method: GET 2023-10-12T00:41:13.474 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:41:13.475 DEBUG:boto:Data: 2023-10-12T00:41:13.476 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.477 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.477 DEBUG:boto:Params: {} 2023-10-12T00:41:13.478 DEBUG:boto:Token: None 2023-10-12T00:41:13.478 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1 2023-10-12T00:41:13.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/SPMp/pd9nMmkIXFuqYln8hJQg= 2023-10-12T00:41:13.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/SPMp/pd9nMmkIXFuqYln8hJQg=', 'Content-Length': '0'} 2023-10-12T00:41:13.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931de27cd18bd194-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.484 DEBUG:boto:path=/wafmoa-48/obj1 2023-10-12T00:41:13.485 DEBUG:boto:auth_path=/wafmoa-48/obj1 2023-10-12T00:41:13.486 DEBUG:boto:Method: GET 2023-10-12T00:41:13.487 DEBUG:boto:Path: /wafmoa-48/obj1 2023-10-12T00:41:13.488 DEBUG:boto:Data: 2023-10-12T00:41:13.488 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.490 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.491 DEBUG:boto:Params: {} 2023-10-12T00:41:13.492 DEBUG:boto:Token: None 2023-10-12T00:41:13.492 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj1 2023-10-12T00:41:13.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/SPMp/pd9nMmkIXFuqYln8hJQg= 2023-10-12T00:41:13.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/SPMp/pd9nMmkIXFuqYln8hJQg=', 'Content-Length': '0'} 2023-10-12T00:41:13.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec61def2e9648fd8-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.497 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.498 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.498 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.499 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.499 DEBUG:boto:Method: GET 2023-10-12T00:41:13.500 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.501 DEBUG:boto:Data: 2023-10-12T00:41:13.501 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.502 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.503 DEBUG:boto:Params: {} 2023-10-12T00:41:13.503 DEBUG:boto:Token: None 2023-10-12T00:41:13.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cb035ca369a511-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.508 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.509 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.510 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.511 DEBUG:boto:path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.511 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.512 DEBUG:boto:Method: GET 2023-10-12T00:41:13.513 DEBUG:boto:Path: /wafmoa-48/?prefix=obj1 2023-10-12T00:41:13.513 DEBUG:boto:Data: 2023-10-12T00:41:13.514 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.515 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.516 DEBUG:boto:Params: {} 2023-10-12T00:41:13.517 DEBUG:boto:Token: None 2023-10-12T00:41:13.518 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee6010be1059d791-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.522 DEBUG:boto:b'wafmoa-48obj11000falseobj12023-10-12T00:40:04.009Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:13.524 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.524 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.525 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.525 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.526 DEBUG:boto:Method: GET 2023-10-12T00:41:13.526 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.527 DEBUG:boto:Data: 2023-10-12T00:41:13.527 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.528 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.529 DEBUG:boto:Params: {} 2023-10-12T00:41:13.529 DEBUG:boto:Token: None 2023-10-12T00:41:13.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls= 2023-10-12T00:41:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls=', 'Content-Length': '0'} 2023-10-12T00:41:13.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000825c93da8e6dddeb-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.578 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.579 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.580 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.580 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.581 DEBUG:boto:Method: GET 2023-10-12T00:41:13.581 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.582 DEBUG:boto:Data: 2023-10-12T00:41:13.583 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.584 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.584 DEBUG:boto:Params: {} 2023-10-12T00:41:13.585 DEBUG:boto:Token: None 2023-10-12T00:41:13.585 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls= 2023-10-12T00:41:13.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls=', 'Content-Length': '0'} 2023-10-12T00:41:13.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091e972c42337d9dd-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.589 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.590 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.591 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.591 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.592 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.592 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.593 DEBUG:boto:Data: 2023-10-12T00:41:13.593 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.594 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.595 DEBUG:boto:Params: {} 2023-10-12T00:41:13.596 DEBUG:boto:Token: None 2023-10-12T00:41:13.597 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bPDA+JZuaZxso7ILqTyLX4lQT1o= 2023-10-12T00:41:13.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bPDA+JZuaZxso7ILqTyLX4lQT1o=', 'Content-Length': '0'} 2023-10-12T00:41:13.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018dd38af63f48c62-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.602 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.603 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.604 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.604 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.605 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.605 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.605 DEBUG:boto:Data: 2023-10-12T00:41:13.606 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.607 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.608 DEBUG:boto:Params: {} 2023-10-12T00:41:13.608 DEBUG:boto:Token: None 2023-10-12T00:41:13.609 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bPDA+JZuaZxso7ILqTyLX4lQT1o= 2023-10-12T00:41:13.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bPDA+JZuaZxso7ILqTyLX4lQT1o=', 'Content-Length': '0'} 2023-10-12T00:41:13.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a2955d376dec897-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:13.614 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.615 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.615 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.616 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.616 DEBUG:boto:Method: GET 2023-10-12T00:41:13.617 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.617 DEBUG:boto:Data: 2023-10-12T00:41:13.618 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.619 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.619 DEBUG:boto:Params: {} 2023-10-12T00:41:13.620 DEBUG:boto:Token: None 2023-10-12T00:41:13.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls= 2023-10-12T00:41:13.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls=', 'Content-Length': '0'} 2023-10-12T00:41:13.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a4da827577f2e8c-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.626 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.627 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.627 DEBUG:boto:path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.628 DEBUG:boto:auth_path=/wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.629 DEBUG:boto:Method: GET 2023-10-12T00:41:13.630 DEBUG:boto:Path: /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.631 DEBUG:boto:Data: 2023-10-12T00:41:13.631 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.633 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.633 DEBUG:boto:Params: {} 2023-10-12T00:41:13.634 DEBUG:boto:Token: None 2023-10-12T00:41:13.634 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2?versionId=null 2023-10-12T00:41:13.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls= 2023-10-12T00:41:13.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9nNRSYbdaeA8wR0mVsGUzz1Wls=', 'Content-Length': '0'} 2023-10-12T00:41:13.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018bba4f7aba8f2b5-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.639 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.640 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.641 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.642 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.643 DEBUG:boto:Method: GET 2023-10-12T00:41:13.644 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.645 DEBUG:boto:Data: 2023-10-12T00:41:13.646 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.647 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.648 DEBUG:boto:Params: {} 2023-10-12T00:41:13.649 DEBUG:boto:Token: None 2023-10-12T00:41:13.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001349c36652504490-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.656 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.657 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.658 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.659 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.661 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.662 DEBUG:boto:Method: GET 2023-10-12T00:41:13.663 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.664 DEBUG:boto:Data: 2023-10-12T00:41:13.665 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.667 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.667 DEBUG:boto:Params: {} 2023-10-12T00:41:13.669 DEBUG:boto:Token: None 2023-10-12T00:41:13.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.670 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13c73c78abb40be-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.718 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.720 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.720 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.721 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.722 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:41:13.723 DEBUG:boto:Data: 2023-10-12T00:41:13.723 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.725 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.725 DEBUG:boto:Params: {} 2023-10-12T00:41:13.726 DEBUG:boto:Token: None 2023-10-12T00:41:13.727 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2 2023-10-12T00:41:13.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VySiBZHW8lmNi6cz/zLBqnz7iJI= 2023-10-12T00:41:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VySiBZHW8lmNi6cz/zLBqnz7iJI=', 'Content-Length': '0'} 2023-10-12T00:41:13.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000855d7189de1e8295-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.736 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.736 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.737 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.738 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:41:13.739 DEBUG:boto:Data: 2023-10-12T00:41:13.739 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.741 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.741 DEBUG:boto:Params: {} 2023-10-12T00:41:13.742 DEBUG:boto:Token: None 2023-10-12T00:41:13.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2 2023-10-12T00:41:13.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VySiBZHW8lmNi6cz/zLBqnz7iJI= 2023-10-12T00:41:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VySiBZHW8lmNi6cz/zLBqnz7iJI=', 'Content-Length': '0'} 2023-10-12T00:41:13.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031d3c1cd4362e44f-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:13.749 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.749 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.750 DEBUG:boto:Method: GET 2023-10-12T00:41:13.751 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:41:13.751 DEBUG:boto:Data: 2023-10-12T00:41:13.752 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.753 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.754 DEBUG:boto:Params: {} 2023-10-12T00:41:13.755 DEBUG:boto:Token: None 2023-10-12T00:41:13.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2 2023-10-12T00:41:13.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y4KGq+B/yoNES9VP9GDXT/oLM4k= 2023-10-12T00:41:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y4KGq+B/yoNES9VP9GDXT/oLM4k=', 'Content-Length': '0'} 2023-10-12T00:41:13.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c13ecea37fe8bc78-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.760 DEBUG:boto:path=/wafmoa-48/obj2 2023-10-12T00:41:13.761 DEBUG:boto:auth_path=/wafmoa-48/obj2 2023-10-12T00:41:13.761 DEBUG:boto:Method: GET 2023-10-12T00:41:13.762 DEBUG:boto:Path: /wafmoa-48/obj2 2023-10-12T00:41:13.762 DEBUG:boto:Data: 2023-10-12T00:41:13.763 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.764 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.764 DEBUG:boto:Params: {} 2023-10-12T00:41:13.765 DEBUG:boto:Token: None 2023-10-12T00:41:13.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj2 2023-10-12T00:41:13.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y4KGq+B/yoNES9VP9GDXT/oLM4k= 2023-10-12T00:41:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y4KGq+B/yoNES9VP9GDXT/oLM4k=', 'Content-Length': '0'} 2023-10-12T00:41:13.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033361e787f3d3028-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.769 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.769 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.769 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.770 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.770 DEBUG:boto:Method: GET 2023-10-12T00:41:13.771 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.771 DEBUG:boto:Data: 2023-10-12T00:41:13.772 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.773 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.773 DEBUG:boto:Params: {} 2023-10-12T00:41:13.773 DEBUG:boto:Token: None 2023-10-12T00:41:13.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b97613030baaba-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.778 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.779 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.779 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.780 DEBUG:boto:path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.780 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.780 DEBUG:boto:Method: GET 2023-10-12T00:41:13.781 DEBUG:boto:Path: /wafmoa-48/?prefix=obj2 2023-10-12T00:41:13.781 DEBUG:boto:Data: 2023-10-12T00:41:13.782 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.783 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.783 DEBUG:boto:Params: {} 2023-10-12T00:41:13.784 DEBUG:boto:Token: None 2023-10-12T00:41:13.784 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004714037bc993cbf-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.788 DEBUG:boto:b'wafmoa-48obj21000falseobj22023-10-12T00:40:04.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:13.789 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.790 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.791 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.791 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.792 DEBUG:boto:Method: GET 2023-10-12T00:41:13.792 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.793 DEBUG:boto:Data: 2023-10-12T00:41:13.793 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.794 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.795 DEBUG:boto:Params: {} 2023-10-12T00:41:13.795 DEBUG:boto:Token: None 2023-10-12T00:41:13.796 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8= 2023-10-12T00:41:13.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8=', 'Content-Length': '0'} 2023-10-12T00:41:13.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f7f670dac910a8a-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.800 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.801 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.802 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.802 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.803 DEBUG:boto:Method: GET 2023-10-12T00:41:13.803 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.804 DEBUG:boto:Data: 2023-10-12T00:41:13.804 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.805 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.806 DEBUG:boto:Params: {} 2023-10-12T00:41:13.807 DEBUG:boto:Token: None 2023-10-12T00:41:13.807 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8= 2023-10-12T00:41:13.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8=', 'Content-Length': '0'} 2023-10-12T00:41:13.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055c8615dd5c09dc2-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.858 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.859 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.860 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.860 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.861 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.861 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.862 DEBUG:boto:Data: 2023-10-12T00:41:13.862 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.863 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.864 DEBUG:boto:Params: {} 2023-10-12T00:41:13.864 DEBUG:boto:Token: None 2023-10-12T00:41:13.865 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aameilbpc1UotTKfHj7MZYF7Asc= 2023-10-12T00:41:13.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aameilbpc1UotTKfHj7MZYF7Asc=', 'Content-Length': '0'} 2023-10-12T00:41:13.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016dc34ffcac6ab19-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.870 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.870 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.871 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.871 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.871 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.872 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.872 DEBUG:boto:Data: 2023-10-12T00:41:13.873 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.873 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.874 DEBUG:boto:Params: {} 2023-10-12T00:41:13.874 DEBUG:boto:Token: None 2023-10-12T00:41:13.875 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aameilbpc1UotTKfHj7MZYF7Asc= 2023-10-12T00:41:13.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aameilbpc1UotTKfHj7MZYF7Asc=', 'Content-Length': '0'} 2023-10-12T00:41:13.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0fbead411ff2002-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:13.879 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.879 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.880 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.880 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.881 DEBUG:boto:Method: GET 2023-10-12T00:41:13.881 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.882 DEBUG:boto:Data: 2023-10-12T00:41:13.882 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.883 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.884 DEBUG:boto:Params: {} 2023-10-12T00:41:13.884 DEBUG:boto:Token: None 2023-10-12T00:41:13.885 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8= 2023-10-12T00:41:13.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8=', 'Content-Length': '0'} 2023-10-12T00:41:13.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4deccf32b4b5423-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.888 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.889 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.889 DEBUG:boto:path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.890 DEBUG:boto:auth_path=/wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.890 DEBUG:boto:Method: GET 2023-10-12T00:41:13.890 DEBUG:boto:Path: /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.891 DEBUG:boto:Data: 2023-10-12T00:41:13.891 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.892 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.893 DEBUG:boto:Params: {} 2023-10-12T00:41:13.893 DEBUG:boto:Token: None 2023-10-12T00:41:13.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3?versionId=null 2023-10-12T00:41:13.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8= 2023-10-12T00:41:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8B4G6hJaDRUbR2Sc40pIAzPPPt8=', 'Content-Length': '0'} 2023-10-12T00:41:13.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1255b8152a3816c-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.897 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.898 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.898 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.898 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.899 DEBUG:boto:Method: GET 2023-10-12T00:41:13.899 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.900 DEBUG:boto:Data: 2023-10-12T00:41:13.900 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.901 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.902 DEBUG:boto:Params: {} 2023-10-12T00:41:13.902 DEBUG:boto:Token: None 2023-10-12T00:41:13.903 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fcca3bde9192d6-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.906 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.907 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.908 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.908 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.909 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.909 DEBUG:boto:Method: GET 2023-10-12T00:41:13.909 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.910 DEBUG:boto:Data: 2023-10-12T00:41:13.910 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.911 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.912 DEBUG:boto:Params: {} 2023-10-12T00:41:13.912 DEBUG:boto:Token: None 2023-10-12T00:41:13.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000961c5be8e73d31de-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.917 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.917 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.918 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.918 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.919 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:41:13.919 DEBUG:boto:Data: 2023-10-12T00:41:13.920 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.920 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.921 DEBUG:boto:Params: {} 2023-10-12T00:41:13.921 DEBUG:boto:Token: None 2023-10-12T00:41:13.922 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3 2023-10-12T00:41:13.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMjkCJKRqT4VFbVNnnErSUc8Jsw= 2023-10-12T00:41:13.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMjkCJKRqT4VFbVNnnErSUc8Jsw=', 'Content-Length': '0'} 2023-10-12T00:41:13.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a6cba6da6d4387e-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.925 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.926 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.926 DEBUG:boto:Method: HEAD 2023-10-12T00:41:13.927 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:41:13.927 DEBUG:boto:Data: 2023-10-12T00:41:13.927 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.928 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.929 DEBUG:boto:Params: {} 2023-10-12T00:41:13.929 DEBUG:boto:Token: None 2023-10-12T00:41:13.930 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3 2023-10-12T00:41:13.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMjkCJKRqT4VFbVNnnErSUc8Jsw= 2023-10-12T00:41:13.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMjkCJKRqT4VFbVNnnErSUc8Jsw=', 'Content-Length': '0'} 2023-10-12T00:41:13.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d02f93d7866b68d3-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:13.933 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.934 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.934 DEBUG:boto:Method: GET 2023-10-12T00:41:13.935 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:41:13.935 DEBUG:boto:Data: 2023-10-12T00:41:13.935 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.936 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.937 DEBUG:boto:Params: {} 2023-10-12T00:41:13.937 DEBUG:boto:Token: None 2023-10-12T00:41:13.938 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3 2023-10-12T00:41:13.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PkVSRYXOZ5xXOo3K6DBODcisuY0= 2023-10-12T00:41:13.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PkVSRYXOZ5xXOo3K6DBODcisuY0=', 'Content-Length': '0'} 2023-10-12T00:41:13.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaa146c359d841d1-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.941 DEBUG:boto:path=/wafmoa-48/obj3 2023-10-12T00:41:13.942 DEBUG:boto:auth_path=/wafmoa-48/obj3 2023-10-12T00:41:13.942 DEBUG:boto:Method: GET 2023-10-12T00:41:13.942 DEBUG:boto:Path: /wafmoa-48/obj3 2023-10-12T00:41:13.943 DEBUG:boto:Data: 2023-10-12T00:41:13.943 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.944 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.945 DEBUG:boto:Params: {} 2023-10-12T00:41:13.945 DEBUG:boto:Token: None 2023-10-12T00:41:13.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj3 2023-10-12T00:41:13.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PkVSRYXOZ5xXOo3K6DBODcisuY0= 2023-10-12T00:41:13.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PkVSRYXOZ5xXOo3K6DBODcisuY0=', 'Content-Length': '0'} 2023-10-12T00:41:13.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049cb4bcf5c906ac5-00652740a9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.949 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.950 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.951 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.951 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.951 DEBUG:boto:Method: GET 2023-10-12T00:41:13.952 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.952 DEBUG:boto:Data: 2023-10-12T00:41:13.953 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.954 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.954 DEBUG:boto:Params: {} 2023-10-12T00:41:13.955 DEBUG:boto:Token: None 2023-10-12T00:41:13.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a15bc1fc98e34a9-00652740a9-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.960 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.961 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:13.961 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:13.962 DEBUG:boto:path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.962 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.963 DEBUG:boto:Method: GET 2023-10-12T00:41:13.964 DEBUG:boto:Path: /wafmoa-48/?prefix=obj3 2023-10-12T00:41:13.964 DEBUG:boto:Data: 2023-10-12T00:41:13.965 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:13.966 DEBUG:boto:Port: 8000 2023-10-12T00:41:13.966 DEBUG:boto:Params: {} 2023-10-12T00:41:13.967 DEBUG:boto:Token: None 2023-10-12T00:41:13.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/ 2023-10-12T00:41:13.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ= 2023-10-12T00:41:13.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/6qU5081ViHu8T78zq2/mCjHpQ=', 'Content-Length': '0'} 2023-10-12T00:41:13.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adda1bd25d4b81dd-00652740a9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:13.971 DEBUG:boto:b'wafmoa-48obj31000falseobj32023-10-12T00:40:04.088Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:13.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:13.972 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:13.973 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:13.973 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:13.973 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:13.974 DEBUG:boto:Method: GET 2023-10-12T00:41:13.974 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:13.975 DEBUG:boto:Data: 2023-10-12T00:41:13.975 DEBUG:boto:Headers: {} 2023-10-12T00:41:13.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:13.976 DEBUG:boto:Port: 8001 2023-10-12T00:41:13.977 DEBUG:boto:Params: {} 2023-10-12T00:41:13.977 DEBUG:boto:Token: None 2023-10-12T00:41:13.978 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:13 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:13.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bIPoQNFUbybVcWGw/OOHa7XkJBM= 2023-10-12T00:41:13.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bIPoQNFUbybVcWGw/OOHa7XkJBM=', 'Content-Length': '0'} 2023-10-12T00:41:13.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094cb349efeacf720-00652740a9-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.027 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.028 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.029 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.030 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.031 DEBUG:boto:Method: GET 2023-10-12T00:41:14.032 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.033 DEBUG:boto:Data: 2023-10-12T00:41:14.033 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.035 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.036 DEBUG:boto:Params: {} 2023-10-12T00:41:14.037 DEBUG:boto:Token: None 2023-10-12T00:41:14.038 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A= 2023-10-12T00:41:14.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A=', 'Content-Length': '0'} 2023-10-12T00:41:14.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f981201932cad0c-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.043 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.043 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.044 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.044 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.044 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.045 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.045 DEBUG:boto:Data: 2023-10-12T00:41:14.046 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.047 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.047 DEBUG:boto:Params: {} 2023-10-12T00:41:14.047 DEBUG:boto:Token: None 2023-10-12T00:41:14.048 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/ogiercbUl5blz0kc/T90PX7Oek= 2023-10-12T00:41:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/ogiercbUl5blz0kc/T90PX7Oek=', 'Content-Length': '0'} 2023-10-12T00:41:14.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b8039d950abc5b3-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.052 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.052 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.053 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.053 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.054 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.054 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.055 DEBUG:boto:Data: 2023-10-12T00:41:14.055 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.056 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.057 DEBUG:boto:Params: {} 2023-10-12T00:41:14.058 DEBUG:boto:Token: None 2023-10-12T00:41:14.058 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/ogiercbUl5blz0kc/T90PX7Oek= 2023-10-12T00:41:14.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/ogiercbUl5blz0kc/T90PX7Oek=', 'Content-Length': '0'} 2023-10-12T00:41:14.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1d0151f40f8b1b8-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:14.062 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.063 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.063 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.064 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.064 DEBUG:boto:Method: GET 2023-10-12T00:41:14.065 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.065 DEBUG:boto:Data: 2023-10-12T00:41:14.066 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.066 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.067 DEBUG:boto:Params: {} 2023-10-12T00:41:14.067 DEBUG:boto:Token: None 2023-10-12T00:41:14.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A= 2023-10-12T00:41:14.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A=', 'Content-Length': '0'} 2023-10-12T00:41:14.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4d6e1ee3bbe27b8-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.072 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.072 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.073 DEBUG:boto:path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.073 DEBUG:boto:auth_path=/wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.074 DEBUG:boto:Method: GET 2023-10-12T00:41:14.074 DEBUG:boto:Path: /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.078 DEBUG:boto:Data: 2023-10-12T00:41:14.078 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.079 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.079 DEBUG:boto:Params: {} 2023-10-12T00:41:14.080 DEBUG:boto:Token: None 2023-10-12T00:41:14.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4?versionId=null 2023-10-12T00:41:14.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A= 2023-10-12T00:41:14.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M0X0hVbaJnC+py4HTXfgi0PQN3A=', 'Content-Length': '0'} 2023-10-12T00:41:14.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000973ec041d487bfc2-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.084 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.085 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.085 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.086 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.086 DEBUG:boto:Method: GET 2023-10-12T00:41:14.086 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.087 DEBUG:boto:Data: 2023-10-12T00:41:14.087 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.088 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.089 DEBUG:boto:Params: {} 2023-10-12T00:41:14.089 DEBUG:boto:Token: None 2023-10-12T00:41:14.090 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007591d9f19eb15525-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.134 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.135 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.136 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.136 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.137 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.137 DEBUG:boto:Method: GET 2023-10-12T00:41:14.138 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.138 DEBUG:boto:Data: 2023-10-12T00:41:14.138 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.139 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.140 DEBUG:boto:Params: {} 2023-10-12T00:41:14.140 DEBUG:boto:Token: None 2023-10-12T00:41:14.141 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1a2e8d116550d5-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.145 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.147 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.148 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.148 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.149 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:41:14.149 DEBUG:boto:Data: 2023-10-12T00:41:14.150 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.151 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.151 DEBUG:boto:Params: {} 2023-10-12T00:41:14.152 DEBUG:boto:Token: None 2023-10-12T00:41:14.152 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4 2023-10-12T00:41:14.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UK47Gg4RjVDKhO8ycwsLkxs8Up0= 2023-10-12T00:41:14.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UK47Gg4RjVDKhO8ycwsLkxs8Up0=', 'Content-Length': '0'} 2023-10-12T00:41:14.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c2a7a88e121977-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.156 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.157 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.157 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.157 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:41:14.158 DEBUG:boto:Data: 2023-10-12T00:41:14.158 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.159 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.160 DEBUG:boto:Params: {} 2023-10-12T00:41:14.160 DEBUG:boto:Token: None 2023-10-12T00:41:14.161 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4 2023-10-12T00:41:14.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UK47Gg4RjVDKhO8ycwsLkxs8Up0= 2023-10-12T00:41:14.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UK47Gg4RjVDKhO8ycwsLkxs8Up0=', 'Content-Length': '0'} 2023-10-12T00:41:14.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ef7763e950acca0-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:14.166 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.166 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.167 DEBUG:boto:Method: GET 2023-10-12T00:41:14.168 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:41:14.169 DEBUG:boto:Data: 2023-10-12T00:41:14.170 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.172 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.172 DEBUG:boto:Params: {} 2023-10-12T00:41:14.173 DEBUG:boto:Token: None 2023-10-12T00:41:14.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4 2023-10-12T00:41:14.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zrlfwi4afZIjSwGBXth9nFvua94= 2023-10-12T00:41:14.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zrlfwi4afZIjSwGBXth9nFvua94=', 'Content-Length': '0'} 2023-10-12T00:41:14.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042224efcac5754eb-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.179 DEBUG:boto:path=/wafmoa-48/obj4 2023-10-12T00:41:14.180 DEBUG:boto:auth_path=/wafmoa-48/obj4 2023-10-12T00:41:14.182 DEBUG:boto:Method: GET 2023-10-12T00:41:14.183 DEBUG:boto:Path: /wafmoa-48/obj4 2023-10-12T00:41:14.185 DEBUG:boto:Data: 2023-10-12T00:41:14.186 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.189 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.190 DEBUG:boto:Params: {} 2023-10-12T00:41:14.192 DEBUG:boto:Token: None 2023-10-12T00:41:14.193 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj4 2023-10-12T00:41:14.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zrlfwi4afZIjSwGBXth9nFvua94= 2023-10-12T00:41:14.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zrlfwi4afZIjSwGBXth9nFvua94=', 'Content-Length': '0'} 2023-10-12T00:41:14.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002af5a1399c81b67b-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.202 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.203 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.204 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.205 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.206 DEBUG:boto:Method: GET 2023-10-12T00:41:14.207 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.208 DEBUG:boto:Data: 2023-10-12T00:41:14.209 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.211 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.213 DEBUG:boto:Params: {} 2023-10-12T00:41:14.214 DEBUG:boto:Token: None 2023-10-12T00:41:14.216 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325aaaad738cd599-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.223 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.224 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.225 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.225 DEBUG:boto:path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.226 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.227 DEBUG:boto:Method: GET 2023-10-12T00:41:14.228 DEBUG:boto:Path: /wafmoa-48/?prefix=obj4 2023-10-12T00:41:14.228 DEBUG:boto:Data: 2023-10-12T00:41:14.229 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.231 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.231 DEBUG:boto:Params: {} 2023-10-12T00:41:14.232 DEBUG:boto:Token: None 2023-10-12T00:41:14.233 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.234 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047efe8d50f7f9cf4-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.239 DEBUG:boto:b'wafmoa-48obj41000falseobj42023-10-12T00:40:04.106Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:14.242 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.244 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.245 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.246 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.246 DEBUG:boto:Method: GET 2023-10-12T00:41:14.247 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.248 DEBUG:boto:Data: 2023-10-12T00:41:14.249 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.250 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.251 DEBUG:boto:Params: {} 2023-10-12T00:41:14.252 DEBUG:boto:Token: None 2023-10-12T00:41:14.253 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ= 2023-10-12T00:41:14.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2e938cf6f8013b8-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.258 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.259 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.260 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.260 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.261 DEBUG:boto:Method: GET 2023-10-12T00:41:14.262 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.263 DEBUG:boto:Data: 2023-10-12T00:41:14.264 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.265 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.266 DEBUG:boto:Params: {} 2023-10-12T00:41:14.267 DEBUG:boto:Token: None 2023-10-12T00:41:14.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.269 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ= 2023-10-12T00:41:14.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000274aac6391cd6adc-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.276 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.277 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.278 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.278 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.279 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.279 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.280 DEBUG:boto:Data: 2023-10-12T00:41:14.280 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.281 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.281 DEBUG:boto:Params: {} 2023-10-12T00:41:14.282 DEBUG:boto:Token: None 2023-10-12T00:41:14.282 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hD/CeVnKrr9hhbyA0ev/VzFe6F4= 2023-10-12T00:41:14.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hD/CeVnKrr9hhbyA0ev/VzFe6F4=', 'Content-Length': '0'} 2023-10-12T00:41:14.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a1e9ed5182341c9-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.286 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.287 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.287 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.288 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.288 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.288 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.289 DEBUG:boto:Data: 2023-10-12T00:41:14.289 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.290 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.290 DEBUG:boto:Params: {} 2023-10-12T00:41:14.291 DEBUG:boto:Token: None 2023-10-12T00:41:14.291 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hD/CeVnKrr9hhbyA0ev/VzFe6F4= 2023-10-12T00:41:14.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hD/CeVnKrr9hhbyA0ev/VzFe6F4=', 'Content-Length': '0'} 2023-10-12T00:41:14.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e59ca38386f74c2b-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:14.295 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.296 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.296 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.296 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.297 DEBUG:boto:Method: GET 2023-10-12T00:41:14.297 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.298 DEBUG:boto:Data: 2023-10-12T00:41:14.298 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.299 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.300 DEBUG:boto:Params: {} 2023-10-12T00:41:14.300 DEBUG:boto:Token: None 2023-10-12T00:41:14.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ= 2023-10-12T00:41:14.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076e731eead637e4b-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.304 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.305 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.305 DEBUG:boto:path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.306 DEBUG:boto:auth_path=/wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.306 DEBUG:boto:Method: GET 2023-10-12T00:41:14.307 DEBUG:boto:Path: /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.307 DEBUG:boto:Data: 2023-10-12T00:41:14.308 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.309 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.309 DEBUG:boto:Params: {} 2023-10-12T00:41:14.310 DEBUG:boto:Token: None 2023-10-12T00:41:14.310 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5?versionId=null 2023-10-12T00:41:14.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ= 2023-10-12T00:41:14.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tEj0gwauFSdBBSVrlto8ysjUbmQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e139370161b32271-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.314 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.315 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.315 DEBUG:boto:path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.316 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.316 DEBUG:boto:Method: GET 2023-10-12T00:41:14.317 DEBUG:boto:Path: /wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.317 DEBUG:boto:Data: 2023-10-12T00:41:14.318 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.319 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.319 DEBUG:boto:Params: {} 2023-10-12T00:41:14.320 DEBUG:boto:Token: None 2023-10-12T00:41:14.321 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd56e51d7284724-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.325 DEBUG:boto:b'wafmoa-48obj51000falseobj52023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.326 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.326 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.326 DEBUG:boto:path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.327 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.327 DEBUG:boto:Method: GET 2023-10-12T00:41:14.328 DEBUG:boto:Path: /wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.328 DEBUG:boto:Data: 2023-10-12T00:41:14.328 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.329 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.330 DEBUG:boto:Params: {} 2023-10-12T00:41:14.330 DEBUG:boto:Token: None 2023-10-12T00:41:14.331 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8245a4254d6c26-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.335 DEBUG:boto:b'wafmoa-48obj51000falseobj52023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.336 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.336 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.337 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.337 DEBUG:boto:Path: /wafmoa-48/obj5 2023-10-12T00:41:14.338 DEBUG:boto:Data: 2023-10-12T00:41:14.338 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.339 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.340 DEBUG:boto:Params: {} 2023-10-12T00:41:14.340 DEBUG:boto:Token: None 2023-10-12T00:41:14.341 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5 2023-10-12T00:41:14.341 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydV2fWqgMxEqZbt7savVEWmNNWQ= 2023-10-12T00:41:14.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydV2fWqgMxEqZbt7savVEWmNNWQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036d27fb9903593a6-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.344 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.345 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.345 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.345 DEBUG:boto:Path: /wafmoa-48/obj5 2023-10-12T00:41:14.346 DEBUG:boto:Data: 2023-10-12T00:41:14.346 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.347 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.347 DEBUG:boto:Params: {} 2023-10-12T00:41:14.348 DEBUG:boto:Token: None 2023-10-12T00:41:14.348 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5 2023-10-12T00:41:14.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydV2fWqgMxEqZbt7savVEWmNNWQ= 2023-10-12T00:41:14.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydV2fWqgMxEqZbt7savVEWmNNWQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb2c296d13fa83de-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:14.352 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.353 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.353 DEBUG:boto:Method: GET 2023-10-12T00:41:14.354 DEBUG:boto:Path: /wafmoa-48/obj5 2023-10-12T00:41:14.354 DEBUG:boto:Data: 2023-10-12T00:41:14.355 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.355 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.356 DEBUG:boto:Params: {} 2023-10-12T00:41:14.356 DEBUG:boto:Token: None 2023-10-12T00:41:14.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5 2023-10-12T00:41:14.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VG3FtuSOiiEU9C92cgetBSMinJI= 2023-10-12T00:41:14.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VG3FtuSOiiEU9C92cgetBSMinJI=', 'Content-Length': '0'} 2023-10-12T00:41:14.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012cae0febfc7dba3-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.361 DEBUG:boto:path=/wafmoa-48/obj5 2023-10-12T00:41:14.361 DEBUG:boto:auth_path=/wafmoa-48/obj5 2023-10-12T00:41:14.362 DEBUG:boto:Method: GET 2023-10-12T00:41:14.362 DEBUG:boto:Path: /wafmoa-48/obj5 2023-10-12T00:41:14.363 DEBUG:boto:Data: 2023-10-12T00:41:14.363 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.364 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.364 DEBUG:boto:Params: {} 2023-10-12T00:41:14.365 DEBUG:boto:Token: None 2023-10-12T00:41:14.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj5 2023-10-12T00:41:14.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VG3FtuSOiiEU9C92cgetBSMinJI= 2023-10-12T00:41:14.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VG3FtuSOiiEU9C92cgetBSMinJI=', 'Content-Length': '0'} 2023-10-12T00:41:14.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070741869ea0f083b-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.414 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.415 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.415 DEBUG:boto:path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.416 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.416 DEBUG:boto:Method: GET 2023-10-12T00:41:14.417 DEBUG:boto:Path: /wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.417 DEBUG:boto:Data: 2023-10-12T00:41:14.418 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.419 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.419 DEBUG:boto:Params: {} 2023-10-12T00:41:14.420 DEBUG:boto:Token: None 2023-10-12T00:41:14.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f324a9080b24363-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.425 DEBUG:boto:b'wafmoa-48obj51000falseobj52023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.425 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.426 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.426 DEBUG:boto:path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.427 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.427 DEBUG:boto:Method: GET 2023-10-12T00:41:14.427 DEBUG:boto:Path: /wafmoa-48/?prefix=obj5 2023-10-12T00:41:14.428 DEBUG:boto:Data: 2023-10-12T00:41:14.429 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.429 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.430 DEBUG:boto:Params: {} 2023-10-12T00:41:14.431 DEBUG:boto:Token: None 2023-10-12T00:41:14.431 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c5f5bfc3fa38db6-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.435 DEBUG:boto:b'wafmoa-48obj51000falseobj52023-10-12T00:41:02.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:14.436 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.436 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.437 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.437 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.438 DEBUG:boto:Method: GET 2023-10-12T00:41:14.438 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.439 DEBUG:boto:Data: 2023-10-12T00:41:14.439 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.440 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.441 DEBUG:boto:Params: {} 2023-10-12T00:41:14.441 DEBUG:boto:Token: None 2023-10-12T00:41:14.442 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM= 2023-10-12T00:41:14.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM=', 'Content-Length': '0'} 2023-10-12T00:41:14.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000767fbafb9d9c80fd-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.445 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.446 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.446 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.447 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.447 DEBUG:boto:Method: GET 2023-10-12T00:41:14.448 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.448 DEBUG:boto:Data: 2023-10-12T00:41:14.448 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.449 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.450 DEBUG:boto:Params: {} 2023-10-12T00:41:14.450 DEBUG:boto:Token: None 2023-10-12T00:41:14.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM= 2023-10-12T00:41:14.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM=', 'Content-Length': '0'} 2023-10-12T00:41:14.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdda1c161d6ec3ff-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.454 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.455 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.455 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.456 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.456 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.456 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.457 DEBUG:boto:Data: 2023-10-12T00:41:14.457 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.458 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.458 DEBUG:boto:Params: {} 2023-10-12T00:41:14.459 DEBUG:boto:Token: None 2023-10-12T00:41:14.459 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zOKslQU1JD4SlEZF7zFcmflGC98= 2023-10-12T00:41:14.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zOKslQU1JD4SlEZF7zFcmflGC98=', 'Content-Length': '0'} 2023-10-12T00:41:14.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfddf9da0d4003a-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.463 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.463 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.464 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.464 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.464 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.465 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.465 DEBUG:boto:Data: 2023-10-12T00:41:14.466 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.467 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.467 DEBUG:boto:Params: {} 2023-10-12T00:41:14.468 DEBUG:boto:Token: None 2023-10-12T00:41:14.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zOKslQU1JD4SlEZF7zFcmflGC98= 2023-10-12T00:41:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zOKslQU1JD4SlEZF7zFcmflGC98=', 'Content-Length': '0'} 2023-10-12T00:41:14.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0aa31e028bacd26-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:14.472 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.473 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.473 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.474 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.474 DEBUG:boto:Method: GET 2023-10-12T00:41:14.475 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.475 DEBUG:boto:Data: 2023-10-12T00:41:14.476 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.476 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.477 DEBUG:boto:Params: {} 2023-10-12T00:41:14.477 DEBUG:boto:Token: None 2023-10-12T00:41:14.478 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM= 2023-10-12T00:41:14.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM=', 'Content-Length': '0'} 2023-10-12T00:41:14.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6f6169210b2a92f-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.522 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.523 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.524 DEBUG:boto:path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.524 DEBUG:boto:auth_path=/wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.524 DEBUG:boto:Method: GET 2023-10-12T00:41:14.525 DEBUG:boto:Path: /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.525 DEBUG:boto:Data: 2023-10-12T00:41:14.526 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.527 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.527 DEBUG:boto:Params: {} 2023-10-12T00:41:14.527 DEBUG:boto:Token: None 2023-10-12T00:41:14.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6?versionId=null 2023-10-12T00:41:14.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM= 2023-10-12T00:41:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHPICL/pxKQDFyT0eJfCYX+PzDM=', 'Content-Length': '0'} 2023-10-12T00:41:14.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000475a23ba4057d18c-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.532 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.532 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.533 DEBUG:boto:path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.533 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.533 DEBUG:boto:Method: GET 2023-10-12T00:41:14.534 DEBUG:boto:Path: /wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.534 DEBUG:boto:Data: 2023-10-12T00:41:14.535 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.536 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.536 DEBUG:boto:Params: {} 2023-10-12T00:41:14.536 DEBUG:boto:Token: None 2023-10-12T00:41:14.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3ee4fd955c1d510-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.541 DEBUG:boto:b'wafmoa-48obj61000falseobj62023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.542 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.542 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.543 DEBUG:boto:path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.543 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.543 DEBUG:boto:Method: GET 2023-10-12T00:41:14.544 DEBUG:boto:Path: /wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.544 DEBUG:boto:Data: 2023-10-12T00:41:14.545 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.546 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.546 DEBUG:boto:Params: {} 2023-10-12T00:41:14.546 DEBUG:boto:Token: None 2023-10-12T00:41:14.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa64b37de9120f4-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.594 DEBUG:boto:b'wafmoa-48obj61000falseobj62023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.595 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.596 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.596 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.596 DEBUG:boto:Path: /wafmoa-48/obj6 2023-10-12T00:41:14.597 DEBUG:boto:Data: 2023-10-12T00:41:14.597 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.598 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.599 DEBUG:boto:Params: {} 2023-10-12T00:41:14.599 DEBUG:boto:Token: None 2023-10-12T00:41:14.599 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6 2023-10-12T00:41:14.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qIhU0c+ACYlNmAsbT7WhsWNUqlw= 2023-10-12T00:41:14.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qIhU0c+ACYlNmAsbT7WhsWNUqlw=', 'Content-Length': '0'} 2023-10-12T00:41:14.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd786ff8861fea94-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.603 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.603 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.604 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.604 DEBUG:boto:Path: /wafmoa-48/obj6 2023-10-12T00:41:14.605 DEBUG:boto:Data: 2023-10-12T00:41:14.605 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.606 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.606 DEBUG:boto:Params: {} 2023-10-12T00:41:14.607 DEBUG:boto:Token: None 2023-10-12T00:41:14.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6 2023-10-12T00:41:14.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qIhU0c+ACYlNmAsbT7WhsWNUqlw= 2023-10-12T00:41:14.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qIhU0c+ACYlNmAsbT7WhsWNUqlw=', 'Content-Length': '0'} 2023-10-12T00:41:14.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7f809f4facdf94d-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:14.611 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.612 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.612 DEBUG:boto:Method: GET 2023-10-12T00:41:14.613 DEBUG:boto:Path: /wafmoa-48/obj6 2023-10-12T00:41:14.613 DEBUG:boto:Data: 2023-10-12T00:41:14.613 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.614 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.615 DEBUG:boto:Params: {} 2023-10-12T00:41:14.615 DEBUG:boto:Token: None 2023-10-12T00:41:14.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6 2023-10-12T00:41:14.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:15M9c+qd7DA0+s9Qik/+8tRjaUc= 2023-10-12T00:41:14.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:15M9c+qd7DA0+s9Qik/+8tRjaUc=', 'Content-Length': '0'} 2023-10-12T00:41:14.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4762536ff35a360-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.620 DEBUG:boto:path=/wafmoa-48/obj6 2023-10-12T00:41:14.620 DEBUG:boto:auth_path=/wafmoa-48/obj6 2023-10-12T00:41:14.621 DEBUG:boto:Method: GET 2023-10-12T00:41:14.621 DEBUG:boto:Path: /wafmoa-48/obj6 2023-10-12T00:41:14.621 DEBUG:boto:Data: 2023-10-12T00:41:14.622 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.623 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.623 DEBUG:boto:Params: {} 2023-10-12T00:41:14.624 DEBUG:boto:Token: None 2023-10-12T00:41:14.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj6 2023-10-12T00:41:14.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:15M9c+qd7DA0+s9Qik/+8tRjaUc= 2023-10-12T00:41:14.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:15M9c+qd7DA0+s9Qik/+8tRjaUc=', 'Content-Length': '0'} 2023-10-12T00:41:14.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005902a66315ea2383-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.628 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.628 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.628 DEBUG:boto:path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.629 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.629 DEBUG:boto:Method: GET 2023-10-12T00:41:14.630 DEBUG:boto:Path: /wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.630 DEBUG:boto:Data: 2023-10-12T00:41:14.631 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.632 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.632 DEBUG:boto:Params: {} 2023-10-12T00:41:14.632 DEBUG:boto:Token: None 2023-10-12T00:41:14.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d6a10bd0767720-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.682 DEBUG:boto:b'wafmoa-48obj61000falseobj62023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.683 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.684 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.684 DEBUG:boto:path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.685 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.685 DEBUG:boto:Method: GET 2023-10-12T00:41:14.686 DEBUG:boto:Path: /wafmoa-48/?prefix=obj6 2023-10-12T00:41:14.686 DEBUG:boto:Data: 2023-10-12T00:41:14.686 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.690 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.690 DEBUG:boto:Params: {} 2023-10-12T00:41:14.691 DEBUG:boto:Token: None 2023-10-12T00:41:14.691 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4b9bb7aa8a35c0-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.696 DEBUG:boto:b'wafmoa-48obj61000falseobj62023-10-12T00:41:02.523Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:14.697 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.697 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.698 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.698 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.698 DEBUG:boto:Method: GET 2023-10-12T00:41:14.699 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.699 DEBUG:boto:Data: 2023-10-12T00:41:14.700 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.701 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.701 DEBUG:boto:Params: {} 2023-10-12T00:41:14.702 DEBUG:boto:Token: None 2023-10-12T00:41:14.702 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94= 2023-10-12T00:41:14.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94=', 'Content-Length': '0'} 2023-10-12T00:41:14.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d4036c210961dd9-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.706 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.707 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.707 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.707 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.708 DEBUG:boto:Method: GET 2023-10-12T00:41:14.708 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.709 DEBUG:boto:Data: 2023-10-12T00:41:14.709 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.710 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.711 DEBUG:boto:Params: {} 2023-10-12T00:41:14.711 DEBUG:boto:Token: None 2023-10-12T00:41:14.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94= 2023-10-12T00:41:14.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94=', 'Content-Length': '0'} 2023-10-12T00:41:14.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c6375c15a02cfac-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.758 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.759 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.759 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.760 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.760 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.760 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.761 DEBUG:boto:Data: 2023-10-12T00:41:14.761 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.762 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.762 DEBUG:boto:Params: {} 2023-10-12T00:41:14.763 DEBUG:boto:Token: None 2023-10-12T00:41:14.763 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5dpZYXTv9u9657Xp64R4EiLwkdo= 2023-10-12T00:41:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5dpZYXTv9u9657Xp64R4EiLwkdo=', 'Content-Length': '0'} 2023-10-12T00:41:14.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ae9d971d1df42be-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.767 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.767 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.767 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.768 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.768 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.769 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.769 DEBUG:boto:Data: 2023-10-12T00:41:14.769 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.770 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.771 DEBUG:boto:Params: {} 2023-10-12T00:41:14.771 DEBUG:boto:Token: None 2023-10-12T00:41:14.772 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5dpZYXTv9u9657Xp64R4EiLwkdo= 2023-10-12T00:41:14.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5dpZYXTv9u9657Xp64R4EiLwkdo=', 'Content-Length': '0'} 2023-10-12T00:41:14.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc4ba61443dd9fd2-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:14.775 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.776 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.776 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.777 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.777 DEBUG:boto:Method: GET 2023-10-12T00:41:14.777 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.778 DEBUG:boto:Data: 2023-10-12T00:41:14.778 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.779 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.779 DEBUG:boto:Params: {} 2023-10-12T00:41:14.780 DEBUG:boto:Token: None 2023-10-12T00:41:14.780 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94= 2023-10-12T00:41:14.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94=', 'Content-Length': '0'} 2023-10-12T00:41:14.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000119a3df7844be3bf-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.784 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.784 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.785 DEBUG:boto:path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.785 DEBUG:boto:auth_path=/wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.786 DEBUG:boto:Method: GET 2023-10-12T00:41:14.786 DEBUG:boto:Path: /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.786 DEBUG:boto:Data: 2023-10-12T00:41:14.787 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.788 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.788 DEBUG:boto:Params: {} 2023-10-12T00:41:14.789 DEBUG:boto:Token: None 2023-10-12T00:41:14.789 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7?versionId=null 2023-10-12T00:41:14.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94= 2023-10-12T00:41:14.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bIFOL6GEM5y+GlUqjRiRZ5EcC94=', 'Content-Length': '0'} 2023-10-12T00:41:14.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028ee9aabee1f6700-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.793 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.793 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.794 DEBUG:boto:path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.794 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.794 DEBUG:boto:Method: GET 2023-10-12T00:41:14.795 DEBUG:boto:Path: /wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.795 DEBUG:boto:Data: 2023-10-12T00:41:14.796 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.796 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.797 DEBUG:boto:Params: {} 2023-10-12T00:41:14.797 DEBUG:boto:Token: None 2023-10-12T00:41:14.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba8999c65593369-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.802 DEBUG:boto:b'wafmoa-48obj71000falseobj72023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.803 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.803 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.803 DEBUG:boto:path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.804 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.804 DEBUG:boto:Method: GET 2023-10-12T00:41:14.805 DEBUG:boto:Path: /wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.805 DEBUG:boto:Data: 2023-10-12T00:41:14.806 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.807 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.807 DEBUG:boto:Params: {} 2023-10-12T00:41:14.807 DEBUG:boto:Token: None 2023-10-12T00:41:14.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe640199b5be1b97-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.812 DEBUG:boto:b'wafmoa-48obj71000falseobj72023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.812 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.813 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.813 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.814 DEBUG:boto:Path: /wafmoa-48/obj7 2023-10-12T00:41:14.814 DEBUG:boto:Data: 2023-10-12T00:41:14.815 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.817 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.817 DEBUG:boto:Params: {} 2023-10-12T00:41:14.818 DEBUG:boto:Token: None 2023-10-12T00:41:14.818 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7 2023-10-12T00:41:14.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6xvcDjm5FrJpozLjnVgR9zFNGDQ= 2023-10-12T00:41:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6xvcDjm5FrJpozLjnVgR9zFNGDQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0225fc800f5c9ab-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.822 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.823 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.823 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.824 DEBUG:boto:Path: /wafmoa-48/obj7 2023-10-12T00:41:14.824 DEBUG:boto:Data: 2023-10-12T00:41:14.825 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.826 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.826 DEBUG:boto:Params: {} 2023-10-12T00:41:14.827 DEBUG:boto:Token: None 2023-10-12T00:41:14.828 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7 2023-10-12T00:41:14.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6xvcDjm5FrJpozLjnVgR9zFNGDQ= 2023-10-12T00:41:14.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6xvcDjm5FrJpozLjnVgR9zFNGDQ=', 'Content-Length': '0'} 2023-10-12T00:41:14.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebf619ffa4750686-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:14.832 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.832 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.833 DEBUG:boto:Method: GET 2023-10-12T00:41:14.833 DEBUG:boto:Path: /wafmoa-48/obj7 2023-10-12T00:41:14.834 DEBUG:boto:Data: 2023-10-12T00:41:14.834 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.835 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.835 DEBUG:boto:Params: {} 2023-10-12T00:41:14.836 DEBUG:boto:Token: None 2023-10-12T00:41:14.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7 2023-10-12T00:41:14.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:arv9VxhFthMtP3/MUUNbJF91taA= 2023-10-12T00:41:14.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:arv9VxhFthMtP3/MUUNbJF91taA=', 'Content-Length': '0'} 2023-10-12T00:41:14.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000308345cd32aac950-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.840 DEBUG:boto:path=/wafmoa-48/obj7 2023-10-12T00:41:14.840 DEBUG:boto:auth_path=/wafmoa-48/obj7 2023-10-12T00:41:14.841 DEBUG:boto:Method: GET 2023-10-12T00:41:14.841 DEBUG:boto:Path: /wafmoa-48/obj7 2023-10-12T00:41:14.842 DEBUG:boto:Data: 2023-10-12T00:41:14.842 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.843 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.843 DEBUG:boto:Params: {} 2023-10-12T00:41:14.844 DEBUG:boto:Token: None 2023-10-12T00:41:14.844 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj7 2023-10-12T00:41:14.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:arv9VxhFthMtP3/MUUNbJF91taA= 2023-10-12T00:41:14.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:arv9VxhFthMtP3/MUUNbJF91taA=', 'Content-Length': '0'} 2023-10-12T00:41:14.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d8178f8e85514f3-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.894 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.895 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.896 DEBUG:boto:path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.896 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.896 DEBUG:boto:Method: GET 2023-10-12T00:41:14.897 DEBUG:boto:Path: /wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.897 DEBUG:boto:Data: 2023-10-12T00:41:14.898 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.899 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.899 DEBUG:boto:Params: {} 2023-10-12T00:41:14.900 DEBUG:boto:Token: None 2023-10-12T00:41:14.901 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.901 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c82f4b0611131f8f-00652740aa-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.905 DEBUG:boto:b'wafmoa-48obj71000falseobj72023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.906 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:14.906 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:14.907 DEBUG:boto:path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.907 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.908 DEBUG:boto:Method: GET 2023-10-12T00:41:14.908 DEBUG:boto:Path: /wafmoa-48/?prefix=obj7 2023-10-12T00:41:14.908 DEBUG:boto:Data: 2023-10-12T00:41:14.909 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.910 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.910 DEBUG:boto:Params: {} 2023-10-12T00:41:14.911 DEBUG:boto:Token: None 2023-10-12T00:41:14.911 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/ 2023-10-12T00:41:14.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8= 2023-10-12T00:41:14.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b8uWgXsJXWarVRW6u/adqV7gHP8=', 'Content-Length': '0'} 2023-10-12T00:41:14.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b097829613a143af-00652740aa-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.916 DEBUG:boto:b'wafmoa-48obj71000falseobj72023-10-12T00:41:02.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:14.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:14.917 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.917 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.918 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.918 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.919 DEBUG:boto:Method: GET 2023-10-12T00:41:14.919 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.920 DEBUG:boto:Data: 2023-10-12T00:41:14.920 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.921 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.921 DEBUG:boto:Params: {} 2023-10-12T00:41:14.922 DEBUG:boto:Token: None 2023-10-12T00:41:14.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8= 2023-10-12T00:41:14.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8=', 'Content-Length': '0'} 2023-10-12T00:41:14.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f83405765df1a6b-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.926 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.927 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.927 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.927 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.928 DEBUG:boto:Method: GET 2023-10-12T00:41:14.928 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.929 DEBUG:boto:Data: 2023-10-12T00:41:14.929 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.930 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.931 DEBUG:boto:Params: {} 2023-10-12T00:41:14.931 DEBUG:boto:Token: None 2023-10-12T00:41:14.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8= 2023-10-12T00:41:14.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8=', 'Content-Length': '0'} 2023-10-12T00:41:14.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c98f905abce7e4-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.936 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.936 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.937 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.937 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.937 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.938 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.939 DEBUG:boto:Data: 2023-10-12T00:41:14.939 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.940 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.941 DEBUG:boto:Params: {} 2023-10-12T00:41:14.941 DEBUG:boto:Token: None 2023-10-12T00:41:14.942 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:veX5fUCMvqrfPHxY7fhkFALTfEk= 2023-10-12T00:41:14.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:veX5fUCMvqrfPHxY7fhkFALTfEk=', 'Content-Length': '0'} 2023-10-12T00:41:14.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068a6c1d615663d72-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.946 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.947 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.947 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.947 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.948 DEBUG:boto:Method: HEAD 2023-10-12T00:41:14.948 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.949 DEBUG:boto:Data: 2023-10-12T00:41:14.949 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.950 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.950 DEBUG:boto:Params: {} 2023-10-12T00:41:14.951 DEBUG:boto:Token: None 2023-10-12T00:41:14.951 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:veX5fUCMvqrfPHxY7fhkFALTfEk= 2023-10-12T00:41:14.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:veX5fUCMvqrfPHxY7fhkFALTfEk=', 'Content-Length': '0'} 2023-10-12T00:41:14.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000046596d1418ad192-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:14.955 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.956 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.956 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.957 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.958 DEBUG:boto:Method: GET 2023-10-12T00:41:14.958 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.959 DEBUG:boto:Data: 2023-10-12T00:41:14.959 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:14.960 DEBUG:boto:Port: 8001 2023-10-12T00:41:14.961 DEBUG:boto:Params: {} 2023-10-12T00:41:14.961 DEBUG:boto:Token: None 2023-10-12T00:41:14.962 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8= 2023-10-12T00:41:14.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8=', 'Content-Length': '0'} 2023-10-12T00:41:14.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aad7f780e6174ab8-00652740aa-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:14.965 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:14.966 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:14.966 DEBUG:boto:path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.967 DEBUG:boto:auth_path=/wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.967 DEBUG:boto:Method: GET 2023-10-12T00:41:14.968 DEBUG:boto:Path: /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.968 DEBUG:boto:Data: 2023-10-12T00:41:14.968 DEBUG:boto:Headers: {} 2023-10-12T00:41:14.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:14.970 DEBUG:boto:Port: 8000 2023-10-12T00:41:14.970 DEBUG:boto:Params: {} 2023-10-12T00:41:14.970 DEBUG:boto:Token: None 2023-10-12T00:41:14.971 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:14 GMT /wafmoa-48/obj8?versionId=null 2023-10-12T00:41:14.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8= 2023-10-12T00:41:14.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9j9lO1MFvvVw/nXLexNdcqM5JS8=', 'Content-Length': '0'} 2023-10-12T00:41:14.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e32417e3a0ac0d62-00652740aa-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.018 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:15.019 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:15.020 DEBUG:boto:path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.020 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.020 DEBUG:boto:Method: GET 2023-10-12T00:41:15.021 DEBUG:boto:Path: /wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.021 DEBUG:boto:Data: 2023-10-12T00:41:15.023 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:15.024 DEBUG:boto:Port: 8001 2023-10-12T00:41:15.024 DEBUG:boto:Params: {} 2023-10-12T00:41:15.025 DEBUG:boto:Token: None 2023-10-12T00:41:15.025 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/ 2023-10-12T00:41:15.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY= 2023-10-12T00:41:15.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY=', 'Content-Length': '0'} 2023-10-12T00:41:15.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b8c979c083a5da7-00652740ab-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.029 DEBUG:boto:b'wafmoa-48obj81000falseobj82023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:15.030 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:15.030 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:15.031 DEBUG:boto:path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.031 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.032 DEBUG:boto:Method: GET 2023-10-12T00:41:15.032 DEBUG:boto:Path: /wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.033 DEBUG:boto:Data: 2023-10-12T00:41:15.033 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:15.034 DEBUG:boto:Port: 8000 2023-10-12T00:41:15.035 DEBUG:boto:Params: {} 2023-10-12T00:41:15.035 DEBUG:boto:Token: None 2023-10-12T00:41:15.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/ 2023-10-12T00:41:15.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY= 2023-10-12T00:41:15.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY=', 'Content-Length': '0'} 2023-10-12T00:41:15.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e8a83a9f56bd72e-00652740ab-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.039 DEBUG:boto:b'wafmoa-48obj81000falseobj82023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:15.040 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:15.040 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:15.041 DEBUG:boto:Method: HEAD 2023-10-12T00:41:15.041 DEBUG:boto:Path: /wafmoa-48/obj8 2023-10-12T00:41:15.042 DEBUG:boto:Data: 2023-10-12T00:41:15.042 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:15.043 DEBUG:boto:Port: 8001 2023-10-12T00:41:15.043 DEBUG:boto:Params: {} 2023-10-12T00:41:15.044 DEBUG:boto:Token: None 2023-10-12T00:41:15.044 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/obj8 2023-10-12T00:41:15.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:36qH4C/PNOr0d76W8NRJFaEtS3U= 2023-10-12T00:41:15.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:36qH4C/PNOr0d76W8NRJFaEtS3U=', 'Content-Length': '0'} 2023-10-12T00:41:15.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d09df685a2d2f990-00652740ab-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.048 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:15.048 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:15.048 DEBUG:boto:Method: HEAD 2023-10-12T00:41:15.049 DEBUG:boto:Path: /wafmoa-48/obj8 2023-10-12T00:41:15.049 DEBUG:boto:Data: 2023-10-12T00:41:15.050 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:15.051 DEBUG:boto:Port: 8000 2023-10-12T00:41:15.051 DEBUG:boto:Params: {} 2023-10-12T00:41:15.051 DEBUG:boto:Token: None 2023-10-12T00:41:15.052 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/obj8 2023-10-12T00:41:15.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:36qH4C/PNOr0d76W8NRJFaEtS3U= 2023-10-12T00:41:15.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:36qH4C/PNOr0d76W8NRJFaEtS3U=', 'Content-Length': '0'} 2023-10-12T00:41:15.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c76f1a1b807cfb81-00652740ab-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:15.056 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:15.056 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:15.057 DEBUG:boto:Method: GET 2023-10-12T00:41:15.057 DEBUG:boto:Path: /wafmoa-48/obj8 2023-10-12T00:41:15.058 DEBUG:boto:Data: 2023-10-12T00:41:15.058 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:15.059 DEBUG:boto:Port: 8001 2023-10-12T00:41:15.059 DEBUG:boto:Params: {} 2023-10-12T00:41:15.060 DEBUG:boto:Token: None 2023-10-12T00:41:15.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/obj8 2023-10-12T00:41:15.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rJ8O1vFqAW29EzEM+fMRnzvyzSI= 2023-10-12T00:41:15.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rJ8O1vFqAW29EzEM+fMRnzvyzSI=', 'Content-Length': '0'} 2023-10-12T00:41:15.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-48:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000199a3638d0533363-00652740ab-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.064 DEBUG:boto:path=/wafmoa-48/obj8 2023-10-12T00:41:15.064 DEBUG:boto:auth_path=/wafmoa-48/obj8 2023-10-12T00:41:15.065 DEBUG:boto:Method: GET 2023-10-12T00:41:15.065 DEBUG:boto:Path: /wafmoa-48/obj8 2023-10-12T00:41:15.066 DEBUG:boto:Data: 2023-10-12T00:41:15.066 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:15.067 DEBUG:boto:Port: 8000 2023-10-12T00:41:15.068 DEBUG:boto:Params: {} 2023-10-12T00:41:15.068 DEBUG:boto:Token: None 2023-10-12T00:41:15.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/obj8 2023-10-12T00:41:15.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rJ8O1vFqAW29EzEM+fMRnzvyzSI= 2023-10-12T00:41:15.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rJ8O1vFqAW29EzEM+fMRnzvyzSI=', 'Content-Length': '0'} 2023-10-12T00:41:15.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a902b87f3f38a35-00652740ab-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.114 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:15.115 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:15.115 DEBUG:boto:path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.116 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.116 DEBUG:boto:Method: GET 2023-10-12T00:41:15.117 DEBUG:boto:Path: /wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.117 DEBUG:boto:Data: 2023-10-12T00:41:15.118 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:15.118 DEBUG:boto:Port: 8001 2023-10-12T00:41:15.119 DEBUG:boto:Params: {} 2023-10-12T00:41:15.120 DEBUG:boto:Token: None 2023-10-12T00:41:15.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/ 2023-10-12T00:41:15.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY= 2023-10-12T00:41:15.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY=', 'Content-Length': '0'} 2023-10-12T00:41:15.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e861f71c4dc88297-00652740ab-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.124 DEBUG:boto:b'wafmoa-48obj81000falseobj82023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:15.125 DEBUG:boto:path=/wafmoa-48/ 2023-10-12T00:41:15.125 DEBUG:boto:auth_path=/wafmoa-48/ 2023-10-12T00:41:15.126 DEBUG:boto:path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.126 DEBUG:boto:auth_path=/wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.127 DEBUG:boto:Method: GET 2023-10-12T00:41:15.127 DEBUG:boto:Path: /wafmoa-48/?prefix=obj8 2023-10-12T00:41:15.128 DEBUG:boto:Data: 2023-10-12T00:41:15.128 DEBUG:boto:Headers: {} 2023-10-12T00:41:15.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:15.129 DEBUG:boto:Port: 8000 2023-10-12T00:41:15.129 DEBUG:boto:Params: {} 2023-10-12T00:41:15.130 DEBUG:boto:Token: None 2023-10-12T00:41:15.130 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:15 GMT /wafmoa-48/ 2023-10-12T00:41:15.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY= 2023-10-12T00:41:15.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNHXAVzrgdn4Bm5o71ysxJtE4jY=', 'Content-Length': '0'} 2023-10-12T00:41:15.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f357c04cbec8c900-00652740ab-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:15.134 DEBUG:boto:b'wafmoa-48obj81000falseobj82023-10-12T00:41:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:15.135 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-48 zones={a2, a1} 2023-10-12T00:41:15.136 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-49 2023-10-12T00:41:15.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:15.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:15.137 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:15.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:15.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:15.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:15.234+0000 7f4d2373a780 1 waiting for incremental sync to catch up: 2023-10-12T00:41:15.235 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.17.1, 00000000002.39.1, 00000000003.28.1, 00000000002.32.1, 00000000003.16.1, 00000000002.16.1, 00000000002.26.1, 00000000003.27.1, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:15.235 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.20.6, 00000000002.39.1, 00000000004.30.6, 00000000002.32.1, 00000000004.18.6, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:20.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:20.236+0000 7f4d2373a780 1 waiting for incremental sync to catch up: 2023-10-12T00:41:20.236 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.17.1, 00000000002.39.1, 00000000003.28.1, 00000000002.32.1, 00000000003.16.1, 00000000002.16.1, 00000000002.26.1, 00000000003.27.1, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:20.236 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.20.6, 00000000002.39.1, 00000000004.30.6, 00000000002.32.1, 00000000004.18.6, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:25.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:25.237+0000 7f4d2373a780 1 waiting for incremental sync to catch up: 2023-10-12T00:41:25.237 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.17.1, 00000000002.39.1, 00000000003.28.1, 00000000002.32.1, 00000000003.16.1, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:25.237 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.20.6, 00000000002.39.1, 00000000004.30.6, 00000000002.32.1, 00000000004.18.6, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:30.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:30.237+0000 7f4d2373a780 1 bucket sync caught up with source: 2023-10-12T00:41:30.238 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.20.6, 00000000002.39.1, 00000000004.30.6, 00000000002.32.1, 00000000004.18.6, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:30.239 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.20.6, 00000000002.39.1, 00000000004.30.6, 00000000002.32.1, 00000000004.18.6, 00000000002.16.1, 00000000002.26.1, 00000000004.29.6, 00000000002.21.1, 00000000002.18.1, 00000000002.28.1] 2023-10-12T00:41:30.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:30.237+0000 7f4d2373a780 0 bucket checkpoint complete 2023-10-12T00:41:30.256 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-49 2023-10-12T00:41:30.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:30.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:30.259 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:30.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:30.325 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:30.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:30.360+0000 7fe8f928e780 1 bucket sync caught up with source: 2023-10-12T00:41:30.361 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.187.1, 00000000002.147.1, 00000000004.304.1, 00000000002.278.1, 00000000004.71.1, 00000000002.121.1, 00000000002.173.1, 00000000004.174.1, 00000000002.68.1, 00000000002.188.1, 00000000002.237.1] 2023-10-12T00:41:30.362 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.187.1, 00000000002.147.1, 00000000004.304.1, 00000000002.278.1, 00000000004.71.1, 00000000002.121.1, 00000000002.173.1, 00000000004.174.1, 00000000002.68.1, 00000000002.188.1, 00000000002.237.1] 2023-10-12T00:41:30.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:30.360+0000 7fe8f928e780 0 bucket checkpoint complete 2023-10-12T00:41:30.375 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-49 zones={a2, a1} 2023-10-12T00:41:30.375 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.376 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.376 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.377 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:30.377 DEBUG:boto:Data: 2023-10-12T00:41:30.377 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.378 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.379 DEBUG:boto:Params: {} 2023-10-12T00:41:30.379 DEBUG:boto:Token: None 2023-10-12T00:41:30.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU4+mMjQAosuXRvA1HCdWAiJ58I= 2023-10-12T00:41:30.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU4+mMjQAosuXRvA1HCdWAiJ58I=', 'Content-Length': '0'} 2023-10-12T00:41:30.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf784f0a093c5c30-00652740ba-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.385 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.385 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.386 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.386 DEBUG:boto:Path: /wafmoa-49/ 2023-10-12T00:41:30.387 DEBUG:boto:Data: 2023-10-12T00:41:30.387 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.388 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.389 DEBUG:boto:Params: {} 2023-10-12T00:41:30.389 DEBUG:boto:Token: None 2023-10-12T00:41:30.390 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU4+mMjQAosuXRvA1HCdWAiJ58I= 2023-10-12T00:41:30.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU4+mMjQAosuXRvA1HCdWAiJ58I=', 'Content-Length': '0'} 2023-10-12T00:41:30.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efb814e67debc627-00652740ba-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:41:30.394 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.394 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.395 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.395 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.396 DEBUG:boto:Method: GET 2023-10-12T00:41:30.396 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.397 DEBUG:boto:Data: 2023-10-12T00:41:30.397 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.398 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.398 DEBUG:boto:Params: {} 2023-10-12T00:41:30.399 DEBUG:boto:Token: None 2023-10-12T00:41:30.399 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/?versions 2023-10-12T00:41:30.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44= 2023-10-12T00:41:30.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44=', 'Content-Length': '0'} 2023-10-12T00:41:30.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017ce544d76c9679-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.404 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:41:30.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:41:30.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:41:30.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:41:30.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:41:30.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:41:30.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:41:30.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:41:30.408 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:41:30.408 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.409 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.409 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.410 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.410 DEBUG:boto:Method: GET 2023-10-12T00:41:30.410 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.411 DEBUG:boto:Data: 2023-10-12T00:41:30.411 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.412 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.413 DEBUG:boto:Params: {} 2023-10-12T00:41:30.413 DEBUG:boto:Token: None 2023-10-12T00:41:30.414 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/?versions 2023-10-12T00:41:30.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44= 2023-10-12T00:41:30.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44=', 'Content-Length': '0'} 2023-10-12T00:41:30.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a15d60356edb92-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.418 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-12T00:41:30.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-12T00:41:30.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-12T00:41:30.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-12T00:41:30.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-12T00:41:30.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-12T00:41:30.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-12T00:41:30.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-12T00:41:30.422 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.423 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.423 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.424 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.424 DEBUG:boto:Method: GET 2023-10-12T00:41:30.424 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.425 DEBUG:boto:Data: 2023-10-12T00:41:30.425 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.426 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.427 DEBUG:boto:Params: {} 2023-10-12T00:41:30.427 DEBUG:boto:Token: None 2023-10-12T00:41:30.428 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/?versions 2023-10-12T00:41:30.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44= 2023-10-12T00:41:30.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44=', 'Content-Length': '0'} 2023-10-12T00:41:30.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e17486f0ccbfcc69-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.432 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.433 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.433 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.434 DEBUG:boto:path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.434 DEBUG:boto:auth_path=/wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.434 DEBUG:boto:Method: GET 2023-10-12T00:41:30.435 DEBUG:boto:Path: /wafmoa-49/?versions&max-keys=999 2023-10-12T00:41:30.435 DEBUG:boto:Data: 2023-10-12T00:41:30.436 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.437 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.437 DEBUG:boto:Params: {} 2023-10-12T00:41:30.437 DEBUG:boto:Token: None 2023-10-12T00:41:30.438 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/?versions 2023-10-12T00:41:30.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44= 2023-10-12T00:41:30.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NQZ9Q3Y/rT2t1/+CpRnlmtqRB44=', 'Content-Length': '0'} 2023-10-12T00:41:30.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc1272d90e35479-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.442 DEBUG:boto:b'wafmoa-49999falseobj1nulltrue2023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:30.444 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.444 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.445 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.445 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.446 DEBUG:boto:Method: GET 2023-10-12T00:41:30.446 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.447 DEBUG:boto:Data: 2023-10-12T00:41:30.447 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.448 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.449 DEBUG:boto:Params: {} 2023-10-12T00:41:30.449 DEBUG:boto:Token: None 2023-10-12T00:41:30.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw= 2023-10-12T00:41:30.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw=', 'Content-Length': '0'} 2023-10-12T00:41:30.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020339af4b7368c80-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.495 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.495 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.496 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.496 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.497 DEBUG:boto:Method: GET 2023-10-12T00:41:30.498 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.498 DEBUG:boto:Data: 2023-10-12T00:41:30.499 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.500 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.500 DEBUG:boto:Params: {} 2023-10-12T00:41:30.501 DEBUG:boto:Token: None 2023-10-12T00:41:30.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.502 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw= 2023-10-12T00:41:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw=', 'Content-Length': '0'} 2023-10-12T00:41:30.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc9e442ac2a52837-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.506 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.507 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.507 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.508 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.508 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.509 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.510 DEBUG:boto:Data: 2023-10-12T00:41:30.511 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.513 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.513 DEBUG:boto:Params: {} 2023-10-12T00:41:30.514 DEBUG:boto:Token: None 2023-10-12T00:41:30.515 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HFGudLNOsolGS1pUcjYOqU1VOw= 2023-10-12T00:41:30.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HFGudLNOsolGS1pUcjYOqU1VOw=', 'Content-Length': '0'} 2023-10-12T00:41:30.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaf4faa1471498ad-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.519 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.520 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.520 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.521 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.522 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.522 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.523 DEBUG:boto:Data: 2023-10-12T00:41:30.523 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.524 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.525 DEBUG:boto:Params: {} 2023-10-12T00:41:30.526 DEBUG:boto:Token: None 2023-10-12T00:41:30.526 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3HFGudLNOsolGS1pUcjYOqU1VOw= 2023-10-12T00:41:30.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3HFGudLNOsolGS1pUcjYOqU1VOw=', 'Content-Length': '0'} 2023-10-12T00:41:30.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5d50dfaf5c13970-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:30.531 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.531 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.532 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.532 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.533 DEBUG:boto:Method: GET 2023-10-12T00:41:30.533 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.534 DEBUG:boto:Data: 2023-10-12T00:41:30.534 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.535 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.536 DEBUG:boto:Params: {} 2023-10-12T00:41:30.537 DEBUG:boto:Token: None 2023-10-12T00:41:30.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw= 2023-10-12T00:41:30.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw=', 'Content-Length': '0'} 2023-10-12T00:41:30.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085d7cd1c1e58f2ec-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.542 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.542 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.543 DEBUG:boto:path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.543 DEBUG:boto:auth_path=/wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.544 DEBUG:boto:Method: GET 2023-10-12T00:41:30.544 DEBUG:boto:Path: /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.545 DEBUG:boto:Data: 2023-10-12T00:41:30.545 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.546 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.547 DEBUG:boto:Params: {} 2023-10-12T00:41:30.547 DEBUG:boto:Token: None 2023-10-12T00:41:30.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1?versionId=null 2023-10-12T00:41:30.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw= 2023-10-12T00:41:30.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDYwJXv0Sgqxe0RYXDbB84JYxvw=', 'Content-Length': '0'} 2023-10-12T00:41:30.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa888ad6ca87fa44-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.552 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.552 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.553 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.554 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.554 DEBUG:boto:Method: GET 2023-10-12T00:41:30.555 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.555 DEBUG:boto:Data: 2023-10-12T00:41:30.556 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.557 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.557 DEBUG:boto:Params: {} 2023-10-12T00:41:30.558 DEBUG:boto:Token: None 2023-10-12T00:41:30.558 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.559 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad24e5c3ee50def-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.563 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.564 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.564 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.565 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.565 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.566 DEBUG:boto:Method: GET 2023-10-12T00:41:30.566 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.567 DEBUG:boto:Data: 2023-10-12T00:41:30.567 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.568 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.569 DEBUG:boto:Params: {} 2023-10-12T00:41:30.569 DEBUG:boto:Token: None 2023-10-12T00:41:30.570 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.571 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6c1e4280c09ac0-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.575 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.576 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.576 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.577 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.577 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:41:30.578 DEBUG:boto:Data: 2023-10-12T00:41:30.578 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.579 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.580 DEBUG:boto:Params: {} 2023-10-12T00:41:30.580 DEBUG:boto:Token: None 2023-10-12T00:41:30.581 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1 2023-10-12T00:41:30.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nDF0PHHvcZpia1ECd5n3EjNn50w= 2023-10-12T00:41:30.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nDF0PHHvcZpia1ECd5n3EjNn50w=', 'Content-Length': '0'} 2023-10-12T00:41:30.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000839fe3eddddcbdb4-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.585 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.585 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.586 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.586 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:41:30.587 DEBUG:boto:Data: 2023-10-12T00:41:30.587 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.588 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.589 DEBUG:boto:Params: {} 2023-10-12T00:41:30.589 DEBUG:boto:Token: None 2023-10-12T00:41:30.590 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1 2023-10-12T00:41:30.590 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nDF0PHHvcZpia1ECd5n3EjNn50w= 2023-10-12T00:41:30.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nDF0PHHvcZpia1ECd5n3EjNn50w=', 'Content-Length': '0'} 2023-10-12T00:41:30.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cf7c39d68b8e5dc-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-12T00:41:30.594 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.594 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.595 DEBUG:boto:Method: GET 2023-10-12T00:41:30.595 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:41:30.596 DEBUG:boto:Data: 2023-10-12T00:41:30.596 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.597 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.598 DEBUG:boto:Params: {} 2023-10-12T00:41:30.598 DEBUG:boto:Token: None 2023-10-12T00:41:30.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1 2023-10-12T00:41:30.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BcBsVTLes0h88Bdstz7a1i5lncY= 2023-10-12T00:41:30.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BcBsVTLes0h88Bdstz7a1i5lncY=', 'Content-Length': '0'} 2023-10-12T00:41:30.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006efd44582f308386-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.603 DEBUG:boto:path=/wafmoa-49/obj1 2023-10-12T00:41:30.603 DEBUG:boto:auth_path=/wafmoa-49/obj1 2023-10-12T00:41:30.604 DEBUG:boto:Method: GET 2023-10-12T00:41:30.604 DEBUG:boto:Path: /wafmoa-49/obj1 2023-10-12T00:41:30.605 DEBUG:boto:Data: 2023-10-12T00:41:30.605 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.606 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.607 DEBUG:boto:Params: {} 2023-10-12T00:41:30.607 DEBUG:boto:Token: None 2023-10-12T00:41:30.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj1 2023-10-12T00:41:30.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BcBsVTLes0h88Bdstz7a1i5lncY= 2023-10-12T00:41:30.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BcBsVTLes0h88Bdstz7a1i5lncY=', 'Content-Length': '0'} 2023-10-12T00:41:30.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aeffabe0eade92bb-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.658 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.659 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.660 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.660 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.661 DEBUG:boto:Method: GET 2023-10-12T00:41:30.661 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.662 DEBUG:boto:Data: 2023-10-12T00:41:30.662 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.663 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.663 DEBUG:boto:Params: {} 2023-10-12T00:41:30.664 DEBUG:boto:Token: None 2023-10-12T00:41:30.664 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af23f805885342ed-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.669 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.670 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.670 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.671 DEBUG:boto:path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.671 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.671 DEBUG:boto:Method: GET 2023-10-12T00:41:30.672 DEBUG:boto:Path: /wafmoa-49/?prefix=obj1 2023-10-12T00:41:30.672 DEBUG:boto:Data: 2023-10-12T00:41:30.673 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.674 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.674 DEBUG:boto:Params: {} 2023-10-12T00:41:30.675 DEBUG:boto:Token: None 2023-10-12T00:41:30.675 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd2358d642eab88-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.680 DEBUG:boto:b'wafmoa-49obj11000falseobj12023-10-12T00:40:04.164Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:30.681 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.682 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.682 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.682 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.683 DEBUG:boto:Method: GET 2023-10-12T00:41:30.683 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.684 DEBUG:boto:Data: 2023-10-12T00:41:30.684 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.685 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.686 DEBUG:boto:Params: {} 2023-10-12T00:41:30.686 DEBUG:boto:Token: None 2023-10-12T00:41:30.687 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.688 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs= 2023-10-12T00:41:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs=', 'Content-Length': '0'} 2023-10-12T00:41:30.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044a0e47cab6f0e7a-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.691 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.692 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.692 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.692 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.693 DEBUG:boto:Method: GET 2023-10-12T00:41:30.694 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.697 DEBUG:boto:Data: 2023-10-12T00:41:30.697 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.698 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.699 DEBUG:boto:Params: {} 2023-10-12T00:41:30.700 DEBUG:boto:Token: None 2023-10-12T00:41:30.700 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs= 2023-10-12T00:41:30.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs=', 'Content-Length': '0'} 2023-10-12T00:41:30.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3474945cd1dcbe9-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.704 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.704 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.705 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.705 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.706 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.706 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.707 DEBUG:boto:Data: 2023-10-12T00:41:30.707 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.708 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.709 DEBUG:boto:Params: {} 2023-10-12T00:41:30.709 DEBUG:boto:Token: None 2023-10-12T00:41:30.710 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4m9V80MbLluKe0A9B9wDVz8HLDI= 2023-10-12T00:41:30.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4m9V80MbLluKe0A9B9wDVz8HLDI=', 'Content-Length': '0'} 2023-10-12T00:41:30.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b266a3c6177c6d6a-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.713 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.714 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.714 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.714 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.715 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.716 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.716 DEBUG:boto:Data: 2023-10-12T00:41:30.717 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.717 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.718 DEBUG:boto:Params: {} 2023-10-12T00:41:30.719 DEBUG:boto:Token: None 2023-10-12T00:41:30.719 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.719 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4m9V80MbLluKe0A9B9wDVz8HLDI= 2023-10-12T00:41:30.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4m9V80MbLluKe0A9B9wDVz8HLDI=', 'Content-Length': '0'} 2023-10-12T00:41:30.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fa80723623b05f2-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:30.723 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.723 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.724 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.724 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.725 DEBUG:boto:Method: GET 2023-10-12T00:41:30.725 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.726 DEBUG:boto:Data: 2023-10-12T00:41:30.726 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.727 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.727 DEBUG:boto:Params: {} 2023-10-12T00:41:30.728 DEBUG:boto:Token: None 2023-10-12T00:41:30.729 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs= 2023-10-12T00:41:30.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs=', 'Content-Length': '0'} 2023-10-12T00:41:30.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c54a36000118f6db-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.774 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.776 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.776 DEBUG:boto:path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.777 DEBUG:boto:auth_path=/wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.778 DEBUG:boto:Method: GET 2023-10-12T00:41:30.779 DEBUG:boto:Path: /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.780 DEBUG:boto:Data: 2023-10-12T00:41:30.780 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.782 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.783 DEBUG:boto:Params: {} 2023-10-12T00:41:30.784 DEBUG:boto:Token: None 2023-10-12T00:41:30.785 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2?versionId=null 2023-10-12T00:41:30.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs= 2023-10-12T00:41:30.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XsYdIPFmdJnV24qw5JQNloyPCfs=', 'Content-Length': '0'} 2023-10-12T00:41:30.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b301f9e84176fb83-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.790 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.791 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.792 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.793 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.793 DEBUG:boto:Method: GET 2023-10-12T00:41:30.794 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.795 DEBUG:boto:Data: 2023-10-12T00:41:30.795 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.796 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.797 DEBUG:boto:Params: {} 2023-10-12T00:41:30.797 DEBUG:boto:Token: None 2023-10-12T00:41:30.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001acd02fca7cab146-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.802 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.803 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.803 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.804 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.805 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.805 DEBUG:boto:Method: GET 2023-10-12T00:41:30.806 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.807 DEBUG:boto:Data: 2023-10-12T00:41:30.807 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.809 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.809 DEBUG:boto:Params: {} 2023-10-12T00:41:30.810 DEBUG:boto:Token: None 2023-10-12T00:41:30.811 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003736cd8901054088-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.816 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.817 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.817 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.819 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.819 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:41:30.820 DEBUG:boto:Data: 2023-10-12T00:41:30.820 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.821 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.821 DEBUG:boto:Params: {} 2023-10-12T00:41:30.822 DEBUG:boto:Token: None 2023-10-12T00:41:30.823 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2 2023-10-12T00:41:30.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DAx9onK2ES7lUfSVaW8py7mTI4I= 2023-10-12T00:41:30.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DAx9onK2ES7lUfSVaW8py7mTI4I=', 'Content-Length': '0'} 2023-10-12T00:41:30.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6dd3f2633480f74-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.830 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.831 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.831 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.832 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:41:30.832 DEBUG:boto:Data: 2023-10-12T00:41:30.832 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.833 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.834 DEBUG:boto:Params: {} 2023-10-12T00:41:30.834 DEBUG:boto:Token: None 2023-10-12T00:41:30.835 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2 2023-10-12T00:41:30.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DAx9onK2ES7lUfSVaW8py7mTI4I= 2023-10-12T00:41:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DAx9onK2ES7lUfSVaW8py7mTI4I=', 'Content-Length': '0'} 2023-10-12T00:41:30.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7c12553f3f18ff4-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-12T00:41:30.841 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.841 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.842 DEBUG:boto:Method: GET 2023-10-12T00:41:30.843 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:41:30.844 DEBUG:boto:Data: 2023-10-12T00:41:30.845 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.846 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.846 DEBUG:boto:Params: {} 2023-10-12T00:41:30.847 DEBUG:boto:Token: None 2023-10-12T00:41:30.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2 2023-10-12T00:41:30.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4IYsgV9DfyBpEKfghQLOTjsB+Fc= 2023-10-12T00:41:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4IYsgV9DfyBpEKfghQLOTjsB+Fc=', 'Content-Length': '0'} 2023-10-12T00:41:30.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f484e6823063362c-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.852 DEBUG:boto:path=/wafmoa-49/obj2 2023-10-12T00:41:30.852 DEBUG:boto:auth_path=/wafmoa-49/obj2 2023-10-12T00:41:30.853 DEBUG:boto:Method: GET 2023-10-12T00:41:30.853 DEBUG:boto:Path: /wafmoa-49/obj2 2023-10-12T00:41:30.854 DEBUG:boto:Data: 2023-10-12T00:41:30.854 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.855 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.855 DEBUG:boto:Params: {} 2023-10-12T00:41:30.856 DEBUG:boto:Token: None 2023-10-12T00:41:30.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj2 2023-10-12T00:41:30.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4IYsgV9DfyBpEKfghQLOTjsB+Fc= 2023-10-12T00:41:30.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4IYsgV9DfyBpEKfghQLOTjsB+Fc=', 'Content-Length': '0'} 2023-10-12T00:41:30.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a95bcbe2ede156d-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.860 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.860 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.861 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.861 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.862 DEBUG:boto:Method: GET 2023-10-12T00:41:30.862 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.863 DEBUG:boto:Data: 2023-10-12T00:41:30.863 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.864 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.864 DEBUG:boto:Params: {} 2023-10-12T00:41:30.865 DEBUG:boto:Token: None 2023-10-12T00:41:30.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae5e7aa4c034f27-00652740ba-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.869 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.870 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:30.870 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:30.871 DEBUG:boto:path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.871 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.872 DEBUG:boto:Method: GET 2023-10-12T00:41:30.872 DEBUG:boto:Path: /wafmoa-49/?prefix=obj2 2023-10-12T00:41:30.872 DEBUG:boto:Data: 2023-10-12T00:41:30.873 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.874 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.874 DEBUG:boto:Params: {} 2023-10-12T00:41:30.875 DEBUG:boto:Token: None 2023-10-12T00:41:30.875 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/ 2023-10-12T00:41:30.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk= 2023-10-12T00:41:30.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfTMWNdPOtIW+EaeA7FwChKOpLk=', 'Content-Length': '0'} 2023-10-12T00:41:30.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5dca9bf1ca929a7-00652740ba-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.923 DEBUG:boto:b'wafmoa-49obj21000falseobj22023-10-12T00:40:04.184Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:30.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:30.925 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:30.925 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:30.926 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.926 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.926 DEBUG:boto:Method: GET 2023-10-12T00:41:30.927 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.927 DEBUG:boto:Data: 2023-10-12T00:41:30.928 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.929 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.929 DEBUG:boto:Params: {} 2023-10-12T00:41:30.930 DEBUG:boto:Token: None 2023-10-12T00:41:30.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk= 2023-10-12T00:41:30.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk=', 'Content-Length': '0'} 2023-10-12T00:41:30.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a9e8d34344ee29-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.935 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:30.935 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:30.936 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.936 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.937 DEBUG:boto:Method: GET 2023-10-12T00:41:30.937 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.937 DEBUG:boto:Data: 2023-10-12T00:41:30.938 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.939 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.939 DEBUG:boto:Params: {} 2023-10-12T00:41:30.940 DEBUG:boto:Token: None 2023-10-12T00:41:30.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk= 2023-10-12T00:41:30.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk=', 'Content-Length': '0'} 2023-10-12T00:41:30.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb66c86f444ad2bd-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.944 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:30.944 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:30.945 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.945 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.946 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.946 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.946 DEBUG:boto:Data: 2023-10-12T00:41:30.947 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.948 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.948 DEBUG:boto:Params: {} 2023-10-12T00:41:30.949 DEBUG:boto:Token: None 2023-10-12T00:41:30.949 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kpf/FRA9Z/5bjUqSddHMEdLYYUI= 2023-10-12T00:41:30.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kpf/FRA9Z/5bjUqSddHMEdLYYUI=', 'Content-Length': '0'} 2023-10-12T00:41:30.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b0f02cdc0de0dc-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.952 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:30.953 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:30.953 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.954 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.954 DEBUG:boto:Method: HEAD 2023-10-12T00:41:30.954 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.955 DEBUG:boto:Data: 2023-10-12T00:41:30.955 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:30.956 DEBUG:boto:Port: 8000 2023-10-12T00:41:30.956 DEBUG:boto:Params: {} 2023-10-12T00:41:30.957 DEBUG:boto:Token: None 2023-10-12T00:41:30.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kpf/FRA9Z/5bjUqSddHMEdLYYUI= 2023-10-12T00:41:30.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kpf/FRA9Z/5bjUqSddHMEdLYYUI=', 'Content-Length': '0'} 2023-10-12T00:41:30.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f077ca1e4f11d11c-00652740ba-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:30.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:30.961 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:30.962 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:30.962 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.962 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.963 DEBUG:boto:Method: GET 2023-10-12T00:41:30.963 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.964 DEBUG:boto:Data: 2023-10-12T00:41:30.964 DEBUG:boto:Headers: {} 2023-10-12T00:41:30.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:30.965 DEBUG:boto:Port: 8001 2023-10-12T00:41:30.966 DEBUG:boto:Params: {} 2023-10-12T00:41:30.966 DEBUG:boto:Token: None 2023-10-12T00:41:30.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:30 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:30.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk= 2023-10-12T00:41:30.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gz2KJl/6MB74VFHqwpIAkq51XMk=', 'Content-Length': '0'} 2023-10-12T00:41:30.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008c2d6473bf89028-00652740ba-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.010 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:31.011 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:31.012 DEBUG:boto:path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:31.012 DEBUG:boto:auth_path=/wafmoa-49/obj3?versionId=null 2023-10-12T00:41:31.012 DEBUG:boto:Method: GET 2023-10-12T00:41:31.013 DEBUG:boto:Path: /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:31.013 DEBUG:boto:Data: 2023-10-12T00:41:31.014 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.015 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.015 DEBUG:boto:Params: {} 2023-10-12T00:41:31.016 DEBUG:boto:Token: None 2023-10-12T00:41:31.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj3?versionId=null 2023-10-12T00:41:31.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/FPCfAqO7h+E4l42/Cy5u243+yk= 2023-10-12T00:41:31.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/FPCfAqO7h+E4l42/Cy5u243+yk=', 'Content-Length': '0'} 2023-10-12T00:41:31.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e4f70bec9a8a48a-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.020 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.020 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.021 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.021 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.022 DEBUG:boto:Method: GET 2023-10-12T00:41:31.022 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.023 DEBUG:boto:Data: 2023-10-12T00:41:31.023 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.024 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.024 DEBUG:boto:Params: {} 2023-10-12T00:41:31.025 DEBUG:boto:Token: None 2023-10-12T00:41:31.025 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed02dcd679d95ec0-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.029 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.030 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.030 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.031 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.031 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.032 DEBUG:boto:Method: GET 2023-10-12T00:41:31.032 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.033 DEBUG:boto:Data: 2023-10-12T00:41:31.033 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.034 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.034 DEBUG:boto:Params: {} 2023-10-12T00:41:31.035 DEBUG:boto:Token: None 2023-10-12T00:41:31.035 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e666a85921d5f331-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.086 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.088 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:31.088 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:31.089 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.089 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:41:31.089 DEBUG:boto:Data: 2023-10-12T00:41:31.090 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.091 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.091 DEBUG:boto:Params: {} 2023-10-12T00:41:31.092 DEBUG:boto:Token: None 2023-10-12T00:41:31.092 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj3 2023-10-12T00:41:31.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sjNx6zvBDzjj6NgKQdmI+2G+SRU= 2023-10-12T00:41:31.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sjNx6zvBDzjj6NgKQdmI+2G+SRU=', 'Content-Length': '0'} 2023-10-12T00:41:31.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006004836c90902d98-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.096 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:31.096 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:31.096 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.097 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:41:31.097 DEBUG:boto:Data: 2023-10-12T00:41:31.098 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.099 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.099 DEBUG:boto:Params: {} 2023-10-12T00:41:31.100 DEBUG:boto:Token: None 2023-10-12T00:41:31.100 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj3 2023-10-12T00:41:31.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sjNx6zvBDzjj6NgKQdmI+2G+SRU= 2023-10-12T00:41:31.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sjNx6zvBDzjj6NgKQdmI+2G+SRU=', 'Content-Length': '0'} 2023-10-12T00:41:31.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000935987a8684ed2e4-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-12T00:41:31.104 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:31.104 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:31.105 DEBUG:boto:Method: GET 2023-10-12T00:41:31.105 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:41:31.106 DEBUG:boto:Data: 2023-10-12T00:41:31.109 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.110 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.110 DEBUG:boto:Params: {} 2023-10-12T00:41:31.111 DEBUG:boto:Token: None 2023-10-12T00:41:31.111 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj3 2023-10-12T00:41:31.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nW3aRFGrCItOLNXilqQ5nok17UM= 2023-10-12T00:41:31.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nW3aRFGrCItOLNXilqQ5nok17UM=', 'Content-Length': '0'} 2023-10-12T00:41:31.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f675dc0d8db05357-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.114 DEBUG:boto:path=/wafmoa-49/obj3 2023-10-12T00:41:31.115 DEBUG:boto:auth_path=/wafmoa-49/obj3 2023-10-12T00:41:31.115 DEBUG:boto:Method: GET 2023-10-12T00:41:31.116 DEBUG:boto:Path: /wafmoa-49/obj3 2023-10-12T00:41:31.116 DEBUG:boto:Data: 2023-10-12T00:41:31.117 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.117 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.118 DEBUG:boto:Params: {} 2023-10-12T00:41:31.118 DEBUG:boto:Token: None 2023-10-12T00:41:31.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj3 2023-10-12T00:41:31.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nW3aRFGrCItOLNXilqQ5nok17UM= 2023-10-12T00:41:31.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nW3aRFGrCItOLNXilqQ5nok17UM=', 'Content-Length': '0'} 2023-10-12T00:41:31.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050c3e01094488fb5-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.123 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.123 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.124 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.124 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.125 DEBUG:boto:Method: GET 2023-10-12T00:41:31.125 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.125 DEBUG:boto:Data: 2023-10-12T00:41:31.126 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.127 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.127 DEBUG:boto:Params: {} 2023-10-12T00:41:31.128 DEBUG:boto:Token: None 2023-10-12T00:41:31.128 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d89403bc58db27a-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.132 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.133 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.134 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.134 DEBUG:boto:path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.135 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.135 DEBUG:boto:Method: GET 2023-10-12T00:41:31.135 DEBUG:boto:Path: /wafmoa-49/?prefix=obj3 2023-10-12T00:41:31.136 DEBUG:boto:Data: 2023-10-12T00:41:31.136 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.137 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.138 DEBUG:boto:Params: {} 2023-10-12T00:41:31.138 DEBUG:boto:Token: None 2023-10-12T00:41:31.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a3153034fc495d-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.143 DEBUG:boto:b'wafmoa-49obj31000falseobj32023-10-12T00:40:04.212Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:31.144 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.145 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.145 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.146 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.146 DEBUG:boto:Method: GET 2023-10-12T00:41:31.147 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.147 DEBUG:boto:Data: 2023-10-12T00:41:31.148 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.149 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.149 DEBUG:boto:Params: {} 2023-10-12T00:41:31.150 DEBUG:boto:Token: None 2023-10-12T00:41:31.150 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.151 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew= 2023-10-12T00:41:31.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew=', 'Content-Length': '0'} 2023-10-12T00:41:31.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b1407f613fa86c-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.154 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.154 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.155 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.155 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.155 DEBUG:boto:Method: GET 2023-10-12T00:41:31.156 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.156 DEBUG:boto:Data: 2023-10-12T00:41:31.157 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.158 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.158 DEBUG:boto:Params: {} 2023-10-12T00:41:31.159 DEBUG:boto:Token: None 2023-10-12T00:41:31.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew= 2023-10-12T00:41:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew=', 'Content-Length': '0'} 2023-10-12T00:41:31.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005284bbc2c6746a9b-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.163 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.163 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.164 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.164 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.164 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.165 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.165 DEBUG:boto:Data: 2023-10-12T00:41:31.166 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.166 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.167 DEBUG:boto:Params: {} 2023-10-12T00:41:31.167 DEBUG:boto:Token: None 2023-10-12T00:41:31.168 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O6vteBXpgDGvW1r+f92YHV/TE8U= 2023-10-12T00:41:31.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O6vteBXpgDGvW1r+f92YHV/TE8U=', 'Content-Length': '0'} 2023-10-12T00:41:31.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000072e4b6fc3fc25be-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.171 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.172 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.172 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.173 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.173 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.174 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.174 DEBUG:boto:Data: 2023-10-12T00:41:31.175 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.176 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.176 DEBUG:boto:Params: {} 2023-10-12T00:41:31.177 DEBUG:boto:Token: None 2023-10-12T00:41:31.177 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.178 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O6vteBXpgDGvW1r+f92YHV/TE8U= 2023-10-12T00:41:31.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O6vteBXpgDGvW1r+f92YHV/TE8U=', 'Content-Length': '0'} 2023-10-12T00:41:31.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d051879c6ff9aee8-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:31.181 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.182 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.182 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.182 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.183 DEBUG:boto:Method: GET 2023-10-12T00:41:31.183 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.184 DEBUG:boto:Data: 2023-10-12T00:41:31.184 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.185 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.186 DEBUG:boto:Params: {} 2023-10-12T00:41:31.186 DEBUG:boto:Token: None 2023-10-12T00:41:31.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew= 2023-10-12T00:41:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew=', 'Content-Length': '0'} 2023-10-12T00:41:31.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa9f497b54a7eac1-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.230 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.231 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.231 DEBUG:boto:path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.232 DEBUG:boto:auth_path=/wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.232 DEBUG:boto:Method: GET 2023-10-12T00:41:31.233 DEBUG:boto:Path: /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.233 DEBUG:boto:Data: 2023-10-12T00:41:31.234 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.235 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.235 DEBUG:boto:Params: {} 2023-10-12T00:41:31.236 DEBUG:boto:Token: None 2023-10-12T00:41:31.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4?versionId=null 2023-10-12T00:41:31.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew= 2023-10-12T00:41:31.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bDTpZLufeC0mRhuyYJPxG7Iw5ew=', 'Content-Length': '0'} 2023-10-12T00:41:31.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049fda77430041605-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.240 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.240 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.241 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.241 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.242 DEBUG:boto:Method: GET 2023-10-12T00:41:31.242 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.242 DEBUG:boto:Data: 2023-10-12T00:41:31.243 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.244 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.244 DEBUG:boto:Params: {} 2023-10-12T00:41:31.245 DEBUG:boto:Token: None 2023-10-12T00:41:31.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1ad5e41b378918-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.249 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.250 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.251 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.251 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.252 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.252 DEBUG:boto:Method: GET 2023-10-12T00:41:31.253 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.253 DEBUG:boto:Data: 2023-10-12T00:41:31.254 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.255 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.255 DEBUG:boto:Params: {} 2023-10-12T00:41:31.256 DEBUG:boto:Token: None 2023-10-12T00:41:31.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068106f957ddb74b3-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.302 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.303 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.304 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.304 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.305 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:41:31.305 DEBUG:boto:Data: 2023-10-12T00:41:31.305 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.306 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.307 DEBUG:boto:Params: {} 2023-10-12T00:41:31.307 DEBUG:boto:Token: None 2023-10-12T00:41:31.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4 2023-10-12T00:41:31.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qyzzowyIcUjvRXkYLKFNrb2cVBE= 2023-10-12T00:41:31.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qyzzowyIcUjvRXkYLKFNrb2cVBE=', 'Content-Length': '0'} 2023-10-12T00:41:31.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000190181bcffd392ee-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.311 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.313 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.313 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.314 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:41:31.314 DEBUG:boto:Data: 2023-10-12T00:41:31.315 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.316 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.316 DEBUG:boto:Params: {} 2023-10-12T00:41:31.316 DEBUG:boto:Token: None 2023-10-12T00:41:31.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4 2023-10-12T00:41:31.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qyzzowyIcUjvRXkYLKFNrb2cVBE= 2023-10-12T00:41:31.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qyzzowyIcUjvRXkYLKFNrb2cVBE=', 'Content-Length': '0'} 2023-10-12T00:41:31.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001321d5be582899d7-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-12T00:41:31.321 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.321 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.322 DEBUG:boto:Method: GET 2023-10-12T00:41:31.322 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:41:31.323 DEBUG:boto:Data: 2023-10-12T00:41:31.323 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.324 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.325 DEBUG:boto:Params: {} 2023-10-12T00:41:31.325 DEBUG:boto:Token: None 2023-10-12T00:41:31.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4 2023-10-12T00:41:31.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b5PN6aTzRoEtRz1nE0IlP1Nu/i8= 2023-10-12T00:41:31.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b5PN6aTzRoEtRz1nE0IlP1Nu/i8=', 'Content-Length': '0'} 2023-10-12T00:41:31.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000432593b72374a35e-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.329 DEBUG:boto:path=/wafmoa-49/obj4 2023-10-12T00:41:31.329 DEBUG:boto:auth_path=/wafmoa-49/obj4 2023-10-12T00:41:31.330 DEBUG:boto:Method: GET 2023-10-12T00:41:31.330 DEBUG:boto:Path: /wafmoa-49/obj4 2023-10-12T00:41:31.331 DEBUG:boto:Data: 2023-10-12T00:41:31.331 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.332 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.332 DEBUG:boto:Params: {} 2023-10-12T00:41:31.333 DEBUG:boto:Token: None 2023-10-12T00:41:31.334 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj4 2023-10-12T00:41:31.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b5PN6aTzRoEtRz1nE0IlP1Nu/i8= 2023-10-12T00:41:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b5PN6aTzRoEtRz1nE0IlP1Nu/i8=', 'Content-Length': '0'} 2023-10-12T00:41:31.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:40:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d056b1b696295e-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.337 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.338 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.338 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.338 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.339 DEBUG:boto:Method: GET 2023-10-12T00:41:31.339 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.340 DEBUG:boto:Data: 2023-10-12T00:41:31.340 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.341 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.341 DEBUG:boto:Params: {} 2023-10-12T00:41:31.342 DEBUG:boto:Token: None 2023-10-12T00:41:31.342 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602b8bd795275dae-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.346 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.347 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.347 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.348 DEBUG:boto:path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.348 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.348 DEBUG:boto:Method: GET 2023-10-12T00:41:31.349 DEBUG:boto:Path: /wafmoa-49/?prefix=obj4 2023-10-12T00:41:31.349 DEBUG:boto:Data: 2023-10-12T00:41:31.350 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.351 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.351 DEBUG:boto:Params: {} 2023-10-12T00:41:31.352 DEBUG:boto:Token: None 2023-10-12T00:41:31.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a05152a1f4e40b8-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.356 DEBUG:boto:b'wafmoa-49obj41000falseobj42023-10-12T00:40:04.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:31.357 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.357 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.358 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.358 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.358 DEBUG:boto:Method: GET 2023-10-12T00:41:31.359 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.359 DEBUG:boto:Data: 2023-10-12T00:41:31.360 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.361 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.361 DEBUG:boto:Params: {} 2023-10-12T00:41:31.362 DEBUG:boto:Token: None 2023-10-12T00:41:31.362 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0= 2023-10-12T00:41:31.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0=', 'Content-Length': '0'} 2023-10-12T00:41:31.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1a4fdaea78452ef-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.366 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.366 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.367 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.367 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.367 DEBUG:boto:Method: GET 2023-10-12T00:41:31.368 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.368 DEBUG:boto:Data: 2023-10-12T00:41:31.369 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.369 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.370 DEBUG:boto:Params: {} 2023-10-12T00:41:31.370 DEBUG:boto:Token: None 2023-10-12T00:41:31.371 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0= 2023-10-12T00:41:31.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0=', 'Content-Length': '0'} 2023-10-12T00:41:31.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b517698553c69808-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.375 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.376 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.376 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.376 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.377 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.377 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.378 DEBUG:boto:Data: 2023-10-12T00:41:31.378 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.379 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.379 DEBUG:boto:Params: {} 2023-10-12T00:41:31.380 DEBUG:boto:Token: None 2023-10-12T00:41:31.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOagkXOhpRS+cT7IyFUpSh5ogKg= 2023-10-12T00:41:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOagkXOhpRS+cT7IyFUpSh5ogKg=', 'Content-Length': '0'} 2023-10-12T00:41:31.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000566d077dc65b03ab-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.383 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.384 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.384 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.385 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.385 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.386 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.386 DEBUG:boto:Data: 2023-10-12T00:41:31.386 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.387 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.388 DEBUG:boto:Params: {} 2023-10-12T00:41:31.388 DEBUG:boto:Token: None 2023-10-12T00:41:31.389 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOagkXOhpRS+cT7IyFUpSh5ogKg= 2023-10-12T00:41:31.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOagkXOhpRS+cT7IyFUpSh5ogKg=', 'Content-Length': '0'} 2023-10-12T00:41:31.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005f029601c985d7d-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:31.392 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.392 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.393 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.393 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.394 DEBUG:boto:Method: GET 2023-10-12T00:41:31.394 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.395 DEBUG:boto:Data: 2023-10-12T00:41:31.395 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.396 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.396 DEBUG:boto:Params: {} 2023-10-12T00:41:31.397 DEBUG:boto:Token: None 2023-10-12T00:41:31.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0= 2023-10-12T00:41:31.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0=', 'Content-Length': '0'} 2023-10-12T00:41:31.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007834b861b99935df-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.442 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.443 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.443 DEBUG:boto:path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.444 DEBUG:boto:auth_path=/wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.444 DEBUG:boto:Method: GET 2023-10-12T00:41:31.445 DEBUG:boto:Path: /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.445 DEBUG:boto:Data: 2023-10-12T00:41:31.446 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.446 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.447 DEBUG:boto:Params: {} 2023-10-12T00:41:31.447 DEBUG:boto:Token: None 2023-10-12T00:41:31.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5?versionId=null 2023-10-12T00:41:31.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0= 2023-10-12T00:41:31.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3WyiADCeVGWhMUArfcj+wtRWjl0=', 'Content-Length': '0'} 2023-10-12T00:41:31.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000930d5506545f49c2-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.452 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.452 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.453 DEBUG:boto:path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.453 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.454 DEBUG:boto:Method: GET 2023-10-12T00:41:31.454 DEBUG:boto:Path: /wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.455 DEBUG:boto:Data: 2023-10-12T00:41:31.455 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.456 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.457 DEBUG:boto:Params: {} 2023-10-12T00:41:31.457 DEBUG:boto:Token: None 2023-10-12T00:41:31.458 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.458 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f633a21f96fee53-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.462 DEBUG:boto:b'wafmoa-49obj51000falseobj52023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.463 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.464 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.464 DEBUG:boto:path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.464 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.465 DEBUG:boto:Method: GET 2023-10-12T00:41:31.465 DEBUG:boto:Path: /wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.466 DEBUG:boto:Data: 2023-10-12T00:41:31.466 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.467 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.467 DEBUG:boto:Params: {} 2023-10-12T00:41:31.468 DEBUG:boto:Token: None 2023-10-12T00:41:31.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ff97ed5aa1e46c-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.472 DEBUG:boto:b'wafmoa-49obj51000falseobj52023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.473 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.473 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.474 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.474 DEBUG:boto:Path: /wafmoa-49/obj5 2023-10-12T00:41:31.475 DEBUG:boto:Data: 2023-10-12T00:41:31.475 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.476 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.476 DEBUG:boto:Params: {} 2023-10-12T00:41:31.477 DEBUG:boto:Token: None 2023-10-12T00:41:31.477 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5 2023-10-12T00:41:31.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JF81rBTKNk+APJfZiTqVHbr1xD0= 2023-10-12T00:41:31.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JF81rBTKNk+APJfZiTqVHbr1xD0=', 'Content-Length': '0'} 2023-10-12T00:41:31.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031f50a11d4fcd5be-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.480 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.481 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.481 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.482 DEBUG:boto:Path: /wafmoa-49/obj5 2023-10-12T00:41:31.482 DEBUG:boto:Data: 2023-10-12T00:41:31.483 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.484 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.484 DEBUG:boto:Params: {} 2023-10-12T00:41:31.484 DEBUG:boto:Token: None 2023-10-12T00:41:31.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5 2023-10-12T00:41:31.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JF81rBTKNk+APJfZiTqVHbr1xD0= 2023-10-12T00:41:31.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JF81rBTKNk+APJfZiTqVHbr1xD0=', 'Content-Length': '0'} 2023-10-12T00:41:31.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c933c559a9cb24-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-12T00:41:31.489 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.489 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.489 DEBUG:boto:Method: GET 2023-10-12T00:41:31.490 DEBUG:boto:Path: /wafmoa-49/obj5 2023-10-12T00:41:31.490 DEBUG:boto:Data: 2023-10-12T00:41:31.491 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.492 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.492 DEBUG:boto:Params: {} 2023-10-12T00:41:31.493 DEBUG:boto:Token: None 2023-10-12T00:41:31.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5 2023-10-12T00:41:31.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QY7qK5i2ROQ4BRCT3+BCQftKGs= 2023-10-12T00:41:31.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QY7qK5i2ROQ4BRCT3+BCQftKGs=', 'Content-Length': '0'} 2023-10-12T00:41:31.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000855522011b06772-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.497 DEBUG:boto:path=/wafmoa-49/obj5 2023-10-12T00:41:31.497 DEBUG:boto:auth_path=/wafmoa-49/obj5 2023-10-12T00:41:31.497 DEBUG:boto:Method: GET 2023-10-12T00:41:31.498 DEBUG:boto:Path: /wafmoa-49/obj5 2023-10-12T00:41:31.498 DEBUG:boto:Data: 2023-10-12T00:41:31.499 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.500 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.500 DEBUG:boto:Params: {} 2023-10-12T00:41:31.501 DEBUG:boto:Token: None 2023-10-12T00:41:31.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj5 2023-10-12T00:41:31.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QY7qK5i2ROQ4BRCT3+BCQftKGs= 2023-10-12T00:41:31.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QY7qK5i2ROQ4BRCT3+BCQftKGs=', 'Content-Length': '0'} 2023-10-12T00:41:31.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009697eea9470e1f82-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.505 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.505 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.506 DEBUG:boto:path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.506 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.506 DEBUG:boto:Method: GET 2023-10-12T00:41:31.507 DEBUG:boto:Path: /wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.507 DEBUG:boto:Data: 2023-10-12T00:41:31.508 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.509 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.509 DEBUG:boto:Params: {} 2023-10-12T00:41:31.509 DEBUG:boto:Token: None 2023-10-12T00:41:31.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ce50dde104f3d1b-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.514 DEBUG:boto:b'wafmoa-49obj51000falseobj52023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.515 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.515 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.516 DEBUG:boto:path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.516 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.516 DEBUG:boto:Method: GET 2023-10-12T00:41:31.517 DEBUG:boto:Path: /wafmoa-49/?prefix=obj5 2023-10-12T00:41:31.517 DEBUG:boto:Data: 2023-10-12T00:41:31.518 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.519 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.519 DEBUG:boto:Params: {} 2023-10-12T00:41:31.521 DEBUG:boto:Token: None 2023-10-12T00:41:31.521 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012be5deed518c8f6-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.525 DEBUG:boto:b'wafmoa-49obj51000falseobj52023-10-12T00:41:02.660Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:31.527 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.527 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.527 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.528 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.528 DEBUG:boto:Method: GET 2023-10-12T00:41:31.529 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.529 DEBUG:boto:Data: 2023-10-12T00:41:31.529 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.530 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.531 DEBUG:boto:Params: {} 2023-10-12T00:41:31.531 DEBUG:boto:Token: None 2023-10-12T00:41:31.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg= 2023-10-12T00:41:31.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg=', 'Content-Length': '0'} 2023-10-12T00:41:31.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adabf5dab2869ffd-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.535 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.535 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.536 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.536 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.537 DEBUG:boto:Method: GET 2023-10-12T00:41:31.537 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.538 DEBUG:boto:Data: 2023-10-12T00:41:31.538 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.539 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.539 DEBUG:boto:Params: {} 2023-10-12T00:41:31.540 DEBUG:boto:Token: None 2023-10-12T00:41:31.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg= 2023-10-12T00:41:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg=', 'Content-Length': '0'} 2023-10-12T00:41:31.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de1cff698b1886f6-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.544 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.544 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.545 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.545 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.546 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.546 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.547 DEBUG:boto:Data: 2023-10-12T00:41:31.547 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.548 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.548 DEBUG:boto:Params: {} 2023-10-12T00:41:31.549 DEBUG:boto:Token: None 2023-10-12T00:41:31.549 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhPGOJOUx+c1qP6BSPpbglqK2+4= 2023-10-12T00:41:31.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhPGOJOUx+c1qP6BSPpbglqK2+4=', 'Content-Length': '0'} 2023-10-12T00:41:31.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008055e0ea0aed10d2-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.552 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.553 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.553 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.554 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.554 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.554 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.555 DEBUG:boto:Data: 2023-10-12T00:41:31.555 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.556 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.556 DEBUG:boto:Params: {} 2023-10-12T00:41:31.557 DEBUG:boto:Token: None 2023-10-12T00:41:31.557 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhPGOJOUx+c1qP6BSPpbglqK2+4= 2023-10-12T00:41:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhPGOJOUx+c1qP6BSPpbglqK2+4=', 'Content-Length': '0'} 2023-10-12T00:41:31.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000332c30cb67bef51c-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:31.561 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.562 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.562 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.562 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.563 DEBUG:boto:Method: GET 2023-10-12T00:41:31.563 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.564 DEBUG:boto:Data: 2023-10-12T00:41:31.564 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.565 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.565 DEBUG:boto:Params: {} 2023-10-12T00:41:31.566 DEBUG:boto:Token: None 2023-10-12T00:41:31.566 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg= 2023-10-12T00:41:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg=', 'Content-Length': '0'} 2023-10-12T00:41:31.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8a0895c84b4d4ce-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.570 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.570 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.571 DEBUG:boto:path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.571 DEBUG:boto:auth_path=/wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.572 DEBUG:boto:Method: GET 2023-10-12T00:41:31.572 DEBUG:boto:Path: /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.572 DEBUG:boto:Data: 2023-10-12T00:41:31.573 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.574 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.574 DEBUG:boto:Params: {} 2023-10-12T00:41:31.575 DEBUG:boto:Token: None 2023-10-12T00:41:31.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6?versionId=null 2023-10-12T00:41:31.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg= 2023-10-12T00:41:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8W0wNPPoxYytjZHwWS6dW9x2tDg=', 'Content-Length': '0'} 2023-10-12T00:41:31.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1e4d63365eb6fe-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.579 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.579 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.579 DEBUG:boto:path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.580 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.580 DEBUG:boto:Method: GET 2023-10-12T00:41:31.581 DEBUG:boto:Path: /wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.581 DEBUG:boto:Data: 2023-10-12T00:41:31.582 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.582 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.583 DEBUG:boto:Params: {} 2023-10-12T00:41:31.583 DEBUG:boto:Token: None 2023-10-12T00:41:31.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58d97fccfb6db7d-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.588 DEBUG:boto:b'wafmoa-49obj61000falseobj62023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.589 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.589 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.589 DEBUG:boto:path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.590 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.590 DEBUG:boto:Method: GET 2023-10-12T00:41:31.591 DEBUG:boto:Path: /wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.591 DEBUG:boto:Data: 2023-10-12T00:41:31.592 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.592 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.593 DEBUG:boto:Params: {} 2023-10-12T00:41:31.593 DEBUG:boto:Token: None 2023-10-12T00:41:31.594 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071d57e67a7673d6c-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.598 DEBUG:boto:b'wafmoa-49obj61000falseobj62023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.598 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.599 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.599 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.600 DEBUG:boto:Path: /wafmoa-49/obj6 2023-10-12T00:41:31.600 DEBUG:boto:Data: 2023-10-12T00:41:31.601 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.601 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.602 DEBUG:boto:Params: {} 2023-10-12T00:41:31.602 DEBUG:boto:Token: None 2023-10-12T00:41:31.603 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6 2023-10-12T00:41:31.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5d7TWn0DATWik+VHJmUL893hor0= 2023-10-12T00:41:31.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5d7TWn0DATWik+VHJmUL893hor0=', 'Content-Length': '0'} 2023-10-12T00:41:31.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d6594d05a861259-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.606 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.606 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.607 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.607 DEBUG:boto:Path: /wafmoa-49/obj6 2023-10-12T00:41:31.608 DEBUG:boto:Data: 2023-10-12T00:41:31.608 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.609 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.609 DEBUG:boto:Params: {} 2023-10-12T00:41:31.610 DEBUG:boto:Token: None 2023-10-12T00:41:31.610 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6 2023-10-12T00:41:31.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5d7TWn0DATWik+VHJmUL893hor0= 2023-10-12T00:41:31.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5d7TWn0DATWik+VHJmUL893hor0=', 'Content-Length': '0'} 2023-10-12T00:41:31.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a200d2e4402a5544-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-12T00:41:31.614 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.614 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.615 DEBUG:boto:Method: GET 2023-10-12T00:41:31.615 DEBUG:boto:Path: /wafmoa-49/obj6 2023-10-12T00:41:31.616 DEBUG:boto:Data: 2023-10-12T00:41:31.616 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.617 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.617 DEBUG:boto:Params: {} 2023-10-12T00:41:31.618 DEBUG:boto:Token: None 2023-10-12T00:41:31.618 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6 2023-10-12T00:41:31.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcKomzN8GSAKKHNBpR/mKyuM4jw= 2023-10-12T00:41:31.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcKomzN8GSAKKHNBpR/mKyuM4jw=', 'Content-Length': '0'} 2023-10-12T00:41:31.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88f19e2139ff26b-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.662 DEBUG:boto:path=/wafmoa-49/obj6 2023-10-12T00:41:31.663 DEBUG:boto:auth_path=/wafmoa-49/obj6 2023-10-12T00:41:31.663 DEBUG:boto:Method: GET 2023-10-12T00:41:31.664 DEBUG:boto:Path: /wafmoa-49/obj6 2023-10-12T00:41:31.664 DEBUG:boto:Data: 2023-10-12T00:41:31.665 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.666 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.666 DEBUG:boto:Params: {} 2023-10-12T00:41:31.666 DEBUG:boto:Token: None 2023-10-12T00:41:31.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj6 2023-10-12T00:41:31.667 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcKomzN8GSAKKHNBpR/mKyuM4jw= 2023-10-12T00:41:31.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcKomzN8GSAKKHNBpR/mKyuM4jw=', 'Content-Length': '0'} 2023-10-12T00:41:31.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1eb09e71405cba4-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.670 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.671 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.671 DEBUG:boto:path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.672 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.672 DEBUG:boto:Method: GET 2023-10-12T00:41:31.673 DEBUG:boto:Path: /wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.673 DEBUG:boto:Data: 2023-10-12T00:41:31.673 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.674 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.675 DEBUG:boto:Params: {} 2023-10-12T00:41:31.675 DEBUG:boto:Token: None 2023-10-12T00:41:31.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068035e1aca227617-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.680 DEBUG:boto:b'wafmoa-49obj61000falseobj62023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.680 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.681 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.681 DEBUG:boto:path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.682 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.682 DEBUG:boto:Method: GET 2023-10-12T00:41:31.683 DEBUG:boto:Path: /wafmoa-49/?prefix=obj6 2023-10-12T00:41:31.683 DEBUG:boto:Data: 2023-10-12T00:41:31.683 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.684 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.685 DEBUG:boto:Params: {} 2023-10-12T00:41:31.685 DEBUG:boto:Token: None 2023-10-12T00:41:31.686 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c222abf5dc07ad15-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.690 DEBUG:boto:b'wafmoa-49obj61000falseobj62023-10-12T00:41:02.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:31.691 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.691 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.692 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.692 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.692 DEBUG:boto:Method: GET 2023-10-12T00:41:31.693 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.693 DEBUG:boto:Data: 2023-10-12T00:41:31.694 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.695 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.695 DEBUG:boto:Params: {} 2023-10-12T00:41:31.696 DEBUG:boto:Token: None 2023-10-12T00:41:31.696 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA= 2023-10-12T00:41:31.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA=', 'Content-Length': '0'} 2023-10-12T00:41:31.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae14737cc90fde1-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.700 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.700 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.701 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.701 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.702 DEBUG:boto:Method: GET 2023-10-12T00:41:31.702 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.702 DEBUG:boto:Data: 2023-10-12T00:41:31.703 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.704 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.704 DEBUG:boto:Params: {} 2023-10-12T00:41:31.705 DEBUG:boto:Token: None 2023-10-12T00:41:31.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA= 2023-10-12T00:41:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA=', 'Content-Length': '0'} 2023-10-12T00:41:31.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ac36327383a4a35-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.709 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.709 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.710 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.710 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.710 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.711 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.711 DEBUG:boto:Data: 2023-10-12T00:41:31.712 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.712 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.713 DEBUG:boto:Params: {} 2023-10-12T00:41:31.713 DEBUG:boto:Token: None 2023-10-12T00:41:31.714 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fNdafJi/W0pduIL6EEfzfawgxgU= 2023-10-12T00:41:31.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fNdafJi/W0pduIL6EEfzfawgxgU=', 'Content-Length': '0'} 2023-10-12T00:41:31.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5aeb58e4b6be2c6-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.717 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.717 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.718 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.718 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.719 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.719 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.721 DEBUG:boto:Data: 2023-10-12T00:41:31.722 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.722 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.723 DEBUG:boto:Params: {} 2023-10-12T00:41:31.723 DEBUG:boto:Token: None 2023-10-12T00:41:31.724 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fNdafJi/W0pduIL6EEfzfawgxgU= 2023-10-12T00:41:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fNdafJi/W0pduIL6EEfzfawgxgU=', 'Content-Length': '0'} 2023-10-12T00:41:31.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adb4f8d8d1a9b918-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:31.728 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.728 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.729 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.729 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.730 DEBUG:boto:Method: GET 2023-10-12T00:41:31.730 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.730 DEBUG:boto:Data: 2023-10-12T00:41:31.731 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.732 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.732 DEBUG:boto:Params: {} 2023-10-12T00:41:31.733 DEBUG:boto:Token: None 2023-10-12T00:41:31.733 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.734 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA= 2023-10-12T00:41:31.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA=', 'Content-Length': '0'} 2023-10-12T00:41:31.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000638438726d1c5545-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.782 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.783 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.783 DEBUG:boto:path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.784 DEBUG:boto:auth_path=/wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.784 DEBUG:boto:Method: GET 2023-10-12T00:41:31.785 DEBUG:boto:Path: /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.785 DEBUG:boto:Data: 2023-10-12T00:41:31.786 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.787 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.787 DEBUG:boto:Params: {} 2023-10-12T00:41:31.787 DEBUG:boto:Token: None 2023-10-12T00:41:31.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7?versionId=null 2023-10-12T00:41:31.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA= 2023-10-12T00:41:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5u2+EBdMfN2eRcir8uGjtaCv0cA=', 'Content-Length': '0'} 2023-10-12T00:41:31.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094be94da85453ac6-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.792 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.792 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.793 DEBUG:boto:path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.793 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.794 DEBUG:boto:Method: GET 2023-10-12T00:41:31.794 DEBUG:boto:Path: /wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.795 DEBUG:boto:Data: 2023-10-12T00:41:31.795 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.796 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.796 DEBUG:boto:Params: {} 2023-10-12T00:41:31.797 DEBUG:boto:Token: None 2023-10-12T00:41:31.797 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098fd7e89f1099a5c-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.801 DEBUG:boto:b'wafmoa-49obj71000falseobj72023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.802 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.802 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.803 DEBUG:boto:path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.803 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.804 DEBUG:boto:Method: GET 2023-10-12T00:41:31.804 DEBUG:boto:Path: /wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.805 DEBUG:boto:Data: 2023-10-12T00:41:31.805 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.806 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.806 DEBUG:boto:Params: {} 2023-10-12T00:41:31.807 DEBUG:boto:Token: None 2023-10-12T00:41:31.807 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e63340ecc82b92-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.811 DEBUG:boto:b'wafmoa-49obj71000falseobj72023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.812 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.812 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.813 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.813 DEBUG:boto:Path: /wafmoa-49/obj7 2023-10-12T00:41:31.813 DEBUG:boto:Data: 2023-10-12T00:41:31.814 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.815 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.815 DEBUG:boto:Params: {} 2023-10-12T00:41:31.816 DEBUG:boto:Token: None 2023-10-12T00:41:31.816 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7 2023-10-12T00:41:31.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M04n9sFv2GxYfsMm/TJ+HCsDZhM= 2023-10-12T00:41:31.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M04n9sFv2GxYfsMm/TJ+HCsDZhM=', 'Content-Length': '0'} 2023-10-12T00:41:31.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c86472c9d3c9a8c8-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.820 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.820 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.820 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.821 DEBUG:boto:Path: /wafmoa-49/obj7 2023-10-12T00:41:31.821 DEBUG:boto:Data: 2023-10-12T00:41:31.822 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.823 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.823 DEBUG:boto:Params: {} 2023-10-12T00:41:31.823 DEBUG:boto:Token: None 2023-10-12T00:41:31.824 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7 2023-10-12T00:41:31.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M04n9sFv2GxYfsMm/TJ+HCsDZhM= 2023-10-12T00:41:31.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M04n9sFv2GxYfsMm/TJ+HCsDZhM=', 'Content-Length': '0'} 2023-10-12T00:41:31.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d62dfe30f261f958-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-12T00:41:31.828 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.828 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.829 DEBUG:boto:Method: GET 2023-10-12T00:41:31.829 DEBUG:boto:Path: /wafmoa-49/obj7 2023-10-12T00:41:31.829 DEBUG:boto:Data: 2023-10-12T00:41:31.830 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.831 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.831 DEBUG:boto:Params: {} 2023-10-12T00:41:31.832 DEBUG:boto:Token: None 2023-10-12T00:41:31.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7 2023-10-12T00:41:31.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d1aQ0yU7FefUBkzJtBkmRZKfhR8= 2023-10-12T00:41:31.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d1aQ0yU7FefUBkzJtBkmRZKfhR8=', 'Content-Length': '0'} 2023-10-12T00:41:31.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c239eeeaca0466ff-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.836 DEBUG:boto:path=/wafmoa-49/obj7 2023-10-12T00:41:31.836 DEBUG:boto:auth_path=/wafmoa-49/obj7 2023-10-12T00:41:31.837 DEBUG:boto:Method: GET 2023-10-12T00:41:31.837 DEBUG:boto:Path: /wafmoa-49/obj7 2023-10-12T00:41:31.837 DEBUG:boto:Data: 2023-10-12T00:41:31.838 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.839 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.839 DEBUG:boto:Params: {} 2023-10-12T00:41:31.840 DEBUG:boto:Token: None 2023-10-12T00:41:31.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj7 2023-10-12T00:41:31.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d1aQ0yU7FefUBkzJtBkmRZKfhR8= 2023-10-12T00:41:31.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d1aQ0yU7FefUBkzJtBkmRZKfhR8=', 'Content-Length': '0'} 2023-10-12T00:41:31.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d31662a77911071-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.886 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.887 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.887 DEBUG:boto:path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.888 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.888 DEBUG:boto:Method: GET 2023-10-12T00:41:31.889 DEBUG:boto:Path: /wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.889 DEBUG:boto:Data: 2023-10-12T00:41:31.890 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.891 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.891 DEBUG:boto:Params: {} 2023-10-12T00:41:31.892 DEBUG:boto:Token: None 2023-10-12T00:41:31.892 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87d7a3ec9d701c3-00652740bb-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.896 DEBUG:boto:b'wafmoa-49obj71000falseobj72023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.897 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:31.897 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:31.898 DEBUG:boto:path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.898 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.898 DEBUG:boto:Method: GET 2023-10-12T00:41:31.899 DEBUG:boto:Path: /wafmoa-49/?prefix=obj7 2023-10-12T00:41:31.899 DEBUG:boto:Data: 2023-10-12T00:41:31.900 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.901 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.901 DEBUG:boto:Params: {} 2023-10-12T00:41:31.901 DEBUG:boto:Token: None 2023-10-12T00:41:31.902 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/ 2023-10-12T00:41:31.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs= 2023-10-12T00:41:31.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hwfH4Atj874rylZpppoNUowGIMs=', 'Content-Length': '0'} 2023-10-12T00:41:31.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e52c0817c98cbe-00652740bb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.906 DEBUG:boto:b'wafmoa-49obj71000falseobj72023-10-12T00:41:02.735Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:31.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:31.907 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.908 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.908 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.909 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.909 DEBUG:boto:Method: GET 2023-10-12T00:41:31.910 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.910 DEBUG:boto:Data: 2023-10-12T00:41:31.910 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.911 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.912 DEBUG:boto:Params: {} 2023-10-12T00:41:31.912 DEBUG:boto:Token: None 2023-10-12T00:41:31.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY= 2023-10-12T00:41:31.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY=', 'Content-Length': '0'} 2023-10-12T00:41:31.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000543bbe446ba36a4c-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.958 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.959 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.959 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.960 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.960 DEBUG:boto:Method: GET 2023-10-12T00:41:31.961 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.961 DEBUG:boto:Data: 2023-10-12T00:41:31.962 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.962 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.963 DEBUG:boto:Params: {} 2023-10-12T00:41:31.963 DEBUG:boto:Token: None 2023-10-12T00:41:31.964 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY= 2023-10-12T00:41:31.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY=', 'Content-Length': '0'} 2023-10-12T00:41:31.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b6abf060c9db4b1-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.968 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.969 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.969 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.970 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.970 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.971 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.971 DEBUG:boto:Data: 2023-10-12T00:41:31.972 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.972 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.973 DEBUG:boto:Params: {} 2023-10-12T00:41:31.973 DEBUG:boto:Token: None 2023-10-12T00:41:31.974 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:26yZwimfwO/sBgFyB/YbzxoFZgU= 2023-10-12T00:41:31.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:26yZwimfwO/sBgFyB/YbzxoFZgU=', 'Content-Length': '0'} 2023-10-12T00:41:31.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024c13244fc927ff1-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.978 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.978 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.979 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.979 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.980 DEBUG:boto:Method: HEAD 2023-10-12T00:41:31.980 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.981 DEBUG:boto:Data: 2023-10-12T00:41:31.981 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:31.982 DEBUG:boto:Port: 8000 2023-10-12T00:41:31.983 DEBUG:boto:Params: {} 2023-10-12T00:41:31.983 DEBUG:boto:Token: None 2023-10-12T00:41:31.984 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:26yZwimfwO/sBgFyB/YbzxoFZgU= 2023-10-12T00:41:31.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:26yZwimfwO/sBgFyB/YbzxoFZgU=', 'Content-Length': '0'} 2023-10-12T00:41:31.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017be49a53cb15458-00652740bb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:31.988 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.988 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.989 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.989 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.990 DEBUG:boto:Method: GET 2023-10-12T00:41:31.990 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.990 DEBUG:boto:Data: 2023-10-12T00:41:31.991 DEBUG:boto:Headers: {} 2023-10-12T00:41:31.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:31.992 DEBUG:boto:Port: 8001 2023-10-12T00:41:31.992 DEBUG:boto:Params: {} 2023-10-12T00:41:31.993 DEBUG:boto:Token: None 2023-10-12T00:41:31.993 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:31 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY= 2023-10-12T00:41:31.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6YTrahRgiXMB9XIo1pvD7ubVfZY=', 'Content-Length': '0'} 2023-10-12T00:41:31.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061748e78509a42e4-00652740bb-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:31.996 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:31.997 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:31.997 DEBUG:boto:path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.998 DEBUG:boto:auth_path=/wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.998 DEBUG:boto:Method: GET 2023-10-12T00:41:31.998 DEBUG:boto:Path: /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:31.999 DEBUG:boto:Data: 2023-10-12T00:41:31.999 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.000 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.001 DEBUG:boto:Params: {} 2023-10-12T00:41:32.001 DEBUG:boto:Token: None 2023-10-12T00:41:32.002 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/obj8?versionId=null 2023-10-12T00:41:32.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DZ0CcC7spBi0VLvehSxb3N+Hu14= 2023-10-12T00:41:32.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DZ0CcC7spBi0VLvehSxb3N+Hu14=', 'Content-Length': '0'} 2023-10-12T00:41:32.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfcb6f2e2ab612ed-00652740bc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.046 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:32.047 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:32.048 DEBUG:boto:path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.048 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.049 DEBUG:boto:Method: GET 2023-10-12T00:41:32.049 DEBUG:boto:Path: /wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.050 DEBUG:boto:Data: 2023-10-12T00:41:32.050 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:32.051 DEBUG:boto:Port: 8001 2023-10-12T00:41:32.052 DEBUG:boto:Params: {} 2023-10-12T00:41:32.052 DEBUG:boto:Token: None 2023-10-12T00:41:32.053 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/ 2023-10-12T00:41:32.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE= 2023-10-12T00:41:32.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE=', 'Content-Length': '0'} 2023-10-12T00:41:32.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3e424ad1eb9d64e-00652740bc-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.057 DEBUG:boto:b'wafmoa-49obj81000falseobj82023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:32.058 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:32.058 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:32.059 DEBUG:boto:path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.059 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.059 DEBUG:boto:Method: GET 2023-10-12T00:41:32.060 DEBUG:boto:Path: /wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.060 DEBUG:boto:Data: 2023-10-12T00:41:32.061 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.062 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.062 DEBUG:boto:Params: {} 2023-10-12T00:41:32.062 DEBUG:boto:Token: None 2023-10-12T00:41:32.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/ 2023-10-12T00:41:32.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE= 2023-10-12T00:41:32.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE=', 'Content-Length': '0'} 2023-10-12T00:41:32.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea3fb1ec4e7d50a-00652740bc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.067 DEBUG:boto:b'wafmoa-49obj81000falseobj82023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:32.068 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:32.068 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:32.068 DEBUG:boto:Method: HEAD 2023-10-12T00:41:32.069 DEBUG:boto:Path: /wafmoa-49/obj8 2023-10-12T00:41:32.069 DEBUG:boto:Data: 2023-10-12T00:41:32.070 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:32.071 DEBUG:boto:Port: 8001 2023-10-12T00:41:32.071 DEBUG:boto:Params: {} 2023-10-12T00:41:32.072 DEBUG:boto:Token: None 2023-10-12T00:41:32.072 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/obj8 2023-10-12T00:41:32.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zPjAvm1LfivnGz7oL84ux5Iz6kc= 2023-10-12T00:41:32.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zPjAvm1LfivnGz7oL84ux5Iz6kc=', 'Content-Length': '0'} 2023-10-12T00:41:32.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b3f88b6c3543c08-00652740bc-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.075 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:32.076 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:32.076 DEBUG:boto:Method: HEAD 2023-10-12T00:41:32.077 DEBUG:boto:Path: /wafmoa-49/obj8 2023-10-12T00:41:32.077 DEBUG:boto:Data: 2023-10-12T00:41:32.077 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.079 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.079 DEBUG:boto:Params: {} 2023-10-12T00:41:32.079 DEBUG:boto:Token: None 2023-10-12T00:41:32.080 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/obj8 2023-10-12T00:41:32.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zPjAvm1LfivnGz7oL84ux5Iz6kc= 2023-10-12T00:41:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zPjAvm1LfivnGz7oL84ux5Iz6kc=', 'Content-Length': '0'} 2023-10-12T00:41:32.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c3e0af4098ab735-00652740bc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-12T00:41:32.084 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:32.085 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:32.085 DEBUG:boto:Method: GET 2023-10-12T00:41:32.086 DEBUG:boto:Path: /wafmoa-49/obj8 2023-10-12T00:41:32.086 DEBUG:boto:Data: 2023-10-12T00:41:32.087 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:32.088 DEBUG:boto:Port: 8001 2023-10-12T00:41:32.088 DEBUG:boto:Params: {} 2023-10-12T00:41:32.089 DEBUG:boto:Token: None 2023-10-12T00:41:32.089 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/obj8 2023-10-12T00:41:32.089 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fcH9c/XPp/ESCWvOsDdC7xk22ZY= 2023-10-12T00:41:32.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fcH9c/XPp/ESCWvOsDdC7xk22ZY=', 'Content-Length': '0'} 2023-10-12T00:41:32.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8862fe981512075-00652740bc-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.092 DEBUG:boto:path=/wafmoa-49/obj8 2023-10-12T00:41:32.093 DEBUG:boto:auth_path=/wafmoa-49/obj8 2023-10-12T00:41:32.093 DEBUG:boto:Method: GET 2023-10-12T00:41:32.094 DEBUG:boto:Path: /wafmoa-49/obj8 2023-10-12T00:41:32.094 DEBUG:boto:Data: 2023-10-12T00:41:32.095 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.096 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.096 DEBUG:boto:Params: {} 2023-10-12T00:41:32.096 DEBUG:boto:Token: None 2023-10-12T00:41:32.097 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/obj8 2023-10-12T00:41:32.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fcH9c/XPp/ESCWvOsDdC7xk22ZY= 2023-10-12T00:41:32.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fcH9c/XPp/ESCWvOsDdC7xk22ZY=', 'Content-Length': '0'} 2023-10-12T00:41:32.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-49:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d22caf919ec6619-00652740bc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.142 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:32.143 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:32.143 DEBUG:boto:path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.144 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.144 DEBUG:boto:Method: GET 2023-10-12T00:41:32.145 DEBUG:boto:Path: /wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.145 DEBUG:boto:Data: 2023-10-12T00:41:32.146 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:32.146 DEBUG:boto:Port: 8001 2023-10-12T00:41:32.147 DEBUG:boto:Params: {} 2023-10-12T00:41:32.147 DEBUG:boto:Token: None 2023-10-12T00:41:32.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/ 2023-10-12T00:41:32.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE= 2023-10-12T00:41:32.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE=', 'Content-Length': '0'} 2023-10-12T00:41:32.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c11865fccd41070a-00652740bc-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.152 DEBUG:boto:b'wafmoa-49obj81000falseobj82023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:32.152 DEBUG:boto:path=/wafmoa-49/ 2023-10-12T00:41:32.153 DEBUG:boto:auth_path=/wafmoa-49/ 2023-10-12T00:41:32.153 DEBUG:boto:path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.154 DEBUG:boto:auth_path=/wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.154 DEBUG:boto:Method: GET 2023-10-12T00:41:32.155 DEBUG:boto:Path: /wafmoa-49/?prefix=obj8 2023-10-12T00:41:32.155 DEBUG:boto:Data: 2023-10-12T00:41:32.156 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.157 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.158 DEBUG:boto:Params: {} 2023-10-12T00:41:32.158 DEBUG:boto:Token: None 2023-10-12T00:41:32.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-49/ 2023-10-12T00:41:32.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE= 2023-10-12T00:41:32.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AbxoOnlbdiOfMb4lR/t3bLLEJVE=', 'Content-Length': '0'} 2023-10-12T00:41:32.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b89d6959ea83e43-00652740bc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.162 DEBUG:boto:b'wafmoa-49obj81000falseobj82023-10-12T00:41:02.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:41:32.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-49 zones={a2, a1} 2023-10-12T00:41:32.163 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T00:41:32.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.165 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:32.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:32.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:32.259 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:41:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-10-12T00:41:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.338363Z" 2023-10-12T00:41:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-12T00:41:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.337804Z" 2023-10-12T00:41:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-10-12T00:41:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:03.012205Z" 2023-10-12T00:41:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-12T00:41:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.292609Z" 2023-10-12T00:41:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.263 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:41:32.264 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-10-12T00:41:32.265 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:41:32.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.266 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:32.319 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:32.319 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:32.361 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:41:32.362 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:41:32.362 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:41:32.362 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:41:32.362 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:41:32.363 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11080321874216560169 2023-10-12T00:41:32.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.363 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:41:32.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:41:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-10-12T00:41:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.365 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.365 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.338363Z" 2023-10-12T00:41:32.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.366 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:41:32.366 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.367 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-12T00:41:32.367 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.368 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.337804Z" 2023-10-12T00:41:32.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.369 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:41:32.369 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.369 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-10-12T00:41:32.370 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.370 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:03.012205Z" 2023-10-12T00:41:32.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.371 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:41:32.371 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.372 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.372 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-12T00:41:32.372 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.372 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.372 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.292609Z" 2023-10-12T00:41:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:41:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:41:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:41:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:41:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:41:32Z" 2023-10-12T00:41:32.375 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:41:32.376 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11080321874216560169 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004595", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.338363Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009807", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.337804Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006851", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:03.012205Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.292609Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:41:32Z" } 2023-10-12T00:41:32.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.338363Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009807', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.337804Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006851', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:03.012205Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.292609Z'}}] 2023-10-12T00:41:32.377 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-10-12T00:41:32.377 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004595', 1: '00000000000000000000:00000000000000009807', 2: '00000000000000000000:00000000000000006851', 3: '00000000000000000000:00000000000000006576'} 2023-10-12T00:41:32.378 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T00:41:32.378 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T00:41:32.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.380 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:32.428 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:32.429 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:32.474 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:41:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-10-12T00:41:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.452890Z" 2023-10-12T00:41:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000008256", 2023-10-12T00:41:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.454209Z" 2023-10-12T00:41:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004174", 2023-10-12T00:41:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.455531Z" 2023-10-12T00:41:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-10-12T00:41:32.478 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:08.456933Z" 2023-10-12T00:41:32.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.478 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:41:32.479 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-10-12T00:41:32.480 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:41:32.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:32.481 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:32.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:32.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:32.576 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:41:32.577 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:41:32.577 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:41:32.577 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:41:32.577 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:41:32.578 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T00:41:32.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.578 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:41:32.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.578 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:41:32.579 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.579 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.579 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003185", 2023-10-12T00:41:32.579 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.580 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.580 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.580 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.452890Z" 2023-10-12T00:41:32.580 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:41:32.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.581 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000008256", 2023-10-12T00:41:32.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.582 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.454209Z" 2023-10-12T00:41:32.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.583 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:41:32.583 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.584 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.584 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004174", 2023-10-12T00:41:32.584 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.585 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.455531Z" 2023-10-12T00:41:32.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:32.586 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:41:32.586 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:32.586 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T00:41:32.586 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-10-12T00:41:32.586 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:32.587 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:32.587 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:32.587 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:08.456933Z" 2023-10-12T00:41:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:41:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:41:32.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T00:41:32.589 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T00:41:32.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:32.589 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:41:32Z" 2023-10-12T00:41:32.589 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:41:32.591 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003185", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.452890Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008256", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.454209Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.455531Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:08.456933Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T00:41:32Z" } 2023-10-12T00:41:32.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003185', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.452890Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008256', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.454209Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.455531Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:08.456933Z'}}] 2023-10-12T00:41:32.592 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-10-12T00:41:32.592 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003185', 1: '00000000000000000000:00000000000000008256', 2: '00000000000000000000:00000000000000004174', 3: '00000000000000000000:00000000000000004597'} 2023-10-12T00:41:32.592 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T00:41:32.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-10-12T00:41:32.595 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-50 2023-10-12T00:41:32.595 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:41:32.595 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:41:32.596 DEBUG:boto:Method: PUT 2023-10-12T00:41:32.596 DEBUG:boto:Path: /wafmoa-50/ 2023-10-12T00:41:32.597 DEBUG:boto:Data: 2023-10-12T00:41:32.597 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.598 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.599 DEBUG:boto:Params: {} 2023-10-12T00:41:32.599 DEBUG:boto:Token: None 2023-10-12T00:41:32.600 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-50/ 2023-10-12T00:41:32.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K7kOEonudbvDot/ixf6VzKHbTC0= 2023-10-12T00:41:32.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K7kOEonudbvDot/ixf6VzKHbTC0=', 'Content-Length': '0'} 2023-10-12T00:41:32.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091a100be7bee137c-00652740bc-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.615 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-51 2023-10-12T00:41:32.615 DEBUG:boto:path=/wafmoa-51/ 2023-10-12T00:41:32.616 DEBUG:boto:auth_path=/wafmoa-51/ 2023-10-12T00:41:32.616 DEBUG:boto:Method: PUT 2023-10-12T00:41:32.617 DEBUG:boto:Path: /wafmoa-51/ 2023-10-12T00:41:32.617 DEBUG:boto:Data: 2023-10-12T00:41:32.617 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:41:32.618 DEBUG:boto:Port: 8001 2023-10-12T00:41:32.619 DEBUG:boto:Params: {} 2023-10-12T00:41:32.619 DEBUG:boto:Token: None 2023-10-12T00:41:32.620 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-51/ 2023-10-12T00:41:32.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CJZMZk91vEgCbhX+unJKODGCjho= 2023-10-12T00:41:32.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CJZMZk91vEgCbhX+unJKODGCjho=', 'Content-Length': '0'} 2023-10-12T00:41:32.643 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000256acac276a0cd66-00652740bc-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:41:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:32.644 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:32.644 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:32.645 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploads 2023-10-12T00:41:32.645 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploads 2023-10-12T00:41:32.645 DEBUG:boto:Method: POST 2023-10-12T00:41:32.646 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploads 2023-10-12T00:41:32.646 DEBUG:boto:Data: 2023-10-12T00:41:32.647 DEBUG:boto:Headers: {} 2023-10-12T00:41:32.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:32.647 DEBUG:boto:Port: 8000 2023-10-12T00:41:32.648 DEBUG:boto:Params: {} 2023-10-12T00:41:32.648 DEBUG:boto:Token: None 2023-10-12T00:41:32.649 DEBUG:boto:StringToSign: POST Thu, 12 Oct 2023 00:41:32 GMT /wafmoa-50/MULTIPART?uploads 2023-10-12T00:41:32.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cgzFENBMlVySWTDwHh1iwqoojCk= 2023-10-12T00:41:32.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cgzFENBMlVySWTDwHh1iwqoojCk=', 'Content-Length': '0'} 2023-10-12T00:41:34.473 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9818939a8adc85c-00652740bc-168d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.474 DEBUG:boto:b'wafmoa-50MULTIPART2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4' 2023-10-12T00:41:34.506 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.536 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.537 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=1 2023-10-12T00:41:34.537 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=1 2023-10-12T00:41:34.538 DEBUG:boto:Method: PUT 2023-10-12T00:41:34.538 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=1 2023-10-12T00:41:34.539 DEBUG:boto:Data: 2023-10-12T00:41:34.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-12T00:41:34.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.540 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.540 DEBUG:boto:Params: {} 2023-10-12T00:41:34.541 DEBUG:boto:Token: None 2023-10-12T00:41:34.541 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?partNumber=1&uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pYNhNV4mlZLwUheQ805oZqp3gVU= 2023-10-12T00:41:34.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pYNhNV4mlZLwUheQ805oZqp3gVU='} 2023-10-12T00:41:34.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abc9485028329485-00652740be-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.621 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.622 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.622 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=2 2023-10-12T00:41:34.623 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=2 2023-10-12T00:41:34.623 DEBUG:boto:Method: PUT 2023-10-12T00:41:34.624 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=2 2023-10-12T00:41:34.624 DEBUG:boto:Data: 2023-10-12T00:41:34.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-12T00:41:34.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.626 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.626 DEBUG:boto:Params: {} 2023-10-12T00:41:34.627 DEBUG:boto:Token: None 2023-10-12T00:41:34.627 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?partNumber=2&uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqTw8Xv5A5DSIyXuyKmccrCLRYU= 2023-10-12T00:41:34.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqTw8Xv5A5DSIyXuyKmccrCLRYU='} 2023-10-12T00:41:34.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004161616d3b023a2a-00652740be-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.687 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.688 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.688 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=3 2023-10-12T00:41:34.689 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=3 2023-10-12T00:41:34.689 DEBUG:boto:Method: PUT 2023-10-12T00:41:34.690 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=3 2023-10-12T00:41:34.690 DEBUG:boto:Data: 2023-10-12T00:41:34.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-12T00:41:34.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.691 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.692 DEBUG:boto:Params: {} 2023-10-12T00:41:34.692 DEBUG:boto:Token: None 2023-10-12T00:41:34.693 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?partNumber=3&uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZiQq4DCdiHQlpnpcLx/KR5BWRcw= 2023-10-12T00:41:34.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZiQq4DCdiHQlpnpcLx/KR5BWRcw='} 2023-10-12T00:41:34.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd50625f7b96b131-00652740be-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.752 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.753 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.753 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=4 2023-10-12T00:41:34.753 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=4 2023-10-12T00:41:34.754 DEBUG:boto:Method: PUT 2023-10-12T00:41:34.754 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4&partNumber=4 2023-10-12T00:41:34.755 DEBUG:boto:Data: 2023-10-12T00:41:34.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-12T00:41:34.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.756 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.757 DEBUG:boto:Params: {} 2023-10-12T00:41:34.757 DEBUG:boto:Token: None 2023-10-12T00:41:34.758 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?partNumber=4&uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9+sfeAR1GcWwMuW+yU2kttzpsyk= 2023-10-12T00:41:34.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9+sfeAR1GcWwMuW+yU2kttzpsyk='} 2023-10-12T00:41:34.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ac3af6dbf57116f-00652740be-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.799 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.799 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.800 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.800 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.800 DEBUG:boto:Method: GET 2023-10-12T00:41:34.801 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.801 DEBUG:boto:Data: 2023-10-12T00:41:34.802 DEBUG:boto:Headers: {} 2023-10-12T00:41:34.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.803 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.803 DEBUG:boto:Params: {} 2023-10-12T00:41:34.803 DEBUG:boto:Token: None 2023-10-12T00:41:34.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7hhyRwK1r4+IvWPPcitWyO5WF8I= 2023-10-12T00:41:34.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7hhyRwK1r4+IvWPPcitWyO5WF8I=', 'Content-Length': '0'} 2023-10-12T00:41:34.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba17334ba9f7840a-00652740be-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.851 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.853 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:41:34.853 DEBUG:boto:path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.854 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.855 DEBUG:boto:Method: POST 2023-10-12T00:41:34.855 DEBUG:boto:Path: /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.856 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-10-12T00:41:34.857 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-10-12T00:41:34.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:41:34.858 DEBUG:boto:Port: 8000 2023-10-12T00:41:34.858 DEBUG:boto:Params: {} 2023-10-12T00:41:34.859 DEBUG:boto:Token: None 2023-10-12T00:41:34.859 DEBUG:boto:StringToSign: POST text/xml Thu, 12 Oct 2023 00:41:34 GMT /wafmoa-50/MULTIPART?uploadId=2~0te1uLArWld9r1Fl9cpiBsy8KvGfZt4 2023-10-12T00:41:34.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yGK0oGQR6OvVCbpoioXh+7/XPeA= 2023-10-12T00:41:34.860 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:41:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yGK0oGQR6OvVCbpoioXh+7/XPeA=', 'Content-Length': '460'} 2023-10-12T00:41:34.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bb64b50433c06e35-00652740be-168d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:41:34.872 DEBUG:boto:http://smithi112.front.sepia.ceph.com:8000/wafmoa-50/MULTIPARTwafmoa-50MULTIPART 2023-10-12T00:41:34.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:34.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:34.873 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:34.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:34.922 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:41:34.962 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:41:34.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:34.963 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:41:34.963 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:41:34.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:34.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:34.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:41:34.964 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:41:34.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:34.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:34.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071292.635469_512.1", 2023-10-12T00:41:34.965 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:41:32.635469Z" 2023-10-12T00:41:34.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:34.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:34.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:41:34.966 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:41:34.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:34.966 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:41:34.967 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071292.635469_512.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:34.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:41:34.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:34.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:34.969 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:35.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:35.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:35.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:35.046+0000 7faa27f0f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:35.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:35.047+0000 7faa27f0f780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:35.059 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:41:35.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:41:35.060 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:41:35.060 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:41:35.060 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:41:35.061 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:41:35.061 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:35.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.061 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:41:35.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:35.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:41:35.062 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:35.062 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:35.063 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:41:35.063 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:35.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:41:35.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:35.063 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:41:35.064 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:35.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:35.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:35.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:41:35.065 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:35.065 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:35.065 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:41:35.065 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:41:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:41:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071292.608059_505.1", 2023-10-12T00:41:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:32.608059Z", 2023-10-12T00:41:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:41:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:41:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:41:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:41:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:41:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:41:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:41:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:41:35Z" 2023-10-12T00:41:35.074 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:41:35.075 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071292.608059_505.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:32.608059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:41:35Z" } 2023-10-12T00:41:35.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071292.608059_505.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:32.608059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:41:35.076 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071292.635469_512.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:35.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071292.608059_505.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:35.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071292.635469_512.1 target=1_1697071292.608059_505.1 2023-10-12T00:41:40.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:40.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:41:40.080 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:41:40.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:40.138 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:40.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:40.198+0000 7f5e31f61780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:40.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:40.199+0000 7f5e31f61780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:41:40.213 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:41:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:41:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:41:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:41:40.214 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:41:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:41:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:41:40.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:41:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:41:40.216 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:41:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:40.217 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:41:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:40.218 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:41:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:41:40.219 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:41:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:40.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:41:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:40.221 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071292.635469_512.1", 2023-10-12T00:41:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:40.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:41:32.635469Z", 2023-10-12T00:41:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:41:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:41:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:41:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:41:40.224 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:41:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:41:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:41:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:41:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:41:40.225 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:41:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:41:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:41:40.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:41:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:41:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:41:40.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:41:40.228 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:41:40Z" 2023-10-12T00:41:40.228 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:41:40.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071292.635469_512.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:41:32.635469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:41:40Z" } 2023-10-12T00:41:40.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071292.635469_512.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:41:32.635469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:41:40.230 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071292.635469_512.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:40.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071292.635469_512.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:41:40.231 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:41:40.231 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-50 2023-10-12T00:41:40.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:40.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:40.234 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:40.291 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:40.292 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:40.326+0000 7ff830c45780 1 waiting to reach incremental sync.. 2023-10-12T00:41:45.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:45.327+0000 7ff830c45780 1 waiting to reach incremental sync.. 2023-10-12T00:41:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:50.328+0000 7ff830c45780 1 bucket sync caught up with source: 2023-10-12T00:41:50.330 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000011.34.6, , , , , ] 2023-10-12T00:41:50.330 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000011.34.6, , , , , ] 2023-10-12T00:41:50.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:50.328+0000 7ff830c45780 0 bucket checkpoint complete 2023-10-12T00:41:50.346 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-50 2023-10-12T00:41:50.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:50.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:41:50.347 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:41:50.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:41:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:41:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:50.445+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:41:55.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:41:55.446+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:00.447+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:05.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:05.448+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:10.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:10.447+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:15.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:15.448+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:20.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:20.449+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:25.450+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:30.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:30.451+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:35.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:35.452+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:40.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:40.452+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:45.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:45.452+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:50.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:50.453+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:42:55.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:42:55.453+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:00.454+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:05.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:05.455+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:10.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:10.456+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:15.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:15.457+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:20.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:20.457+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:25.457+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:30.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:30.458+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:35.458+0000 7fd94f05a780 1 waiting to reach incremental sync.. 2023-10-12T00:43:40.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:40.460+0000 7fd94f05a780 1 bucket sync caught up with source: 2023-10-12T00:43:40.461 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.266.6, , , , , ] 2023-10-12T00:43:40.462 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.266.6, , , , , ] 2023-10-12T00:43:40.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:40.460+0000 7fd94f05a780 0 bucket checkpoint complete 2023-10-12T00:43:40.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-50 zones={a2, a1} 2023-10-12T00:43:40.478 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.479 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.479 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.479 DEBUG:boto:Path: /wafmoa-50/ 2023-10-12T00:43:40.480 DEBUG:boto:Data: 2023-10-12T00:43:40.480 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.481 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.481 DEBUG:boto:Params: {} 2023-10-12T00:43:40.482 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:43:40.483 DEBUG:boto:Token: None 2023-10-12T00:43:40.483 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/ 2023-10-12T00:43:40.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rPDlkZ4dYMsW/qpTOZVG++2r7HM= 2023-10-12T00:43:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rPDlkZ4dYMsW/qpTOZVG++2r7HM=', 'Content-Length': '0'} 2023-10-12T00:43:40.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3d9501a10d2315d-006527413c-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.489 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.490 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.490 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.490 DEBUG:boto:Path: /wafmoa-50/ 2023-10-12T00:43:40.491 DEBUG:boto:Data: 2023-10-12T00:43:40.491 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.492 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.492 DEBUG:boto:Params: {} 2023-10-12T00:43:40.493 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:43:40.493 DEBUG:boto:Token: None 2023-10-12T00:43:40.494 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/ 2023-10-12T00:43:40.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rPDlkZ4dYMsW/qpTOZVG++2r7HM= 2023-10-12T00:43:40.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rPDlkZ4dYMsW/qpTOZVG++2r7HM=', 'Content-Length': '0'} 2023-10-12T00:43:40.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dec69eda08ccfbc-006527413c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:43:40.499 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.500 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.500 DEBUG:boto:path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.500 DEBUG:boto:auth_path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.501 DEBUG:boto:Method: GET 2023-10-12T00:43:40.501 DEBUG:boto:Path: /wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.502 DEBUG:boto:Data: 2023-10-12T00:43:40.502 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.503 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.503 DEBUG:boto:Params: {} 2023-10-12T00:43:40.504 DEBUG:boto:Token: None 2023-10-12T00:43:40.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/?versions 2023-10-12T00:43:40.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0= 2023-10-12T00:43:40.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0=', 'Content-Length': '0'} 2023-10-12T00:43:40.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009838dfc29af39d67-006527413c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.508 DEBUG:boto:b'wafmoa-50999falseMULTIPARTnulltrue2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.509 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-12T00:43:40.509 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:43:40.510 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.510 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.511 DEBUG:boto:path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.511 DEBUG:boto:auth_path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.512 DEBUG:boto:Method: GET 2023-10-12T00:43:40.512 DEBUG:boto:Path: /wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.512 DEBUG:boto:Data: 2023-10-12T00:43:40.513 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.514 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.514 DEBUG:boto:Params: {} 2023-10-12T00:43:40.515 DEBUG:boto:Token: None 2023-10-12T00:43:40.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/?versions 2023-10-12T00:43:40.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0= 2023-10-12T00:43:40.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0=', 'Content-Length': '0'} 2023-10-12T00:43:40.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159e9c7669ea3594-006527413c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.521 DEBUG:boto:b'wafmoa-50999falseMULTIPARTnulltrue2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.522 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-12T00:43:40.522 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.523 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.523 DEBUG:boto:path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.523 DEBUG:boto:auth_path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.524 DEBUG:boto:Method: GET 2023-10-12T00:43:40.524 DEBUG:boto:Path: /wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.525 DEBUG:boto:Data: 2023-10-12T00:43:40.525 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.526 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.526 DEBUG:boto:Params: {} 2023-10-12T00:43:40.527 DEBUG:boto:Token: None 2023-10-12T00:43:40.527 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/?versions 2023-10-12T00:43:40.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0= 2023-10-12T00:43:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0=', 'Content-Length': '0'} 2023-10-12T00:43:40.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c146668b64d61bea-006527413c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.531 DEBUG:boto:b'wafmoa-50999falseMULTIPARTnulltrue2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.532 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.532 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.533 DEBUG:boto:path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.533 DEBUG:boto:auth_path=/wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.533 DEBUG:boto:Method: GET 2023-10-12T00:43:40.534 DEBUG:boto:Path: /wafmoa-50/?versions&max-keys=999 2023-10-12T00:43:40.534 DEBUG:boto:Data: 2023-10-12T00:43:40.535 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.535 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.536 DEBUG:boto:Params: {} 2023-10-12T00:43:40.536 DEBUG:boto:Token: None 2023-10-12T00:43:40.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/?versions 2023-10-12T00:43:40.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0= 2023-10-12T00:43:40.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQAaJ7HXd1ZNzsrm7y/ULSQAwU0=', 'Content-Length': '0'} 2023-10-12T00:43:40.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973a503a8deb8712-006527413c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.540 DEBUG:boto:b'wafmoa-50999falseMULTIPARTnulltrue2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-12T00:43:40.541 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.542 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.542 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.543 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.543 DEBUG:boto:Method: GET 2023-10-12T00:43:40.543 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.544 DEBUG:boto:Data: 2023-10-12T00:43:40.544 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.545 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.545 DEBUG:boto:Params: {} 2023-10-12T00:43:40.546 DEBUG:boto:Token: None 2023-10-12T00:43:40.546 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE= 2023-10-12T00:43:40.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE=', 'Content-Length': '0'} 2023-10-12T00:43:40.553 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-50:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006e572762fe5ac50a-006527413c-1689-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.626 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.627 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.628 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.628 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.628 DEBUG:boto:Method: GET 2023-10-12T00:43:40.629 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.629 DEBUG:boto:Data: 2023-10-12T00:43:40.630 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.631 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.631 DEBUG:boto:Params: {} 2023-10-12T00:43:40.631 DEBUG:boto:Token: None 2023-10-12T00:43:40.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE= 2023-10-12T00:43:40.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE=', 'Content-Length': '0'} 2023-10-12T00:43:40.639 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008bd7638ca4360718-006527413c-168d-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.725 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.726 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.726 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.727 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.727 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.728 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.728 DEBUG:boto:Data: 2023-10-12T00:43:40.728 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.729 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.729 DEBUG:boto:Params: {} 2023-10-12T00:43:40.730 DEBUG:boto:Token: None 2023-10-12T00:43:40.730 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRCcXVc/ChPgtYg7Y/8I8B5HHnE= 2023-10-12T00:43:40.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRCcXVc/ChPgtYg7Y/8I8B5HHnE=', 'Content-Length': '0'} 2023-10-12T00:43:40.734 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-50:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f0ff98f2752a16ef-006527413c-1689-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.734 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.735 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.735 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.736 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.736 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.736 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.737 DEBUG:boto:Data: 2023-10-12T00:43:40.737 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.738 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.738 DEBUG:boto:Params: {} 2023-10-12T00:43:40.739 DEBUG:boto:Token: None 2023-10-12T00:43:40.739 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRCcXVc/ChPgtYg7Y/8I8B5HHnE= 2023-10-12T00:43:40.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRCcXVc/ChPgtYg7Y/8I8B5HHnE=', 'Content-Length': '0'} 2023-10-12T00:43:40.742 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000043029606b185fcb5-006527413c-168d-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-12T00:43:40.743 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.743 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.744 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.744 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.744 DEBUG:boto:Method: GET 2023-10-12T00:43:40.745 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.745 DEBUG:boto:Data: 2023-10-12T00:43:40.746 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.747 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.747 DEBUG:boto:Params: {} 2023-10-12T00:43:40.747 DEBUG:boto:Token: None 2023-10-12T00:43:40.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE= 2023-10-12T00:43:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE=', 'Content-Length': '0'} 2023-10-12T00:43:40.753 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-50:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d59e2d1ab0d14669-006527413c-1689-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.834 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.835 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.835 DEBUG:boto:path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.836 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.836 DEBUG:boto:Method: GET 2023-10-12T00:43:40.837 DEBUG:boto:Path: /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.837 DEBUG:boto:Data: 2023-10-12T00:43:40.837 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.838 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.839 DEBUG:boto:Params: {} 2023-10-12T00:43:40.839 DEBUG:boto:Token: None 2023-10-12T00:43:40.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART?versionId=null 2023-10-12T00:43:40.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE= 2023-10-12T00:43:40.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOlA14w5nK25l09PZHhUIXuRhzE=', 'Content-Length': '0'} 2023-10-12T00:43:40.845 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ec92d53c05931c24-006527413c-168d-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.935 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.936 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.936 DEBUG:boto:path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.937 DEBUG:boto:auth_path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.937 DEBUG:boto:Method: GET 2023-10-12T00:43:40.938 DEBUG:boto:Path: /wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.938 DEBUG:boto:Data: 2023-10-12T00:43:40.939 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.939 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.940 DEBUG:boto:Params: {} 2023-10-12T00:43:40.940 DEBUG:boto:Token: None 2023-10-12T00:43:40.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/ 2023-10-12T00:43:40.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P1PzhaOsbe7s3JX0bdXUgCse+us= 2023-10-12T00:43:40.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P1PzhaOsbe7s3JX0bdXUgCse+us=', 'Content-Length': '0'} 2023-10-12T00:43:40.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d3e4ecb1655ab71-006527413c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.945 DEBUG:boto:b'wafmoa-50MULTIPART1000falseMULTIPART2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.946 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:40.946 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:40.947 DEBUG:boto:path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.947 DEBUG:boto:auth_path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.948 DEBUG:boto:Method: GET 2023-10-12T00:43:40.948 DEBUG:boto:Path: /wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:40.948 DEBUG:boto:Data: 2023-10-12T00:43:40.949 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.949 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.950 DEBUG:boto:Params: {} 2023-10-12T00:43:40.950 DEBUG:boto:Token: None 2023-10-12T00:43:40.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/ 2023-10-12T00:43:40.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P1PzhaOsbe7s3JX0bdXUgCse+us= 2023-10-12T00:43:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P1PzhaOsbe7s3JX0bdXUgCse+us=', 'Content-Length': '0'} 2023-10-12T00:43:40.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b61f2706964910-006527413c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.955 DEBUG:boto:b'wafmoa-50MULTIPART1000falseMULTIPART2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:40.955 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.956 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.956 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.956 DEBUG:boto:Path: /wafmoa-50/MULTIPART 2023-10-12T00:43:40.957 DEBUG:boto:Data: 2023-10-12T00:43:40.957 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.958 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.958 DEBUG:boto:Params: {} 2023-10-12T00:43:40.959 DEBUG:boto:Token: None 2023-10-12T00:43:40.959 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART 2023-10-12T00:43:40.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0ZqF2rqUP3QC5VAsDgPwgxIXsQs= 2023-10-12T00:43:40.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0ZqF2rqUP3QC5VAsDgPwgxIXsQs=', 'Content-Length': '0'} 2023-10-12T00:43:40.962 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-50:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000fca29fbc3ec37743-006527413c-1689-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.962 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.963 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.963 DEBUG:boto:Method: HEAD 2023-10-12T00:43:40.964 DEBUG:boto:Path: /wafmoa-50/MULTIPART 2023-10-12T00:43:40.964 DEBUG:boto:Data: 2023-10-12T00:43:40.964 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:40.965 DEBUG:boto:Port: 8000 2023-10-12T00:43:40.966 DEBUG:boto:Params: {} 2023-10-12T00:43:40.966 DEBUG:boto:Token: None 2023-10-12T00:43:40.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART 2023-10-12T00:43:40.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0ZqF2rqUP3QC5VAsDgPwgxIXsQs= 2023-10-12T00:43:40.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0ZqF2rqUP3QC5VAsDgPwgxIXsQs=', 'Content-Length': '0'} 2023-10-12T00:43:40.969 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009b28b6892f2f8af1-006527413c-168d-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:40.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-12T00:43:40.970 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.971 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:40.971 DEBUG:boto:Method: GET 2023-10-12T00:43:40.972 DEBUG:boto:Path: /wafmoa-50/MULTIPART 2023-10-12T00:43:40.972 DEBUG:boto:Data: 2023-10-12T00:43:40.972 DEBUG:boto:Headers: {} 2023-10-12T00:43:40.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:40.973 DEBUG:boto:Port: 8001 2023-10-12T00:43:40.974 DEBUG:boto:Params: {} 2023-10-12T00:43:40.974 DEBUG:boto:Token: None 2023-10-12T00:43:40.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:40 GMT /wafmoa-50/MULTIPART 2023-10-12T00:43:40.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z3qdLSusBvam8R0R4bZsnhkNAoo= 2023-10-12T00:43:40.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z3qdLSusBvam8R0R4bZsnhkNAoo=', 'Content-Length': '0'} 2023-10-12T00:43:40.980 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-50:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000007c0a5aa52762abd-006527413c-1689-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.058 DEBUG:boto:path=/wafmoa-50/MULTIPART 2023-10-12T00:43:41.059 DEBUG:boto:auth_path=/wafmoa-50/MULTIPART 2023-10-12T00:43:41.059 DEBUG:boto:Method: GET 2023-10-12T00:43:41.060 DEBUG:boto:Path: /wafmoa-50/MULTIPART 2023-10-12T00:43:41.060 DEBUG:boto:Data: 2023-10-12T00:43:41.061 DEBUG:boto:Headers: {} 2023-10-12T00:43:41.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:41.061 DEBUG:boto:Port: 8000 2023-10-12T00:43:41.062 DEBUG:boto:Params: {} 2023-10-12T00:43:41.062 DEBUG:boto:Token: None 2023-10-12T00:43:41.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:41 GMT /wafmoa-50/MULTIPART 2023-10-12T00:43:41.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pla2/k9Bg2eY1heu+dDgM+xfPy0= 2023-10-12T00:43:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pla2/k9Bg2eY1heu+dDgM+xfPy0=', 'Content-Length': '0'} 2023-10-12T00:43:41.068 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:41:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000aa68ec60f34cb306-006527413d-168d-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.154 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:41.155 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:41.155 DEBUG:boto:path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.155 DEBUG:boto:auth_path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.156 DEBUG:boto:Method: GET 2023-10-12T00:43:41.156 DEBUG:boto:Path: /wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.157 DEBUG:boto:Data: 2023-10-12T00:43:41.157 DEBUG:boto:Headers: {} 2023-10-12T00:43:41.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:41.158 DEBUG:boto:Port: 8001 2023-10-12T00:43:41.158 DEBUG:boto:Params: {} 2023-10-12T00:43:41.159 DEBUG:boto:Token: None 2023-10-12T00:43:41.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:41 GMT /wafmoa-50/ 2023-10-12T00:43:41.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e7dt8PiJ4CccvtPh58doLlVZcQE= 2023-10-12T00:43:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e7dt8PiJ4CccvtPh58doLlVZcQE=', 'Content-Length': '0'} 2023-10-12T00:43:41.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce00cf0df64661d-006527413d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.163 DEBUG:boto:b'wafmoa-50MULTIPART1000falseMULTIPART2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:41.164 DEBUG:boto:path=/wafmoa-50/ 2023-10-12T00:43:41.164 DEBUG:boto:auth_path=/wafmoa-50/ 2023-10-12T00:43:41.165 DEBUG:boto:path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.165 DEBUG:boto:auth_path=/wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.166 DEBUG:boto:Method: GET 2023-10-12T00:43:41.166 DEBUG:boto:Path: /wafmoa-50/?prefix=MULTIPART 2023-10-12T00:43:41.167 DEBUG:boto:Data: 2023-10-12T00:43:41.167 DEBUG:boto:Headers: {} 2023-10-12T00:43:41.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:41.168 DEBUG:boto:Port: 8000 2023-10-12T00:43:41.168 DEBUG:boto:Params: {} 2023-10-12T00:43:41.169 DEBUG:boto:Token: None 2023-10-12T00:43:41.169 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:41 GMT /wafmoa-50/ 2023-10-12T00:43:41.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e7dt8PiJ4CccvtPh58doLlVZcQE= 2023-10-12T00:43:41.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e7dt8PiJ4CccvtPh58doLlVZcQE=', 'Content-Length': '0'} 2023-10-12T00:43:41.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6840f2af5f2bf8b-006527413d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.173 DEBUG:boto:b'wafmoa-50MULTIPART1000falseMULTIPART2023-10-12T00:41:34.865Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:43:41.173 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-50 zones={a2, a1} 2023-10-12T00:43:41.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-10-12T00:43:41.175 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-52 2023-10-12T00:43:41.176 DEBUG:boto:path=/wafmoa-52/ 2023-10-12T00:43:41.176 DEBUG:boto:auth_path=/wafmoa-52/ 2023-10-12T00:43:41.176 DEBUG:boto:Method: PUT 2023-10-12T00:43:41.177 DEBUG:boto:Path: /wafmoa-52/ 2023-10-12T00:43:41.177 DEBUG:boto:Data: 2023-10-12T00:43:41.177 DEBUG:boto:Headers: {} 2023-10-12T00:43:41.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:41.178 DEBUG:boto:Port: 8000 2023-10-12T00:43:41.179 DEBUG:boto:Params: {} 2023-10-12T00:43:41.179 DEBUG:boto:Token: None 2023-10-12T00:43:41.179 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:43:41 GMT /wafmoa-52/ 2023-10-12T00:43:41.180 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8047A7mDl465NWM+R36C9xMZ68Q= 2023-10-12T00:43:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8047A7mDl465NWM+R36C9xMZ68Q=', 'Content-Length': '0'} 2023-10-12T00:43:41.195 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a77dbffd439aaf1f-006527413d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.196 DEBUG:boto:path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:41.196 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:41.197 DEBUG:boto:Method: PUT 2023-10-12T00:43:41.197 DEBUG:boto:Path: /wafmoa-52/testobj-sse-c 2023-10-12T00:43:41.198 DEBUG:boto:Data: 2023-10-12T00:43:41.198 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:43:41.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:41.199 DEBUG:boto:Port: 8000 2023-10-12T00:43:41.199 DEBUG:boto:Params: {} 2023-10-12T00:43:41.200 DEBUG:boto:Token: None 2023-10-12T00:43:41.200 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 12 Oct 2023 00:43:41 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /wafmoa-52/testobj-sse-c 2023-10-12T00:43:41.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7PY4OsqnWpNvljPgPLJGccCsaiI= 2023-10-12T00:43:41.201 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7PY4OsqnWpNvljPgPLJGccCsaiI='} 2023-10-12T00:43:41.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000c8a49d253c02b9fd-006527413d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.208 DEBUG:boto:path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:41.208 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:41.209 DEBUG:boto:Method: PUT 2023-10-12T00:43:41.209 DEBUG:boto:Path: /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:41.210 DEBUG:boto:Data: 2023-10-12T00:43:41.210 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-12T00:43:41.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:41.211 DEBUG:boto:Port: 8000 2023-10-12T00:43:41.211 DEBUG:boto:Params: {} 2023-10-12T00:43:41.212 DEBUG:boto:Token: None 2023-10-12T00:43:41.212 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Thu, 12 Oct 2023 00:43:41 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:41.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aSYYCmH4s0F+SDQAHeJgDEmuSRQ= 2023-10-12T00:43:41.213 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aSYYCmH4s0F+SDQAHeJgDEmuSRQ='} 2023-10-12T00:43:41.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6d2FmbW9hLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000073dc59dcbfafcff8-006527413d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:41.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:41.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:41.264 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:43:41.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:41.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:41.346 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:43:41.359 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:43:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:43:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:43:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:43:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:43:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071421.194579_516.1", 2023-10-12T00:43:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:43:41.194579Z" 2023-10-12T00:43:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:43:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:43:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.362 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:43:41.363 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071421.194579_516.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:41.364 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:43:41.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:41.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:41.365 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:43:41.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:41.415 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:41.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:41.443+0000 7f98b21ed780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:41.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:41.444+0000 7f98b21ed780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:41.457 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:43:41.457 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:43:41.458 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:43:41.458 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:43:41.458 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:43:41.458 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:43:41.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:41.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.459 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:43:41.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:43:41.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:41.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:41.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:43:41.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:41.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:43:41.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:41.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:43:41.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:41.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:43:41.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:41.462 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:41.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:43:41.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:41.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:41.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:41.464 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:43:41.464 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:41.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.465 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:43:41.465 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:41.465 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:41.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071421.188563_513.1", 2023-10-12T00:43:41.466 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:41.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:41.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:41.466 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:41.188563Z", 2023-10-12T00:43:41.466 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:41.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:41.467 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:43:41.468 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:41.468 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:41.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:43:41.468 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:41.468 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:41.469 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:41.469 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:43:41.469 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:41.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:41.470 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:43:41.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.470 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:43:41.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:43:41.471 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:43:41.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:41.471 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:43:41Z" 2023-10-12T00:43:41.471 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:43:41.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071421.188563_513.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:43:41.188563Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:43:41Z" } 2023-10-12T00:43:41.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071421.188563_513.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:43:41.188563Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:43:41.473 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071421.194579_516.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:41.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071421.188563_513.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:41.474 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071421.194579_516.1 target=1_1697071421.188563_513.1 2023-10-12T00:43:46.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.477 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:43:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:46.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:46.565+0000 7fd6c5f55780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:46.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:46.566+0000 7fd6c5f55780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:46.579 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:43:46.579 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:43:46.580 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:43:46.580 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:43:46.580 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:43:46.580 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:43:46.581 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:46.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.581 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:43:46.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.582 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:43:46.582 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:46.582 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:46.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:43:46.583 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:46.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:43:46.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:46.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:43:46.583 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:46.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:43:46.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:46.585 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:46.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:43:46.585 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:46.585 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:46.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:46.586 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:43:46.586 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:46.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.587 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:43:46.587 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:46.587 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:46.587 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071421.194579_516.1", 2023-10-12T00:43:46.588 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:46.588 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:46.588 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:46.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:41.194579Z", 2023-10-12T00:43:46.588 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:46.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.589 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:43:46.589 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:46.590 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:46.590 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:43:46.590 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:46.590 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:46.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:46.591 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:43:46.591 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:46.591 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.592 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:43:46.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.592 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:43:46.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:43:46.593 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:43:46.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.593 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:43:46Z" 2023-10-12T00:43:46.593 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:43:46.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071421.194579_516.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:43:41.194579Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:43:46Z" } 2023-10-12T00:43:46.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071421.194579_516.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:43:41.194579Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:43:46.595 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071421.194579_516.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:46.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071421.194579_516.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:46.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:43:46.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:43:46.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:43:46.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:43:46.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:46.655 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:46.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:46.689+0000 7f7abfe6a780 1 bucket sync caught up with source: 2023-10-12T00:43:46.689 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.12.6, , , , 00000000001.22.6, , , , , ] 2023-10-12T00:43:46.689 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.12.6, , , , 00000000001.22.6, , , , , ] 2023-10-12T00:43:46.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:46.689+0000 7f7abfe6a780 0 bucket checkpoint complete 2023-10-12T00:43:46.702 DEBUG:boto:path=/wafmoa-52/ 2023-10-12T00:43:46.703 DEBUG:boto:auth_path=/wafmoa-52/ 2023-10-12T00:43:46.703 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.704 DEBUG:boto:Path: /wafmoa-52/ 2023-10-12T00:43:46.704 DEBUG:boto:Data: 2023-10-12T00:43:46.705 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:46.705 DEBUG:boto:Port: 8001 2023-10-12T00:43:46.706 DEBUG:boto:Params: {} 2023-10-12T00:43:46.706 DEBUG:boto:Token: None 2023-10-12T00:43:46.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-52/ 2023-10-12T00:43:46.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yJDXsy+rpevLQ4wYJ9SuOhPDhH0= 2023-10-12T00:43:46.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yJDXsy+rpevLQ4wYJ9SuOhPDhH0=', 'Content-Length': '0'} 2023-10-12T00:43:46.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000f18b1b6001496b-0065274142-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.711 DEBUG:boto:path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.711 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.711 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.712 DEBUG:boto:Path: /wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.712 DEBUG:boto:Data: 2023-10-12T00:43:46.712 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-12T00:43:46.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:46.713 DEBUG:boto:Port: 8001 2023-10-12T00:43:46.713 DEBUG:boto:Params: {} 2023-10-12T00:43:46.714 DEBUG:boto:Token: None 2023-10-12T00:43:46.714 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nh6SfCgn0Ao/arTJEHpl8S9Fch8= 2023-10-12T00:43:46.715 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nh6SfCgn0Ao/arTJEHpl8S9Fch8=', 'Content-Length': '0'} 2023-10-12T00:43:46.717 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-52:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000498bc52fc84fa043-0065274142-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.717 DEBUG:boto:path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.718 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.718 DEBUG:boto:Method: GET 2023-10-12T00:43:46.719 DEBUG:boto:Path: /wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.719 DEBUG:boto:Data: 2023-10-12T00:43:46.719 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-12T00:43:46.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:46.720 DEBUG:boto:Port: 8001 2023-10-12T00:43:46.721 DEBUG:boto:Params: {} 2023-10-12T00:43:46.721 DEBUG:boto:Token: None 2023-10-12T00:43:46.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /wafmoa-52/testobj-sse-c 2023-10-12T00:43:46.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i3mv3fSBW/WVgkPBkTEB8GcwZlg= 2023-10-12T00:43:46.722 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i3mv3fSBW/WVgkPBkTEB8GcwZlg=', 'Content-Length': '0'} 2023-10-12T00:43:46.725 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-52:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000da9324edfa71e77c-0065274142-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.725 DEBUG:boto:path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.726 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.726 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.727 DEBUG:boto:Path: /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.727 DEBUG:boto:Data: 2023-10-12T00:43:46.728 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:46.728 DEBUG:boto:Port: 8001 2023-10-12T00:43:46.729 DEBUG:boto:Params: {} 2023-10-12T00:43:46.729 DEBUG:boto:Token: None 2023-10-12T00:43:46.729 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Yu000aanqHmaAUM4DKEVAFzwS0= 2023-10-12T00:43:46.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Yu000aanqHmaAUM4DKEVAFzwS0=', 'Content-Length': '0'} 2023-10-12T00:43:46.732 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-52:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000f4cec56c92cf641b-0065274142-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.733 DEBUG:boto:path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.733 DEBUG:boto:auth_path=/wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.733 DEBUG:boto:Method: GET 2023-10-12T00:43:46.734 DEBUG:boto:Path: /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.734 DEBUG:boto:Data: 2023-10-12T00:43:46.735 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:43:46.736 DEBUG:boto:Port: 8001 2023-10-12T00:43:46.736 DEBUG:boto:Params: {} 2023-10-12T00:43:46.737 DEBUG:boto:Token: None 2023-10-12T00:43:46.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-52/testobj-sse-kms 2023-10-12T00:43:46.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:91JS3Xf4UJrSpC9LQX7NWWdEK7k= 2023-10-12T00:43:46.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:91JS3Xf4UJrSpC9LQX7NWWdEK7k=', 'Content-Length': '0'} 2023-10-12T00:43:46.740 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-52:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000ed66fd2f535843e1-0065274142-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-10-12T00:43:46.741 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-53 2023-10-12T00:43:46.742 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:43:46.742 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:43:46.743 DEBUG:boto:Method: PUT 2023-10-12T00:43:46.743 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:43:46.744 DEBUG:boto:Data: 2023-10-12T00:43:46.744 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.745 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.745 DEBUG:boto:Params: {} 2023-10-12T00:43:46.746 DEBUG:boto:Token: None 2023-10-12T00:43:46.746 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/ 2023-10-12T00:43:46.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3I0aQWHcuZm+kDZ38sMrgcxkKXs= 2023-10-12T00:43:46.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3I0aQWHcuZm+kDZ38sMrgcxkKXs=', 'Content-Length': '0'} 2023-10-12T00:43:46.761 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ea56c85625b065b-0065274142-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.762 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:43:46.762 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:43:46.763 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.763 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:43:46.763 DEBUG:boto:Data: 2023-10-12T00:43:46.764 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.765 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.765 DEBUG:boto:Params: {} 2023-10-12T00:43:46.765 DEBUG:boto:Token: None 2023-10-12T00:43:46.766 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/ 2023-10-12T00:43:46.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY= 2023-10-12T00:43:46.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY=', 'Content-Length': '0'} 2023-10-12T00:43:46.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039722e9288d72c11-0065274142-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.770 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:43:46.770 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:43:46.771 DEBUG:boto:Method: PUT 2023-10-12T00:43:46.771 DEBUG:boto:Path: /wafmoa-53/a 2023-10-12T00:43:46.772 DEBUG:boto:Data: 2023-10-12T00:43:46.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:43:46.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.773 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.773 DEBUG:boto:Params: {} 2023-10-12T00:43:46.773 DEBUG:boto:Token: None 2023-10-12T00:43:46.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/a 2023-10-12T00:43:46.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MGmF8FUCH8K1Zvrk25NZ/WHZRR8= 2023-10-12T00:43:46.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MGmF8FUCH8K1Zvrk25NZ/WHZRR8='} 2023-10-12T00:43:46.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060089bf3aa4c59a2-0065274142-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.782 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:43:46.782 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:43:46.783 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.783 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:43:46.783 DEBUG:boto:Data: 2023-10-12T00:43:46.784 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.784 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.785 DEBUG:boto:Params: {} 2023-10-12T00:43:46.786 DEBUG:boto:Token: None 2023-10-12T00:43:46.786 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/ 2023-10-12T00:43:46.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY= 2023-10-12T00:43:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY=', 'Content-Length': '0'} 2023-10-12T00:43:46.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000537c31d5af680471-0065274142-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.791 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:43:46.792 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:43:46.792 DEBUG:boto:Method: PUT 2023-10-12T00:43:46.793 DEBUG:boto:Path: /wafmoa-53/b 2023-10-12T00:43:46.793 DEBUG:boto:Data: 2023-10-12T00:43:46.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:43:46.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.794 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.795 DEBUG:boto:Params: {} 2023-10-12T00:43:46.795 DEBUG:boto:Token: None 2023-10-12T00:43:46.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/b 2023-10-12T00:43:46.796 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iPpz8tm14MbBclTipYAh0J4Lc9w= 2023-10-12T00:43:46.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iPpz8tm14MbBclTipYAh0J4Lc9w='} 2023-10-12T00:43:46.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d13cf0aeaaed66e-0065274142-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.839 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:43:46.839 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:43:46.840 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.840 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:43:46.840 DEBUG:boto:Data: 2023-10-12T00:43:46.841 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.842 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.842 DEBUG:boto:Params: {} 2023-10-12T00:43:46.843 DEBUG:boto:Token: None 2023-10-12T00:43:46.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/ 2023-10-12T00:43:46.843 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY= 2023-10-12T00:43:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY=', 'Content-Length': '0'} 2023-10-12T00:43:46.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024a70c4e13962170-0065274142-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.847 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:43:46.847 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:43:46.848 DEBUG:boto:Method: PUT 2023-10-12T00:43:46.848 DEBUG:boto:Path: /wafmoa-53/c 2023-10-12T00:43:46.848 DEBUG:boto:Data: 2023-10-12T00:43:46.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:43:46.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.850 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.850 DEBUG:boto:Params: {} 2023-10-12T00:43:46.851 DEBUG:boto:Token: None 2023-10-12T00:43:46.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/c 2023-10-12T00:43:46.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5C0bA7BEal2zl3QD4HrLDmUexQo= 2023-10-12T00:43:46.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5C0bA7BEal2zl3QD4HrLDmUexQo='} 2023-10-12T00:43:46.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8ed1bddc9d079a4-0065274142-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.858 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:43:46.858 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:43:46.859 DEBUG:boto:Method: HEAD 2023-10-12T00:43:46.859 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:43:46.861 DEBUG:boto:Data: 2023-10-12T00:43:46.862 DEBUG:boto:Headers: {} 2023-10-12T00:43:46.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.863 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.864 DEBUG:boto:Params: {} 2023-10-12T00:43:46.865 DEBUG:boto:Token: None 2023-10-12T00:43:46.865 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/ 2023-10-12T00:43:46.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY= 2023-10-12T00:43:46.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mcwzBlH17zN2ptIsKqrQQa7BUhY=', 'Content-Length': '0'} 2023-10-12T00:43:46.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016e32c7b5d6e6753-0065274142-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.869 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:43:46.870 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:43:46.870 DEBUG:boto:Method: PUT 2023-10-12T00:43:46.871 DEBUG:boto:Path: /wafmoa-53/d 2023-10-12T00:43:46.871 DEBUG:boto:Data: 2023-10-12T00:43:46.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:43:46.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:43:46.872 DEBUG:boto:Port: 8000 2023-10-12T00:43:46.873 DEBUG:boto:Params: {} 2023-10-12T00:43:46.873 DEBUG:boto:Token: None 2023-10-12T00:43:46.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:43:46 GMT /wafmoa-53/d 2023-10-12T00:43:46.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RwP1M8YTAnsmnwCFUqG+dwXSyhM= 2023-10-12T00:43:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:43:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RwP1M8YTAnsmnwCFUqG+dwXSyhM='} 2023-10-12T00:43:46.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009377fed588194d05-0065274142-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:43:46.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.881 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:43:46.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:46.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:46.959 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:43:46.970 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:43:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:43:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:43:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:43:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:43:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071426.760611_520.1", 2023-10-12T00:43:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:43:46.760611Z" 2023-10-12T00:43:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:43:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:43:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:46.974 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:43:46.975 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071426.760611_520.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:46.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:43:46.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:43:46.977 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:43:47.027 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:47.028 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:47.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:47.055+0000 7f219bf79780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:47.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:47.056+0000 7f219bf79780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:43:47.067 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:43:47.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:43:47.068 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:43:47.068 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:43:47.068 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:43:47.069 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:43:47.069 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:47.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.069 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:43:47.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:47.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:43:47.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:47.070 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:47.070 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:43:47.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:47.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:43:47.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:47.071 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:43:47.071 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:47.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:47.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:47.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:43:47.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:47.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:47.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:43:47.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:47.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:47.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:47.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:43:47.074 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:47.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:47.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:47.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:43:47.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:47.075 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:47.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071426.760611_520.1", 2023-10-12T00:43:47.076 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:47.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:47.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:47.076 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.760611Z", 2023-10-12T00:43:47.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:47.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:47.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:43:47.077 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:43:47.077 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:43:47.078 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:43:47.078 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:43:47.078 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:43:47.078 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:43:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:43:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:43:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:43:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:47.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:43:47.080 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:43:47.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.080 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:43:47.080 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:43:47.081 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:43:47.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:43:47.081 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:43:47Z" 2023-10-12T00:43:47.081 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:43:47.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071426.760611_520.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:43:46.760611Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:43:47Z" } 2023-10-12T00:43:47.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071426.760611_520.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:43:46.760611Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:43:47.083 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071426.760611_520.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:47.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071426.760611_520.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:43:47.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:43:47.085 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-53 2023-10-12T00:43:47.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:43:47.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:43:47.086 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:43:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:43:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:43:47.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:47.177+0000 7f0dc0f02780 1 waiting to reach incremental sync.. 2023-10-12T00:43:52.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:52.177+0000 7f0dc0f02780 1 waiting to reach incremental sync.. 2023-10-12T00:43:55.872 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:43:55.872+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T00:43:57.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:43:57.178+0000 7f0dc0f02780 1 waiting to reach incremental sync.. 2023-10-12T00:44:02.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:02.177+0000 7f0dc0f02780 1 waiting to reach incremental sync.. 2023-10-12T00:44:07.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:07.179+0000 7f0dc0f02780 1 bucket sync caught up with source: 2023-10-12T00:44:07.180 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.36.6, , , , 00000000001.22.6, , , 00000000001.33.6, , 00000000001.31.6] 2023-10-12T00:44:07.180 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.36.6, , , , 00000000001.22.6, , , 00000000001.33.6, , 00000000001.31.6] 2023-10-12T00:44:07.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:07.179+0000 7f0dc0f02780 0 bucket checkpoint complete 2023-10-12T00:44:07.197 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-53 2023-10-12T00:44:07.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:07.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:07.198 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:44:07.260 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:07.261 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:07.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:07.296+0000 7fe5f94db780 1 waiting to reach incremental sync.. 2023-10-12T00:44:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:12.298+0000 7fe5f94db780 1 bucket sync caught up with source: 2023-10-12T00:44:12.299 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.290.6, , , , 00000000001.163.6, , , 00000000001.248.6, , 00000000001.171.6] 2023-10-12T00:44:12.299 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.290.6, , , , 00000000001.163.6, , , 00000000001.248.6, , 00000000001.171.6] 2023-10-12T00:44:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:12.298+0000 7fe5f94db780 0 bucket checkpoint complete 2023-10-12T00:44:12.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-53 zones={a2, a1} 2023-10-12T00:44:12.317 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.318 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.318 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.319 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:44:12.319 DEBUG:boto:Data: 2023-10-12T00:44:12.319 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.320 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.321 DEBUG:boto:Params: {} 2023-10-12T00:44:12.321 DEBUG:boto:Token: None 2023-10-12T00:44:12.322 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/a/BGpxfsUWlq9FB3FBiN9V07UE= 2023-10-12T00:44:12.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/a/BGpxfsUWlq9FB3FBiN9V07UE=', 'Content-Length': '0'} 2023-10-12T00:44:12.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f11ca53ddada4760-006527415c-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.326 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.326 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.327 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.327 DEBUG:boto:Path: /wafmoa-53/ 2023-10-12T00:44:12.327 DEBUG:boto:Data: 2023-10-12T00:44:12.328 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.329 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.329 DEBUG:boto:Params: {} 2023-10-12T00:44:12.329 DEBUG:boto:Token: None 2023-10-12T00:44:12.330 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/a/BGpxfsUWlq9FB3FBiN9V07UE= 2023-10-12T00:44:12.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/a/BGpxfsUWlq9FB3FBiN9V07UE=', 'Content-Length': '0'} 2023-10-12T00:44:12.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f225058a708d440c-006527415c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.334 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:44:12.334 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.335 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.335 DEBUG:boto:path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.336 DEBUG:boto:auth_path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.336 DEBUG:boto:Method: GET 2023-10-12T00:44:12.336 DEBUG:boto:Path: /wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.337 DEBUG:boto:Data: 2023-10-12T00:44:12.337 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.338 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.339 DEBUG:boto:Params: {} 2023-10-12T00:44:12.339 DEBUG:boto:Token: None 2023-10-12T00:44:12.340 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/?versions 2023-10-12T00:44:12.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo= 2023-10-12T00:44:12.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo=', 'Content-Length': '0'} 2023-10-12T00:44:12.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6b0833f2231f5ca-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.344 DEBUG:boto:b'wafmoa-53999falseanulltrue2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.344 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:44:12.345 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:44:12.345 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:44:12.346 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:44:12.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:44:12.347 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.347 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.347 DEBUG:boto:path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.348 DEBUG:boto:auth_path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.348 DEBUG:boto:Method: GET 2023-10-12T00:44:12.349 DEBUG:boto:Path: /wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.349 DEBUG:boto:Data: 2023-10-12T00:44:12.349 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.350 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.351 DEBUG:boto:Params: {} 2023-10-12T00:44:12.351 DEBUG:boto:Token: None 2023-10-12T00:44:12.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/?versions 2023-10-12T00:44:12.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo= 2023-10-12T00:44:12.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo=', 'Content-Length': '0'} 2023-10-12T00:44:12.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002687ebb1b0e6a0a0-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.356 DEBUG:boto:b'wafmoa-53999falseanulltrue2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.357 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:44:12.357 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:44:12.357 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:44:12.358 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:44:12.358 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.359 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.359 DEBUG:boto:path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.359 DEBUG:boto:auth_path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.360 DEBUG:boto:Method: GET 2023-10-12T00:44:12.360 DEBUG:boto:Path: /wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.361 DEBUG:boto:Data: 2023-10-12T00:44:12.361 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.362 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.362 DEBUG:boto:Params: {} 2023-10-12T00:44:12.363 DEBUG:boto:Token: None 2023-10-12T00:44:12.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/?versions 2023-10-12T00:44:12.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo= 2023-10-12T00:44:12.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo=', 'Content-Length': '0'} 2023-10-12T00:44:12.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a321abe962b82ac-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.367 DEBUG:boto:b'wafmoa-53999falseanulltrue2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.368 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.368 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.369 DEBUG:boto:path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.369 DEBUG:boto:auth_path=/wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.369 DEBUG:boto:Method: GET 2023-10-12T00:44:12.370 DEBUG:boto:Path: /wafmoa-53/?versions&max-keys=999 2023-10-12T00:44:12.370 DEBUG:boto:Data: 2023-10-12T00:44:12.371 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.371 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.372 DEBUG:boto:Params: {} 2023-10-12T00:44:12.372 DEBUG:boto:Token: None 2023-10-12T00:44:12.373 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/?versions 2023-10-12T00:44:12.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo= 2023-10-12T00:44:12.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AcgyCSvBxWndT8MmIaKEz1ydWFo=', 'Content-Length': '0'} 2023-10-12T00:44:12.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e932e4bc73c65553-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.377 DEBUG:boto:b'wafmoa-53999falseanulltrue2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:12.378 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.378 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.379 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.379 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.380 DEBUG:boto:Method: GET 2023-10-12T00:44:12.380 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.380 DEBUG:boto:Data: 2023-10-12T00:44:12.381 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.382 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.382 DEBUG:boto:Params: {} 2023-10-12T00:44:12.383 DEBUG:boto:Token: None 2023-10-12T00:44:12.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto= 2023-10-12T00:44:12.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto=', 'Content-Length': '0'} 2023-10-12T00:44:12.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5caa930412c87c6-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.387 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.387 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.388 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.388 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.389 DEBUG:boto:Method: GET 2023-10-12T00:44:12.389 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.389 DEBUG:boto:Data: 2023-10-12T00:44:12.390 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.391 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.391 DEBUG:boto:Params: {} 2023-10-12T00:44:12.391 DEBUG:boto:Token: None 2023-10-12T00:44:12.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto= 2023-10-12T00:44:12.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto=', 'Content-Length': '0'} 2023-10-12T00:44:12.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080822ca6c27c6afc-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.438 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.439 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.439 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.442 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.442 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.443 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.443 DEBUG:boto:Data: 2023-10-12T00:44:12.444 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.445 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.445 DEBUG:boto:Params: {} 2023-10-12T00:44:12.446 DEBUG:boto:Token: None 2023-10-12T00:44:12.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWKXMRdz3yPQyFVgU3lneVgapcA= 2023-10-12T00:44:12.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWKXMRdz3yPQyFVgU3lneVgapcA=', 'Content-Length': '0'} 2023-10-12T00:44:12.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8ff7a94b5abc1d-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.450 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.450 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.451 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.451 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.451 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.452 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.452 DEBUG:boto:Data: 2023-10-12T00:44:12.453 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.454 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.454 DEBUG:boto:Params: {} 2023-10-12T00:44:12.454 DEBUG:boto:Token: None 2023-10-12T00:44:12.455 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWKXMRdz3yPQyFVgU3lneVgapcA= 2023-10-12T00:44:12.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWKXMRdz3yPQyFVgU3lneVgapcA=', 'Content-Length': '0'} 2023-10-12T00:44:12.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007919379ae8c02843-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:12.459 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.459 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.460 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.460 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.460 DEBUG:boto:Method: GET 2023-10-12T00:44:12.461 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.461 DEBUG:boto:Data: 2023-10-12T00:44:12.462 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.462 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.463 DEBUG:boto:Params: {} 2023-10-12T00:44:12.463 DEBUG:boto:Token: None 2023-10-12T00:44:12.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto= 2023-10-12T00:44:12.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto=', 'Content-Length': '0'} 2023-10-12T00:44:12.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212c587ac818e83b-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.468 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.468 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.468 DEBUG:boto:path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.469 DEBUG:boto:auth_path=/wafmoa-53/a?versionId=null 2023-10-12T00:44:12.469 DEBUG:boto:Method: GET 2023-10-12T00:44:12.470 DEBUG:boto:Path: /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.470 DEBUG:boto:Data: 2023-10-12T00:44:12.471 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.471 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.472 DEBUG:boto:Params: {} 2023-10-12T00:44:12.472 DEBUG:boto:Token: None 2023-10-12T00:44:12.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a?versionId=null 2023-10-12T00:44:12.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto= 2023-10-12T00:44:12.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCEUYLMm4xYLiqcGo+AtHxkisto=', 'Content-Length': '0'} 2023-10-12T00:44:12.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347b79805ade99c6-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.477 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.477 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.477 DEBUG:boto:path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.478 DEBUG:boto:auth_path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.478 DEBUG:boto:Method: GET 2023-10-12T00:44:12.479 DEBUG:boto:Path: /wafmoa-53/?prefix=a 2023-10-12T00:44:12.479 DEBUG:boto:Data: 2023-10-12T00:44:12.480 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.480 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.481 DEBUG:boto:Params: {} 2023-10-12T00:44:12.481 DEBUG:boto:Token: None 2023-10-12T00:44:12.482 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6dd263d86814f09-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.526 DEBUG:boto:b'wafmoa-53a1000falsea2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.527 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.528 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.528 DEBUG:boto:path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.528 DEBUG:boto:auth_path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.529 DEBUG:boto:Method: GET 2023-10-12T00:44:12.529 DEBUG:boto:Path: /wafmoa-53/?prefix=a 2023-10-12T00:44:12.529 DEBUG:boto:Data: 2023-10-12T00:44:12.530 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.531 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.531 DEBUG:boto:Params: {} 2023-10-12T00:44:12.532 DEBUG:boto:Token: None 2023-10-12T00:44:12.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92114f7514e35a9-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.536 DEBUG:boto:b'wafmoa-53a1000falsea2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.537 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.537 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.538 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.538 DEBUG:boto:Path: /wafmoa-53/a 2023-10-12T00:44:12.539 DEBUG:boto:Data: 2023-10-12T00:44:12.539 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.540 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.540 DEBUG:boto:Params: {} 2023-10-12T00:44:12.541 DEBUG:boto:Token: None 2023-10-12T00:44:12.541 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a 2023-10-12T00:44:12.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Uuml+pXM3LAn+mqYBiJloW6Ee4= 2023-10-12T00:44:12.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Uuml+pXM3LAn+mqYBiJloW6Ee4=', 'Content-Length': '0'} 2023-10-12T00:44:12.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630c13384f54e406-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.544 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.545 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.545 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.546 DEBUG:boto:Path: /wafmoa-53/a 2023-10-12T00:44:12.546 DEBUG:boto:Data: 2023-10-12T00:44:12.547 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.547 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.548 DEBUG:boto:Params: {} 2023-10-12T00:44:12.548 DEBUG:boto:Token: None 2023-10-12T00:44:12.549 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a 2023-10-12T00:44:12.549 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Uuml+pXM3LAn+mqYBiJloW6Ee4= 2023-10-12T00:44:12.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Uuml+pXM3LAn+mqYBiJloW6Ee4=', 'Content-Length': '0'} 2023-10-12T00:44:12.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f4ca4292443eee-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:12.552 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.553 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.553 DEBUG:boto:Method: GET 2023-10-12T00:44:12.554 DEBUG:boto:Path: /wafmoa-53/a 2023-10-12T00:44:12.554 DEBUG:boto:Data: 2023-10-12T00:44:12.555 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.556 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.556 DEBUG:boto:Params: {} 2023-10-12T00:44:12.556 DEBUG:boto:Token: None 2023-10-12T00:44:12.557 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a 2023-10-12T00:44:12.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zp2N8RX2gvHsnwwkuTqxaRsWEzs= 2023-10-12T00:44:12.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zp2N8RX2gvHsnwwkuTqxaRsWEzs=', 'Content-Length': '0'} 2023-10-12T00:44:12.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046505127ecc03c48-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.561 DEBUG:boto:path=/wafmoa-53/a 2023-10-12T00:44:12.561 DEBUG:boto:auth_path=/wafmoa-53/a 2023-10-12T00:44:12.562 DEBUG:boto:Method: GET 2023-10-12T00:44:12.562 DEBUG:boto:Path: /wafmoa-53/a 2023-10-12T00:44:12.563 DEBUG:boto:Data: 2023-10-12T00:44:12.563 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.564 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.564 DEBUG:boto:Params: {} 2023-10-12T00:44:12.565 DEBUG:boto:Token: None 2023-10-12T00:44:12.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/a 2023-10-12T00:44:12.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zp2N8RX2gvHsnwwkuTqxaRsWEzs= 2023-10-12T00:44:12.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zp2N8RX2gvHsnwwkuTqxaRsWEzs=', 'Content-Length': '0'} 2023-10-12T00:44:12.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140ca3ae4369f228-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.569 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.569 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.570 DEBUG:boto:path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.570 DEBUG:boto:auth_path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.570 DEBUG:boto:Method: GET 2023-10-12T00:44:12.571 DEBUG:boto:Path: /wafmoa-53/?prefix=a 2023-10-12T00:44:12.571 DEBUG:boto:Data: 2023-10-12T00:44:12.572 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.572 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.573 DEBUG:boto:Params: {} 2023-10-12T00:44:12.573 DEBUG:boto:Token: None 2023-10-12T00:44:12.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.574 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1678c2ea4e1125-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.577 DEBUG:boto:b'wafmoa-53a1000falsea2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.578 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.578 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.579 DEBUG:boto:path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.579 DEBUG:boto:auth_path=/wafmoa-53/?prefix=a 2023-10-12T00:44:12.580 DEBUG:boto:Method: GET 2023-10-12T00:44:12.580 DEBUG:boto:Path: /wafmoa-53/?prefix=a 2023-10-12T00:44:12.580 DEBUG:boto:Data: 2023-10-12T00:44:12.581 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.581 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.582 DEBUG:boto:Params: {} 2023-10-12T00:44:12.582 DEBUG:boto:Token: None 2023-10-12T00:44:12.583 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055eb9b1166762649-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.587 DEBUG:boto:b'wafmoa-53a1000falsea2023-10-12T00:43:46.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:12.588 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.588 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.589 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.589 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.589 DEBUG:boto:Method: GET 2023-10-12T00:44:12.590 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.590 DEBUG:boto:Data: 2023-10-12T00:44:12.591 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.591 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.592 DEBUG:boto:Params: {} 2023-10-12T00:44:12.592 DEBUG:boto:Token: None 2023-10-12T00:44:12.593 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc= 2023-10-12T00:44:12.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc=', 'Content-Length': '0'} 2023-10-12T00:44:12.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076404fdc9776acc6-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.596 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.597 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.597 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.598 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.598 DEBUG:boto:Method: GET 2023-10-12T00:44:12.599 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.599 DEBUG:boto:Data: 2023-10-12T00:44:12.600 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.600 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.601 DEBUG:boto:Params: {} 2023-10-12T00:44:12.601 DEBUG:boto:Token: None 2023-10-12T00:44:12.602 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.602 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc= 2023-10-12T00:44:12.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc=', 'Content-Length': '0'} 2023-10-12T00:44:12.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a2f587e2c1e902-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.646 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.647 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.647 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.648 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.648 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.648 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.649 DEBUG:boto:Data: 2023-10-12T00:44:12.649 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.650 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.650 DEBUG:boto:Params: {} 2023-10-12T00:44:12.651 DEBUG:boto:Token: None 2023-10-12T00:44:12.651 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E5O3kb2PLn6V57j5xkNZXNPDMEg= 2023-10-12T00:44:12.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E5O3kb2PLn6V57j5xkNZXNPDMEg=', 'Content-Length': '0'} 2023-10-12T00:44:12.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090827c748b69f6f-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.655 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.655 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.656 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.656 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.656 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.657 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.657 DEBUG:boto:Data: 2023-10-12T00:44:12.658 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.659 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.659 DEBUG:boto:Params: {} 2023-10-12T00:44:12.659 DEBUG:boto:Token: None 2023-10-12T00:44:12.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E5O3kb2PLn6V57j5xkNZXNPDMEg= 2023-10-12T00:44:12.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E5O3kb2PLn6V57j5xkNZXNPDMEg=', 'Content-Length': '0'} 2023-10-12T00:44:12.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ecdaad305ddaad3-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:12.664 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.664 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.664 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.665 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.665 DEBUG:boto:Method: GET 2023-10-12T00:44:12.666 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.666 DEBUG:boto:Data: 2023-10-12T00:44:12.666 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.667 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.668 DEBUG:boto:Params: {} 2023-10-12T00:44:12.668 DEBUG:boto:Token: None 2023-10-12T00:44:12.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc= 2023-10-12T00:44:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc=', 'Content-Length': '0'} 2023-10-12T00:44:12.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005935e7b05f287ec-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.672 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.672 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.673 DEBUG:boto:path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.673 DEBUG:boto:auth_path=/wafmoa-53/b?versionId=null 2023-10-12T00:44:12.674 DEBUG:boto:Method: GET 2023-10-12T00:44:12.674 DEBUG:boto:Path: /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.675 DEBUG:boto:Data: 2023-10-12T00:44:12.675 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.676 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.676 DEBUG:boto:Params: {} 2023-10-12T00:44:12.677 DEBUG:boto:Token: None 2023-10-12T00:44:12.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b?versionId=null 2023-10-12T00:44:12.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc= 2023-10-12T00:44:12.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2yRG4GNt8UIGQQqpuCPKb0FeYXc=', 'Content-Length': '0'} 2023-10-12T00:44:12.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa98a78c8de43b1-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.681 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.681 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.682 DEBUG:boto:path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.682 DEBUG:boto:auth_path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.683 DEBUG:boto:Method: GET 2023-10-12T00:44:12.683 DEBUG:boto:Path: /wafmoa-53/?prefix=b 2023-10-12T00:44:12.683 DEBUG:boto:Data: 2023-10-12T00:44:12.684 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.685 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.685 DEBUG:boto:Params: {} 2023-10-12T00:44:12.685 DEBUG:boto:Token: None 2023-10-12T00:44:12.686 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eb6e541d8316902-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.690 DEBUG:boto:b'wafmoa-53b1000falseb2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.691 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.691 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.692 DEBUG:boto:path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.692 DEBUG:boto:auth_path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.693 DEBUG:boto:Method: GET 2023-10-12T00:44:12.693 DEBUG:boto:Path: /wafmoa-53/?prefix=b 2023-10-12T00:44:12.694 DEBUG:boto:Data: 2023-10-12T00:44:12.694 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.695 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.695 DEBUG:boto:Params: {} 2023-10-12T00:44:12.696 DEBUG:boto:Token: None 2023-10-12T00:44:12.696 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf2c75ca43507e9-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.701 DEBUG:boto:b'wafmoa-53b1000falseb2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.701 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.702 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.702 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.703 DEBUG:boto:Path: /wafmoa-53/b 2023-10-12T00:44:12.703 DEBUG:boto:Data: 2023-10-12T00:44:12.703 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.704 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.705 DEBUG:boto:Params: {} 2023-10-12T00:44:12.705 DEBUG:boto:Token: None 2023-10-12T00:44:12.705 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b 2023-10-12T00:44:12.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZtPj9JjUFvJ652pRbeFnbakrs2U= 2023-10-12T00:44:12.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZtPj9JjUFvJ652pRbeFnbakrs2U=', 'Content-Length': '0'} 2023-10-12T00:44:12.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636a38631790a3d5-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.709 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.709 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.710 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.710 DEBUG:boto:Path: /wafmoa-53/b 2023-10-12T00:44:12.711 DEBUG:boto:Data: 2023-10-12T00:44:12.711 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.712 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.712 DEBUG:boto:Params: {} 2023-10-12T00:44:12.713 DEBUG:boto:Token: None 2023-10-12T00:44:12.713 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b 2023-10-12T00:44:12.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZtPj9JjUFvJ652pRbeFnbakrs2U= 2023-10-12T00:44:12.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZtPj9JjUFvJ652pRbeFnbakrs2U=', 'Content-Length': '0'} 2023-10-12T00:44:12.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000395be8a89ca079fc-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:12.717 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.718 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.718 DEBUG:boto:Method: GET 2023-10-12T00:44:12.719 DEBUG:boto:Path: /wafmoa-53/b 2023-10-12T00:44:12.719 DEBUG:boto:Data: 2023-10-12T00:44:12.720 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.720 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.721 DEBUG:boto:Params: {} 2023-10-12T00:44:12.721 DEBUG:boto:Token: None 2023-10-12T00:44:12.722 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b 2023-10-12T00:44:12.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCKCHQCOFbdE/6P9zQT0yDynQFU= 2023-10-12T00:44:12.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCKCHQCOFbdE/6P9zQT0yDynQFU=', 'Content-Length': '0'} 2023-10-12T00:44:12.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfbdc2758772e74-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.726 DEBUG:boto:path=/wafmoa-53/b 2023-10-12T00:44:12.726 DEBUG:boto:auth_path=/wafmoa-53/b 2023-10-12T00:44:12.727 DEBUG:boto:Method: GET 2023-10-12T00:44:12.727 DEBUG:boto:Path: /wafmoa-53/b 2023-10-12T00:44:12.728 DEBUG:boto:Data: 2023-10-12T00:44:12.728 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.729 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.729 DEBUG:boto:Params: {} 2023-10-12T00:44:12.730 DEBUG:boto:Token: None 2023-10-12T00:44:12.730 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/b 2023-10-12T00:44:12.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yCKCHQCOFbdE/6P9zQT0yDynQFU= 2023-10-12T00:44:12.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yCKCHQCOFbdE/6P9zQT0yDynQFU=', 'Content-Length': '0'} 2023-10-12T00:44:12.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4ac461c41bad85-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.774 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.775 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.775 DEBUG:boto:path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.776 DEBUG:boto:auth_path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.776 DEBUG:boto:Method: GET 2023-10-12T00:44:12.777 DEBUG:boto:Path: /wafmoa-53/?prefix=b 2023-10-12T00:44:12.777 DEBUG:boto:Data: 2023-10-12T00:44:12.778 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.778 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.779 DEBUG:boto:Params: {} 2023-10-12T00:44:12.779 DEBUG:boto:Token: None 2023-10-12T00:44:12.780 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d59771d43d1ed6-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.784 DEBUG:boto:b'wafmoa-53b1000falseb2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.785 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.786 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.786 DEBUG:boto:path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.786 DEBUG:boto:auth_path=/wafmoa-53/?prefix=b 2023-10-12T00:44:12.787 DEBUG:boto:Method: GET 2023-10-12T00:44:12.787 DEBUG:boto:Path: /wafmoa-53/?prefix=b 2023-10-12T00:44:12.788 DEBUG:boto:Data: 2023-10-12T00:44:12.788 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.789 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.789 DEBUG:boto:Params: {} 2023-10-12T00:44:12.790 DEBUG:boto:Token: None 2023-10-12T00:44:12.791 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985642182cc40c62-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.795 DEBUG:boto:b'wafmoa-53b1000falseb2023-10-12T00:43:46.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:12.796 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.796 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.797 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.797 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.798 DEBUG:boto:Method: GET 2023-10-12T00:44:12.798 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.799 DEBUG:boto:Data: 2023-10-12T00:44:12.799 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.800 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.800 DEBUG:boto:Params: {} 2023-10-12T00:44:12.801 DEBUG:boto:Token: None 2023-10-12T00:44:12.801 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.802 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs= 2023-10-12T00:44:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs=', 'Content-Length': '0'} 2023-10-12T00:44:12.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7eb8997f189ad5-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.805 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.806 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.806 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.807 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.807 DEBUG:boto:Method: GET 2023-10-12T00:44:12.808 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.808 DEBUG:boto:Data: 2023-10-12T00:44:12.809 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.809 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.810 DEBUG:boto:Params: {} 2023-10-12T00:44:12.810 DEBUG:boto:Token: None 2023-10-12T00:44:12.811 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs= 2023-10-12T00:44:12.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs=', 'Content-Length': '0'} 2023-10-12T00:44:12.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae860a0309e6d25-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.815 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.815 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.816 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.816 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.816 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.817 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.817 DEBUG:boto:Data: 2023-10-12T00:44:12.818 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.818 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.819 DEBUG:boto:Params: {} 2023-10-12T00:44:12.819 DEBUG:boto:Token: None 2023-10-12T00:44:12.820 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.820 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c683g78TElLjOkOXl2+Ckespo0I= 2023-10-12T00:44:12.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c683g78TElLjOkOXl2+Ckespo0I=', 'Content-Length': '0'} 2023-10-12T00:44:12.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac30a7c8ac0212e-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.823 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.824 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.824 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.825 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.825 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.826 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.826 DEBUG:boto:Data: 2023-10-12T00:44:12.826 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.827 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.828 DEBUG:boto:Params: {} 2023-10-12T00:44:12.828 DEBUG:boto:Token: None 2023-10-12T00:44:12.829 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c683g78TElLjOkOXl2+Ckespo0I= 2023-10-12T00:44:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c683g78TElLjOkOXl2+Ckespo0I=', 'Content-Length': '0'} 2023-10-12T00:44:12.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f49ef3020c5b9bc-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:12.833 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.833 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.833 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.834 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.834 DEBUG:boto:Method: GET 2023-10-12T00:44:12.835 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.835 DEBUG:boto:Data: 2023-10-12T00:44:12.836 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.837 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.837 DEBUG:boto:Params: {} 2023-10-12T00:44:12.838 DEBUG:boto:Token: None 2023-10-12T00:44:12.838 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs= 2023-10-12T00:44:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs=', 'Content-Length': '0'} 2023-10-12T00:44:12.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c81426d425d562b-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.882 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.883 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.883 DEBUG:boto:path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.884 DEBUG:boto:auth_path=/wafmoa-53/c?versionId=null 2023-10-12T00:44:12.884 DEBUG:boto:Method: GET 2023-10-12T00:44:12.885 DEBUG:boto:Path: /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.885 DEBUG:boto:Data: 2023-10-12T00:44:12.886 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.887 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.887 DEBUG:boto:Params: {} 2023-10-12T00:44:12.888 DEBUG:boto:Token: None 2023-10-12T00:44:12.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c?versionId=null 2023-10-12T00:44:12.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs= 2023-10-12T00:44:12.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WGUfs/gsC4zuVF6srldQKNf+jOs=', 'Content-Length': '0'} 2023-10-12T00:44:12.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d47862b707e621-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.892 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.892 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.892 DEBUG:boto:path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.893 DEBUG:boto:auth_path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.893 DEBUG:boto:Method: GET 2023-10-12T00:44:12.894 DEBUG:boto:Path: /wafmoa-53/?prefix=c 2023-10-12T00:44:12.894 DEBUG:boto:Data: 2023-10-12T00:44:12.895 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.895 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.896 DEBUG:boto:Params: {} 2023-10-12T00:44:12.896 DEBUG:boto:Token: None 2023-10-12T00:44:12.897 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3fbea5e2fc34319-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.901 DEBUG:boto:b'wafmoa-53c1000falsec2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.902 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.902 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.903 DEBUG:boto:path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.903 DEBUG:boto:auth_path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.904 DEBUG:boto:Method: GET 2023-10-12T00:44:12.904 DEBUG:boto:Path: /wafmoa-53/?prefix=c 2023-10-12T00:44:12.904 DEBUG:boto:Data: 2023-10-12T00:44:12.905 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.906 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.906 DEBUG:boto:Params: {} 2023-10-12T00:44:12.906 DEBUG:boto:Token: None 2023-10-12T00:44:12.907 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698210048dcca344-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.911 DEBUG:boto:b'wafmoa-53c1000falsec2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.911 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.912 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.912 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.912 DEBUG:boto:Path: /wafmoa-53/c 2023-10-12T00:44:12.913 DEBUG:boto:Data: 2023-10-12T00:44:12.913 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.914 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.914 DEBUG:boto:Params: {} 2023-10-12T00:44:12.915 DEBUG:boto:Token: None 2023-10-12T00:44:12.915 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c 2023-10-12T00:44:12.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/AeKU2qKxd5NkVMEkviOXbjcxMk= 2023-10-12T00:44:12.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/AeKU2qKxd5NkVMEkviOXbjcxMk=', 'Content-Length': '0'} 2023-10-12T00:44:12.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc792f0dfe1f9927-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.919 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.919 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.920 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.920 DEBUG:boto:Path: /wafmoa-53/c 2023-10-12T00:44:12.920 DEBUG:boto:Data: 2023-10-12T00:44:12.921 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.922 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.922 DEBUG:boto:Params: {} 2023-10-12T00:44:12.923 DEBUG:boto:Token: None 2023-10-12T00:44:12.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c 2023-10-12T00:44:12.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/AeKU2qKxd5NkVMEkviOXbjcxMk= 2023-10-12T00:44:12.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/AeKU2qKxd5NkVMEkviOXbjcxMk=', 'Content-Length': '0'} 2023-10-12T00:44:12.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f03c78a864e278-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:12.927 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.927 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.928 DEBUG:boto:Method: GET 2023-10-12T00:44:12.928 DEBUG:boto:Path: /wafmoa-53/c 2023-10-12T00:44:12.928 DEBUG:boto:Data: 2023-10-12T00:44:12.929 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.929 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.930 DEBUG:boto:Params: {} 2023-10-12T00:44:12.930 DEBUG:boto:Token: None 2023-10-12T00:44:12.931 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c 2023-10-12T00:44:12.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sjid2B9JoZViKDJlqY7MnEgLyNY= 2023-10-12T00:44:12.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sjid2B9JoZViKDJlqY7MnEgLyNY=', 'Content-Length': '0'} 2023-10-12T00:44:12.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f1350d4de25c62-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.934 DEBUG:boto:path=/wafmoa-53/c 2023-10-12T00:44:12.935 DEBUG:boto:auth_path=/wafmoa-53/c 2023-10-12T00:44:12.935 DEBUG:boto:Method: GET 2023-10-12T00:44:12.935 DEBUG:boto:Path: /wafmoa-53/c 2023-10-12T00:44:12.936 DEBUG:boto:Data: 2023-10-12T00:44:12.936 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.937 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.938 DEBUG:boto:Params: {} 2023-10-12T00:44:12.938 DEBUG:boto:Token: None 2023-10-12T00:44:12.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/c 2023-10-12T00:44:12.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sjid2B9JoZViKDJlqY7MnEgLyNY= 2023-10-12T00:44:12.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sjid2B9JoZViKDJlqY7MnEgLyNY=', 'Content-Length': '0'} 2023-10-12T00:44:12.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ee1cd9c598f0ab-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.942 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.942 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.943 DEBUG:boto:path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.943 DEBUG:boto:auth_path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.944 DEBUG:boto:Method: GET 2023-10-12T00:44:12.944 DEBUG:boto:Path: /wafmoa-53/?prefix=c 2023-10-12T00:44:12.945 DEBUG:boto:Data: 2023-10-12T00:44:12.945 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.946 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.946 DEBUG:boto:Params: {} 2023-10-12T00:44:12.947 DEBUG:boto:Token: None 2023-10-12T00:44:12.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b0de187cc81a0c-006527415c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.951 DEBUG:boto:b'wafmoa-53c1000falsec2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.952 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:12.952 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:12.952 DEBUG:boto:path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.953 DEBUG:boto:auth_path=/wafmoa-53/?prefix=c 2023-10-12T00:44:12.953 DEBUG:boto:Method: GET 2023-10-12T00:44:12.954 DEBUG:boto:Path: /wafmoa-53/?prefix=c 2023-10-12T00:44:12.954 DEBUG:boto:Data: 2023-10-12T00:44:12.954 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.955 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.956 DEBUG:boto:Params: {} 2023-10-12T00:44:12.956 DEBUG:boto:Token: None 2023-10-12T00:44:12.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/ 2023-10-12T00:44:12.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI= 2023-10-12T00:44:12.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wcc9futW3LIZWbinNadcJb5r7EI=', 'Content-Length': '0'} 2023-10-12T00:44:12.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07f9d17d24a79c3-006527415c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.961 DEBUG:boto:b'wafmoa-53c1000falsec2023-10-12T00:43:46.854Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:12.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:12.962 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:12.962 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:12.963 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.963 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.963 DEBUG:boto:Method: GET 2023-10-12T00:44:12.964 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.964 DEBUG:boto:Data: 2023-10-12T00:44:12.965 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.965 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.966 DEBUG:boto:Params: {} 2023-10-12T00:44:12.966 DEBUG:boto:Token: None 2023-10-12T00:44:12.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e/ivpORBgQ9nW9Dnt0tFcRTWkRk= 2023-10-12T00:44:12.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e/ivpORBgQ9nW9Dnt0tFcRTWkRk=', 'Content-Length': '0'} 2023-10-12T00:44:12.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006975908da4f5e5d-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.971 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:12.971 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:12.972 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.972 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.972 DEBUG:boto:Method: GET 2023-10-12T00:44:12.973 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.973 DEBUG:boto:Data: 2023-10-12T00:44:12.974 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.975 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.975 DEBUG:boto:Params: {} 2023-10-12T00:44:12.976 DEBUG:boto:Token: None 2023-10-12T00:44:12.976 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e/ivpORBgQ9nW9Dnt0tFcRTWkRk= 2023-10-12T00:44:12.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e/ivpORBgQ9nW9Dnt0tFcRTWkRk=', 'Content-Length': '0'} 2023-10-12T00:44:12.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a43dead2a9124e01-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.980 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:12.980 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:12.981 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.981 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.982 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.982 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.982 DEBUG:boto:Data: 2023-10-12T00:44:12.983 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:12.984 DEBUG:boto:Port: 8001 2023-10-12T00:44:12.984 DEBUG:boto:Params: {} 2023-10-12T00:44:12.984 DEBUG:boto:Token: None 2023-10-12T00:44:12.985 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EkUgP/NTWBK4DS2m/DPeaPRoXCE= 2023-10-12T00:44:12.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EkUgP/NTWBK4DS2m/DPeaPRoXCE=', 'Content-Length': '0'} 2023-10-12T00:44:12.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916a2c8e5d802d64-006527415c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.988 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:12.989 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:12.989 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.989 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.990 DEBUG:boto:Method: HEAD 2023-10-12T00:44:12.990 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.991 DEBUG:boto:Data: 2023-10-12T00:44:12.991 DEBUG:boto:Headers: {} 2023-10-12T00:44:12.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:12.992 DEBUG:boto:Port: 8000 2023-10-12T00:44:12.992 DEBUG:boto:Params: {} 2023-10-12T00:44:12.993 DEBUG:boto:Token: None 2023-10-12T00:44:12.993 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:12 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EkUgP/NTWBK4DS2m/DPeaPRoXCE= 2023-10-12T00:44:12.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EkUgP/NTWBK4DS2m/DPeaPRoXCE=', 'Content-Length': '0'} 2023-10-12T00:44:12.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000570de7494760d7a3-006527415c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:12.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:12.997 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:12.997 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:12.998 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.998 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:12.999 DEBUG:boto:Method: GET 2023-10-12T00:44:12.999 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:12.999 DEBUG:boto:Data: 2023-10-12T00:44:13.000 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:13.000 DEBUG:boto:Port: 8001 2023-10-12T00:44:13.001 DEBUG:boto:Params: {} 2023-10-12T00:44:13.001 DEBUG:boto:Token: None 2023-10-12T00:44:13.002 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:13.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:88H9h1hp4wVDjYLdYE22kX4mals= 2023-10-12T00:44:13.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:88H9h1hp4wVDjYLdYE22kX4mals=', 'Content-Length': '0'} 2023-10-12T00:44:13.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d737d217c6ba639-006527415d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.046 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:13.047 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:13.047 DEBUG:boto:path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:13.048 DEBUG:boto:auth_path=/wafmoa-53/d?versionId=null 2023-10-12T00:44:13.048 DEBUG:boto:Method: GET 2023-10-12T00:44:13.049 DEBUG:boto:Path: /wafmoa-53/d?versionId=null 2023-10-12T00:44:13.049 DEBUG:boto:Data: 2023-10-12T00:44:13.049 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.050 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.051 DEBUG:boto:Params: {} 2023-10-12T00:44:13.051 DEBUG:boto:Token: None 2023-10-12T00:44:13.053 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d?versionId=null 2023-10-12T00:44:13.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:88H9h1hp4wVDjYLdYE22kX4mals= 2023-10-12T00:44:13.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:88H9h1hp4wVDjYLdYE22kX4mals=', 'Content-Length': '0'} 2023-10-12T00:44:13.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac88cfbbe1dbfe8-006527415d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.057 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:13.058 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:13.058 DEBUG:boto:path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.059 DEBUG:boto:auth_path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.059 DEBUG:boto:Method: GET 2023-10-12T00:44:13.060 DEBUG:boto:Path: /wafmoa-53/?prefix=d 2023-10-12T00:44:13.060 DEBUG:boto:Data: 2023-10-12T00:44:13.061 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:13.061 DEBUG:boto:Port: 8001 2023-10-12T00:44:13.062 DEBUG:boto:Params: {} 2023-10-12T00:44:13.062 DEBUG:boto:Token: None 2023-10-12T00:44:13.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/ 2023-10-12T00:44:13.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04= 2023-10-12T00:44:13.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04=', 'Content-Length': '0'} 2023-10-12T00:44:13.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601ca76171f0afdd-006527415d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.067 DEBUG:boto:b'wafmoa-53d1000falsed2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:13.068 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:13.068 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:13.069 DEBUG:boto:path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.069 DEBUG:boto:auth_path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.070 DEBUG:boto:Method: GET 2023-10-12T00:44:13.070 DEBUG:boto:Path: /wafmoa-53/?prefix=d 2023-10-12T00:44:13.071 DEBUG:boto:Data: 2023-10-12T00:44:13.071 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.072 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.072 DEBUG:boto:Params: {} 2023-10-12T00:44:13.073 DEBUG:boto:Token: None 2023-10-12T00:44:13.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/ 2023-10-12T00:44:13.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04= 2023-10-12T00:44:13.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04=', 'Content-Length': '0'} 2023-10-12T00:44:13.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6216a143ef4f607-006527415d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.077 DEBUG:boto:b'wafmoa-53d1000falsed2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:13.078 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:13.078 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:13.079 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.079 DEBUG:boto:Path: /wafmoa-53/d 2023-10-12T00:44:13.080 DEBUG:boto:Data: 2023-10-12T00:44:13.080 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:13.081 DEBUG:boto:Port: 8001 2023-10-12T00:44:13.081 DEBUG:boto:Params: {} 2023-10-12T00:44:13.082 DEBUG:boto:Token: None 2023-10-12T00:44:13.082 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d 2023-10-12T00:44:13.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:66jDD0Vz72KRilgEMK9xMmsIVbI= 2023-10-12T00:44:13.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:66jDD0Vz72KRilgEMK9xMmsIVbI=', 'Content-Length': '0'} 2023-10-12T00:44:13.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002732a4d53efd5012-006527415d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.086 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:13.086 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:13.087 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.087 DEBUG:boto:Path: /wafmoa-53/d 2023-10-12T00:44:13.087 DEBUG:boto:Data: 2023-10-12T00:44:13.088 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.089 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.089 DEBUG:boto:Params: {} 2023-10-12T00:44:13.090 DEBUG:boto:Token: None 2023-10-12T00:44:13.090 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d 2023-10-12T00:44:13.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:66jDD0Vz72KRilgEMK9xMmsIVbI= 2023-10-12T00:44:13.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:66jDD0Vz72KRilgEMK9xMmsIVbI=', 'Content-Length': '0'} 2023-10-12T00:44:13.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849817922808d9c0-006527415d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:13.094 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:13.094 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:13.095 DEBUG:boto:Method: GET 2023-10-12T00:44:13.095 DEBUG:boto:Path: /wafmoa-53/d 2023-10-12T00:44:13.096 DEBUG:boto:Data: 2023-10-12T00:44:13.096 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:13.097 DEBUG:boto:Port: 8001 2023-10-12T00:44:13.097 DEBUG:boto:Params: {} 2023-10-12T00:44:13.098 DEBUG:boto:Token: None 2023-10-12T00:44:13.098 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d 2023-10-12T00:44:13.099 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RDMar2SQrcsE7l1/VUXa2VBcv98= 2023-10-12T00:44:13.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RDMar2SQrcsE7l1/VUXa2VBcv98=', 'Content-Length': '0'} 2023-10-12T00:44:13.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003421cd1addbad393-006527415d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.102 DEBUG:boto:path=/wafmoa-53/d 2023-10-12T00:44:13.102 DEBUG:boto:auth_path=/wafmoa-53/d 2023-10-12T00:44:13.103 DEBUG:boto:Method: GET 2023-10-12T00:44:13.103 DEBUG:boto:Path: /wafmoa-53/d 2023-10-12T00:44:13.104 DEBUG:boto:Data: 2023-10-12T00:44:13.104 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.105 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.105 DEBUG:boto:Params: {} 2023-10-12T00:44:13.106 DEBUG:boto:Token: None 2023-10-12T00:44:13.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/d 2023-10-12T00:44:13.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RDMar2SQrcsE7l1/VUXa2VBcv98= 2023-10-12T00:44:13.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RDMar2SQrcsE7l1/VUXa2VBcv98=', 'Content-Length': '0'} 2023-10-12T00:44:13.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:43:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567fe2b571fbc849-006527415d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.110 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:13.111 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:13.111 DEBUG:boto:path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.111 DEBUG:boto:auth_path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.112 DEBUG:boto:Method: GET 2023-10-12T00:44:13.112 DEBUG:boto:Path: /wafmoa-53/?prefix=d 2023-10-12T00:44:13.113 DEBUG:boto:Data: 2023-10-12T00:44:13.113 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:13.114 DEBUG:boto:Port: 8001 2023-10-12T00:44:13.114 DEBUG:boto:Params: {} 2023-10-12T00:44:13.115 DEBUG:boto:Token: None 2023-10-12T00:44:13.115 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/ 2023-10-12T00:44:13.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04= 2023-10-12T00:44:13.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04=', 'Content-Length': '0'} 2023-10-12T00:44:13.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036d0e57cbdd208a8-006527415d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.119 DEBUG:boto:b'wafmoa-53d1000falsed2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:13.120 DEBUG:boto:path=/wafmoa-53/ 2023-10-12T00:44:13.121 DEBUG:boto:auth_path=/wafmoa-53/ 2023-10-12T00:44:13.121 DEBUG:boto:path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.121 DEBUG:boto:auth_path=/wafmoa-53/?prefix=d 2023-10-12T00:44:13.122 DEBUG:boto:Method: GET 2023-10-12T00:44:13.122 DEBUG:boto:Path: /wafmoa-53/?prefix=d 2023-10-12T00:44:13.123 DEBUG:boto:Data: 2023-10-12T00:44:13.123 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.124 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.124 DEBUG:boto:Params: {} 2023-10-12T00:44:13.125 DEBUG:boto:Token: None 2023-10-12T00:44:13.125 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-53/ 2023-10-12T00:44:13.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04= 2023-10-12T00:44:13.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bszVa0iKnX10flGpGH9dNHCUz04=', 'Content-Length': '0'} 2023-10-12T00:44:13.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c797f9fc76ed7a4-006527415d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.130 DEBUG:boto:b'wafmoa-53d1000falsed2023-10-12T00:43:46.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:13.130 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-53 zones={a2, a1} 2023-10-12T00:44:13.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:13.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:13.132 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:44:13.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:13.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:13.232 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-54 2023-10-12T00:44:13.233 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:13.233 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:13.234 DEBUG:boto:Method: PUT 2023-10-12T00:44:13.234 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:13.234 DEBUG:boto:Data: 2023-10-12T00:44:13.235 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.236 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.236 DEBUG:boto:Params: {} 2023-10-12T00:44:13.236 DEBUG:boto:Token: None 2023-10-12T00:44:13.237 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/ 2023-10-12T00:44:13.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycKsSdQmPr2wtd71+4FBMJqO5Wk= 2023-10-12T00:44:13.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycKsSdQmPr2wtd71+4FBMJqO5Wk=', 'Content-Length': '0'} 2023-10-12T00:44:13.252 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a06b6fd5595aea8-006527415d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.253 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:13.253 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:13.254 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.254 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:13.255 DEBUG:boto:Data: 2023-10-12T00:44:13.255 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.256 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.256 DEBUG:boto:Params: {} 2023-10-12T00:44:13.257 DEBUG:boto:Token: None 2023-10-12T00:44:13.257 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/ 2023-10-12T00:44:13.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM= 2023-10-12T00:44:13.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM=', 'Content-Length': '0'} 2023-10-12T00:44:13.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003173734e5f613636-006527415d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.262 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:13.262 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:13.263 DEBUG:boto:Method: PUT 2023-10-12T00:44:13.263 DEBUG:boto:Path: /wafmoa-54/a 2023-10-12T00:44:13.264 DEBUG:boto:Data: 2023-10-12T00:44:13.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:13.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.265 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.265 DEBUG:boto:Params: {} 2023-10-12T00:44:13.266 DEBUG:boto:Token: None 2023-10-12T00:44:13.266 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/a 2023-10-12T00:44:13.267 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:10d4y3sTWNJwO0/o2swYHJod3bM= 2023-10-12T00:44:13.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:10d4y3sTWNJwO0/o2swYHJod3bM='} 2023-10-12T00:44:13.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021e7b0af5c6345a8-006527415d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.273 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:13.274 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:13.274 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.275 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:13.275 DEBUG:boto:Data: 2023-10-12T00:44:13.275 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.276 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.277 DEBUG:boto:Params: {} 2023-10-12T00:44:13.277 DEBUG:boto:Token: None 2023-10-12T00:44:13.278 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/ 2023-10-12T00:44:13.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM= 2023-10-12T00:44:13.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM=', 'Content-Length': '0'} 2023-10-12T00:44:13.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff1ca33665cfe911-006527415d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.281 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:13.281 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:13.282 DEBUG:boto:Method: PUT 2023-10-12T00:44:13.282 DEBUG:boto:Path: /wafmoa-54/b 2023-10-12T00:44:13.283 DEBUG:boto:Data: 2023-10-12T00:44:13.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:13.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.284 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.284 DEBUG:boto:Params: {} 2023-10-12T00:44:13.285 DEBUG:boto:Token: None 2023-10-12T00:44:13.285 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/b 2023-10-12T00:44:13.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hz/MbIMWPPrvMUS1vqQs3yOoiCM= 2023-10-12T00:44:13.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hz/MbIMWPPrvMUS1vqQs3yOoiCM='} 2023-10-12T00:44:13.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a05919f0f20a42ae-006527415d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.292 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:13.292 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:13.293 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.293 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:13.293 DEBUG:boto:Data: 2023-10-12T00:44:13.294 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.295 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.295 DEBUG:boto:Params: {} 2023-10-12T00:44:13.296 DEBUG:boto:Token: None 2023-10-12T00:44:13.296 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/ 2023-10-12T00:44:13.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM= 2023-10-12T00:44:13.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM=', 'Content-Length': '0'} 2023-10-12T00:44:13.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079e3bd38c5aaa2bf-006527415d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.300 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:13.300 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:13.301 DEBUG:boto:Method: PUT 2023-10-12T00:44:13.301 DEBUG:boto:Path: /wafmoa-54/c 2023-10-12T00:44:13.301 DEBUG:boto:Data: 2023-10-12T00:44:13.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:13.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.303 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.303 DEBUG:boto:Params: {} 2023-10-12T00:44:13.304 DEBUG:boto:Token: None 2023-10-12T00:44:13.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/c 2023-10-12T00:44:13.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:COaNdj4lnQIBM7/JCHF0oMbasdE= 2023-10-12T00:44:13.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:COaNdj4lnQIBM7/JCHF0oMbasdE='} 2023-10-12T00:44:13.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f10106f5344dfc-006527415d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.351 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:13.351 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:13.352 DEBUG:boto:Method: HEAD 2023-10-12T00:44:13.352 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:13.353 DEBUG:boto:Data: 2023-10-12T00:44:13.353 DEBUG:boto:Headers: {} 2023-10-12T00:44:13.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.354 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.355 DEBUG:boto:Params: {} 2023-10-12T00:44:13.355 DEBUG:boto:Token: None 2023-10-12T00:44:13.356 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/ 2023-10-12T00:44:13.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM= 2023-10-12T00:44:13.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W6xR2srfG6H5Wc1uskRd2c5JHyM=', 'Content-Length': '0'} 2023-10-12T00:44:13.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b332e2295673617f-006527415d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.359 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:13.360 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:13.360 DEBUG:boto:Method: PUT 2023-10-12T00:44:13.361 DEBUG:boto:Path: /wafmoa-54/d 2023-10-12T00:44:13.361 DEBUG:boto:Data: 2023-10-12T00:44:13.362 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:13.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:13.363 DEBUG:boto:Port: 8000 2023-10-12T00:44:13.363 DEBUG:boto:Params: {} 2023-10-12T00:44:13.363 DEBUG:boto:Token: None 2023-10-12T00:44:13.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:13 GMT /wafmoa-54/d 2023-10-12T00:44:13.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eGyEjW2YVH863hQX6mbif1yDtzo= 2023-10-12T00:44:13.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eGyEjW2YVH863hQX6mbif1yDtzo='} 2023-10-12T00:44:13.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005683b703e8bb527e-006527415d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:13.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:13.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:13.372 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:13.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:13.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:13.450 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:44:13.462 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:44:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:44:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:44:13.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:44:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:44:13.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071453.252001_525.1", 2023-10-12T00:44:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:44:13.252001Z" 2023-10-12T00:44:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:44:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:44:13.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.466 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:44:13.467 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071453.252001_525.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:13.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:44:13.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:13.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:13.469 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:13.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:13.520 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:13.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:13.547+0000 7f5ebade4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:13.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:13.549+0000 7f5ebade4780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:13.561 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:44:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:44:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:44:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:44:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:44:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:44:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:44:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:44:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:44:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:44:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:44:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:44:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:13.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:44:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:13.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:44:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:44:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071426.760611_520.1", 2023-10-12T00:44:13.570 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.760611Z", 2023-10-12T00:44:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:13.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:44:13.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:44:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:13.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:44:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:13.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:13.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.575 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:44:13.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:44:13.576 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:44:13.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:13.576 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:44:13Z" 2023-10-12T00:44:13.576 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:44:13.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071426.760611_520.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:43:46.760611Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:44:13Z" } 2023-10-12T00:44:13.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071426.760611_520.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:43:46.760611Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:44:13.578 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071453.252001_525.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:13.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071426.760611_520.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:13.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071453.252001_525.1 target=1_1697071426.760611_520.1 2023-10-12T00:44:18.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:18.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:18.581 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:18.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:18.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:18.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:18.666+0000 7fbd89fee780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:18.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:18.667+0000 7fbd89fee780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:18.680 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:44:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:44:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:44:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:44:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:44:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:44:18.682 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:18.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.682 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:44:18.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:18.683 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:44:18.683 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:18.683 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:18.684 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:44:18.684 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:18.684 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:44:18.685 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:18.685 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:44:18.685 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:18.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:18.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:18.687 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:44:18.688 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:18.688 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:18.688 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:44:18.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:18.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:18.689 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:18.690 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:44:18.690 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:18.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:18.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:18.692 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:44:18.692 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071453.252001_525.1", 2023-10-12T00:44:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:18.693 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:13.252001Z", 2023-10-12T00:44:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:18.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:44:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:18.695 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:44:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:18.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:44:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:18.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:44:18.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:44:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:44:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:18.699 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:44:18Z" 2023-10-12T00:44:18.699 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:44:18.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071453.252001_525.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:44:13.252001Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:44:18Z" } 2023-10-12T00:44:18.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071453.252001_525.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:44:13.252001Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:44:18.701 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071453.252001_525.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:18.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071453.252001_525.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:18.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:44:18.703 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-54 2023-10-12T00:44:18.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:18.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:18.704 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:44:18.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:18.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:18.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:18.795+0000 7f39b1e2e780 1 waiting to reach incremental sync.. 2023-10-12T00:44:23.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:23.796+0000 7f39b1e2e780 1 waiting to reach incremental sync.. 2023-10-12T00:44:28.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:28.798+0000 7f39b1e2e780 1 bucket sync caught up with source: 2023-10-12T00:44:28.798 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.34.6, , , , 00000000001.36.6, , , 00000000001.25.6, , 00000000001.24.6] 2023-10-12T00:44:28.798 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.34.6, , , , 00000000001.36.6, , , 00000000001.25.6, , 00000000001.24.6] 2023-10-12T00:44:28.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:28.798+0000 7f39b1e2e780 0 bucket checkpoint complete 2023-10-12T00:44:28.815 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-54 2023-10-12T00:44:28.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:28.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:28.816 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:28.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:28.913+0000 7f95e86ae780 1 waiting to reach incremental sync.. 2023-10-12T00:44:33.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:33.915+0000 7f95e86ae780 1 bucket sync caught up with source: 2023-10-12T00:44:33.915 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.158.6, , , , 00000000001.251.6, , , 00000000001.409.6, , 00000000001.82.6] 2023-10-12T00:44:33.915 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.158.6, , , , 00000000001.251.6, , , 00000000001.409.6, , 00000000001.82.6] 2023-10-12T00:44:33.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:33.915+0000 7f95e86ae780 0 bucket checkpoint complete 2023-10-12T00:44:33.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-54 zones={a2, a1} 2023-10-12T00:44:33.932 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.932 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.932 DEBUG:boto:Method: HEAD 2023-10-12T00:44:33.933 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:33.933 DEBUG:boto:Data: 2023-10-12T00:44:33.934 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:33.934 DEBUG:boto:Port: 8001 2023-10-12T00:44:33.935 DEBUG:boto:Params: {} 2023-10-12T00:44:33.935 DEBUG:boto:Token: None 2023-10-12T00:44:33.936 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/ 2023-10-12T00:44:33.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eORW2HDhEjcVtNIBZfzbGgavtq4= 2023-10-12T00:44:33.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eORW2HDhEjcVtNIBZfzbGgavtq4=', 'Content-Length': '0'} 2023-10-12T00:44:33.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000073402f6328d0625-0065274171-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.940 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.940 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.941 DEBUG:boto:Method: HEAD 2023-10-12T00:44:33.941 DEBUG:boto:Path: /wafmoa-54/ 2023-10-12T00:44:33.942 DEBUG:boto:Data: 2023-10-12T00:44:33.942 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:33.943 DEBUG:boto:Port: 8000 2023-10-12T00:44:33.943 DEBUG:boto:Params: {} 2023-10-12T00:44:33.944 DEBUG:boto:Token: None 2023-10-12T00:44:33.944 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/ 2023-10-12T00:44:33.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eORW2HDhEjcVtNIBZfzbGgavtq4= 2023-10-12T00:44:33.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eORW2HDhEjcVtNIBZfzbGgavtq4=', 'Content-Length': '0'} 2023-10-12T00:44:33.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000199c4fae63f5afdc-0065274171-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:44:33.948 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.949 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.949 DEBUG:boto:path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.950 DEBUG:boto:auth_path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.950 DEBUG:boto:Method: GET 2023-10-12T00:44:33.950 DEBUG:boto:Path: /wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.951 DEBUG:boto:Data: 2023-10-12T00:44:33.951 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:33.952 DEBUG:boto:Port: 8001 2023-10-12T00:44:33.953 DEBUG:boto:Params: {} 2023-10-12T00:44:33.953 DEBUG:boto:Token: None 2023-10-12T00:44:33.954 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/?versions 2023-10-12T00:44:33.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0= 2023-10-12T00:44:33.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0=', 'Content-Length': '0'} 2023-10-12T00:44:33.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076c6c5df8cb8532e-0065274171-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.958 DEBUG:boto:b'wafmoa-54999falseanulltrue2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:33.959 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:44:33.959 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:44:33.960 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:44:33.960 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:44:33.960 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:44:33.961 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.961 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.961 DEBUG:boto:path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.962 DEBUG:boto:auth_path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.962 DEBUG:boto:Method: GET 2023-10-12T00:44:33.963 DEBUG:boto:Path: /wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.963 DEBUG:boto:Data: 2023-10-12T00:44:33.964 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:33.964 DEBUG:boto:Port: 8000 2023-10-12T00:44:33.965 DEBUG:boto:Params: {} 2023-10-12T00:44:33.965 DEBUG:boto:Token: None 2023-10-12T00:44:33.966 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/?versions 2023-10-12T00:44:33.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0= 2023-10-12T00:44:33.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0=', 'Content-Length': '0'} 2023-10-12T00:44:33.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254c7a96ead75e34-0065274171-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.970 DEBUG:boto:b'wafmoa-54999falseanulltrue2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:33.971 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:44:33.971 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:44:33.971 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:44:33.972 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:44:33.972 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.973 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.973 DEBUG:boto:path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.973 DEBUG:boto:auth_path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.974 DEBUG:boto:Method: GET 2023-10-12T00:44:33.974 DEBUG:boto:Path: /wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.975 DEBUG:boto:Data: 2023-10-12T00:44:33.975 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:33.976 DEBUG:boto:Port: 8001 2023-10-12T00:44:33.976 DEBUG:boto:Params: {} 2023-10-12T00:44:33.977 DEBUG:boto:Token: None 2023-10-12T00:44:33.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/?versions 2023-10-12T00:44:33.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0= 2023-10-12T00:44:33.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0=', 'Content-Length': '0'} 2023-10-12T00:44:33.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eea0b49636e0bc43-0065274171-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.981 DEBUG:boto:b'wafmoa-54999falseanulltrue2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:33.982 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:33.983 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:33.983 DEBUG:boto:path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.984 DEBUG:boto:auth_path=/wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.984 DEBUG:boto:Method: GET 2023-10-12T00:44:33.984 DEBUG:boto:Path: /wafmoa-54/?versions&max-keys=999 2023-10-12T00:44:33.985 DEBUG:boto:Data: 2023-10-12T00:44:33.985 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:33.986 DEBUG:boto:Port: 8000 2023-10-12T00:44:33.986 DEBUG:boto:Params: {} 2023-10-12T00:44:33.987 DEBUG:boto:Token: None 2023-10-12T00:44:33.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/?versions 2023-10-12T00:44:33.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0= 2023-10-12T00:44:33.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6hlYxrdb0XTjgMfz5ONvohifJI0=', 'Content-Length': '0'} 2023-10-12T00:44:33.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7ae3c9d216e95d-0065274171-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:33.991 DEBUG:boto:b'wafmoa-54999falseanulltrue2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:33.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:33.992 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:33.993 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:33.993 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:33.994 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:33.994 DEBUG:boto:Method: GET 2023-10-12T00:44:33.995 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:33.995 DEBUG:boto:Data: 2023-10-12T00:44:33.995 DEBUG:boto:Headers: {} 2023-10-12T00:44:33.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:33.996 DEBUG:boto:Port: 8001 2023-10-12T00:44:33.997 DEBUG:boto:Params: {} 2023-10-12T00:44:33.997 DEBUG:boto:Token: None 2023-10-12T00:44:33.998 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:33 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:33.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cux1Bar+NM2dLrv9HjFKLXXw52A= 2023-10-12T00:44:33.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cux1Bar+NM2dLrv9HjFKLXXw52A=', 'Content-Length': '0'} 2023-10-12T00:44:34.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f7e8466186f713-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.042 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.043 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.043 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.044 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.044 DEBUG:boto:Method: GET 2023-10-12T00:44:34.045 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.045 DEBUG:boto:Data: 2023-10-12T00:44:34.046 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.047 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.047 DEBUG:boto:Params: {} 2023-10-12T00:44:34.048 DEBUG:boto:Token: None 2023-10-12T00:44:34.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4= 2023-10-12T00:44:34.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4=', 'Content-Length': '0'} 2023-10-12T00:44:34.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8cac2628c2da1b-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.052 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.053 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.053 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.053 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.054 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.054 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.055 DEBUG:boto:Data: 2023-10-12T00:44:34.055 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.056 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.056 DEBUG:boto:Params: {} 2023-10-12T00:44:34.057 DEBUG:boto:Token: None 2023-10-12T00:44:34.057 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKqZlU0jChRK6zgzCLkAIz2YOg4= 2023-10-12T00:44:34.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKqZlU0jChRK6zgzCLkAIz2YOg4=', 'Content-Length': '0'} 2023-10-12T00:44:34.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cc69557efde997-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.061 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.061 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.062 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.062 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.062 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.063 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.063 DEBUG:boto:Data: 2023-10-12T00:44:34.064 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.065 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.065 DEBUG:boto:Params: {} 2023-10-12T00:44:34.065 DEBUG:boto:Token: None 2023-10-12T00:44:34.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKqZlU0jChRK6zgzCLkAIz2YOg4= 2023-10-12T00:44:34.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKqZlU0jChRK6zgzCLkAIz2YOg4=', 'Content-Length': '0'} 2023-10-12T00:44:34.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de869bb334995121-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:34.070 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.070 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.071 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.071 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.072 DEBUG:boto:Method: GET 2023-10-12T00:44:34.072 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.072 DEBUG:boto:Data: 2023-10-12T00:44:34.073 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.074 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.074 DEBUG:boto:Params: {} 2023-10-12T00:44:34.075 DEBUG:boto:Token: None 2023-10-12T00:44:34.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4= 2023-10-12T00:44:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4=', 'Content-Length': '0'} 2023-10-12T00:44:34.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be59cb34f8f3acc2-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.079 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.079 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.080 DEBUG:boto:path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.080 DEBUG:boto:auth_path=/wafmoa-54/a?versionId=null 2023-10-12T00:44:34.081 DEBUG:boto:Method: GET 2023-10-12T00:44:34.081 DEBUG:boto:Path: /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.082 DEBUG:boto:Data: 2023-10-12T00:44:34.082 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.083 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.084 DEBUG:boto:Params: {} 2023-10-12T00:44:34.084 DEBUG:boto:Token: None 2023-10-12T00:44:34.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a?versionId=null 2023-10-12T00:44:34.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4= 2023-10-12T00:44:34.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QMkXVlc3L+8WjY+dGIA32g/pTI4=', 'Content-Length': '0'} 2023-10-12T00:44:34.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013de4694fc5efc19-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.088 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.089 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.089 DEBUG:boto:path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.090 DEBUG:boto:auth_path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.090 DEBUG:boto:Method: GET 2023-10-12T00:44:34.091 DEBUG:boto:Path: /wafmoa-54/?prefix=a 2023-10-12T00:44:34.091 DEBUG:boto:Data: 2023-10-12T00:44:34.091 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.092 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.093 DEBUG:boto:Params: {} 2023-10-12T00:44:34.093 DEBUG:boto:Token: None 2023-10-12T00:44:34.094 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd8223425fe32ab-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.098 DEBUG:boto:b'wafmoa-54a1000falsea2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.099 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.099 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.100 DEBUG:boto:path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.100 DEBUG:boto:auth_path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.101 DEBUG:boto:Method: GET 2023-10-12T00:44:34.101 DEBUG:boto:Path: /wafmoa-54/?prefix=a 2023-10-12T00:44:34.101 DEBUG:boto:Data: 2023-10-12T00:44:34.102 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.103 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.103 DEBUG:boto:Params: {} 2023-10-12T00:44:34.104 DEBUG:boto:Token: None 2023-10-12T00:44:34.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a4d9def324246e5-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.109 DEBUG:boto:b'wafmoa-54a1000falsea2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.109 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.110 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.110 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.111 DEBUG:boto:Path: /wafmoa-54/a 2023-10-12T00:44:34.111 DEBUG:boto:Data: 2023-10-12T00:44:34.111 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.112 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.113 DEBUG:boto:Params: {} 2023-10-12T00:44:34.113 DEBUG:boto:Token: None 2023-10-12T00:44:34.114 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a 2023-10-12T00:44:34.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfhVAGcdL5kqpYZHCK9Kj+SJrYk= 2023-10-12T00:44:34.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfhVAGcdL5kqpYZHCK9Kj+SJrYk=', 'Content-Length': '0'} 2023-10-12T00:44:34.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac2e5f978089780-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.117 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.118 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.118 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.119 DEBUG:boto:Path: /wafmoa-54/a 2023-10-12T00:44:34.119 DEBUG:boto:Data: 2023-10-12T00:44:34.120 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.120 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.121 DEBUG:boto:Params: {} 2023-10-12T00:44:34.121 DEBUG:boto:Token: None 2023-10-12T00:44:34.122 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a 2023-10-12T00:44:34.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfhVAGcdL5kqpYZHCK9Kj+SJrYk= 2023-10-12T00:44:34.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfhVAGcdL5kqpYZHCK9Kj+SJrYk=', 'Content-Length': '0'} 2023-10-12T00:44:34.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044afdb6bd81ebc9f-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:44:34.126 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.126 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.127 DEBUG:boto:Method: GET 2023-10-12T00:44:34.127 DEBUG:boto:Path: /wafmoa-54/a 2023-10-12T00:44:34.127 DEBUG:boto:Data: 2023-10-12T00:44:34.128 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.129 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.129 DEBUG:boto:Params: {} 2023-10-12T00:44:34.130 DEBUG:boto:Token: None 2023-10-12T00:44:34.130 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a 2023-10-12T00:44:34.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ITaua1FQtx0otSlbN8LjxkdHXS4= 2023-10-12T00:44:34.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ITaua1FQtx0otSlbN8LjxkdHXS4=', 'Content-Length': '0'} 2023-10-12T00:44:34.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05521aa08d981cb-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.134 DEBUG:boto:path=/wafmoa-54/a 2023-10-12T00:44:34.134 DEBUG:boto:auth_path=/wafmoa-54/a 2023-10-12T00:44:34.135 DEBUG:boto:Method: GET 2023-10-12T00:44:34.135 DEBUG:boto:Path: /wafmoa-54/a 2023-10-12T00:44:34.136 DEBUG:boto:Data: 2023-10-12T00:44:34.136 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.137 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.137 DEBUG:boto:Params: {} 2023-10-12T00:44:34.138 DEBUG:boto:Token: None 2023-10-12T00:44:34.138 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/a 2023-10-12T00:44:34.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ITaua1FQtx0otSlbN8LjxkdHXS4= 2023-10-12T00:44:34.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ITaua1FQtx0otSlbN8LjxkdHXS4=', 'Content-Length': '0'} 2023-10-12T00:44:34.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f5440df2ba932e-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.142 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.143 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.143 DEBUG:boto:path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.143 DEBUG:boto:auth_path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.144 DEBUG:boto:Method: GET 2023-10-12T00:44:34.144 DEBUG:boto:Path: /wafmoa-54/?prefix=a 2023-10-12T00:44:34.145 DEBUG:boto:Data: 2023-10-12T00:44:34.145 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.146 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.147 DEBUG:boto:Params: {} 2023-10-12T00:44:34.147 DEBUG:boto:Token: None 2023-10-12T00:44:34.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac4273c7cc9a6de-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.152 DEBUG:boto:b'wafmoa-54a1000falsea2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.152 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.153 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.153 DEBUG:boto:path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.154 DEBUG:boto:auth_path=/wafmoa-54/?prefix=a 2023-10-12T00:44:34.154 DEBUG:boto:Method: GET 2023-10-12T00:44:34.155 DEBUG:boto:Path: /wafmoa-54/?prefix=a 2023-10-12T00:44:34.155 DEBUG:boto:Data: 2023-10-12T00:44:34.156 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.156 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.157 DEBUG:boto:Params: {} 2023-10-12T00:44:34.157 DEBUG:boto:Token: None 2023-10-12T00:44:34.158 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048e18ca57c2ca9b1-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.162 DEBUG:boto:b'wafmoa-54a1000falsea2023-10-12T00:44:13.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:34.163 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.164 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.164 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.164 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.165 DEBUG:boto:Method: GET 2023-10-12T00:44:34.165 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.166 DEBUG:boto:Data: 2023-10-12T00:44:34.166 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.167 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.168 DEBUG:boto:Params: {} 2023-10-12T00:44:34.168 DEBUG:boto:Token: None 2023-10-12T00:44:34.169 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw= 2023-10-12T00:44:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw=', 'Content-Length': '0'} 2023-10-12T00:44:34.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000489ae3dcbf61044e-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.218 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.219 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.219 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.220 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.220 DEBUG:boto:Method: GET 2023-10-12T00:44:34.220 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.221 DEBUG:boto:Data: 2023-10-12T00:44:34.221 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.222 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.222 DEBUG:boto:Params: {} 2023-10-12T00:44:34.223 DEBUG:boto:Token: None 2023-10-12T00:44:34.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw= 2023-10-12T00:44:34.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw=', 'Content-Length': '0'} 2023-10-12T00:44:34.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5769dfda7252b19-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.227 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.227 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.228 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.228 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.228 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.229 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.229 DEBUG:boto:Data: 2023-10-12T00:44:34.230 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.230 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.231 DEBUG:boto:Params: {} 2023-10-12T00:44:34.231 DEBUG:boto:Token: None 2023-10-12T00:44:34.232 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kGUL/olKNYXutJs1TZK43qw1Y9k= 2023-10-12T00:44:34.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kGUL/olKNYXutJs1TZK43qw1Y9k=', 'Content-Length': '0'} 2023-10-12T00:44:34.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007391e900126a18c9-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.235 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.236 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.236 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.236 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.237 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.237 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.238 DEBUG:boto:Data: 2023-10-12T00:44:34.238 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.239 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.239 DEBUG:boto:Params: {} 2023-10-12T00:44:34.239 DEBUG:boto:Token: None 2023-10-12T00:44:34.240 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kGUL/olKNYXutJs1TZK43qw1Y9k= 2023-10-12T00:44:34.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kGUL/olKNYXutJs1TZK43qw1Y9k=', 'Content-Length': '0'} 2023-10-12T00:44:34.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459e0526fc51c047-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:34.243 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.244 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.244 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.245 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.245 DEBUG:boto:Method: GET 2023-10-12T00:44:34.245 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.246 DEBUG:boto:Data: 2023-10-12T00:44:34.246 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.247 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.248 DEBUG:boto:Params: {} 2023-10-12T00:44:34.248 DEBUG:boto:Token: None 2023-10-12T00:44:34.248 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw= 2023-10-12T00:44:34.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw=', 'Content-Length': '0'} 2023-10-12T00:44:34.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691355c9447e5b1f-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.252 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.252 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.253 DEBUG:boto:path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.253 DEBUG:boto:auth_path=/wafmoa-54/b?versionId=null 2023-10-12T00:44:34.253 DEBUG:boto:Method: GET 2023-10-12T00:44:34.254 DEBUG:boto:Path: /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.254 DEBUG:boto:Data: 2023-10-12T00:44:34.255 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.256 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.256 DEBUG:boto:Params: {} 2023-10-12T00:44:34.256 DEBUG:boto:Token: None 2023-10-12T00:44:34.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b?versionId=null 2023-10-12T00:44:34.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw= 2023-10-12T00:44:34.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nVIexjPvw8YAndxkzXfxe9D5uBw=', 'Content-Length': '0'} 2023-10-12T00:44:34.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007500a644a746dd8e-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.302 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.303 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.303 DEBUG:boto:path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.304 DEBUG:boto:auth_path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.304 DEBUG:boto:Method: GET 2023-10-12T00:44:34.304 DEBUG:boto:Path: /wafmoa-54/?prefix=b 2023-10-12T00:44:34.305 DEBUG:boto:Data: 2023-10-12T00:44:34.305 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.306 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.307 DEBUG:boto:Params: {} 2023-10-12T00:44:34.307 DEBUG:boto:Token: None 2023-10-12T00:44:34.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680c232ec984a9e8-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.311 DEBUG:boto:b'wafmoa-54b1000falseb2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.312 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.313 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.313 DEBUG:boto:path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.313 DEBUG:boto:auth_path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.314 DEBUG:boto:Method: GET 2023-10-12T00:44:34.314 DEBUG:boto:Path: /wafmoa-54/?prefix=b 2023-10-12T00:44:34.315 DEBUG:boto:Data: 2023-10-12T00:44:34.315 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.316 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.317 DEBUG:boto:Params: {} 2023-10-12T00:44:34.317 DEBUG:boto:Token: None 2023-10-12T00:44:34.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b844a90e66787da-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.322 DEBUG:boto:b'wafmoa-54b1000falseb2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.322 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.323 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.323 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.324 DEBUG:boto:Path: /wafmoa-54/b 2023-10-12T00:44:34.324 DEBUG:boto:Data: 2023-10-12T00:44:34.325 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.325 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.326 DEBUG:boto:Params: {} 2023-10-12T00:44:34.326 DEBUG:boto:Token: None 2023-10-12T00:44:34.327 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b 2023-10-12T00:44:34.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNsuQe6gD9sx3t1wWKLp2EwYOSw= 2023-10-12T00:44:34.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNsuQe6gD9sx3t1wWKLp2EwYOSw=', 'Content-Length': '0'} 2023-10-12T00:44:34.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007778964f3dc7e445-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.330 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.331 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.331 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.332 DEBUG:boto:Path: /wafmoa-54/b 2023-10-12T00:44:34.332 DEBUG:boto:Data: 2023-10-12T00:44:34.332 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.333 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.334 DEBUG:boto:Params: {} 2023-10-12T00:44:34.334 DEBUG:boto:Token: None 2023-10-12T00:44:34.335 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b 2023-10-12T00:44:34.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNsuQe6gD9sx3t1wWKLp2EwYOSw= 2023-10-12T00:44:34.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNsuQe6gD9sx3t1wWKLp2EwYOSw=', 'Content-Length': '0'} 2023-10-12T00:44:34.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e91132ff2c16c77-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:44:34.339 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.339 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.339 DEBUG:boto:Method: GET 2023-10-12T00:44:34.340 DEBUG:boto:Path: /wafmoa-54/b 2023-10-12T00:44:34.340 DEBUG:boto:Data: 2023-10-12T00:44:34.341 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.341 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.342 DEBUG:boto:Params: {} 2023-10-12T00:44:34.342 DEBUG:boto:Token: None 2023-10-12T00:44:34.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b 2023-10-12T00:44:34.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8xBS0nwf/qcsWchq3Fn+Ry50/qE= 2023-10-12T00:44:34.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8xBS0nwf/qcsWchq3Fn+Ry50/qE=', 'Content-Length': '0'} 2023-10-12T00:44:34.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9ad26b727139fc-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.386 DEBUG:boto:path=/wafmoa-54/b 2023-10-12T00:44:34.387 DEBUG:boto:auth_path=/wafmoa-54/b 2023-10-12T00:44:34.387 DEBUG:boto:Method: GET 2023-10-12T00:44:34.388 DEBUG:boto:Path: /wafmoa-54/b 2023-10-12T00:44:34.389 DEBUG:boto:Data: 2023-10-12T00:44:34.389 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.390 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.390 DEBUG:boto:Params: {} 2023-10-12T00:44:34.391 DEBUG:boto:Token: None 2023-10-12T00:44:34.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/b 2023-10-12T00:44:34.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8xBS0nwf/qcsWchq3Fn+Ry50/qE= 2023-10-12T00:44:34.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8xBS0nwf/qcsWchq3Fn+Ry50/qE=', 'Content-Length': '0'} 2023-10-12T00:44:34.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a45b049415a9c48e-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.395 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.396 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.396 DEBUG:boto:path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.397 DEBUG:boto:auth_path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.397 DEBUG:boto:Method: GET 2023-10-12T00:44:34.397 DEBUG:boto:Path: /wafmoa-54/?prefix=b 2023-10-12T00:44:34.398 DEBUG:boto:Data: 2023-10-12T00:44:34.398 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.399 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.400 DEBUG:boto:Params: {} 2023-10-12T00:44:34.400 DEBUG:boto:Token: None 2023-10-12T00:44:34.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003183b404b532552-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.405 DEBUG:boto:b'wafmoa-54b1000falseb2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.405 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.406 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.406 DEBUG:boto:path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.407 DEBUG:boto:auth_path=/wafmoa-54/?prefix=b 2023-10-12T00:44:34.407 DEBUG:boto:Method: GET 2023-10-12T00:44:34.408 DEBUG:boto:Path: /wafmoa-54/?prefix=b 2023-10-12T00:44:34.408 DEBUG:boto:Data: 2023-10-12T00:44:34.409 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.410 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.410 DEBUG:boto:Params: {} 2023-10-12T00:44:34.411 DEBUG:boto:Token: None 2023-10-12T00:44:34.411 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d60d1f2616bedf08-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.415 DEBUG:boto:b'wafmoa-54b1000falseb2023-10-12T00:44:13.289Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:34.416 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.417 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.417 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.418 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.418 DEBUG:boto:Method: GET 2023-10-12T00:44:34.419 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.419 DEBUG:boto:Data: 2023-10-12T00:44:34.420 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.421 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.421 DEBUG:boto:Params: {} 2023-10-12T00:44:34.421 DEBUG:boto:Token: None 2023-10-12T00:44:34.422 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas= 2023-10-12T00:44:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas=', 'Content-Length': '0'} 2023-10-12T00:44:34.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed6ca8813bd35e6-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.426 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.426 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.427 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.427 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.427 DEBUG:boto:Method: GET 2023-10-12T00:44:34.428 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.428 DEBUG:boto:Data: 2023-10-12T00:44:34.429 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.430 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.430 DEBUG:boto:Params: {} 2023-10-12T00:44:34.431 DEBUG:boto:Token: None 2023-10-12T00:44:34.431 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas= 2023-10-12T00:44:34.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas=', 'Content-Length': '0'} 2023-10-12T00:44:34.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aeeb948d1d1f551-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.482 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.483 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.483 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.484 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.484 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.485 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.485 DEBUG:boto:Data: 2023-10-12T00:44:34.485 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.486 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.487 DEBUG:boto:Params: {} 2023-10-12T00:44:34.487 DEBUG:boto:Token: None 2023-10-12T00:44:34.488 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkQvZqMcHuLBevEUBpd598iTCXA= 2023-10-12T00:44:34.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkQvZqMcHuLBevEUBpd598iTCXA=', 'Content-Length': '0'} 2023-10-12T00:44:34.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645817a815937dbb-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.491 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.492 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.492 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.493 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.493 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.493 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.494 DEBUG:boto:Data: 2023-10-12T00:44:34.494 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.495 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.495 DEBUG:boto:Params: {} 2023-10-12T00:44:34.496 DEBUG:boto:Token: None 2023-10-12T00:44:34.496 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkQvZqMcHuLBevEUBpd598iTCXA= 2023-10-12T00:44:34.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkQvZqMcHuLBevEUBpd598iTCXA=', 'Content-Length': '0'} 2023-10-12T00:44:34.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe8d4db9e3ae072-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:34.500 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.501 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.501 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.502 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.502 DEBUG:boto:Method: GET 2023-10-12T00:44:34.503 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.503 DEBUG:boto:Data: 2023-10-12T00:44:34.503 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.504 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.505 DEBUG:boto:Params: {} 2023-10-12T00:44:34.505 DEBUG:boto:Token: None 2023-10-12T00:44:34.506 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas= 2023-10-12T00:44:34.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas=', 'Content-Length': '0'} 2023-10-12T00:44:34.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f372889dfb082385-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.509 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.510 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.510 DEBUG:boto:path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.511 DEBUG:boto:auth_path=/wafmoa-54/c?versionId=null 2023-10-12T00:44:34.511 DEBUG:boto:Method: GET 2023-10-12T00:44:34.511 DEBUG:boto:Path: /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.512 DEBUG:boto:Data: 2023-10-12T00:44:34.512 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.513 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.513 DEBUG:boto:Params: {} 2023-10-12T00:44:34.514 DEBUG:boto:Token: None 2023-10-12T00:44:34.514 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c?versionId=null 2023-10-12T00:44:34.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas= 2023-10-12T00:44:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HvDROKCb/uWcj41HVZEyvJ1Vpas=', 'Content-Length': '0'} 2023-10-12T00:44:34.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e0c62a226225d9-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.518 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.518 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.519 DEBUG:boto:path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.519 DEBUG:boto:auth_path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.520 DEBUG:boto:Method: GET 2023-10-12T00:44:34.520 DEBUG:boto:Path: /wafmoa-54/?prefix=c 2023-10-12T00:44:34.520 DEBUG:boto:Data: 2023-10-12T00:44:34.521 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.522 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.522 DEBUG:boto:Params: {} 2023-10-12T00:44:34.523 DEBUG:boto:Token: None 2023-10-12T00:44:34.523 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd07453c5c8bceaa-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.527 DEBUG:boto:b'wafmoa-54c1000falsec2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.527 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.528 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.528 DEBUG:boto:path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.529 DEBUG:boto:auth_path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.529 DEBUG:boto:Method: GET 2023-10-12T00:44:34.529 DEBUG:boto:Path: /wafmoa-54/?prefix=c 2023-10-12T00:44:34.530 DEBUG:boto:Data: 2023-10-12T00:44:34.530 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.531 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.531 DEBUG:boto:Params: {} 2023-10-12T00:44:34.532 DEBUG:boto:Token: None 2023-10-12T00:44:34.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb9135b9869c5cf-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.536 DEBUG:boto:b'wafmoa-54c1000falsec2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.537 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.537 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.538 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.538 DEBUG:boto:Path: /wafmoa-54/c 2023-10-12T00:44:34.538 DEBUG:boto:Data: 2023-10-12T00:44:34.539 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.539 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.540 DEBUG:boto:Params: {} 2023-10-12T00:44:34.540 DEBUG:boto:Token: None 2023-10-12T00:44:34.541 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c 2023-10-12T00:44:34.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hziowz6czhuYGo2qfaTfKVx9TQ4= 2023-10-12T00:44:34.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hziowz6czhuYGo2qfaTfKVx9TQ4=', 'Content-Length': '0'} 2023-10-12T00:44:34.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6859e74a2b83395-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.544 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.544 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.545 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.545 DEBUG:boto:Path: /wafmoa-54/c 2023-10-12T00:44:34.546 DEBUG:boto:Data: 2023-10-12T00:44:34.546 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.547 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.547 DEBUG:boto:Params: {} 2023-10-12T00:44:34.549 DEBUG:boto:Token: None 2023-10-12T00:44:34.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c 2023-10-12T00:44:34.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hziowz6czhuYGo2qfaTfKVx9TQ4= 2023-10-12T00:44:34.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hziowz6czhuYGo2qfaTfKVx9TQ4=', 'Content-Length': '0'} 2023-10-12T00:44:34.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab4ce03ddb14eda-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:44:34.554 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.555 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.555 DEBUG:boto:Method: GET 2023-10-12T00:44:34.555 DEBUG:boto:Path: /wafmoa-54/c 2023-10-12T00:44:34.556 DEBUG:boto:Data: 2023-10-12T00:44:34.556 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.557 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.557 DEBUG:boto:Params: {} 2023-10-12T00:44:34.558 DEBUG:boto:Token: None 2023-10-12T00:44:34.558 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c 2023-10-12T00:44:34.559 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HaPV+UdZ4yKhF+xVh1C1Mbz3qyA= 2023-10-12T00:44:34.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HaPV+UdZ4yKhF+xVh1C1Mbz3qyA=', 'Content-Length': '0'} 2023-10-12T00:44:34.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bee31bab64ed86b-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.602 DEBUG:boto:path=/wafmoa-54/c 2023-10-12T00:44:34.603 DEBUG:boto:auth_path=/wafmoa-54/c 2023-10-12T00:44:34.603 DEBUG:boto:Method: GET 2023-10-12T00:44:34.604 DEBUG:boto:Path: /wafmoa-54/c 2023-10-12T00:44:34.604 DEBUG:boto:Data: 2023-10-12T00:44:34.605 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.605 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.606 DEBUG:boto:Params: {} 2023-10-12T00:44:34.606 DEBUG:boto:Token: None 2023-10-12T00:44:34.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/c 2023-10-12T00:44:34.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HaPV+UdZ4yKhF+xVh1C1Mbz3qyA= 2023-10-12T00:44:34.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HaPV+UdZ4yKhF+xVh1C1Mbz3qyA=', 'Content-Length': '0'} 2023-10-12T00:44:34.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5330a3e1f0ac2b-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.611 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.611 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.611 DEBUG:boto:path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.612 DEBUG:boto:auth_path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.612 DEBUG:boto:Method: GET 2023-10-12T00:44:34.613 DEBUG:boto:Path: /wafmoa-54/?prefix=c 2023-10-12T00:44:34.613 DEBUG:boto:Data: 2023-10-12T00:44:34.614 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.615 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.615 DEBUG:boto:Params: {} 2023-10-12T00:44:34.615 DEBUG:boto:Token: None 2023-10-12T00:44:34.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c035f17f2a47bf9-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.620 DEBUG:boto:b'wafmoa-54c1000falsec2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.621 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.621 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.622 DEBUG:boto:path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.622 DEBUG:boto:auth_path=/wafmoa-54/?prefix=c 2023-10-12T00:44:34.622 DEBUG:boto:Method: GET 2023-10-12T00:44:34.623 DEBUG:boto:Path: /wafmoa-54/?prefix=c 2023-10-12T00:44:34.623 DEBUG:boto:Data: 2023-10-12T00:44:34.624 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.625 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.625 DEBUG:boto:Params: {} 2023-10-12T00:44:34.625 DEBUG:boto:Token: None 2023-10-12T00:44:34.626 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9392ebe6d7ed3d-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.630 DEBUG:boto:b'wafmoa-54c1000falsec2023-10-12T00:44:13.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:34.631 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.631 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.632 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.632 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.633 DEBUG:boto:Method: GET 2023-10-12T00:44:34.633 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.633 DEBUG:boto:Data: 2023-10-12T00:44:34.634 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.635 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.635 DEBUG:boto:Params: {} 2023-10-12T00:44:34.636 DEBUG:boto:Token: None 2023-10-12T00:44:34.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8= 2023-10-12T00:44:34.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8=', 'Content-Length': '0'} 2023-10-12T00:44:34.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076675ace9d59ace2-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.640 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.640 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.641 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.641 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.641 DEBUG:boto:Method: GET 2023-10-12T00:44:34.642 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.642 DEBUG:boto:Data: 2023-10-12T00:44:34.643 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.644 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.644 DEBUG:boto:Params: {} 2023-10-12T00:44:34.644 DEBUG:boto:Token: None 2023-10-12T00:44:34.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8= 2023-10-12T00:44:34.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8=', 'Content-Length': '0'} 2023-10-12T00:44:34.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deeace4c17743cc2-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.648 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.649 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.649 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.650 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.650 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.651 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.651 DEBUG:boto:Data: 2023-10-12T00:44:34.651 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.652 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.653 DEBUG:boto:Params: {} 2023-10-12T00:44:34.653 DEBUG:boto:Token: None 2023-10-12T00:44:34.653 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/4Rqqi0WShXcWnvNkN9np+AhBs= 2023-10-12T00:44:34.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/4Rqqi0WShXcWnvNkN9np+AhBs=', 'Content-Length': '0'} 2023-10-12T00:44:34.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240ad6e377193087-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.657 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.657 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.658 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.658 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.658 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.659 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.659 DEBUG:boto:Data: 2023-10-12T00:44:34.660 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.661 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.661 DEBUG:boto:Params: {} 2023-10-12T00:44:34.662 DEBUG:boto:Token: None 2023-10-12T00:44:34.662 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K/4Rqqi0WShXcWnvNkN9np+AhBs= 2023-10-12T00:44:34.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K/4Rqqi0WShXcWnvNkN9np+AhBs=', 'Content-Length': '0'} 2023-10-12T00:44:34.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b46b9283332b3d7-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:34.666 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.666 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.667 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.667 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.668 DEBUG:boto:Method: GET 2023-10-12T00:44:34.668 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.669 DEBUG:boto:Data: 2023-10-12T00:44:34.669 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.670 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.670 DEBUG:boto:Params: {} 2023-10-12T00:44:34.671 DEBUG:boto:Token: None 2023-10-12T00:44:34.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8= 2023-10-12T00:44:34.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8=', 'Content-Length': '0'} 2023-10-12T00:44:34.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cd302c6ef7074c-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.722 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.723 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.723 DEBUG:boto:path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.724 DEBUG:boto:auth_path=/wafmoa-54/d?versionId=null 2023-10-12T00:44:34.724 DEBUG:boto:Method: GET 2023-10-12T00:44:34.725 DEBUG:boto:Path: /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.725 DEBUG:boto:Data: 2023-10-12T00:44:34.726 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.726 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.727 DEBUG:boto:Params: {} 2023-10-12T00:44:34.727 DEBUG:boto:Token: None 2023-10-12T00:44:34.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d?versionId=null 2023-10-12T00:44:34.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8= 2023-10-12T00:44:34.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xUoFhmg8zJUL9FdAD3H2hzIvbH8=', 'Content-Length': '0'} 2023-10-12T00:44:34.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001922530554682b85-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.731 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.732 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.732 DEBUG:boto:path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.733 DEBUG:boto:auth_path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.733 DEBUG:boto:Method: GET 2023-10-12T00:44:34.734 DEBUG:boto:Path: /wafmoa-54/?prefix=d 2023-10-12T00:44:34.734 DEBUG:boto:Data: 2023-10-12T00:44:34.735 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.736 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.736 DEBUG:boto:Params: {} 2023-10-12T00:44:34.737 DEBUG:boto:Token: None 2023-10-12T00:44:34.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf442b8eac0195b7-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.741 DEBUG:boto:b'wafmoa-54d1000falsed2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.742 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.742 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.743 DEBUG:boto:path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.743 DEBUG:boto:auth_path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.744 DEBUG:boto:Method: GET 2023-10-12T00:44:34.744 DEBUG:boto:Path: /wafmoa-54/?prefix=d 2023-10-12T00:44:34.745 DEBUG:boto:Data: 2023-10-12T00:44:34.745 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.746 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.746 DEBUG:boto:Params: {} 2023-10-12T00:44:34.747 DEBUG:boto:Token: None 2023-10-12T00:44:34.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c44175ca3a3c6f0-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.754 DEBUG:boto:b'wafmoa-54d1000falsed2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.755 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.755 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.756 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.756 DEBUG:boto:Path: /wafmoa-54/d 2023-10-12T00:44:34.758 DEBUG:boto:Data: 2023-10-12T00:44:34.758 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.759 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.760 DEBUG:boto:Params: {} 2023-10-12T00:44:34.761 DEBUG:boto:Token: None 2023-10-12T00:44:34.761 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d 2023-10-12T00:44:34.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gsJoM7Y+pFZM0fcWJrs3762jYZs= 2023-10-12T00:44:34.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gsJoM7Y+pFZM0fcWJrs3762jYZs=', 'Content-Length': '0'} 2023-10-12T00:44:34.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beeed6a5e8e3cf05-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.765 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.765 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.766 DEBUG:boto:Method: HEAD 2023-10-12T00:44:34.766 DEBUG:boto:Path: /wafmoa-54/d 2023-10-12T00:44:34.767 DEBUG:boto:Data: 2023-10-12T00:44:34.767 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.768 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.769 DEBUG:boto:Params: {} 2023-10-12T00:44:34.769 DEBUG:boto:Token: None 2023-10-12T00:44:34.770 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d 2023-10-12T00:44:34.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gsJoM7Y+pFZM0fcWJrs3762jYZs= 2023-10-12T00:44:34.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gsJoM7Y+pFZM0fcWJrs3762jYZs=', 'Content-Length': '0'} 2023-10-12T00:44:34.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4647bd4906c4f26-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:44:34.774 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.774 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.775 DEBUG:boto:Method: GET 2023-10-12T00:44:34.775 DEBUG:boto:Path: /wafmoa-54/d 2023-10-12T00:44:34.775 DEBUG:boto:Data: 2023-10-12T00:44:34.776 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.777 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.777 DEBUG:boto:Params: {} 2023-10-12T00:44:34.778 DEBUG:boto:Token: None 2023-10-12T00:44:34.778 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d 2023-10-12T00:44:34.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B4P+hQjHA8Bq22JT453VIAoORFY= 2023-10-12T00:44:34.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4P+hQjHA8Bq22JT453VIAoORFY=', 'Content-Length': '0'} 2023-10-12T00:44:34.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-54:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93a37b9d55b14f3-0065274172-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.782 DEBUG:boto:path=/wafmoa-54/d 2023-10-12T00:44:34.783 DEBUG:boto:auth_path=/wafmoa-54/d 2023-10-12T00:44:34.783 DEBUG:boto:Method: GET 2023-10-12T00:44:34.783 DEBUG:boto:Path: /wafmoa-54/d 2023-10-12T00:44:34.784 DEBUG:boto:Data: 2023-10-12T00:44:34.784 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.785 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.786 DEBUG:boto:Params: {} 2023-10-12T00:44:34.786 DEBUG:boto:Token: None 2023-10-12T00:44:34.787 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/d 2023-10-12T00:44:34.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B4P+hQjHA8Bq22JT453VIAoORFY= 2023-10-12T00:44:34.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4P+hQjHA8Bq22JT453VIAoORFY=', 'Content-Length': '0'} 2023-10-12T00:44:34.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006535a7beb65aa89f-0065274172-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.790 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.791 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.791 DEBUG:boto:path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.792 DEBUG:boto:auth_path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.792 DEBUG:boto:Method: GET 2023-10-12T00:44:34.793 DEBUG:boto:Path: /wafmoa-54/?prefix=d 2023-10-12T00:44:34.793 DEBUG:boto:Data: 2023-10-12T00:44:34.793 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:44:34.794 DEBUG:boto:Port: 8001 2023-10-12T00:44:34.795 DEBUG:boto:Params: {} 2023-10-12T00:44:34.795 DEBUG:boto:Token: None 2023-10-12T00:44:34.796 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e7619219d14bfd-0065274172-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.801 DEBUG:boto:b'wafmoa-54d1000falsed2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.803 DEBUG:boto:path=/wafmoa-54/ 2023-10-12T00:44:34.803 DEBUG:boto:auth_path=/wafmoa-54/ 2023-10-12T00:44:34.803 DEBUG:boto:path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.804 DEBUG:boto:auth_path=/wafmoa-54/?prefix=d 2023-10-12T00:44:34.804 DEBUG:boto:Method: GET 2023-10-12T00:44:34.805 DEBUG:boto:Path: /wafmoa-54/?prefix=d 2023-10-12T00:44:34.805 DEBUG:boto:Data: 2023-10-12T00:44:34.805 DEBUG:boto:Headers: {} 2023-10-12T00:44:34.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:34.806 DEBUG:boto:Port: 8000 2023-10-12T00:44:34.807 DEBUG:boto:Params: {} 2023-10-12T00:44:34.807 DEBUG:boto:Token: None 2023-10-12T00:44:34.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:44:34 GMT /wafmoa-54/ 2023-10-12T00:44:34.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI= 2023-10-12T00:44:34.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cu1OUv5H0xcOugiS+ghAlO1EdXI=', 'Content-Length': '0'} 2023-10-12T00:44:34.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f26aca86b8d31e-0065274172-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:34.812 DEBUG:boto:b'wafmoa-54d1000falsed2023-10-12T00:44:13.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:44:34.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-54 zones={a2, a1} 2023-10-12T00:44:34.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:34.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:34.814 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:44:34.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:34.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:34.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:34.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:34.923 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:34.973 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:34.973 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:35.013 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:44:35.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:35.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:35.016 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:35.104 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:44:35.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.104 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "1#00000000001.36.6", 2023-10-12T00:44:35.105 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6919952298291255714", 2023-10-12T00:44:35.105 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:44:35.105 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "a", 2023-10-12T00:44:35.105 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:44:35.105 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:44:35.106 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:44:35.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.777806088Z", 2023-10-12T00:44:35.106 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:44:35.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:44:35.107 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:44:35.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.107 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:44:35.107 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:44:35.107 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:44:35.108 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:44:35.108 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:44:35.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.108 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18" 2023-10-12T00:44:35.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:35.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:35.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.110 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "5#00000000001.22.6", 2023-10-12T00:44:35.110 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10626242038868823301", 2023-10-12T00:44:35.110 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:44:35.110 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "d", 2023-10-12T00:44:35.111 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:44:35.111 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:44:35.111 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:44:35.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.877310826Z", 2023-10-12T00:44:35.111 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:44:35.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:44:35.112 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:44:35.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.112 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:44:35.113 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:44:35.113 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:44:35.113 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:44:35.113 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:44:35.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.114 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18" 2023-10-12T00:44:35.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:35.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:35.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.115 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "8#00000000001.33.6", 2023-10-12T00:44:35.115 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17937023530274159012", 2023-10-12T00:44:35.115 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:44:35.116 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "c", 2023-10-12T00:44:35.116 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:44:35.116 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:44:35.116 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:44:35.117 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.854837579Z", 2023-10-12T00:44:35.117 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:44:35.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:44:35.117 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T00:44:35.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.118 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:44:35.118 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:44:35.118 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:44:35.118 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:44:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:44:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18" 2023-10-12T00:44:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "10#00000000001.31.6", 2023-10-12T00:44:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10165696429699290734", 2023-10-12T00:44:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:44:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "b", 2023-10-12T00:44:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:44:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:44:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:44:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:43:46.799375917Z", 2023-10-12T00:44:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:44:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:44:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:44:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:44:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:44:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:44:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:44:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:44:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-53:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.18" 2023-10-12T00:44:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:35.125 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:35.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:35.125 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:44:35.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:35.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:44:35.127 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:44:35.174 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:35.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:35.216 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.216+0000 7fbe80e17700 -1 req 1111594471696073198 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-37:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.2,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-37:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.2,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.217 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.217+0000 7fbe79608700 -1 req 11661083890879769687 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-27:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.27,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-27:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.27,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.217 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.217+0000 7fbe90636700 -1 req 17531710451633915301 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-19:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.19,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-19:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.19,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.220 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.221+0000 7fbe6adeb700 -1 req 4583287081590152857 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-23:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.23,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-23:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.23,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.223 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.223+0000 7fbe8a62a700 -1 req 9803023909601375559 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-43:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.8,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-43:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.8,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.265 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.265+0000 7fbe455a0700 -1 req 10248238066355922572 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-5:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.5,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-5:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.5,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.267 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.267+0000 7fbe53dbd700 -1 req 2740765986238034524 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-1:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.1,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-1:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.1,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.311 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.311+0000 7fbe4c5ae700 -1 req 5982592213524380154 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-4:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.4,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-4:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.4,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.312 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.313+0000 7fbe45da1700 -1 req 5499482331758679311 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-21:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.21,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-21:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.21,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.355 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.355+0000 7fbe2355c700 -1 req 15177617304527955270 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-24:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.24,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-24:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.24,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.357 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.357+0000 7fbe1ed53700 -1 req 5065253107976998292 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-20:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.20,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-20:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.20,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.400 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.400+0000 7fbe28d67700 -1 req 349221696107235694 0.002999921s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-6:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.6,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-6:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.6,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.405 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.405+0000 7fbe06d23700 -1 req 185157422496490007 0.001999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-2:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.2,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-2:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.2,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.444 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.444+0000 7fbdffd15700 -1 req 3502359942870035271 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-26:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.26,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-26:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.26,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.496 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.496+0000 7fbedfed5700 -1 req 7243449378530486252 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-29:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.29,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-29:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.29,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.497 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.498+0000 7fbed8ec7700 -1 req 8110279857540322220 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-38:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.3,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-38:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.3,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.499 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.499+0000 7fbef16f8700 -1 req 9624970414308845602 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-28:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.28,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-28:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.28,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.531 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.531+0000 7fbed36bc700 -1 req 8736974989030603358 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-25:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.25,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-25:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.25,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.539 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.539+0000 7fbee16d8700 -1 req 18139545173279567043 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-36:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.1,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-36:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.1,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.542 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.542+0000 7fbec8ea7700 -1 req 17545693084784309767 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-41:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.6,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-41:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.6,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.587 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.587+0000 7fbea0656700 -1 req 6344136847056791622 0.002999921s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-42:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.7,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-42:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.7,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.622 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.622+0000 7fbeae672700 -1 req 6996202961220464694 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-51:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.16,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-51:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.16,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.627 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.627+0000 7fbe95e41700 -1 req 13326658179527873401 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-22:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.22,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-22:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.22,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.675 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.676+0000 7fbe78606700 -1 req 8071943184113619835 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-39:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.4,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-39:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.4,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.679 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.679+0000 7fbe66de3700 -1 req 6053748343948614296 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-40:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.5,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-40:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.5,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.720 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.720+0000 7fbe71df9700 -1 req 9492996684501838988 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-3:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.3,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-3:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.3,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.765 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:44:35.765+0000 7fbe455a0700 -1 req 7450323659206821246 0.001999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-30:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.30,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-30:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.30,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:44:35.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:35.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:35.823 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:35.870 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:35.871 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:35.912 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:44:35.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-10-12T00:44:35.916 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-55 2023-10-12T00:44:35.916 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:44:35.917 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:44:35.917 DEBUG:boto:Method: PUT 2023-10-12T00:44:35.917 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:44:35.918 DEBUG:boto:Data: 2023-10-12T00:44:35.918 DEBUG:boto:Headers: {} 2023-10-12T00:44:35.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:35.919 DEBUG:boto:Port: 8000 2023-10-12T00:44:35.919 DEBUG:boto:Params: {} 2023-10-12T00:44:35.920 DEBUG:boto:Token: None 2023-10-12T00:44:35.920 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:44:35 GMT /wafmoa-55/ 2023-10-12T00:44:35.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:prFWrRCLGoNZk7sjDvHTF0jk1ps= 2023-10-12T00:44:35.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:prFWrRCLGoNZk7sjDvHTF0jk1ps=', 'Content-Length': '0'} 2023-10-12T00:44:35.936 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd29153033bac0bd-0065274173-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:35.936 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:44:35.937 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:44:35.937 DEBUG:boto:Method: HEAD 2023-10-12T00:44:35.938 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:44:35.938 DEBUG:boto:Data: 2023-10-12T00:44:35.939 DEBUG:boto:Headers: {} 2023-10-12T00:44:35.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:35.939 DEBUG:boto:Port: 8000 2023-10-12T00:44:35.940 DEBUG:boto:Params: {} 2023-10-12T00:44:35.940 DEBUG:boto:Token: None 2023-10-12T00:44:35.941 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:35 GMT /wafmoa-55/ 2023-10-12T00:44:35.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cDNCc0bcp4Ns8wYWNPcIoo+JEy4= 2023-10-12T00:44:35.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cDNCc0bcp4Ns8wYWNPcIoo+JEy4=', 'Content-Length': '0'} 2023-10-12T00:44:35.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000057a517a2d0d4288-0065274173-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:35.945 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:44:35.945 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:44:35.946 DEBUG:boto:Method: PUT 2023-10-12T00:44:35.947 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:44:35.947 DEBUG:boto:Data: 2023-10-12T00:44:35.948 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:35.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:35.949 DEBUG:boto:Port: 8000 2023-10-12T00:44:35.949 DEBUG:boto:Params: {} 2023-10-12T00:44:35.949 DEBUG:boto:Token: None 2023-10-12T00:44:35.950 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:35 GMT /wafmoa-55/a 2023-10-12T00:44:35.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:341hwWliwR+mzkWHMfxJKVefDbA= 2023-10-12T00:44:35.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:341hwWliwR+mzkWHMfxJKVefDbA='} 2023-10-12T00:44:35.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001dfa567fc4a4ba2-0065274173-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:35.995 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:44:35.995 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:44:35.996 DEBUG:boto:Method: HEAD 2023-10-12T00:44:35.996 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:44:35.997 DEBUG:boto:Data: 2023-10-12T00:44:35.997 DEBUG:boto:Headers: {} 2023-10-12T00:44:35.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:35.998 DEBUG:boto:Port: 8000 2023-10-12T00:44:35.999 DEBUG:boto:Params: {} 2023-10-12T00:44:35.999 DEBUG:boto:Token: None 2023-10-12T00:44:36.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/ 2023-10-12T00:44:36.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8= 2023-10-12T00:44:36.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8=', 'Content-Length': '0'} 2023-10-12T00:44:36.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b72ab6a04dcfb4f7-0065274174-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.003 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:44:36.004 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:44:36.004 DEBUG:boto:Method: PUT 2023-10-12T00:44:36.005 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:44:36.005 DEBUG:boto:Data: 2023-10-12T00:44:36.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:36.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:36.006 DEBUG:boto:Port: 8000 2023-10-12T00:44:36.007 DEBUG:boto:Params: {} 2023-10-12T00:44:36.007 DEBUG:boto:Token: None 2023-10-12T00:44:36.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/b 2023-10-12T00:44:36.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:53jc+/7F8sGuwoMsUfOr5vD7tVw= 2023-10-12T00:44:36.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:53jc+/7F8sGuwoMsUfOr5vD7tVw='} 2023-10-12T00:44:36.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0ed4e7493c21470-0065274174-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.014 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:44:36.014 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:44:36.015 DEBUG:boto:Method: HEAD 2023-10-12T00:44:36.015 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:44:36.016 DEBUG:boto:Data: 2023-10-12T00:44:36.016 DEBUG:boto:Headers: {} 2023-10-12T00:44:36.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:36.017 DEBUG:boto:Port: 8000 2023-10-12T00:44:36.017 DEBUG:boto:Params: {} 2023-10-12T00:44:36.018 DEBUG:boto:Token: None 2023-10-12T00:44:36.018 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/ 2023-10-12T00:44:36.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8= 2023-10-12T00:44:36.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8=', 'Content-Length': '0'} 2023-10-12T00:44:36.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf95d1e88bbbba0f-0065274174-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.021 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:44:36.022 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:44:36.022 DEBUG:boto:Method: PUT 2023-10-12T00:44:36.023 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:44:36.023 DEBUG:boto:Data: 2023-10-12T00:44:36.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:36.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:36.025 DEBUG:boto:Port: 8000 2023-10-12T00:44:36.025 DEBUG:boto:Params: {} 2023-10-12T00:44:36.025 DEBUG:boto:Token: None 2023-10-12T00:44:36.026 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/c 2023-10-12T00:44:36.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LCsMQMLkw9Mj2mNcWjvK1Bs4jek= 2023-10-12T00:44:36.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LCsMQMLkw9Mj2mNcWjvK1Bs4jek='} 2023-10-12T00:44:36.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aff101fd2fb1cbca-0065274174-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.071 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:44:36.071 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:44:36.072 DEBUG:boto:Method: HEAD 2023-10-12T00:44:36.072 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:44:36.073 DEBUG:boto:Data: 2023-10-12T00:44:36.073 DEBUG:boto:Headers: {} 2023-10-12T00:44:36.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:36.074 DEBUG:boto:Port: 8000 2023-10-12T00:44:36.074 DEBUG:boto:Params: {} 2023-10-12T00:44:36.075 DEBUG:boto:Token: None 2023-10-12T00:44:36.075 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/ 2023-10-12T00:44:36.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8= 2023-10-12T00:44:36.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x/v1fQn8u3TFroPo4xC0WiERNb8=', 'Content-Length': '0'} 2023-10-12T00:44:36.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004a4c6c2a9574222-0065274174-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.079 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:44:36.079 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:44:36.080 DEBUG:boto:Method: PUT 2023-10-12T00:44:36.080 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:44:36.080 DEBUG:boto:Data: 2023-10-12T00:44:36.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:44:36.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:44:36.082 DEBUG:boto:Port: 8000 2023-10-12T00:44:36.082 DEBUG:boto:Params: {} 2023-10-12T00:44:36.082 DEBUG:boto:Token: None 2023-10-12T00:44:36.083 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:44:36 GMT /wafmoa-55/d 2023-10-12T00:44:36.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YuokiI+MXsHMX/8e1v+KJn6fKqs= 2023-10-12T00:44:36.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:44:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YuokiI+MXsHMX/8e1v+KJn6fKqs='} 2023-10-12T00:44:36.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc87d43040db9a37-0065274174-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:44:36.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:36.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:36.090 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:36.138 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:36.138 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:36.170 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:44:36.182 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:44:36.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.183 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:44:36.183 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:44:36.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.184 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:44:36.184 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:44:36.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.185 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071475.935467_529.1", 2023-10-12T00:44:36.186 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:44:35.935467Z" 2023-10-12T00:44:36.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.187 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:44:36.187 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:44:36.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.187 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:44:36.188 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071475.935467_529.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:36.189 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:44:36.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:36.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:44:36.190 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:44:36.239 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:36.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:36.267+0000 7f2abfac2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:36.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:36.268+0000 7f2abfac2780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:44:36.280 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:44:36.280 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:44:36.280 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:44:36.281 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:44:36.281 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:44:36.281 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:44:36.281 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:36.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.282 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:44:36.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.282 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:44:36.282 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:36.283 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:36.283 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:44:36.283 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:36.283 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:44:36.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:36.284 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:44:36.284 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:36.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.285 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:44:36.285 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:36.285 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:36.285 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:44:36.286 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:36.286 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:36.286 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:36.286 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:44:36.287 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:36.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.288 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:44:36.288 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:36.288 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:36.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071475.935467_529.1", 2023-10-12T00:44:36.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:36.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:36.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:36.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:35.935467Z", 2023-10-12T00:44:36.289 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:36.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:44:36.290 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:44:36.290 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:44:36.291 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:44:36.291 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:44:36.291 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:44:36.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:44:36.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:44:36.292 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:44:36.292 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:44:36.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:44:36.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:44:36.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.293 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:44:36.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:44:36.293 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:44:36.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:44:36.294 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:44:36Z" 2023-10-12T00:44:36.294 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:44:36.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071475.935467_529.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:44:35.935467Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:44:36Z" } 2023-10-12T00:44:36.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071475.935467_529.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:44:35.935467Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:44:36.296 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071475.935467_529.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:36.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071475.935467_529.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:44:36.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:44:36.297 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-55 2023-10-12T00:44:36.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:36.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:44:36.299 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:44:36.356 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:44:36.356 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:44:36.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:36.390+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:44:41.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:41.389+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:44:46.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:46.390+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:44:51.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:51.390+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:44:56.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:44:56.391+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:01.393+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:06.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:06.393+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:11.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:11.394+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:16.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:16.394+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:21.394+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:26.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:26.395+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:31.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:31.395+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:36.395+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:41.397+0000 7fd2026a1780 1 waiting to reach incremental sync.. 2023-10-12T00:45:46.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:46.399+0000 7fd2026a1780 1 bucket sync caught up with source: 2023-10-12T00:45:46.399 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.45.6, , , , 00000000001.36.6, , , 00000000001.63.6, , 00000000001.61.6] 2023-10-12T00:45:46.399 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.45.6, , , , 00000000001.36.6, , , 00000000001.63.6, , 00000000001.61.6] 2023-10-12T00:45:46.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:46.399+0000 7fd2026a1780 0 bucket checkpoint complete 2023-10-12T00:45:46.415 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-55 2023-10-12T00:45:46.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:45:46.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:45:46.417 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:45:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:46.515+0000 7f1ba5893780 1 waiting for incremental sync to catch up: 2023-10-12T00:45:46.515 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.294.6, , , , 00000000001.201.6, , , , , 00000000001.342.6] 2023-10-12T00:45:46.515 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.294.6, , , , 00000000001.201.6, , , 00000000001.344.6, , 00000000001.342.6] 2023-10-12T00:45:51.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:51.516+0000 7f1ba5893780 1 waiting for incremental sync to catch up: 2023-10-12T00:45:51.517 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.294.6, , , , 00000000001.201.6, , , , , 00000000001.342.6] 2023-10-12T00:45:51.517 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.294.6, , , , 00000000001.201.6, , , 00000000001.344.6, , 00000000001.342.6] 2023-10-12T00:45:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:56.516+0000 7f1ba5893780 1 bucket sync caught up with source: 2023-10-12T00:45:56.518 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.294.6, , , , 00000000001.201.6, , , 00000000001.344.6, , 00000000001.342.6] 2023-10-12T00:45:56.518 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.294.6, , , , 00000000001.201.6, , , 00000000001.344.6, , 00000000001.342.6] 2023-10-12T00:45:56.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:56.516+0000 7f1ba5893780 0 bucket checkpoint complete 2023-10-12T00:45:56.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:45:56.535 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.535 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.536 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.536 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:56.537 DEBUG:boto:Data: 2023-10-12T00:45:56.537 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.538 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.539 DEBUG:boto:Params: {} 2023-10-12T00:45:56.539 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:45:56.540 DEBUG:boto:Token: None 2023-10-12T00:45:56.540 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hojq7ZoyoM5C3V9pjLymh8qLp38= 2023-10-12T00:45:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hojq7ZoyoM5C3V9pjLymh8qLp38=', 'Content-Length': '0'} 2023-10-12T00:45:56.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfcae777c51c229b-00652741c4-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.549 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.549 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.550 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.550 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:56.551 DEBUG:boto:Data: 2023-10-12T00:45:56.551 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.552 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.552 DEBUG:boto:Params: {} 2023-10-12T00:45:56.553 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:45:56.553 DEBUG:boto:Token: None 2023-10-12T00:45:56.554 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hojq7ZoyoM5C3V9pjLymh8qLp38= 2023-10-12T00:45:56.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hojq7ZoyoM5C3V9pjLymh8qLp38=', 'Content-Length': '0'} 2023-10-12T00:45:56.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050879fad2deb0834-00652741c4-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:45:56.563 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.564 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.564 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.565 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.565 DEBUG:boto:Method: GET 2023-10-12T00:45:56.565 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.566 DEBUG:boto:Data: 2023-10-12T00:45:56.566 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.567 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.568 DEBUG:boto:Params: {} 2023-10-12T00:45:56.568 DEBUG:boto:Token: None 2023-10-12T00:45:56.569 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/?versions 2023-10-12T00:45:56.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo= 2023-10-12T00:45:56.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo=', 'Content-Length': '0'} 2023-10-12T00:45:56.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30bd3b2aec3441f-00652741c4-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.573 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.574 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:45:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:45:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:45:56.576 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:45:56.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:45:56.577 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.577 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.578 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.578 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.579 DEBUG:boto:Method: GET 2023-10-12T00:45:56.579 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.580 DEBUG:boto:Data: 2023-10-12T00:45:56.580 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.581 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.581 DEBUG:boto:Params: {} 2023-10-12T00:45:56.582 DEBUG:boto:Token: None 2023-10-12T00:45:56.582 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/?versions 2023-10-12T00:45:56.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo= 2023-10-12T00:45:56.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo=', 'Content-Length': '0'} 2023-10-12T00:45:56.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d314068f4328bcfb-00652741c4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.587 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.588 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:45:56.589 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:45:56.589 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:45:56.589 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:45:56.590 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.590 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.591 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.591 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.592 DEBUG:boto:Method: GET 2023-10-12T00:45:56.592 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.593 DEBUG:boto:Data: 2023-10-12T00:45:56.593 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.594 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.595 DEBUG:boto:Params: {} 2023-10-12T00:45:56.595 DEBUG:boto:Token: None 2023-10-12T00:45:56.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/?versions 2023-10-12T00:45:56.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo= 2023-10-12T00:45:56.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo=', 'Content-Length': '0'} 2023-10-12T00:45:56.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004382bad7696aea1-00652741c4-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.602 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.603 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.603 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.604 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.604 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.605 DEBUG:boto:Method: GET 2023-10-12T00:45:56.605 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:45:56.606 DEBUG:boto:Data: 2023-10-12T00:45:56.606 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.607 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.607 DEBUG:boto:Params: {} 2023-10-12T00:45:56.608 DEBUG:boto:Token: None 2023-10-12T00:45:56.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/?versions 2023-10-12T00:45:56.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo= 2023-10-12T00:45:56.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YhVSxohuLSDMiCSiT/2T6veIHGo=', 'Content-Length': '0'} 2023-10-12T00:45:56.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005520dfc76e052415-00652741c4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.614 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:45:56.615 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.616 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.616 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.617 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.617 DEBUG:boto:Method: GET 2023-10-12T00:45:56.618 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.618 DEBUG:boto:Data: 2023-10-12T00:45:56.619 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.619 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.620 DEBUG:boto:Params: {} 2023-10-12T00:45:56.620 DEBUG:boto:Token: None 2023-10-12T00:45:56.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY= 2023-10-12T00:45:56.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084945d353e9763da-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.627 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.628 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.628 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.629 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.629 DEBUG:boto:Method: GET 2023-10-12T00:45:56.630 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.630 DEBUG:boto:Data: 2023-10-12T00:45:56.630 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.631 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.632 DEBUG:boto:Params: {} 2023-10-12T00:45:56.632 DEBUG:boto:Token: None 2023-10-12T00:45:56.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY= 2023-10-12T00:45:56.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0452d02fa1b4b1-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.683 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.684 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.685 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.685 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.685 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.686 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.686 DEBUG:boto:Data: 2023-10-12T00:45:56.687 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.688 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.688 DEBUG:boto:Params: {} 2023-10-12T00:45:56.688 DEBUG:boto:Token: None 2023-10-12T00:45:56.689 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uSk2W3xPdQU+Ma6yZeH29FcY0o= 2023-10-12T00:45:56.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uSk2W3xPdQU+Ma6yZeH29FcY0o=', 'Content-Length': '0'} 2023-10-12T00:45:56.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a97cad79cad20ca8-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.693 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.693 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.694 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.694 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.695 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.695 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.696 DEBUG:boto:Data: 2023-10-12T00:45:56.696 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.697 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.697 DEBUG:boto:Params: {} 2023-10-12T00:45:56.698 DEBUG:boto:Token: None 2023-10-12T00:45:56.698 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uSk2W3xPdQU+Ma6yZeH29FcY0o= 2023-10-12T00:45:56.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uSk2W3xPdQU+Ma6yZeH29FcY0o=', 'Content-Length': '0'} 2023-10-12T00:45:56.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2311790a17cbb6-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:45:56.703 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.704 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.704 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.705 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.705 DEBUG:boto:Method: GET 2023-10-12T00:45:56.706 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.706 DEBUG:boto:Data: 2023-10-12T00:45:56.706 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.707 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.708 DEBUG:boto:Params: {} 2023-10-12T00:45:56.708 DEBUG:boto:Token: None 2023-10-12T00:45:56.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY= 2023-10-12T00:45:56.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5107e434763d0ed-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.713 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.714 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.714 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.714 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:45:56.715 DEBUG:boto:Method: GET 2023-10-12T00:45:56.715 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.716 DEBUG:boto:Data: 2023-10-12T00:45:56.716 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.717 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.718 DEBUG:boto:Params: {} 2023-10-12T00:45:56.718 DEBUG:boto:Token: None 2023-10-12T00:45:56.719 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:45:56.719 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY= 2023-10-12T00:45:56.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z9I7FqhYsmlPI7uNMabhlRyvLVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746e5f02a2877de4-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.725 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.726 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.729 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.729 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.730 DEBUG:boto:Method: GET 2023-10-12T00:45:56.730 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:45:56.731 DEBUG:boto:Data: 2023-10-12T00:45:56.731 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.732 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.732 DEBUG:boto:Params: {} 2023-10-12T00:45:56.733 DEBUG:boto:Token: None 2023-10-12T00:45:56.733 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.734 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af27e4f78bb596e5-00652741c4-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.738 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.739 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.739 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.739 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.740 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.740 DEBUG:boto:Method: GET 2023-10-12T00:45:56.741 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:45:56.741 DEBUG:boto:Data: 2023-10-12T00:45:56.742 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.743 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.743 DEBUG:boto:Params: {} 2023-10-12T00:45:56.744 DEBUG:boto:Token: None 2023-10-12T00:45:56.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a0773b73683d48-00652741c4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.749 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.749 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.750 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.751 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.751 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:45:56.752 DEBUG:boto:Data: 2023-10-12T00:45:56.752 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.754 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.754 DEBUG:boto:Params: {} 2023-10-12T00:45:56.755 DEBUG:boto:Token: None 2023-10-12T00:45:56.756 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a 2023-10-12T00:45:56.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h8j2U59Ubt1rUx+LucEbzfnoIoU= 2023-10-12T00:45:56.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h8j2U59Ubt1rUx+LucEbzfnoIoU=', 'Content-Length': '0'} 2023-10-12T00:45:56.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252aaa2e936ba65c-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.760 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.760 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.761 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.761 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:45:56.762 DEBUG:boto:Data: 2023-10-12T00:45:56.762 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.763 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.764 DEBUG:boto:Params: {} 2023-10-12T00:45:56.764 DEBUG:boto:Token: None 2023-10-12T00:45:56.765 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a 2023-10-12T00:45:56.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h8j2U59Ubt1rUx+LucEbzfnoIoU= 2023-10-12T00:45:56.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h8j2U59Ubt1rUx+LucEbzfnoIoU=', 'Content-Length': '0'} 2023-10-12T00:45:56.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b883989fad26c8b4-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:45:56.769 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.769 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.770 DEBUG:boto:Method: GET 2023-10-12T00:45:56.770 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:45:56.771 DEBUG:boto:Data: 2023-10-12T00:45:56.771 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.772 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.773 DEBUG:boto:Params: {} 2023-10-12T00:45:56.773 DEBUG:boto:Token: None 2023-10-12T00:45:56.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a 2023-10-12T00:45:56.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bV8Mthux9/f3svNg2g1NQy95iUg= 2023-10-12T00:45:56.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bV8Mthux9/f3svNg2g1NQy95iUg=', 'Content-Length': '0'} 2023-10-12T00:45:56.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942003c233300134-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.778 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:45:56.778 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:45:56.779 DEBUG:boto:Method: GET 2023-10-12T00:45:56.780 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:45:56.780 DEBUG:boto:Data: 2023-10-12T00:45:56.781 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.783 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.784 DEBUG:boto:Params: {} 2023-10-12T00:45:56.784 DEBUG:boto:Token: None 2023-10-12T00:45:56.785 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/a 2023-10-12T00:45:56.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bV8Mthux9/f3svNg2g1NQy95iUg= 2023-10-12T00:45:56.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bV8Mthux9/f3svNg2g1NQy95iUg=', 'Content-Length': '0'} 2023-10-12T00:45:56.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca4f64011b343b6-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.789 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.789 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.790 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.790 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.791 DEBUG:boto:Method: GET 2023-10-12T00:45:56.791 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:45:56.791 DEBUG:boto:Data: 2023-10-12T00:45:56.792 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.793 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.793 DEBUG:boto:Params: {} 2023-10-12T00:45:56.794 DEBUG:boto:Token: None 2023-10-12T00:45:56.794 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009467e2d30f1bedc2-00652741c4-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.798 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.799 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.800 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.800 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.800 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:45:56.801 DEBUG:boto:Method: GET 2023-10-12T00:45:56.801 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:45:56.802 DEBUG:boto:Data: 2023-10-12T00:45:56.802 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.803 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.803 DEBUG:boto:Params: {} 2023-10-12T00:45:56.804 DEBUG:boto:Token: None 2023-10-12T00:45:56.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.805 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000364dbf50fb607b69-00652741c4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.809 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:45:56.810 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.811 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.812 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.812 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.813 DEBUG:boto:Method: GET 2023-10-12T00:45:56.813 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.814 DEBUG:boto:Data: 2023-10-12T00:45:56.815 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.815 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.816 DEBUG:boto:Params: {} 2023-10-12T00:45:56.817 DEBUG:boto:Token: None 2023-10-12T00:45:56.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.818 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg= 2023-10-12T00:45:56.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg=', 'Content-Length': '0'} 2023-10-12T00:45:56.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc43bb565474dced-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.821 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.821 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.822 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.822 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.823 DEBUG:boto:Method: GET 2023-10-12T00:45:56.823 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.824 DEBUG:boto:Data: 2023-10-12T00:45:56.825 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.826 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.827 DEBUG:boto:Params: {} 2023-10-12T00:45:56.827 DEBUG:boto:Token: None 2023-10-12T00:45:56.828 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg= 2023-10-12T00:45:56.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg=', 'Content-Length': '0'} 2023-10-12T00:45:56.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6eec634708508f-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.832 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.833 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.833 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.834 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.834 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.835 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.835 DEBUG:boto:Data: 2023-10-12T00:45:56.836 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.836 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.837 DEBUG:boto:Params: {} 2023-10-12T00:45:56.837 DEBUG:boto:Token: None 2023-10-12T00:45:56.838 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EIrgQHJsihnM1WhctpXr4TN99dQ= 2023-10-12T00:45:56.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EIrgQHJsihnM1WhctpXr4TN99dQ=', 'Content-Length': '0'} 2023-10-12T00:45:56.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aaf10c02c2d641b-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.841 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.842 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.842 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.842 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.843 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.843 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.844 DEBUG:boto:Data: 2023-10-12T00:45:56.844 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.845 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.845 DEBUG:boto:Params: {} 2023-10-12T00:45:56.846 DEBUG:boto:Token: None 2023-10-12T00:45:56.846 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EIrgQHJsihnM1WhctpXr4TN99dQ= 2023-10-12T00:45:56.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EIrgQHJsihnM1WhctpXr4TN99dQ=', 'Content-Length': '0'} 2023-10-12T00:45:56.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000876bd8cad51ea07b-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:45:56.850 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.850 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.851 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.851 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.852 DEBUG:boto:Method: GET 2023-10-12T00:45:56.852 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.852 DEBUG:boto:Data: 2023-10-12T00:45:56.853 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.854 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.854 DEBUG:boto:Params: {} 2023-10-12T00:45:56.855 DEBUG:boto:Token: None 2023-10-12T00:45:56.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.855 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg= 2023-10-12T00:45:56.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg=', 'Content-Length': '0'} 2023-10-12T00:45:56.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072a60f6d1de6fa59-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.859 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.859 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.859 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.860 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:45:56.860 DEBUG:boto:Method: GET 2023-10-12T00:45:56.861 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.861 DEBUG:boto:Data: 2023-10-12T00:45:56.862 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.862 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.863 DEBUG:boto:Params: {} 2023-10-12T00:45:56.863 DEBUG:boto:Token: None 2023-10-12T00:45:56.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:45:56.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg= 2023-10-12T00:45:56.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Woa4XMulCt5tNtIWvVEc1t4WMPg=', 'Content-Length': '0'} 2023-10-12T00:45:56.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be81bd8175896d4-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.868 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.868 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.868 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:45:56.869 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:45:56.869 DEBUG:boto:Method: GET 2023-10-12T00:45:56.870 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:45:56.870 DEBUG:boto:Data: 2023-10-12T00:45:56.871 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.871 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.872 DEBUG:boto:Params: {} 2023-10-12T00:45:56.872 DEBUG:boto:Token: None 2023-10-12T00:45:56.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5079ef53a8e5a58-00652741c4-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.877 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.877 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:56.878 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:56.878 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:45:56.879 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:45:56.879 DEBUG:boto:Method: GET 2023-10-12T00:45:56.879 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:45:56.880 DEBUG:boto:Data: 2023-10-12T00:45:56.880 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.881 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.881 DEBUG:boto:Params: {} 2023-10-12T00:45:56.882 DEBUG:boto:Token: None 2023-10-12T00:45:56.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/ 2023-10-12T00:45:56.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY= 2023-10-12T00:45:56.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zRMKyl4UhPRphShGBwwvzlZHiVY=', 'Content-Length': '0'} 2023-10-12T00:45:56.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee20bbe2959efd5d-00652741c4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.930 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:56.932 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.933 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.933 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.934 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:45:56.935 DEBUG:boto:Data: 2023-10-12T00:45:56.935 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.937 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.938 DEBUG:boto:Params: {} 2023-10-12T00:45:56.939 DEBUG:boto:Token: None 2023-10-12T00:45:56.939 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b 2023-10-12T00:45:56.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+7Mf3Ew5zEmpbD+T4EwsUlxkvGY= 2023-10-12T00:45:56.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+7Mf3Ew5zEmpbD+T4EwsUlxkvGY=', 'Content-Length': '0'} 2023-10-12T00:45:56.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233920adea3ce213-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.945 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.945 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.946 DEBUG:boto:Method: HEAD 2023-10-12T00:45:56.947 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:45:56.947 DEBUG:boto:Data: 2023-10-12T00:45:56.948 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:56.949 DEBUG:boto:Port: 8000 2023-10-12T00:45:56.949 DEBUG:boto:Params: {} 2023-10-12T00:45:56.950 DEBUG:boto:Token: None 2023-10-12T00:45:56.951 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b 2023-10-12T00:45:56.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+7Mf3Ew5zEmpbD+T4EwsUlxkvGY= 2023-10-12T00:45:56.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+7Mf3Ew5zEmpbD+T4EwsUlxkvGY=', 'Content-Length': '0'} 2023-10-12T00:45:56.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029448cda81e53449-00652741c4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:56.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:45:56.955 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:56.955 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:56.956 DEBUG:boto:Method: GET 2023-10-12T00:45:56.956 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:45:56.956 DEBUG:boto:Data: 2023-10-12T00:45:56.957 DEBUG:boto:Headers: {} 2023-10-12T00:45:56.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:56.958 DEBUG:boto:Port: 8001 2023-10-12T00:45:56.958 DEBUG:boto:Params: {} 2023-10-12T00:45:56.959 DEBUG:boto:Token: None 2023-10-12T00:45:56.959 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:56 GMT /wafmoa-55/b 2023-10-12T00:45:56.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Wa/udb5VCIUODd1oLqCg4HniSU= 2023-10-12T00:45:56.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Wa/udb5VCIUODd1oLqCg4HniSU=', 'Content-Length': '0'} 2023-10-12T00:45:56.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015a48e21764fa5d-00652741c4-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.002 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:45:57.003 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:45:57.003 DEBUG:boto:Method: GET 2023-10-12T00:45:57.004 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:45:57.005 DEBUG:boto:Data: 2023-10-12T00:45:57.005 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.006 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.006 DEBUG:boto:Params: {} 2023-10-12T00:45:57.007 DEBUG:boto:Token: None 2023-10-12T00:45:57.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/b 2023-10-12T00:45:57.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:16Q1qQuLlp7zqi0FgYOxKQXdZxg= 2023-10-12T00:45:57.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:16Q1qQuLlp7zqi0FgYOxKQXdZxg=', 'Content-Length': '0'} 2023-10-12T00:45:57.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d675d00b4787724b-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.011 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.012 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.012 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:45:57.013 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:45:57.013 DEBUG:boto:Method: GET 2023-10-12T00:45:57.014 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:45:57.014 DEBUG:boto:Data: 2023-10-12T00:45:57.015 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.016 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.016 DEBUG:boto:Params: {} 2023-10-12T00:45:57.017 DEBUG:boto:Token: None 2023-10-12T00:45:57.017 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3ca2217c81621d-00652741c5-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.021 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.022 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.023 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.023 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:45:57.024 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:45:57.024 DEBUG:boto:Method: GET 2023-10-12T00:45:57.024 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:45:57.025 DEBUG:boto:Data: 2023-10-12T00:45:57.025 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.026 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.027 DEBUG:boto:Params: {} 2023-10-12T00:45:57.027 DEBUG:boto:Token: None 2023-10-12T00:45:57.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097612e047ca6019e-00652741c5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.032 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:45:57.033 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.034 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.034 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.035 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.035 DEBUG:boto:Method: GET 2023-10-12T00:45:57.036 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.036 DEBUG:boto:Data: 2023-10-12T00:45:57.036 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.037 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.038 DEBUG:boto:Params: {} 2023-10-12T00:45:57.038 DEBUG:boto:Token: None 2023-10-12T00:45:57.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI= 2023-10-12T00:45:57.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI=', 'Content-Length': '0'} 2023-10-12T00:45:57.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ae355c9faa1919-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.043 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.043 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.044 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.045 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.045 DEBUG:boto:Method: GET 2023-10-12T00:45:57.045 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.046 DEBUG:boto:Data: 2023-10-12T00:45:57.047 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.047 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.048 DEBUG:boto:Params: {} 2023-10-12T00:45:57.048 DEBUG:boto:Token: None 2023-10-12T00:45:57.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI= 2023-10-12T00:45:57.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI=', 'Content-Length': '0'} 2023-10-12T00:45:57.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870556a342e2ca27-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.094 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.095 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.095 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.096 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.096 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.097 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.097 DEBUG:boto:Data: 2023-10-12T00:45:57.098 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.098 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.099 DEBUG:boto:Params: {} 2023-10-12T00:45:57.099 DEBUG:boto:Token: None 2023-10-12T00:45:57.100 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R8/6NcVCLlBn57VIhfmuGmcElos= 2023-10-12T00:45:57.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R8/6NcVCLlBn57VIhfmuGmcElos=', 'Content-Length': '0'} 2023-10-12T00:45:57.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a786bec26093231a-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.103 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.104 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.104 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.105 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.105 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.106 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.106 DEBUG:boto:Data: 2023-10-12T00:45:57.106 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.107 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.108 DEBUG:boto:Params: {} 2023-10-12T00:45:57.108 DEBUG:boto:Token: None 2023-10-12T00:45:57.109 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.109 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R8/6NcVCLlBn57VIhfmuGmcElos= 2023-10-12T00:45:57.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R8/6NcVCLlBn57VIhfmuGmcElos=', 'Content-Length': '0'} 2023-10-12T00:45:57.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57cc04a4c7982fa-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:45:57.113 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.114 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.114 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.114 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.115 DEBUG:boto:Method: GET 2023-10-12T00:45:57.115 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.116 DEBUG:boto:Data: 2023-10-12T00:45:57.116 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.117 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.118 DEBUG:boto:Params: {} 2023-10-12T00:45:57.118 DEBUG:boto:Token: None 2023-10-12T00:45:57.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI= 2023-10-12T00:45:57.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI=', 'Content-Length': '0'} 2023-10-12T00:45:57.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81c3a598a086a97-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.122 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.123 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.123 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.124 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:45:57.124 DEBUG:boto:Method: GET 2023-10-12T00:45:57.124 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.125 DEBUG:boto:Data: 2023-10-12T00:45:57.125 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.126 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.127 DEBUG:boto:Params: {} 2023-10-12T00:45:57.127 DEBUG:boto:Token: None 2023-10-12T00:45:57.128 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:45:57.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI= 2023-10-12T00:45:57.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wre9qOGS/uR3d5HL6uwyKgurwlI=', 'Content-Length': '0'} 2023-10-12T00:45:57.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3cf9fce98f75f1-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.131 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.132 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.132 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.133 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.133 DEBUG:boto:Method: GET 2023-10-12T00:45:57.133 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:45:57.134 DEBUG:boto:Data: 2023-10-12T00:45:57.134 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.135 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.136 DEBUG:boto:Params: {} 2023-10-12T00:45:57.136 DEBUG:boto:Token: None 2023-10-12T00:45:57.137 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef81acca16eb1832-00652741c5-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.141 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.142 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.142 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.143 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.143 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.143 DEBUG:boto:Method: GET 2023-10-12T00:45:57.144 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:45:57.144 DEBUG:boto:Data: 2023-10-12T00:45:57.145 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.146 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.146 DEBUG:boto:Params: {} 2023-10-12T00:45:57.147 DEBUG:boto:Token: None 2023-10-12T00:45:57.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed7640a827486e45-00652741c5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.151 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.152 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.152 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.153 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.153 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:45:57.154 DEBUG:boto:Data: 2023-10-12T00:45:57.154 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.155 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.155 DEBUG:boto:Params: {} 2023-10-12T00:45:57.156 DEBUG:boto:Token: None 2023-10-12T00:45:57.156 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c 2023-10-12T00:45:57.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u5ZZsrEiyfPtlPH/ZHipHlxUWKc= 2023-10-12T00:45:57.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u5ZZsrEiyfPtlPH/ZHipHlxUWKc=', 'Content-Length': '0'} 2023-10-12T00:45:57.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000498be27fd75b5ed-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.160 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.160 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.161 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.161 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:45:57.161 DEBUG:boto:Data: 2023-10-12T00:45:57.162 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.163 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.163 DEBUG:boto:Params: {} 2023-10-12T00:45:57.164 DEBUG:boto:Token: None 2023-10-12T00:45:57.164 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c 2023-10-12T00:45:57.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u5ZZsrEiyfPtlPH/ZHipHlxUWKc= 2023-10-12T00:45:57.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u5ZZsrEiyfPtlPH/ZHipHlxUWKc=', 'Content-Length': '0'} 2023-10-12T00:45:57.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45f483f05867920-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:45:57.168 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.168 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.169 DEBUG:boto:Method: GET 2023-10-12T00:45:57.169 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:45:57.170 DEBUG:boto:Data: 2023-10-12T00:45:57.170 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.171 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.171 DEBUG:boto:Params: {} 2023-10-12T00:45:57.172 DEBUG:boto:Token: None 2023-10-12T00:45:57.172 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c 2023-10-12T00:45:57.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w0CEz6n/qYu33vMGZhwl93tdkYo= 2023-10-12T00:45:57.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w0CEz6n/qYu33vMGZhwl93tdkYo=', 'Content-Length': '0'} 2023-10-12T00:45:57.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd04a060016c580-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.218 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:45:57.219 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:45:57.220 DEBUG:boto:Method: GET 2023-10-12T00:45:57.220 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:45:57.220 DEBUG:boto:Data: 2023-10-12T00:45:57.221 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.222 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.222 DEBUG:boto:Params: {} 2023-10-12T00:45:57.223 DEBUG:boto:Token: None 2023-10-12T00:45:57.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/c 2023-10-12T00:45:57.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w0CEz6n/qYu33vMGZhwl93tdkYo= 2023-10-12T00:45:57.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w0CEz6n/qYu33vMGZhwl93tdkYo=', 'Content-Length': '0'} 2023-10-12T00:45:57.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d7bd279e2bf7f9-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.227 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.228 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.228 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.229 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.229 DEBUG:boto:Method: GET 2023-10-12T00:45:57.230 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:45:57.230 DEBUG:boto:Data: 2023-10-12T00:45:57.231 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.231 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.232 DEBUG:boto:Params: {} 2023-10-12T00:45:57.232 DEBUG:boto:Token: None 2023-10-12T00:45:57.233 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037192d99cf9db854-00652741c5-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.237 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.238 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.238 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.239 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.239 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:45:57.240 DEBUG:boto:Method: GET 2023-10-12T00:45:57.240 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:45:57.240 DEBUG:boto:Data: 2023-10-12T00:45:57.241 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.242 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.242 DEBUG:boto:Params: {} 2023-10-12T00:45:57.243 DEBUG:boto:Token: None 2023-10-12T00:45:57.243 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed026b58bfcd177-00652741c5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.247 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:45:57.249 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.249 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.249 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.250 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.250 DEBUG:boto:Method: GET 2023-10-12T00:45:57.251 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.251 DEBUG:boto:Data: 2023-10-12T00:45:57.252 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.253 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.253 DEBUG:boto:Params: {} 2023-10-12T00:45:57.254 DEBUG:boto:Token: None 2023-10-12T00:45:57.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs= 2023-10-12T00:45:57.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs=', 'Content-Length': '0'} 2023-10-12T00:45:57.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d0cfcf4ab3956e-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.258 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.258 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.259 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.259 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.260 DEBUG:boto:Method: GET 2023-10-12T00:45:57.260 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.260 DEBUG:boto:Data: 2023-10-12T00:45:57.261 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.262 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.262 DEBUG:boto:Params: {} 2023-10-12T00:45:57.263 DEBUG:boto:Token: None 2023-10-12T00:45:57.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs= 2023-10-12T00:45:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs=', 'Content-Length': '0'} 2023-10-12T00:45:57.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d410be37de4a2fb-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.267 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.267 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.268 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.268 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.269 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.269 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.270 DEBUG:boto:Data: 2023-10-12T00:45:57.270 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.271 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.271 DEBUG:boto:Params: {} 2023-10-12T00:45:57.272 DEBUG:boto:Token: None 2023-10-12T00:45:57.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lB/PNVH2WUQWYodWM7DWnVvyXcA= 2023-10-12T00:45:57.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lB/PNVH2WUQWYodWM7DWnVvyXcA=', 'Content-Length': '0'} 2023-10-12T00:45:57.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953391b8047babc5-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.276 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.276 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.277 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.277 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.278 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.278 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.278 DEBUG:boto:Data: 2023-10-12T00:45:57.279 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.280 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.280 DEBUG:boto:Params: {} 2023-10-12T00:45:57.280 DEBUG:boto:Token: None 2023-10-12T00:45:57.281 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lB/PNVH2WUQWYodWM7DWnVvyXcA= 2023-10-12T00:45:57.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lB/PNVH2WUQWYodWM7DWnVvyXcA=', 'Content-Length': '0'} 2023-10-12T00:45:57.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e47066838f5772-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:45:57.285 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.286 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.286 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.287 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.287 DEBUG:boto:Method: GET 2023-10-12T00:45:57.287 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.288 DEBUG:boto:Data: 2023-10-12T00:45:57.288 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.289 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.290 DEBUG:boto:Params: {} 2023-10-12T00:45:57.290 DEBUG:boto:Token: None 2023-10-12T00:45:57.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs= 2023-10-12T00:45:57.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs=', 'Content-Length': '0'} 2023-10-12T00:45:57.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a249e3acbc2178-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.294 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.295 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.295 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.296 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:45:57.296 DEBUG:boto:Method: GET 2023-10-12T00:45:57.297 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.297 DEBUG:boto:Data: 2023-10-12T00:45:57.298 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.298 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.299 DEBUG:boto:Params: {} 2023-10-12T00:45:57.299 DEBUG:boto:Token: None 2023-10-12T00:45:57.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:45:57.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs= 2023-10-12T00:45:57.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:onbQyiAaycRIFAVvsx9oBbHp9Vs=', 'Content-Length': '0'} 2023-10-12T00:45:57.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64a9a763e9dbfde-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.347 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.347 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.348 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.348 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.349 DEBUG:boto:Method: GET 2023-10-12T00:45:57.349 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:45:57.349 DEBUG:boto:Data: 2023-10-12T00:45:57.350 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.351 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.351 DEBUG:boto:Params: {} 2023-10-12T00:45:57.352 DEBUG:boto:Token: None 2023-10-12T00:45:57.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f42c3437370e93-00652741c5-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.357 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.357 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.358 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.358 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.359 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.359 DEBUG:boto:Method: GET 2023-10-12T00:45:57.360 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:45:57.360 DEBUG:boto:Data: 2023-10-12T00:45:57.361 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.361 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.362 DEBUG:boto:Params: {} 2023-10-12T00:45:57.362 DEBUG:boto:Token: None 2023-10-12T00:45:57.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e75ebb645b5bc8-00652741c5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.367 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.368 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.368 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.369 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.369 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:45:57.369 DEBUG:boto:Data: 2023-10-12T00:45:57.370 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.371 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.371 DEBUG:boto:Params: {} 2023-10-12T00:45:57.372 DEBUG:boto:Token: None 2023-10-12T00:45:57.372 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d 2023-10-12T00:45:57.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8HszlQ9qnnX1Y/kxavAjskmsqNc= 2023-10-12T00:45:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8HszlQ9qnnX1Y/kxavAjskmsqNc=', 'Content-Length': '0'} 2023-10-12T00:45:57.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d93f72a24424ce7-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.376 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.376 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.376 DEBUG:boto:Method: HEAD 2023-10-12T00:45:57.377 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:45:57.377 DEBUG:boto:Data: 2023-10-12T00:45:57.378 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.378 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.379 DEBUG:boto:Params: {} 2023-10-12T00:45:57.379 DEBUG:boto:Token: None 2023-10-12T00:45:57.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d 2023-10-12T00:45:57.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8HszlQ9qnnX1Y/kxavAjskmsqNc= 2023-10-12T00:45:57.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8HszlQ9qnnX1Y/kxavAjskmsqNc=', 'Content-Length': '0'} 2023-10-12T00:45:57.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ba87b719604819-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:45:57.384 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.384 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.385 DEBUG:boto:Method: GET 2023-10-12T00:45:57.385 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:45:57.386 DEBUG:boto:Data: 2023-10-12T00:45:57.386 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.387 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.387 DEBUG:boto:Params: {} 2023-10-12T00:45:57.388 DEBUG:boto:Token: None 2023-10-12T00:45:57.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d 2023-10-12T00:45:57.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WS66yGmkNn+UgnFnVoNjdW07Yaw= 2023-10-12T00:45:57.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WS66yGmkNn+UgnFnVoNjdW07Yaw=', 'Content-Length': '0'} 2023-10-12T00:45:57.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed83de3b22e019b-00652741c5-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.434 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:45:57.435 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:45:57.435 DEBUG:boto:Method: GET 2023-10-12T00:45:57.436 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:45:57.436 DEBUG:boto:Data: 2023-10-12T00:45:57.437 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.438 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.438 DEBUG:boto:Params: {} 2023-10-12T00:45:57.439 DEBUG:boto:Token: None 2023-10-12T00:45:57.439 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/d 2023-10-12T00:45:57.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WS66yGmkNn+UgnFnVoNjdW07Yaw= 2023-10-12T00:45:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WS66yGmkNn+UgnFnVoNjdW07Yaw=', 'Content-Length': '0'} 2023-10-12T00:45:57.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e506716f7fec95c9-00652741c5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.443 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.443 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.444 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.444 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.445 DEBUG:boto:Method: GET 2023-10-12T00:45:57.445 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:45:57.445 DEBUG:boto:Data: 2023-10-12T00:45:57.446 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:45:57.447 DEBUG:boto:Port: 8001 2023-10-12T00:45:57.447 DEBUG:boto:Params: {} 2023-10-12T00:45:57.448 DEBUG:boto:Token: None 2023-10-12T00:45:57.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428e46fb0b74406b-00652741c5-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.453 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.453 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:57.454 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:57.454 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.455 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:45:57.455 DEBUG:boto:Method: GET 2023-10-12T00:45:57.456 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:45:57.456 DEBUG:boto:Data: 2023-10-12T00:45:57.456 DEBUG:boto:Headers: {} 2023-10-12T00:45:57.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:57.457 DEBUG:boto:Port: 8000 2023-10-12T00:45:57.458 DEBUG:boto:Params: {} 2023-10-12T00:45:57.458 DEBUG:boto:Token: None 2023-10-12T00:45:57.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:45:57 GMT /wafmoa-55/ 2023-10-12T00:45:57.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144= 2023-10-12T00:45:57.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MMEfG4x8SNsHJRqT6ih7Oji9144=', 'Content-Length': '0'} 2023-10-12T00:45:57.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058fb2fe1421d9d8d-00652741c5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:45:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:57.502 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:45:57.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:45:57.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.505 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:45:57.554 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:57.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:57.596 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:45:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-10-12T00:45:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-10-12T00:45:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:45:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-10-12T00:45:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-10-12T00:45:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-10-12T00:45:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:45:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-10-12T00:45:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:45:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:45:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:45:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:45:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-10-12T00:45:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:45:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.602 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.603 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:45:57.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:45:57.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:45:57.605 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:45:57.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:57.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:57.696 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:45:57.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.696 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "1#00000000001.45.6", 2023-10-12T00:45:57.696 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.135008379877346210", 2023-10-12T00:45:57.697 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:45:57.697 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "a", 2023-10-12T00:45:57.697 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:45:57.697 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:45:57.698 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:45:57.698 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:35.953880370Z", 2023-10-12T00:45:57.698 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:45:57.698 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:45:57.698 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:45:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:45:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:45:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:45:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:45:57.700 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:45:57.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.700 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:45:57.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.700 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.701 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "5#00000000001.36.6", 2023-10-12T00:45:57.701 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15890903113911736887", 2023-10-12T00:45:57.701 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:45:57.702 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "d", 2023-10-12T00:45:57.702 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:45:57.702 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:45:57.702 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:45:57.702 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:36.086552547Z", 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:45:57.703 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:45:57.704 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:45:57.704 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:45:57.704 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:45:57.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.705 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:45:57.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.706 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "8#00000000001.63.6", 2023-10-12T00:45:57.706 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12677916612963191754", 2023-10-12T00:45:57.706 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:45:57.706 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "c", 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:36.029895689Z", 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:45:57.707 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:45:57.708 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:45:57.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.708 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:45:57.708 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:45:57.708 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:45:57.709 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:45:57.709 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:45:57.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.709 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:45:57.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.710 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "10#00000000001.61.6", 2023-10-12T00:45:57.710 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17360618401185272944", 2023-10-12T00:45:57.711 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:45:57.711 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "b", 2023-10-12T00:45:57.711 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:45:57.711 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:45:57.712 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-10-12T00:45:57.712 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:44:36.011557039Z", 2023-10-12T00:45:57.712 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:45:57.712 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:45:57.712 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:45:57.713 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:45:57.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.714 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:45:57.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.715 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:45:57.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'wafmoa-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'wafmoa-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.717 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket wafmoa-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:45:57.765 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:57.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:57.802 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:45:57.802 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-55 2023-10-12T00:45:57.806 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-10-12T00:45:57.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:45:57.842 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:45:57.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:57.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:57.928 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:45:57.928 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.929 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-10-12T00:45:57.929 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-10-12T00:45:57.929 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:45:57.929 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-10-12T00:45:57.930 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-10-12T00:45:57.930 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:45:57.930 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:45:57.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.931 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-10-12T00:45:57.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.932 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:45:57.932 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:45:57.932 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:45:57.933 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:45:57.933 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.933 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-10-12T00:45:57.933 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:45:57.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:45:57.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:45:57.935 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:45:57.935 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:45:57.935 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:45:57.936 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:45:57.936 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:45:57.936 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:45:57.936 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:45:57.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:45:57.937 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:45:57.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:45:57.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:45:57.939 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:45:57.990 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:57.990 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:58.033 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-10-12T00:45:58.035 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:58.036 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:58.036 DEBUG:boto:Method: HEAD 2023-10-12T00:45:58.037 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:58.037 DEBUG:boto:Data: 2023-10-12T00:45:58.037 DEBUG:boto:Headers: {} 2023-10-12T00:45:58.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.038 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.039 DEBUG:boto:Params: {} 2023-10-12T00:45:58.039 DEBUG:boto:Token: None 2023-10-12T00:45:58.040 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/ 2023-10-12T00:45:58.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ= 2023-10-12T00:45:58.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ=', 'Content-Length': '0'} 2023-10-12T00:45:58.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e6c92dd9a8bd818-00652741c6-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.043 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:45:58.044 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:45:58.044 DEBUG:boto:Method: PUT 2023-10-12T00:45:58.044 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:45:58.045 DEBUG:boto:Data: 2023-10-12T00:45:58.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:45:58.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.046 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.047 DEBUG:boto:Params: {} 2023-10-12T00:45:58.047 DEBUG:boto:Token: None 2023-10-12T00:45:58.047 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/e 2023-10-12T00:45:58.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CNRSRwfaDiqxPhX9GMb8wr2KyeY= 2023-10-12T00:45:58.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CNRSRwfaDiqxPhX9GMb8wr2KyeY='} 2023-10-12T00:45:58.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f50cfb287a514ed-00652741c6-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.054 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:58.054 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:58.055 DEBUG:boto:Method: HEAD 2023-10-12T00:45:58.055 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:58.056 DEBUG:boto:Data: 2023-10-12T00:45:58.056 DEBUG:boto:Headers: {} 2023-10-12T00:45:58.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.057 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.057 DEBUG:boto:Params: {} 2023-10-12T00:45:58.058 DEBUG:boto:Token: None 2023-10-12T00:45:58.058 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/ 2023-10-12T00:45:58.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ= 2023-10-12T00:45:58.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ=', 'Content-Length': '0'} 2023-10-12T00:45:58.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a421b462b11b4ef-00652741c6-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.061 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:45:58.062 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:45:58.062 DEBUG:boto:Method: PUT 2023-10-12T00:45:58.062 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:45:58.063 DEBUG:boto:Data: 2023-10-12T00:45:58.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:45:58.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.064 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.064 DEBUG:boto:Params: {} 2023-10-12T00:45:58.065 DEBUG:boto:Token: None 2023-10-12T00:45:58.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/f 2023-10-12T00:45:58.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/MwE3KyIGqo7T4E/pulLuTcwX0= 2023-10-12T00:45:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/MwE3KyIGqo7T4E/pulLuTcwX0='} 2023-10-12T00:45:58.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030c43085fd8b4e0d-00652741c6-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.071 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:58.072 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:58.072 DEBUG:boto:Method: HEAD 2023-10-12T00:45:58.072 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:58.073 DEBUG:boto:Data: 2023-10-12T00:45:58.073 DEBUG:boto:Headers: {} 2023-10-12T00:45:58.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.074 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.074 DEBUG:boto:Params: {} 2023-10-12T00:45:58.075 DEBUG:boto:Token: None 2023-10-12T00:45:58.075 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/ 2023-10-12T00:45:58.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ= 2023-10-12T00:45:58.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ=', 'Content-Length': '0'} 2023-10-12T00:45:58.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005500830ad535dff7-00652741c6-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.078 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:45:58.079 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:45:58.079 DEBUG:boto:Method: PUT 2023-10-12T00:45:58.079 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:45:58.080 DEBUG:boto:Data: 2023-10-12T00:45:58.080 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:45:58.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.081 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.081 DEBUG:boto:Params: {} 2023-10-12T00:45:58.082 DEBUG:boto:Token: None 2023-10-12T00:45:58.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/g 2023-10-12T00:45:58.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J3hryhqR7MndTm+Oo0UycLGeliY= 2023-10-12T00:45:58.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J3hryhqR7MndTm+Oo0UycLGeliY='} 2023-10-12T00:45:58.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3146bd497179d64-00652741c6-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.127 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:45:58.127 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:45:58.128 DEBUG:boto:Method: HEAD 2023-10-12T00:45:58.128 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:45:58.128 DEBUG:boto:Data: 2023-10-12T00:45:58.129 DEBUG:boto:Headers: {} 2023-10-12T00:45:58.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.130 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.130 DEBUG:boto:Params: {} 2023-10-12T00:45:58.131 DEBUG:boto:Token: None 2023-10-12T00:45:58.131 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/ 2023-10-12T00:45:58.132 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ= 2023-10-12T00:45:58.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QaXCYla2snWpGTpP5RxzBwSWyuQ=', 'Content-Length': '0'} 2023-10-12T00:45:58.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004caa9a76f65ce0a2-00652741c6-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.134 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:45:58.135 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:45:58.135 DEBUG:boto:Method: PUT 2023-10-12T00:45:58.136 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:45:58.136 DEBUG:boto:Data: 2023-10-12T00:45:58.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:45:58.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:45:58.138 DEBUG:boto:Port: 8000 2023-10-12T00:45:58.138 DEBUG:boto:Params: {} 2023-10-12T00:45:58.139 DEBUG:boto:Token: None 2023-10-12T00:45:58.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:45:58 GMT /wafmoa-55/h 2023-10-12T00:45:58.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jPzgxpm6IS+ZJJ2RHNwL6b27VR0= 2023-10-12T00:45:58.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:45:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jPzgxpm6IS+ZJJ2RHNwL6b27VR0='} 2023-10-12T00:45:58.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5e60936cb399d43-00652741c6-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:45:58.146 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-55 2023-10-12T00:45:58.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:45:58.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:45:58.148 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:45:58.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:45:58.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:45:58.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:45:58.240+0000 7f08527d9780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:46:03.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:03.240+0000 7f08527d9780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:46:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:08.241+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:08.243 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:46:08.243 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:13.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:13.242+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:13.244 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:46:13.244 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:18.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:18.244+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:18.246 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:46:18.246 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:23.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:23.245+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:23.245 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:46:23.245 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:28.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:28.245+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:28.246 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, , ] 2023-10-12T00:46:28.246 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:33.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:33.245+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:33.247 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, , ] 2023-10-12T00:46:33.247 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:38.246+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:38.247 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, , ] 2023-10-12T00:46:38.247 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:43.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:43.247+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:43.248 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, , ] 2023-10-12T00:46:43.248 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:48.248+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:48.249 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:46:48.249 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:53.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:53.249+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:53.250 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:46:53.250 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:46:58.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:46:58.251+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:46:58.251 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:46:58.251 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:03.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:03.251+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:03.252 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:03.252 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:08.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:08.252+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:08.252 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:08.252 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:13.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:13.252+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:13.253 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:13.254 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:18.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:18.253+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:18.254 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:18.254 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:23.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:23.254+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:23.255 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:23.255 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:28.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:28.254+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:28.256 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:28.256 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:33.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:33.256+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:33.256 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:33.256 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:38.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:38.257+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:38.257 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:38.257 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:43.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:43.258+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:43.258 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:43.258 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:48.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:48.258+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:48.259 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:48.259 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:53.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:53.258+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:53.260 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:53.260 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:47:58.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:47:58.259+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:47:58.261 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:47:58.261 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:48:03.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:03.260+0000 7f08527d9780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:03.261 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, ] 2023-10-12T00:48:03.262 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:48:08.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:08.261+0000 7f08527d9780 1 bucket sync caught up with source: 2023-10-12T00:48:08.262 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:48:08.262 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.51.6, 00000000002.41.6, 00000000002.40.6] 2023-10-12T00:48:08.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:08.261+0000 7f08527d9780 0 bucket checkpoint complete 2023-10-12T00:48:08.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-55 2023-10-12T00:48:08.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:08.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:08.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:48:08.345 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:08.345 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:08.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:08.383+0000 7fd7f657f780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:08.384 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000001.344.6, , 00000000002.350.6] 2023-10-12T00:48:08.385 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:13.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:13.385+0000 7fd7f657f780 1 bucket sync caught up with source: 2023-10-12T00:48:13.386 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:13.386 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:13.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:13.385+0000 7fd7f657f780 0 bucket checkpoint complete 2023-10-12T00:48:13.400 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:48:13.401 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.402 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.402 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.403 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:13.403 DEBUG:boto:Data: 2023-10-12T00:48:13.404 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.404 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.405 DEBUG:boto:Params: {} 2023-10-12T00:48:13.406 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:48:13.406 DEBUG:boto:Token: None 2023-10-12T00:48:13.407 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XMZblr99XDnXMaQU4kaFlCOzHiY= 2023-10-12T00:48:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XMZblr99XDnXMaQU4kaFlCOzHiY=', 'Content-Length': '0'} 2023-10-12T00:48:13.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ee9ce5f94322fb4-006527424d-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.416 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.417 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.418 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.418 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:13.419 DEBUG:boto:Data: 2023-10-12T00:48:13.420 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.421 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.421 DEBUG:boto:Params: {} 2023-10-12T00:48:13.422 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:48:13.427 DEBUG:boto:Token: None 2023-10-12T00:48:13.427 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XMZblr99XDnXMaQU4kaFlCOzHiY= 2023-10-12T00:48:13.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XMZblr99XDnXMaQU4kaFlCOzHiY=', 'Content-Length': '0'} 2023-10-12T00:48:13.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096e4298adf38897f-006527424d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:48:13.435 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.436 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.436 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.437 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.437 DEBUG:boto:Method: GET 2023-10-12T00:48:13.437 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.438 DEBUG:boto:Data: 2023-10-12T00:48:13.438 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.439 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.439 DEBUG:boto:Params: {} 2023-10-12T00:48:13.440 DEBUG:boto:Token: None 2023-10-12T00:48:13.440 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/?versions 2023-10-12T00:48:13.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs= 2023-10-12T00:48:13.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs=', 'Content-Length': '0'} 2023-10-12T00:48:13.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008feec6404a7aaf53-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.444 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.445 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:13.446 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:13.446 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:13.447 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:13.447 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:13.447 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:13.448 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:13.448 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:13.449 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:48:13.449 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.450 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.450 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.451 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.451 DEBUG:boto:Method: GET 2023-10-12T00:48:13.452 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.452 DEBUG:boto:Data: 2023-10-12T00:48:13.452 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.453 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.454 DEBUG:boto:Params: {} 2023-10-12T00:48:13.455 DEBUG:boto:Token: None 2023-10-12T00:48:13.455 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/?versions 2023-10-12T00:48:13.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs= 2023-10-12T00:48:13.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs=', 'Content-Length': '0'} 2023-10-12T00:48:13.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a01d161caabbab29-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.459 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.460 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:13.460 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:13.460 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:13.461 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:13.461 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:13.462 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:13.462 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:13.463 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:13.463 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.464 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.464 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.465 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.465 DEBUG:boto:Method: GET 2023-10-12T00:48:13.465 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.466 DEBUG:boto:Data: 2023-10-12T00:48:13.466 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.467 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.467 DEBUG:boto:Params: {} 2023-10-12T00:48:13.468 DEBUG:boto:Token: None 2023-10-12T00:48:13.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/?versions 2023-10-12T00:48:13.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs= 2023-10-12T00:48:13.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs=', 'Content-Length': '0'} 2023-10-12T00:48:13.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002045f61521268578-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.472 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.473 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.474 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.474 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.475 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.475 DEBUG:boto:Method: GET 2023-10-12T00:48:13.475 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:13.476 DEBUG:boto:Data: 2023-10-12T00:48:13.476 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.477 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.477 DEBUG:boto:Params: {} 2023-10-12T00:48:13.478 DEBUG:boto:Token: None 2023-10-12T00:48:13.478 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/?versions 2023-10-12T00:48:13.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs= 2023-10-12T00:48:13.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Izne1i7Rtkck+7CPMsXjPmVkJOs=', 'Content-Length': '0'} 2023-10-12T00:48:13.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce1648d6f9e4855-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.482 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:13.483 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.484 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.484 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.485 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.485 DEBUG:boto:Method: GET 2023-10-12T00:48:13.486 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.486 DEBUG:boto:Data: 2023-10-12T00:48:13.486 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.487 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.488 DEBUG:boto:Params: {} 2023-10-12T00:48:13.488 DEBUG:boto:Token: None 2023-10-12T00:48:13.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ= 2023-10-12T00:48:13.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5fbbba82bcee430-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.492 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.493 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.493 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.494 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.494 DEBUG:boto:Method: GET 2023-10-12T00:48:13.495 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.495 DEBUG:boto:Data: 2023-10-12T00:48:13.496 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.497 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.497 DEBUG:boto:Params: {} 2023-10-12T00:48:13.497 DEBUG:boto:Token: None 2023-10-12T00:48:13.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ= 2023-10-12T00:48:13.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63bf141f9cbdee7-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.502 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.502 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.503 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.503 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.503 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.504 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.504 DEBUG:boto:Data: 2023-10-12T00:48:13.505 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.505 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.506 DEBUG:boto:Params: {} 2023-10-12T00:48:13.506 DEBUG:boto:Token: None 2023-10-12T00:48:13.507 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.507 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hz3En/WF5VnnJ//e1GQbQ0rrFKQ= 2023-10-12T00:48:13.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hz3En/WF5VnnJ//e1GQbQ0rrFKQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d770e65c8c8759b6-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.510 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.511 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.511 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.511 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.512 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.512 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.513 DEBUG:boto:Data: 2023-10-12T00:48:13.513 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.514 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.514 DEBUG:boto:Params: {} 2023-10-12T00:48:13.515 DEBUG:boto:Token: None 2023-10-12T00:48:13.515 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hz3En/WF5VnnJ//e1GQbQ0rrFKQ= 2023-10-12T00:48:13.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hz3En/WF5VnnJ//e1GQbQ0rrFKQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0f95c5ac26af2d-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:13.519 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.519 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.520 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.520 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.521 DEBUG:boto:Method: GET 2023-10-12T00:48:13.521 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.521 DEBUG:boto:Data: 2023-10-12T00:48:13.522 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.523 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.523 DEBUG:boto:Params: {} 2023-10-12T00:48:13.524 DEBUG:boto:Token: None 2023-10-12T00:48:13.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ= 2023-10-12T00:48:13.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc661d1ab672b41-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.528 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.528 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.528 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.529 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:13.529 DEBUG:boto:Method: GET 2023-10-12T00:48:13.530 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.530 DEBUG:boto:Data: 2023-10-12T00:48:13.531 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.531 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.532 DEBUG:boto:Params: {} 2023-10-12T00:48:13.532 DEBUG:boto:Token: None 2023-10-12T00:48:13.533 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:13.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ= 2023-10-12T00:48:13.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbSTeYDECnqqhdO4l+VV+quQIbQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193f42412b3106fb-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.538 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.538 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.539 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.539 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.540 DEBUG:boto:Method: GET 2023-10-12T00:48:13.540 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:13.540 DEBUG:boto:Data: 2023-10-12T00:48:13.541 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.542 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.542 DEBUG:boto:Params: {} 2023-10-12T00:48:13.543 DEBUG:boto:Token: None 2023-10-12T00:48:13.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf9c248a4a0521b-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.547 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.548 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.548 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.549 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.549 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.550 DEBUG:boto:Method: GET 2023-10-12T00:48:13.550 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:13.551 DEBUG:boto:Data: 2023-10-12T00:48:13.551 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.552 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.552 DEBUG:boto:Params: {} 2023-10-12T00:48:13.553 DEBUG:boto:Token: None 2023-10-12T00:48:13.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c818f4bb9c760e6f-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.557 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.557 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.558 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.558 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.559 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:13.559 DEBUG:boto:Data: 2023-10-12T00:48:13.559 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.560 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.561 DEBUG:boto:Params: {} 2023-10-12T00:48:13.561 DEBUG:boto:Token: None 2023-10-12T00:48:13.562 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a 2023-10-12T00:48:13.562 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:38GtT1zESEgJvqSE/K5W0KVDTpw= 2023-10-12T00:48:13.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:38GtT1zESEgJvqSE/K5W0KVDTpw=', 'Content-Length': '0'} 2023-10-12T00:48:13.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5c4f172f2ab1b1-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.565 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.565 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.566 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.566 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:13.566 DEBUG:boto:Data: 2023-10-12T00:48:13.567 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.568 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.568 DEBUG:boto:Params: {} 2023-10-12T00:48:13.568 DEBUG:boto:Token: None 2023-10-12T00:48:13.569 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a 2023-10-12T00:48:13.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:38GtT1zESEgJvqSE/K5W0KVDTpw= 2023-10-12T00:48:13.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:38GtT1zESEgJvqSE/K5W0KVDTpw=', 'Content-Length': '0'} 2023-10-12T00:48:13.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e2b9aae48c92e3-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:13.573 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.573 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.574 DEBUG:boto:Method: GET 2023-10-12T00:48:13.574 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:13.575 DEBUG:boto:Data: 2023-10-12T00:48:13.575 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.576 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.576 DEBUG:boto:Params: {} 2023-10-12T00:48:13.577 DEBUG:boto:Token: None 2023-10-12T00:48:13.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a 2023-10-12T00:48:13.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y/yS3J5nicULvNYcBxwnJrH3XRM= 2023-10-12T00:48:13.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y/yS3J5nicULvNYcBxwnJrH3XRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26abef7d29731f3-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.626 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:13.627 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:13.627 DEBUG:boto:Method: GET 2023-10-12T00:48:13.628 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:13.628 DEBUG:boto:Data: 2023-10-12T00:48:13.629 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.629 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.630 DEBUG:boto:Params: {} 2023-10-12T00:48:13.630 DEBUG:boto:Token: None 2023-10-12T00:48:13.631 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/a 2023-10-12T00:48:13.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y/yS3J5nicULvNYcBxwnJrH3XRM= 2023-10-12T00:48:13.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y/yS3J5nicULvNYcBxwnJrH3XRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050526dd36608529f-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.634 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.635 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.635 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.636 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.636 DEBUG:boto:Method: GET 2023-10-12T00:48:13.637 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:13.637 DEBUG:boto:Data: 2023-10-12T00:48:13.637 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.638 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.639 DEBUG:boto:Params: {} 2023-10-12T00:48:13.639 DEBUG:boto:Token: None 2023-10-12T00:48:13.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000439c494b84b01e45-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.644 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.645 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.645 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.645 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.646 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:13.646 DEBUG:boto:Method: GET 2023-10-12T00:48:13.647 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:13.647 DEBUG:boto:Data: 2023-10-12T00:48:13.648 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.648 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.649 DEBUG:boto:Params: {} 2023-10-12T00:48:13.649 DEBUG:boto:Token: None 2023-10-12T00:48:13.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1a8f82c1215a9f-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.654 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:13.655 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.655 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.656 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.656 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.657 DEBUG:boto:Method: GET 2023-10-12T00:48:13.657 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.657 DEBUG:boto:Data: 2023-10-12T00:48:13.658 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.659 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.659 DEBUG:boto:Params: {} 2023-10-12T00:48:13.660 DEBUG:boto:Token: None 2023-10-12T00:48:13.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM= 2023-10-12T00:48:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM=', 'Content-Length': '0'} 2023-10-12T00:48:13.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e84e81bd531cd0c-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.664 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.664 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.664 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.665 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.665 DEBUG:boto:Method: GET 2023-10-12T00:48:13.666 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.666 DEBUG:boto:Data: 2023-10-12T00:48:13.667 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.667 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.668 DEBUG:boto:Params: {} 2023-10-12T00:48:13.668 DEBUG:boto:Token: None 2023-10-12T00:48:13.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM= 2023-10-12T00:48:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM=', 'Content-Length': '0'} 2023-10-12T00:48:13.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8a7f801d480f0f-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.672 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.673 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.673 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.674 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.674 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.674 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.675 DEBUG:boto:Data: 2023-10-12T00:48:13.675 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.676 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.676 DEBUG:boto:Params: {} 2023-10-12T00:48:13.677 DEBUG:boto:Token: None 2023-10-12T00:48:13.677 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9gEO0mqQskO6yqI5EIKEQ57EpWc= 2023-10-12T00:48:13.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9gEO0mqQskO6yqI5EIKEQ57EpWc=', 'Content-Length': '0'} 2023-10-12T00:48:13.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359e6c9b915fb1be-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.681 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.681 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.682 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.682 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.682 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.683 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.683 DEBUG:boto:Data: 2023-10-12T00:48:13.684 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.684 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.685 DEBUG:boto:Params: {} 2023-10-12T00:48:13.685 DEBUG:boto:Token: None 2023-10-12T00:48:13.686 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9gEO0mqQskO6yqI5EIKEQ57EpWc= 2023-10-12T00:48:13.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9gEO0mqQskO6yqI5EIKEQ57EpWc=', 'Content-Length': '0'} 2023-10-12T00:48:13.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc4d29adeadf5c7-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:13.689 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.690 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.690 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.691 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.691 DEBUG:boto:Method: GET 2023-10-12T00:48:13.692 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.692 DEBUG:boto:Data: 2023-10-12T00:48:13.692 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.693 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.694 DEBUG:boto:Params: {} 2023-10-12T00:48:13.694 DEBUG:boto:Token: None 2023-10-12T00:48:13.695 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM= 2023-10-12T00:48:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM=', 'Content-Length': '0'} 2023-10-12T00:48:13.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f283d1e10e8498ee-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.742 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.743 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.743 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.744 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:13.744 DEBUG:boto:Method: GET 2023-10-12T00:48:13.745 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.745 DEBUG:boto:Data: 2023-10-12T00:48:13.746 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.746 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.747 DEBUG:boto:Params: {} 2023-10-12T00:48:13.747 DEBUG:boto:Token: None 2023-10-12T00:48:13.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:13.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM= 2023-10-12T00:48:13.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:go8vHPuSgGHosJmcVmXB0+wJErM=', 'Content-Length': '0'} 2023-10-12T00:48:13.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2556c82033b4bd0-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.751 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.752 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.753 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.753 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.753 DEBUG:boto:Method: GET 2023-10-12T00:48:13.754 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:13.754 DEBUG:boto:Data: 2023-10-12T00:48:13.755 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.757 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.757 DEBUG:boto:Params: {} 2023-10-12T00:48:13.758 DEBUG:boto:Token: None 2023-10-12T00:48:13.759 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9412ef727e3a961-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.764 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.764 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.765 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.765 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.766 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.766 DEBUG:boto:Method: GET 2023-10-12T00:48:13.767 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:13.767 DEBUG:boto:Data: 2023-10-12T00:48:13.767 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.768 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.769 DEBUG:boto:Params: {} 2023-10-12T00:48:13.769 DEBUG:boto:Token: None 2023-10-12T00:48:13.770 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc60ae95ad4cabe6-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.774 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.775 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.775 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.775 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.776 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:13.776 DEBUG:boto:Data: 2023-10-12T00:48:13.777 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.777 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.778 DEBUG:boto:Params: {} 2023-10-12T00:48:13.778 DEBUG:boto:Token: None 2023-10-12T00:48:13.779 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b 2023-10-12T00:48:13.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xPGRufvPegTwwUsjKG4txcLaDk8= 2023-10-12T00:48:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xPGRufvPegTwwUsjKG4txcLaDk8=', 'Content-Length': '0'} 2023-10-12T00:48:13.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959ea2ce6d2d2034-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.785 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.786 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.786 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.787 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:13.787 DEBUG:boto:Data: 2023-10-12T00:48:13.787 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.789 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.789 DEBUG:boto:Params: {} 2023-10-12T00:48:13.789 DEBUG:boto:Token: None 2023-10-12T00:48:13.790 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b 2023-10-12T00:48:13.790 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xPGRufvPegTwwUsjKG4txcLaDk8= 2023-10-12T00:48:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xPGRufvPegTwwUsjKG4txcLaDk8=', 'Content-Length': '0'} 2023-10-12T00:48:13.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df45ff9a5c0b67ab-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:13.794 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.794 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.795 DEBUG:boto:Method: GET 2023-10-12T00:48:13.795 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:13.796 DEBUG:boto:Data: 2023-10-12T00:48:13.796 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.797 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.798 DEBUG:boto:Params: {} 2023-10-12T00:48:13.798 DEBUG:boto:Token: None 2023-10-12T00:48:13.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b 2023-10-12T00:48:13.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rFx+bWx1awoJtPR3T0EtibGK8PU= 2023-10-12T00:48:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rFx+bWx1awoJtPR3T0EtibGK8PU=', 'Content-Length': '0'} 2023-10-12T00:48:13.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3de361d53efd4c7-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.802 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:13.803 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:13.803 DEBUG:boto:Method: GET 2023-10-12T00:48:13.803 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:13.804 DEBUG:boto:Data: 2023-10-12T00:48:13.804 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.805 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.806 DEBUG:boto:Params: {} 2023-10-12T00:48:13.806 DEBUG:boto:Token: None 2023-10-12T00:48:13.807 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/b 2023-10-12T00:48:13.807 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rFx+bWx1awoJtPR3T0EtibGK8PU= 2023-10-12T00:48:13.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rFx+bWx1awoJtPR3T0EtibGK8PU=', 'Content-Length': '0'} 2023-10-12T00:48:13.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e92cd60d589583b-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.810 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.811 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.811 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.812 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.812 DEBUG:boto:Method: GET 2023-10-12T00:48:13.813 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:13.813 DEBUG:boto:Data: 2023-10-12T00:48:13.813 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.814 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.815 DEBUG:boto:Params: {} 2023-10-12T00:48:13.815 DEBUG:boto:Token: None 2023-10-12T00:48:13.816 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.816 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000853bf896338cf190-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.820 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.820 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.821 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.821 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.822 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:13.822 DEBUG:boto:Method: GET 2023-10-12T00:48:13.823 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:13.823 DEBUG:boto:Data: 2023-10-12T00:48:13.824 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.824 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.825 DEBUG:boto:Params: {} 2023-10-12T00:48:13.825 DEBUG:boto:Token: None 2023-10-12T00:48:13.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c279bc2c830109-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.829 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:13.830 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.831 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.831 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.832 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.832 DEBUG:boto:Method: GET 2023-10-12T00:48:13.833 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.833 DEBUG:boto:Data: 2023-10-12T00:48:13.833 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.834 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.835 DEBUG:boto:Params: {} 2023-10-12T00:48:13.835 DEBUG:boto:Token: None 2023-10-12T00:48:13.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU= 2023-10-12T00:48:13.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU=', 'Content-Length': '0'} 2023-10-12T00:48:13.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf6730a081e1b71-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.886 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.887 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.887 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.888 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.888 DEBUG:boto:Method: GET 2023-10-12T00:48:13.889 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.889 DEBUG:boto:Data: 2023-10-12T00:48:13.890 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.891 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.891 DEBUG:boto:Params: {} 2023-10-12T00:48:13.892 DEBUG:boto:Token: None 2023-10-12T00:48:13.892 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU= 2023-10-12T00:48:13.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU=', 'Content-Length': '0'} 2023-10-12T00:48:13.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004804384afa07f725-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.896 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.896 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.897 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.897 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.898 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.898 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.899 DEBUG:boto:Data: 2023-10-12T00:48:13.899 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.900 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.900 DEBUG:boto:Params: {} 2023-10-12T00:48:13.901 DEBUG:boto:Token: None 2023-10-12T00:48:13.901 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tP9AAQ+RYvU0LDIP/XOQQvrKkZQ= 2023-10-12T00:48:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tP9AAQ+RYvU0LDIP/XOQQvrKkZQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056160cfeab079d9c-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.904 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.905 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.905 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.906 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.906 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.907 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.907 DEBUG:boto:Data: 2023-10-12T00:48:13.907 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.908 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.909 DEBUG:boto:Params: {} 2023-10-12T00:48:13.909 DEBUG:boto:Token: None 2023-10-12T00:48:13.910 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.910 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tP9AAQ+RYvU0LDIP/XOQQvrKkZQ= 2023-10-12T00:48:13.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tP9AAQ+RYvU0LDIP/XOQQvrKkZQ=', 'Content-Length': '0'} 2023-10-12T00:48:13.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fe91a0f41f4c0d-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:13.914 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.914 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.915 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.915 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.916 DEBUG:boto:Method: GET 2023-10-12T00:48:13.916 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.916 DEBUG:boto:Data: 2023-10-12T00:48:13.917 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.918 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.918 DEBUG:boto:Params: {} 2023-10-12T00:48:13.919 DEBUG:boto:Token: None 2023-10-12T00:48:13.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU= 2023-10-12T00:48:13.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU=', 'Content-Length': '0'} 2023-10-12T00:48:13.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aad80a17efb36ca-006527424d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.923 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.923 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.924 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.924 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:13.925 DEBUG:boto:Method: GET 2023-10-12T00:48:13.925 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.925 DEBUG:boto:Data: 2023-10-12T00:48:13.926 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.927 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.927 DEBUG:boto:Params: {} 2023-10-12T00:48:13.928 DEBUG:boto:Token: None 2023-10-12T00:48:13.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:13.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU= 2023-10-12T00:48:13.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KEuQuLS2z809hGhU9jo/yi1vbEU=', 'Content-Length': '0'} 2023-10-12T00:48:13.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3ad3aa8154d3d6-006527424d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.974 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.975 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.976 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:13.976 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:13.976 DEBUG:boto:Method: GET 2023-10-12T00:48:13.977 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:13.977 DEBUG:boto:Data: 2023-10-12T00:48:13.978 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.979 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.979 DEBUG:boto:Params: {} 2023-10-12T00:48:13.980 DEBUG:boto:Token: None 2023-10-12T00:48:13.980 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883dedc492949f97-006527424d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.985 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.985 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:13.986 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:13.986 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:13.987 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:13.987 DEBUG:boto:Method: GET 2023-10-12T00:48:13.988 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:13.988 DEBUG:boto:Data: 2023-10-12T00:48:13.988 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:13.989 DEBUG:boto:Port: 8000 2023-10-12T00:48:13.990 DEBUG:boto:Params: {} 2023-10-12T00:48:13.991 DEBUG:boto:Token: None 2023-10-12T00:48:13.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:13 GMT /wafmoa-55/ 2023-10-12T00:48:13.992 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM= 2023-10-12T00:48:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p12t68MykC53+i2lZbQOjYRiNRM=', 'Content-Length': '0'} 2023-10-12T00:48:13.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f8b80b80146431-006527424d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:13.995 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:13.995 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:13.996 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:13.997 DEBUG:boto:Method: HEAD 2023-10-12T00:48:13.997 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:13.997 DEBUG:boto:Data: 2023-10-12T00:48:13.998 DEBUG:boto:Headers: {} 2023-10-12T00:48:13.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:13.999 DEBUG:boto:Port: 8001 2023-10-12T00:48:13.999 DEBUG:boto:Params: {} 2023-10-12T00:48:13.999 DEBUG:boto:Token: None 2023-10-12T00:48:14.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/c 2023-10-12T00:48:14.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8qSHZ++Qh4+8RecHpreIFM+dAV8= 2023-10-12T00:48:14.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8qSHZ++Qh4+8RecHpreIFM+dAV8=', 'Content-Length': '0'} 2023-10-12T00:48:14.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6e90d25812432b-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.004 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:14.005 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:14.005 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.006 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:14.007 DEBUG:boto:Data: 2023-10-12T00:48:14.008 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.009 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.010 DEBUG:boto:Params: {} 2023-10-12T00:48:14.011 DEBUG:boto:Token: None 2023-10-12T00:48:14.011 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/c 2023-10-12T00:48:14.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8qSHZ++Qh4+8RecHpreIFM+dAV8= 2023-10-12T00:48:14.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8qSHZ++Qh4+8RecHpreIFM+dAV8=', 'Content-Length': '0'} 2023-10-12T00:48:14.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aa1fbffb6c74561-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:14.016 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:14.016 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:14.017 DEBUG:boto:Method: GET 2023-10-12T00:48:14.017 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:14.017 DEBUG:boto:Data: 2023-10-12T00:48:14.018 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.019 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.019 DEBUG:boto:Params: {} 2023-10-12T00:48:14.020 DEBUG:boto:Token: None 2023-10-12T00:48:14.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/c 2023-10-12T00:48:14.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GBuYI5D4OAqWNiQcNkdgeFZVCUo= 2023-10-12T00:48:14.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GBuYI5D4OAqWNiQcNkdgeFZVCUo=', 'Content-Length': '0'} 2023-10-12T00:48:14.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd48691644b7b359-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.066 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:14.067 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:14.068 DEBUG:boto:Method: GET 2023-10-12T00:48:14.068 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:14.069 DEBUG:boto:Data: 2023-10-12T00:48:14.069 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.070 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.071 DEBUG:boto:Params: {} 2023-10-12T00:48:14.071 DEBUG:boto:Token: None 2023-10-12T00:48:14.072 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/c 2023-10-12T00:48:14.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GBuYI5D4OAqWNiQcNkdgeFZVCUo= 2023-10-12T00:48:14.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GBuYI5D4OAqWNiQcNkdgeFZVCUo=', 'Content-Length': '0'} 2023-10-12T00:48:14.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950cdd28aae60ba2-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.076 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.076 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.077 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:14.077 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:14.078 DEBUG:boto:Method: GET 2023-10-12T00:48:14.078 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:14.079 DEBUG:boto:Data: 2023-10-12T00:48:14.079 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.080 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.081 DEBUG:boto:Params: {} 2023-10-12T00:48:14.081 DEBUG:boto:Token: None 2023-10-12T00:48:14.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004569dd1c3cc08be3-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.092 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.094 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.095 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.097 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:14.098 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:14.098 DEBUG:boto:Method: GET 2023-10-12T00:48:14.099 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:14.100 DEBUG:boto:Data: 2023-10-12T00:48:14.101 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.102 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.103 DEBUG:boto:Params: {} 2023-10-12T00:48:14.104 DEBUG:boto:Token: None 2023-10-12T00:48:14.105 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edd1a7fc079448eb-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.150 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:14.152 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.152 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.153 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.153 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.153 DEBUG:boto:Method: GET 2023-10-12T00:48:14.154 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.154 DEBUG:boto:Data: 2023-10-12T00:48:14.155 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.156 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.156 DEBUG:boto:Params: {} 2023-10-12T00:48:14.157 DEBUG:boto:Token: None 2023-10-12T00:48:14.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M= 2023-10-12T00:48:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M=', 'Content-Length': '0'} 2023-10-12T00:48:14.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007624a10965002aac-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.161 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.162 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.162 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.163 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.163 DEBUG:boto:Method: GET 2023-10-12T00:48:14.164 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.164 DEBUG:boto:Data: 2023-10-12T00:48:14.164 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.165 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.166 DEBUG:boto:Params: {} 2023-10-12T00:48:14.167 DEBUG:boto:Token: None 2023-10-12T00:48:14.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M= 2023-10-12T00:48:14.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M=', 'Content-Length': '0'} 2023-10-12T00:48:14.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001896704116ee10a8-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.172 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.173 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.173 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.174 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.175 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.175 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.175 DEBUG:boto:Data: 2023-10-12T00:48:14.176 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.177 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.177 DEBUG:boto:Params: {} 2023-10-12T00:48:14.178 DEBUG:boto:Token: None 2023-10-12T00:48:14.178 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D4cA0KYdILrHWK0rFLhJcTwB2js= 2023-10-12T00:48:14.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D4cA0KYdILrHWK0rFLhJcTwB2js=', 'Content-Length': '0'} 2023-10-12T00:48:14.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d595d6ab6a321af-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.182 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.182 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.182 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.183 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.183 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.184 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.184 DEBUG:boto:Data: 2023-10-12T00:48:14.185 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.185 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.186 DEBUG:boto:Params: {} 2023-10-12T00:48:14.186 DEBUG:boto:Token: None 2023-10-12T00:48:14.187 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D4cA0KYdILrHWK0rFLhJcTwB2js= 2023-10-12T00:48:14.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D4cA0KYdILrHWK0rFLhJcTwB2js=', 'Content-Length': '0'} 2023-10-12T00:48:14.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0c25a316c58ddd7-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:14.191 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.191 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.192 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.192 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.192 DEBUG:boto:Method: GET 2023-10-12T00:48:14.193 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.193 DEBUG:boto:Data: 2023-10-12T00:48:14.194 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.195 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.195 DEBUG:boto:Params: {} 2023-10-12T00:48:14.196 DEBUG:boto:Token: None 2023-10-12T00:48:14.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M= 2023-10-12T00:48:14.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M=', 'Content-Length': '0'} 2023-10-12T00:48:14.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468b18d511eb7b86-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.200 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.200 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.201 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.201 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:14.201 DEBUG:boto:Method: GET 2023-10-12T00:48:14.202 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.202 DEBUG:boto:Data: 2023-10-12T00:48:14.203 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.204 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.204 DEBUG:boto:Params: {} 2023-10-12T00:48:14.204 DEBUG:boto:Token: None 2023-10-12T00:48:14.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:14.205 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M= 2023-10-12T00:48:14.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fG66m9BJA/R50rC9cuRc15Q46M=', 'Content-Length': '0'} 2023-10-12T00:48:14.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d55dc9a0ab64ae8-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.209 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.209 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.209 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.210 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.210 DEBUG:boto:Method: GET 2023-10-12T00:48:14.211 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:14.211 DEBUG:boto:Data: 2023-10-12T00:48:14.212 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.212 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.213 DEBUG:boto:Params: {} 2023-10-12T00:48:14.213 DEBUG:boto:Token: None 2023-10-12T00:48:14.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088de96eff796c94f-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.218 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.219 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.219 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.220 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.220 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.220 DEBUG:boto:Method: GET 2023-10-12T00:48:14.221 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:14.221 DEBUG:boto:Data: 2023-10-12T00:48:14.222 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.223 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.223 DEBUG:boto:Params: {} 2023-10-12T00:48:14.224 DEBUG:boto:Token: None 2023-10-12T00:48:14.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c77cfce79f8eee87-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.228 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.229 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.229 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.229 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.230 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:14.230 DEBUG:boto:Data: 2023-10-12T00:48:14.231 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.231 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.232 DEBUG:boto:Params: {} 2023-10-12T00:48:14.232 DEBUG:boto:Token: None 2023-10-12T00:48:14.233 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d 2023-10-12T00:48:14.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uj+OR5ZWN/ao6arXmPIRSfHXde0= 2023-10-12T00:48:14.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uj+OR5ZWN/ao6arXmPIRSfHXde0=', 'Content-Length': '0'} 2023-10-12T00:48:14.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840a279e5d37a0a6-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.236 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.237 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.237 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.237 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:14.238 DEBUG:boto:Data: 2023-10-12T00:48:14.238 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.239 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.240 DEBUG:boto:Params: {} 2023-10-12T00:48:14.240 DEBUG:boto:Token: None 2023-10-12T00:48:14.240 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d 2023-10-12T00:48:14.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uj+OR5ZWN/ao6arXmPIRSfHXde0= 2023-10-12T00:48:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uj+OR5ZWN/ao6arXmPIRSfHXde0=', 'Content-Length': '0'} 2023-10-12T00:48:14.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f26a670c85a5e74-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:14.246 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.246 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.247 DEBUG:boto:Method: GET 2023-10-12T00:48:14.247 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:14.247 DEBUG:boto:Data: 2023-10-12T00:48:14.248 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.249 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.249 DEBUG:boto:Params: {} 2023-10-12T00:48:14.250 DEBUG:boto:Token: None 2023-10-12T00:48:14.250 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d 2023-10-12T00:48:14.251 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8qT6s0/aTksXjvrY91c0oklPijs= 2023-10-12T00:48:14.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8qT6s0/aTksXjvrY91c0oklPijs=', 'Content-Length': '0'} 2023-10-12T00:48:14.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3093dd53b53892c-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.254 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:14.255 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:14.255 DEBUG:boto:Method: GET 2023-10-12T00:48:14.256 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:14.256 DEBUG:boto:Data: 2023-10-12T00:48:14.257 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.257 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.258 DEBUG:boto:Params: {} 2023-10-12T00:48:14.258 DEBUG:boto:Token: None 2023-10-12T00:48:14.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/d 2023-10-12T00:48:14.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8qT6s0/aTksXjvrY91c0oklPijs= 2023-10-12T00:48:14.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8qT6s0/aTksXjvrY91c0oklPijs=', 'Content-Length': '0'} 2023-10-12T00:48:14.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2354f4f6b01b187-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.302 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.303 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.303 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.304 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.304 DEBUG:boto:Method: GET 2023-10-12T00:48:14.305 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:14.305 DEBUG:boto:Data: 2023-10-12T00:48:14.306 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.307 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.307 DEBUG:boto:Params: {} 2023-10-12T00:48:14.308 DEBUG:boto:Token: None 2023-10-12T00:48:14.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009864c18b9c2cbcae-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.312 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.313 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.313 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.314 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.314 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:14.315 DEBUG:boto:Method: GET 2023-10-12T00:48:14.315 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:14.316 DEBUG:boto:Data: 2023-10-12T00:48:14.316 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.317 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.317 DEBUG:boto:Params: {} 2023-10-12T00:48:14.318 DEBUG:boto:Token: None 2023-10-12T00:48:14.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b5969d70902f108-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.322 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:14.323 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.323 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.324 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.324 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.324 DEBUG:boto:Method: GET 2023-10-12T00:48:14.325 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.325 DEBUG:boto:Data: 2023-10-12T00:48:14.326 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.327 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.327 DEBUG:boto:Params: {} 2023-10-12T00:48:14.327 DEBUG:boto:Token: None 2023-10-12T00:48:14.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA= 2023-10-12T00:48:14.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA=', 'Content-Length': '0'} 2023-10-12T00:48:14.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007252d099ecb0d806-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.374 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.375 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.375 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.376 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.376 DEBUG:boto:Method: GET 2023-10-12T00:48:14.377 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.377 DEBUG:boto:Data: 2023-10-12T00:48:14.378 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.378 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.379 DEBUG:boto:Params: {} 2023-10-12T00:48:14.379 DEBUG:boto:Token: None 2023-10-12T00:48:14.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA= 2023-10-12T00:48:14.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA=', 'Content-Length': '0'} 2023-10-12T00:48:14.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6a0c906717fd33-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.384 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.384 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.385 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.385 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.386 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.386 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.386 DEBUG:boto:Data: 2023-10-12T00:48:14.387 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.388 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.388 DEBUG:boto:Params: {} 2023-10-12T00:48:14.388 DEBUG:boto:Token: None 2023-10-12T00:48:14.389 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lrPApit3EOfWDl6mpR/rpdhrfs4= 2023-10-12T00:48:14.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lrPApit3EOfWDl6mpR/rpdhrfs4=', 'Content-Length': '0'} 2023-10-12T00:48:14.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019f593f02bb4c86-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.392 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.393 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.393 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.394 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.394 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.395 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.395 DEBUG:boto:Data: 2023-10-12T00:48:14.395 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.396 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.397 DEBUG:boto:Params: {} 2023-10-12T00:48:14.397 DEBUG:boto:Token: None 2023-10-12T00:48:14.398 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lrPApit3EOfWDl6mpR/rpdhrfs4= 2023-10-12T00:48:14.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lrPApit3EOfWDl6mpR/rpdhrfs4=', 'Content-Length': '0'} 2023-10-12T00:48:14.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009398db85e28769ac-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:14.402 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.402 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.402 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.403 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.403 DEBUG:boto:Method: GET 2023-10-12T00:48:14.404 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.404 DEBUG:boto:Data: 2023-10-12T00:48:14.405 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.405 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.406 DEBUG:boto:Params: {} 2023-10-12T00:48:14.406 DEBUG:boto:Token: None 2023-10-12T00:48:14.407 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA= 2023-10-12T00:48:14.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA=', 'Content-Length': '0'} 2023-10-12T00:48:14.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e2a95edc31ff83-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.410 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.411 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.411 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.412 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:14.412 DEBUG:boto:Method: GET 2023-10-12T00:48:14.412 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.413 DEBUG:boto:Data: 2023-10-12T00:48:14.413 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.414 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.414 DEBUG:boto:Params: {} 2023-10-12T00:48:14.415 DEBUG:boto:Token: None 2023-10-12T00:48:14.415 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:14.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA= 2023-10-12T00:48:14.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w3kt9rT+GquGMe8YrAcwfDB8ySA=', 'Content-Length': '0'} 2023-10-12T00:48:14.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca63cc6d928cc27-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.419 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.420 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.420 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.420 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.421 DEBUG:boto:Method: GET 2023-10-12T00:48:14.421 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:14.422 DEBUG:boto:Data: 2023-10-12T00:48:14.422 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.423 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.423 DEBUG:boto:Params: {} 2023-10-12T00:48:14.424 DEBUG:boto:Token: None 2023-10-12T00:48:14.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8819e7909cc688-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.429 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.429 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.430 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.430 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.431 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.431 DEBUG:boto:Method: GET 2023-10-12T00:48:14.431 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:14.432 DEBUG:boto:Data: 2023-10-12T00:48:14.432 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.433 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.434 DEBUG:boto:Params: {} 2023-10-12T00:48:14.434 DEBUG:boto:Token: None 2023-10-12T00:48:14.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.435 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5b88a92b644b334-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.478 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.480 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.480 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.481 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.482 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:14.482 DEBUG:boto:Data: 2023-10-12T00:48:14.483 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.484 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.484 DEBUG:boto:Params: {} 2023-10-12T00:48:14.484 DEBUG:boto:Token: None 2023-10-12T00:48:14.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e 2023-10-12T00:48:14.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C8D8uJLK6SgCplMuytk3iX2Hf3U= 2023-10-12T00:48:14.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C8D8uJLK6SgCplMuytk3iX2Hf3U=', 'Content-Length': '0'} 2023-10-12T00:48:14.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fde0eb797e4014-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.489 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.489 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.489 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.490 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:14.490 DEBUG:boto:Data: 2023-10-12T00:48:14.491 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.492 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.492 DEBUG:boto:Params: {} 2023-10-12T00:48:14.492 DEBUG:boto:Token: None 2023-10-12T00:48:14.493 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e 2023-10-12T00:48:14.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C8D8uJLK6SgCplMuytk3iX2Hf3U= 2023-10-12T00:48:14.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C8D8uJLK6SgCplMuytk3iX2Hf3U=', 'Content-Length': '0'} 2023-10-12T00:48:14.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000826882a92d165cb-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:14.497 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.498 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.498 DEBUG:boto:Method: GET 2023-10-12T00:48:14.499 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:14.499 DEBUG:boto:Data: 2023-10-12T00:48:14.500 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.500 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.501 DEBUG:boto:Params: {} 2023-10-12T00:48:14.501 DEBUG:boto:Token: None 2023-10-12T00:48:14.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e 2023-10-12T00:48:14.503 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zckQ6ob/nWYO3lweai3Iji5XlmQ= 2023-10-12T00:48:14.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zckQ6ob/nWYO3lweai3Iji5XlmQ=', 'Content-Length': '0'} 2023-10-12T00:48:14.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286816cc71c1e95b-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.506 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:14.506 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:14.507 DEBUG:boto:Method: GET 2023-10-12T00:48:14.507 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:14.508 DEBUG:boto:Data: 2023-10-12T00:48:14.508 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.509 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.509 DEBUG:boto:Params: {} 2023-10-12T00:48:14.510 DEBUG:boto:Token: None 2023-10-12T00:48:14.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/e 2023-10-12T00:48:14.511 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zckQ6ob/nWYO3lweai3Iji5XlmQ= 2023-10-12T00:48:14.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zckQ6ob/nWYO3lweai3Iji5XlmQ=', 'Content-Length': '0'} 2023-10-12T00:48:14.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d311c9431d2b06e-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.514 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.514 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.515 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.515 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.516 DEBUG:boto:Method: GET 2023-10-12T00:48:14.516 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:14.517 DEBUG:boto:Data: 2023-10-12T00:48:14.517 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.518 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.518 DEBUG:boto:Params: {} 2023-10-12T00:48:14.519 DEBUG:boto:Token: None 2023-10-12T00:48:14.519 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf8c1072aaf0209-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.523 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.524 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.524 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.525 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.525 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:14.526 DEBUG:boto:Method: GET 2023-10-12T00:48:14.526 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:14.527 DEBUG:boto:Data: 2023-10-12T00:48:14.527 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.528 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.528 DEBUG:boto:Params: {} 2023-10-12T00:48:14.529 DEBUG:boto:Token: None 2023-10-12T00:48:14.529 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf5f00302ec5e07-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.533 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:14.534 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.535 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.535 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.536 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.536 DEBUG:boto:Method: GET 2023-10-12T00:48:14.536 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.537 DEBUG:boto:Data: 2023-10-12T00:48:14.537 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.538 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.539 DEBUG:boto:Params: {} 2023-10-12T00:48:14.539 DEBUG:boto:Token: None 2023-10-12T00:48:14.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0= 2023-10-12T00:48:14.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0=', 'Content-Length': '0'} 2023-10-12T00:48:14.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ccd44767d602599-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.543 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.544 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.544 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.545 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.545 DEBUG:boto:Method: GET 2023-10-12T00:48:14.546 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.546 DEBUG:boto:Data: 2023-10-12T00:48:14.546 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.547 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.548 DEBUG:boto:Params: {} 2023-10-12T00:48:14.548 DEBUG:boto:Token: None 2023-10-12T00:48:14.549 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.549 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0= 2023-10-12T00:48:14.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0=', 'Content-Length': '0'} 2023-10-12T00:48:14.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1c0c54586ae5a1b-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.553 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.553 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.553 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.554 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.554 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.555 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.555 DEBUG:boto:Data: 2023-10-12T00:48:14.555 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.556 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.557 DEBUG:boto:Params: {} 2023-10-12T00:48:14.557 DEBUG:boto:Token: None 2023-10-12T00:48:14.558 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ug5FVRj2vyoNyr1voCZNCkBvK3o= 2023-10-12T00:48:14.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ug5FVRj2vyoNyr1voCZNCkBvK3o=', 'Content-Length': '0'} 2023-10-12T00:48:14.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df2900fa7d6e987-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.561 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.561 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.562 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.562 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.563 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.563 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.563 DEBUG:boto:Data: 2023-10-12T00:48:14.564 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.565 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.565 DEBUG:boto:Params: {} 2023-10-12T00:48:14.566 DEBUG:boto:Token: None 2023-10-12T00:48:14.566 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ug5FVRj2vyoNyr1voCZNCkBvK3o= 2023-10-12T00:48:14.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ug5FVRj2vyoNyr1voCZNCkBvK3o=', 'Content-Length': '0'} 2023-10-12T00:48:14.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951ff220f425450b-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:14.570 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.570 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.571 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.571 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.572 DEBUG:boto:Method: GET 2023-10-12T00:48:14.572 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.572 DEBUG:boto:Data: 2023-10-12T00:48:14.573 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.574 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.574 DEBUG:boto:Params: {} 2023-10-12T00:48:14.575 DEBUG:boto:Token: None 2023-10-12T00:48:14.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0= 2023-10-12T00:48:14.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0=', 'Content-Length': '0'} 2023-10-12T00:48:14.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009149d125455f05e9-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.618 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.619 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.619 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.620 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:14.620 DEBUG:boto:Method: GET 2023-10-12T00:48:14.621 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.621 DEBUG:boto:Data: 2023-10-12T00:48:14.622 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.622 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.623 DEBUG:boto:Params: {} 2023-10-12T00:48:14.624 DEBUG:boto:Token: None 2023-10-12T00:48:14.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:14.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0= 2023-10-12T00:48:14.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yb6eaGtHLwfkDBHGxm7xMzYR4D0=', 'Content-Length': '0'} 2023-10-12T00:48:14.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983c4997db1875a2-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.628 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.628 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.629 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.629 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.629 DEBUG:boto:Method: GET 2023-10-12T00:48:14.630 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:14.630 DEBUG:boto:Data: 2023-10-12T00:48:14.631 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.632 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.632 DEBUG:boto:Params: {} 2023-10-12T00:48:14.633 DEBUG:boto:Token: None 2023-10-12T00:48:14.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d440c718009e3f24-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.638 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.638 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.639 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.639 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.640 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.640 DEBUG:boto:Method: GET 2023-10-12T00:48:14.641 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:14.641 DEBUG:boto:Data: 2023-10-12T00:48:14.641 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.642 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.643 DEBUG:boto:Params: {} 2023-10-12T00:48:14.643 DEBUG:boto:Token: None 2023-10-12T00:48:14.644 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff1fc67501cb14e-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.647 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.648 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.648 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.649 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.649 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:14.649 DEBUG:boto:Data: 2023-10-12T00:48:14.650 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.651 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.651 DEBUG:boto:Params: {} 2023-10-12T00:48:14.653 DEBUG:boto:Token: None 2023-10-12T00:48:14.654 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f 2023-10-12T00:48:14.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rp5vxesgCaeE6JrqkZCLLBffbNU= 2023-10-12T00:48:14.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rp5vxesgCaeE6JrqkZCLLBffbNU=', 'Content-Length': '0'} 2023-10-12T00:48:14.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f112953c32f082-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.658 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.658 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.658 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.659 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:14.659 DEBUG:boto:Data: 2023-10-12T00:48:14.660 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.661 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.661 DEBUG:boto:Params: {} 2023-10-12T00:48:14.661 DEBUG:boto:Token: None 2023-10-12T00:48:14.662 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f 2023-10-12T00:48:14.662 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rp5vxesgCaeE6JrqkZCLLBffbNU= 2023-10-12T00:48:14.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rp5vxesgCaeE6JrqkZCLLBffbNU=', 'Content-Length': '0'} 2023-10-12T00:48:14.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003bebb33aeedbce0-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:14.666 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.666 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.667 DEBUG:boto:Method: GET 2023-10-12T00:48:14.667 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:14.667 DEBUG:boto:Data: 2023-10-12T00:48:14.668 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.669 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.669 DEBUG:boto:Params: {} 2023-10-12T00:48:14.670 DEBUG:boto:Token: None 2023-10-12T00:48:14.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f 2023-10-12T00:48:14.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eUZN9IMBuNZECyq3mVSa3XfgENU= 2023-10-12T00:48:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eUZN9IMBuNZECyq3mVSa3XfgENU=', 'Content-Length': '0'} 2023-10-12T00:48:14.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017864cc0861ca20-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.674 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:14.674 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:14.674 DEBUG:boto:Method: GET 2023-10-12T00:48:14.675 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:14.675 DEBUG:boto:Data: 2023-10-12T00:48:14.676 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.677 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.677 DEBUG:boto:Params: {} 2023-10-12T00:48:14.678 DEBUG:boto:Token: None 2023-10-12T00:48:14.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/f 2023-10-12T00:48:14.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eUZN9IMBuNZECyq3mVSa3XfgENU= 2023-10-12T00:48:14.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eUZN9IMBuNZECyq3mVSa3XfgENU=', 'Content-Length': '0'} 2023-10-12T00:48:14.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be67d07029aea284-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.722 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.723 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.723 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.724 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.724 DEBUG:boto:Method: GET 2023-10-12T00:48:14.725 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:14.725 DEBUG:boto:Data: 2023-10-12T00:48:14.726 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.726 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.727 DEBUG:boto:Params: {} 2023-10-12T00:48:14.727 DEBUG:boto:Token: None 2023-10-12T00:48:14.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000264adb125fbe33c6-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.732 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.733 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.733 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.734 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.734 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:14.735 DEBUG:boto:Method: GET 2023-10-12T00:48:14.735 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:14.735 DEBUG:boto:Data: 2023-10-12T00:48:14.736 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.737 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.737 DEBUG:boto:Params: {} 2023-10-12T00:48:14.738 DEBUG:boto:Token: None 2023-10-12T00:48:14.738 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efcdf761b7242ecb-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.742 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:14.743 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.743 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.744 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.744 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.745 DEBUG:boto:Method: GET 2023-10-12T00:48:14.745 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.745 DEBUG:boto:Data: 2023-10-12T00:48:14.746 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.747 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.747 DEBUG:boto:Params: {} 2023-10-12T00:48:14.748 DEBUG:boto:Token: None 2023-10-12T00:48:14.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw= 2023-10-12T00:48:14.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw=', 'Content-Length': '0'} 2023-10-12T00:48:14.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f95e0d7539845f-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.752 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.752 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.753 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.753 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.754 DEBUG:boto:Method: GET 2023-10-12T00:48:14.754 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.755 DEBUG:boto:Data: 2023-10-12T00:48:14.755 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.756 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.756 DEBUG:boto:Params: {} 2023-10-12T00:48:14.757 DEBUG:boto:Token: None 2023-10-12T00:48:14.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw= 2023-10-12T00:48:14.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw=', 'Content-Length': '0'} 2023-10-12T00:48:14.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c24d12a59ad83e4-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.761 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.761 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.762 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.762 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.763 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.763 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.763 DEBUG:boto:Data: 2023-10-12T00:48:14.764 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.765 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.765 DEBUG:boto:Params: {} 2023-10-12T00:48:14.766 DEBUG:boto:Token: None 2023-10-12T00:48:14.766 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1SAab7FKWkSOfdunsDonQZgSH04= 2023-10-12T00:48:14.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1SAab7FKWkSOfdunsDonQZgSH04=', 'Content-Length': '0'} 2023-10-12T00:48:14.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a104491b3ca8a527-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.769 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.770 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.770 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.771 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.771 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.771 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.772 DEBUG:boto:Data: 2023-10-12T00:48:14.772 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.773 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.773 DEBUG:boto:Params: {} 2023-10-12T00:48:14.774 DEBUG:boto:Token: None 2023-10-12T00:48:14.774 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1SAab7FKWkSOfdunsDonQZgSH04= 2023-10-12T00:48:14.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1SAab7FKWkSOfdunsDonQZgSH04=', 'Content-Length': '0'} 2023-10-12T00:48:14.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f906ae98674a5327-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:14.778 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.779 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.779 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.780 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.780 DEBUG:boto:Method: GET 2023-10-12T00:48:14.780 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.781 DEBUG:boto:Data: 2023-10-12T00:48:14.781 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.782 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.783 DEBUG:boto:Params: {} 2023-10-12T00:48:14.783 DEBUG:boto:Token: None 2023-10-12T00:48:14.784 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw= 2023-10-12T00:48:14.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw=', 'Content-Length': '0'} 2023-10-12T00:48:14.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd73397ff9c74aa-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.830 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.831 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.832 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.833 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:14.833 DEBUG:boto:Method: GET 2023-10-12T00:48:14.834 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.835 DEBUG:boto:Data: 2023-10-12T00:48:14.836 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.837 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.837 DEBUG:boto:Params: {} 2023-10-12T00:48:14.838 DEBUG:boto:Token: None 2023-10-12T00:48:14.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:14.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw= 2023-10-12T00:48:14.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEM/eV180lFk8SiUlziON9dD4Kw=', 'Content-Length': '0'} 2023-10-12T00:48:14.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cffc6f844c0c6fa1-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.843 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.844 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.844 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.845 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.845 DEBUG:boto:Method: GET 2023-10-12T00:48:14.845 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:14.846 DEBUG:boto:Data: 2023-10-12T00:48:14.847 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.847 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.848 DEBUG:boto:Params: {} 2023-10-12T00:48:14.848 DEBUG:boto:Token: None 2023-10-12T00:48:14.849 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ec9fe3ae394ff9-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.853 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.854 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.857 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.858 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.858 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.859 DEBUG:boto:Method: GET 2023-10-12T00:48:14.859 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:14.860 DEBUG:boto:Data: 2023-10-12T00:48:14.861 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.862 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.863 DEBUG:boto:Params: {} 2023-10-12T00:48:14.863 DEBUG:boto:Token: None 2023-10-12T00:48:14.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb039ec6e98982a-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.868 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.868 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.869 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.870 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.870 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:14.871 DEBUG:boto:Data: 2023-10-12T00:48:14.872 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.873 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.874 DEBUG:boto:Params: {} 2023-10-12T00:48:14.875 DEBUG:boto:Token: None 2023-10-12T00:48:14.876 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g 2023-10-12T00:48:14.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XByTZMuzqaplVFqNx2PJXFhuWsY= 2023-10-12T00:48:14.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XByTZMuzqaplVFqNx2PJXFhuWsY=', 'Content-Length': '0'} 2023-10-12T00:48:14.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d299d47d1836cba4-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.880 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.881 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.881 DEBUG:boto:Method: HEAD 2023-10-12T00:48:14.882 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:14.882 DEBUG:boto:Data: 2023-10-12T00:48:14.883 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.884 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.885 DEBUG:boto:Params: {} 2023-10-12T00:48:14.885 DEBUG:boto:Token: None 2023-10-12T00:48:14.886 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g 2023-10-12T00:48:14.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XByTZMuzqaplVFqNx2PJXFhuWsY= 2023-10-12T00:48:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XByTZMuzqaplVFqNx2PJXFhuWsY=', 'Content-Length': '0'} 2023-10-12T00:48:14.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f6e55898babc060-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:14.892 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.893 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.893 DEBUG:boto:Method: GET 2023-10-12T00:48:14.894 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:14.895 DEBUG:boto:Data: 2023-10-12T00:48:14.896 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.897 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.898 DEBUG:boto:Params: {} 2023-10-12T00:48:14.899 DEBUG:boto:Token: None 2023-10-12T00:48:14.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g 2023-10-12T00:48:14.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xWwPc+b7TSCkczQWvnPAzB/QvtM= 2023-10-12T00:48:14.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xWwPc+b7TSCkczQWvnPAzB/QvtM=', 'Content-Length': '0'} 2023-10-12T00:48:14.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094aefd258621c164-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.904 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:14.904 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:14.905 DEBUG:boto:Method: GET 2023-10-12T00:48:14.905 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:14.906 DEBUG:boto:Data: 2023-10-12T00:48:14.906 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.907 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.908 DEBUG:boto:Params: {} 2023-10-12T00:48:14.909 DEBUG:boto:Token: None 2023-10-12T00:48:14.910 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/g 2023-10-12T00:48:14.910 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xWwPc+b7TSCkczQWvnPAzB/QvtM= 2023-10-12T00:48:14.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xWwPc+b7TSCkczQWvnPAzB/QvtM=', 'Content-Length': '0'} 2023-10-12T00:48:14.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6885c4c0aa960fb-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.954 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.955 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.956 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.956 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.956 DEBUG:boto:Method: GET 2023-10-12T00:48:14.957 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:14.958 DEBUG:boto:Data: 2023-10-12T00:48:14.958 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.959 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.960 DEBUG:boto:Params: {} 2023-10-12T00:48:14.960 DEBUG:boto:Token: None 2023-10-12T00:48:14.961 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8557de488251bd1-006527424e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.965 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.966 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:14.967 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:14.967 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.968 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:14.969 DEBUG:boto:Method: GET 2023-10-12T00:48:14.969 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:14.970 DEBUG:boto:Data: 2023-10-12T00:48:14.971 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.972 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.973 DEBUG:boto:Params: {} 2023-10-12T00:48:14.973 DEBUG:boto:Token: None 2023-10-12T00:48:14.974 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/ 2023-10-12T00:48:14.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU= 2023-10-12T00:48:14.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYrkVSqrLdZ+fzE+23acQ4ylTPU=', 'Content-Length': '0'} 2023-10-12T00:48:14.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207c363c73a905f5-006527424e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.978 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:14.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:14.980 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:14.980 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:14.980 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:14.981 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:14.981 DEBUG:boto:Method: GET 2023-10-12T00:48:14.982 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:14.982 DEBUG:boto:Data: 2023-10-12T00:48:14.983 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:14.983 DEBUG:boto:Port: 8001 2023-10-12T00:48:14.984 DEBUG:boto:Params: {} 2023-10-12T00:48:14.984 DEBUG:boto:Token: None 2023-10-12T00:48:14.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:14.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c/RPAx2GFhR7e5+EjdUVsT7zzSI= 2023-10-12T00:48:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c/RPAx2GFhR7e5+EjdUVsT7zzSI=', 'Content-Length': '0'} 2023-10-12T00:48:14.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e63b6126f011a9-006527424e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.989 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:14.989 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:14.990 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:14.990 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:14.991 DEBUG:boto:Method: GET 2023-10-12T00:48:14.991 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:14.992 DEBUG:boto:Data: 2023-10-12T00:48:14.992 DEBUG:boto:Headers: {} 2023-10-12T00:48:14.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:14.993 DEBUG:boto:Port: 8000 2023-10-12T00:48:14.993 DEBUG:boto:Params: {} 2023-10-12T00:48:14.994 DEBUG:boto:Token: None 2023-10-12T00:48:14.994 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:14 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:14.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c/RPAx2GFhR7e5+EjdUVsT7zzSI= 2023-10-12T00:48:14.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c/RPAx2GFhR7e5+EjdUVsT7zzSI=', 'Content-Length': '0'} 2023-10-12T00:48:14.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebbba0a827762c7f-006527424e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:14.998 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:14.998 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:14.999 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:14.999 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.000 DEBUG:boto:Method: HEAD 2023-10-12T00:48:15.000 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.001 DEBUG:boto:Data: 2023-10-12T00:48:15.001 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.002 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.002 DEBUG:boto:Params: {} 2023-10-12T00:48:15.003 DEBUG:boto:Token: None 2023-10-12T00:48:15.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u4jw7Qc3rf/BJzzYY2b6zmrvrSM= 2023-10-12T00:48:15.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u4jw7Qc3rf/BJzzYY2b6zmrvrSM=', 'Content-Length': '0'} 2023-10-12T00:48:15.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b279f32afeb20fb0-006527424f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.006 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.007 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.007 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.008 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.008 DEBUG:boto:Method: HEAD 2023-10-12T00:48:15.008 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.009 DEBUG:boto:Data: 2023-10-12T00:48:15.009 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.010 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.010 DEBUG:boto:Params: {} 2023-10-12T00:48:15.011 DEBUG:boto:Token: None 2023-10-12T00:48:15.011 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u4jw7Qc3rf/BJzzYY2b6zmrvrSM= 2023-10-12T00:48:15.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u4jw7Qc3rf/BJzzYY2b6zmrvrSM=', 'Content-Length': '0'} 2023-10-12T00:48:15.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d41bd4138d94b231-006527424f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:15.015 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.016 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.016 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.016 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.017 DEBUG:boto:Method: GET 2023-10-12T00:48:15.017 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.018 DEBUG:boto:Data: 2023-10-12T00:48:15.018 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.019 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.020 DEBUG:boto:Params: {} 2023-10-12T00:48:15.020 DEBUG:boto:Token: None 2023-10-12T00:48:15.020 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xHlNN6+Z+y9xoejb52rg6pxWS6k= 2023-10-12T00:48:15.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xHlNN6+Z+y9xoejb52rg6pxWS6k=', 'Content-Length': '0'} 2023-10-12T00:48:15.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fd59158dd31a62-006527424f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.024 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.025 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.025 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.026 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:15.026 DEBUG:boto:Method: GET 2023-10-12T00:48:15.026 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.027 DEBUG:boto:Data: 2023-10-12T00:48:15.027 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.028 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.029 DEBUG:boto:Params: {} 2023-10-12T00:48:15.029 DEBUG:boto:Token: None 2023-10-12T00:48:15.029 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:15.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xHlNN6+Z+y9xoejb52rg6pxWS6k= 2023-10-12T00:48:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xHlNN6+Z+y9xoejb52rg6pxWS6k=', 'Content-Length': '0'} 2023-10-12T00:48:15.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de344b2b7d98717c-006527424f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.033 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:15.034 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:15.034 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.035 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.035 DEBUG:boto:Method: GET 2023-10-12T00:48:15.035 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:15.036 DEBUG:boto:Data: 2023-10-12T00:48:15.036 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.037 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.038 DEBUG:boto:Params: {} 2023-10-12T00:48:15.038 DEBUG:boto:Token: None 2023-10-12T00:48:15.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/ 2023-10-12T00:48:15.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs= 2023-10-12T00:48:15.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs=', 'Content-Length': '0'} 2023-10-12T00:48:15.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7deefcc3b297ec-006527424f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.043 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:15.043 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:15.044 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:15.044 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.045 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.045 DEBUG:boto:Method: GET 2023-10-12T00:48:15.046 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:15.046 DEBUG:boto:Data: 2023-10-12T00:48:15.047 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.047 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.048 DEBUG:boto:Params: {} 2023-10-12T00:48:15.048 DEBUG:boto:Token: None 2023-10-12T00:48:15.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/ 2023-10-12T00:48:15.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs= 2023-10-12T00:48:15.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs=', 'Content-Length': '0'} 2023-10-12T00:48:15.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832aa0aaba8c02de-006527424f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.052 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:15.053 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.053 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.054 DEBUG:boto:Method: HEAD 2023-10-12T00:48:15.054 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:15.055 DEBUG:boto:Data: 2023-10-12T00:48:15.055 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.056 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.056 DEBUG:boto:Params: {} 2023-10-12T00:48:15.057 DEBUG:boto:Token: None 2023-10-12T00:48:15.057 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h 2023-10-12T00:48:15.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N3How32rF7Mk6Qo65IkTmW6MBkM= 2023-10-12T00:48:15.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N3How32rF7Mk6Qo65IkTmW6MBkM=', 'Content-Length': '0'} 2023-10-12T00:48:15.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cab8f0213c80a78-006527424f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.060 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.061 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.061 DEBUG:boto:Method: HEAD 2023-10-12T00:48:15.062 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:15.062 DEBUG:boto:Data: 2023-10-12T00:48:15.062 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.063 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.064 DEBUG:boto:Params: {} 2023-10-12T00:48:15.065 DEBUG:boto:Token: None 2023-10-12T00:48:15.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h 2023-10-12T00:48:15.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N3How32rF7Mk6Qo65IkTmW6MBkM= 2023-10-12T00:48:15.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N3How32rF7Mk6Qo65IkTmW6MBkM=', 'Content-Length': '0'} 2023-10-12T00:48:15.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435527d70a01d4cf-006527424f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:15.070 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.070 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.070 DEBUG:boto:Method: GET 2023-10-12T00:48:15.071 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:15.071 DEBUG:boto:Data: 2023-10-12T00:48:15.072 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.073 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.073 DEBUG:boto:Params: {} 2023-10-12T00:48:15.074 DEBUG:boto:Token: None 2023-10-12T00:48:15.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h 2023-10-12T00:48:15.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9cY/A7erQsTAipqEojTeN3LIdE= 2023-10-12T00:48:15.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9cY/A7erQsTAipqEojTeN3LIdE=', 'Content-Length': '0'} 2023-10-12T00:48:15.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a47471c91fe8057-006527424f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.078 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:15.078 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:15.079 DEBUG:boto:Method: GET 2023-10-12T00:48:15.079 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:15.080 DEBUG:boto:Data: 2023-10-12T00:48:15.080 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.081 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.082 DEBUG:boto:Params: {} 2023-10-12T00:48:15.082 DEBUG:boto:Token: None 2023-10-12T00:48:15.083 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/h 2023-10-12T00:48:15.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N9cY/A7erQsTAipqEojTeN3LIdE= 2023-10-12T00:48:15.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N9cY/A7erQsTAipqEojTeN3LIdE=', 'Content-Length': '0'} 2023-10-12T00:48:15.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018665636ebb36b93-006527424f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.087 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:15.087 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:15.088 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.088 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.089 DEBUG:boto:Method: GET 2023-10-12T00:48:15.089 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:15.090 DEBUG:boto:Data: 2023-10-12T00:48:15.090 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:15.091 DEBUG:boto:Port: 8001 2023-10-12T00:48:15.091 DEBUG:boto:Params: {} 2023-10-12T00:48:15.092 DEBUG:boto:Token: None 2023-10-12T00:48:15.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/ 2023-10-12T00:48:15.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs= 2023-10-12T00:48:15.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs=', 'Content-Length': '0'} 2023-10-12T00:48:15.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ebbc3bcb77215df-006527424f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.097 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:15.098 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:15.099 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:15.100 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.100 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:15.101 DEBUG:boto:Method: GET 2023-10-12T00:48:15.101 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:15.102 DEBUG:boto:Data: 2023-10-12T00:48:15.102 DEBUG:boto:Headers: {} 2023-10-12T00:48:15.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:15.103 DEBUG:boto:Port: 8000 2023-10-12T00:48:15.103 DEBUG:boto:Params: {} 2023-10-12T00:48:15.104 DEBUG:boto:Token: None 2023-10-12T00:48:15.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:15 GMT /wafmoa-55/ 2023-10-12T00:48:15.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs= 2023-10-12T00:48:15.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:15 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im6SlFrf13P2YN3KmFDb0CaP2Qs=', 'Content-Length': '0'} 2023-10-12T00:48:15.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388d0b4f058f6f23-006527424f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:15.108 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:15.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:48:15.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'wafmoa-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:15.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'wafmoa-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:15.110 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket wafmoa-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:15.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:15.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:15.204 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:48:15.204 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-55 2023-10-12T00:48:15.205 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 8 2023-10-12T00:48:15.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:15.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:15.234 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:15.281 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:15.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:15.325 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:48:15.325 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.326 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-10-12T00:48:15.326 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-10-12T00:48:15.326 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:15.326 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.327 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-10-12T00:48:15.327 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-10-12T00:48:15.327 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:15.327 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:15.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:15.328 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-10-12T00:48:15.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:15.329 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:48:15.329 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.329 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:15.330 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:15.330 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-10-12T00:48:15.330 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.331 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-10-12T00:48:15.331 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:15.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:15.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:15.333 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:48:15.333 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.334 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:15.334 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:15.334 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:48:15.335 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.335 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:15.335 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:15.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:15.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:15.337 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:15.338 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:15.339 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:15.339 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:15.339 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:15.340 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:15.340 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:15.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:48:15.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:15.342 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:48:15.344 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-55 2023-10-12T00:48:15.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:15.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:15.345 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:48:15.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:15.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:15.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:15.436+0000 7f0beed50780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:48:20.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:20.436+0000 7f0beed50780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:48:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:25.436+0000 7f0beed50780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:48:30.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:30.436+0000 7f0beed50780 1 bucket sync caught up with empty source 2023-10-12T00:48:30.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:30.436+0000 7f0beed50780 0 bucket checkpoint complete 2023-10-12T00:48:30.455 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-55 2023-10-12T00:48:30.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:30.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:30.456 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:48:30.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:30.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:30.556+0000 7fed5f0bd780 1 bucket sync caught up with source: 2023-10-12T00:48:30.557 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:30.558 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:30.556+0000 7fed5f0bd780 0 bucket checkpoint complete 2023-10-12T00:48:30.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:48:30.572 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.573 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.573 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.574 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:30.575 DEBUG:boto:Data: 2023-10-12T00:48:30.576 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.577 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.578 DEBUG:boto:Params: {} 2023-10-12T00:48:30.579 DEBUG:boto:Token: None 2023-10-12T00:48:30.580 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1KjrmIY8g6pnFl7GzTyfdQOGLZo= 2023-10-12T00:48:30.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1KjrmIY8g6pnFl7GzTyfdQOGLZo=', 'Content-Length': '0'} 2023-10-12T00:48:30.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009822ebd558c8d3cf-006527425e-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.585 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.585 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.586 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.586 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:30.587 DEBUG:boto:Data: 2023-10-12T00:48:30.587 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.588 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.588 DEBUG:boto:Params: {} 2023-10-12T00:48:30.589 DEBUG:boto:Token: None 2023-10-12T00:48:30.589 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.590 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1KjrmIY8g6pnFl7GzTyfdQOGLZo= 2023-10-12T00:48:30.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1KjrmIY8g6pnFl7GzTyfdQOGLZo=', 'Content-Length': '0'} 2023-10-12T00:48:30.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005162e641a26b6d82-006527425e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:48:30.593 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.594 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.594 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.595 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.595 DEBUG:boto:Method: GET 2023-10-12T00:48:30.596 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.596 DEBUG:boto:Data: 2023-10-12T00:48:30.597 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.597 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.598 DEBUG:boto:Params: {} 2023-10-12T00:48:30.598 DEBUG:boto:Token: None 2023-10-12T00:48:30.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/?versions 2023-10-12T00:48:30.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo= 2023-10-12T00:48:30.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo=', 'Content-Length': '0'} 2023-10-12T00:48:30.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bfb9f9099651a55-006527425e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.603 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.604 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:30.604 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:30.605 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:30.605 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:30.606 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:30.607 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:30.607 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:30.608 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:48:30.609 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.610 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.610 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.611 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.612 DEBUG:boto:Method: GET 2023-10-12T00:48:30.612 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.613 DEBUG:boto:Data: 2023-10-12T00:48:30.613 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.614 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.614 DEBUG:boto:Params: {} 2023-10-12T00:48:30.615 DEBUG:boto:Token: None 2023-10-12T00:48:30.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/?versions 2023-10-12T00:48:30.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo= 2023-10-12T00:48:30.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo=', 'Content-Length': '0'} 2023-10-12T00:48:30.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c07796c8d4d55ea-006527425e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.619 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.622 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:30.622 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:30.623 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:30.623 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:30.623 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:30.624 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:30.624 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:30.625 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:30.625 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.625 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.626 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.626 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.627 DEBUG:boto:Method: GET 2023-10-12T00:48:30.627 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.628 DEBUG:boto:Data: 2023-10-12T00:48:30.628 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.629 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.629 DEBUG:boto:Params: {} 2023-10-12T00:48:30.630 DEBUG:boto:Token: None 2023-10-12T00:48:30.630 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/?versions 2023-10-12T00:48:30.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo= 2023-10-12T00:48:30.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo=', 'Content-Length': '0'} 2023-10-12T00:48:30.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009794673a41bb2d6f-006527425e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.637 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.638 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.638 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.639 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.639 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.639 DEBUG:boto:Method: GET 2023-10-12T00:48:30.640 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:30.640 DEBUG:boto:Data: 2023-10-12T00:48:30.641 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.641 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.642 DEBUG:boto:Params: {} 2023-10-12T00:48:30.642 DEBUG:boto:Token: None 2023-10-12T00:48:30.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/?versions 2023-10-12T00:48:30.643 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo= 2023-10-12T00:48:30.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Top+S/eSnAqiKph4OW6KSQEZcKo=', 'Content-Length': '0'} 2023-10-12T00:48:30.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a9c2e479d0ab19-006527425e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.646 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:30.648 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.648 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.649 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.649 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.649 DEBUG:boto:Method: GET 2023-10-12T00:48:30.650 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.650 DEBUG:boto:Data: 2023-10-12T00:48:30.651 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.652 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.652 DEBUG:boto:Params: {} 2023-10-12T00:48:30.653 DEBUG:boto:Token: None 2023-10-12T00:48:30.653 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ= 2023-10-12T00:48:30.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ=', 'Content-Length': '0'} 2023-10-12T00:48:30.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed1cddc650017bf-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.698 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.699 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.699 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.700 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.700 DEBUG:boto:Method: GET 2023-10-12T00:48:30.701 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.701 DEBUG:boto:Data: 2023-10-12T00:48:30.702 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.702 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.703 DEBUG:boto:Params: {} 2023-10-12T00:48:30.703 DEBUG:boto:Token: None 2023-10-12T00:48:30.704 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ= 2023-10-12T00:48:30.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ=', 'Content-Length': '0'} 2023-10-12T00:48:30.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd84407775547f1-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.708 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.708 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.708 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.709 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.709 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.710 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.710 DEBUG:boto:Data: 2023-10-12T00:48:30.711 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.711 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.712 DEBUG:boto:Params: {} 2023-10-12T00:48:30.712 DEBUG:boto:Token: None 2023-10-12T00:48:30.713 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LiQZBEu+bdnItwG9H7/936fG848= 2023-10-12T00:48:30.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LiQZBEu+bdnItwG9H7/936fG848=', 'Content-Length': '0'} 2023-10-12T00:48:30.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739a446bdb9cca0a-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.716 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.717 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.717 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.717 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.718 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.718 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.719 DEBUG:boto:Data: 2023-10-12T00:48:30.719 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.720 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.720 DEBUG:boto:Params: {} 2023-10-12T00:48:30.721 DEBUG:boto:Token: None 2023-10-12T00:48:30.721 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LiQZBEu+bdnItwG9H7/936fG848= 2023-10-12T00:48:30.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LiQZBEu+bdnItwG9H7/936fG848=', 'Content-Length': '0'} 2023-10-12T00:48:30.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec2ff146ce5fd19-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:30.725 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.725 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.726 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.726 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.727 DEBUG:boto:Method: GET 2023-10-12T00:48:30.727 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.728 DEBUG:boto:Data: 2023-10-12T00:48:30.728 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.729 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.729 DEBUG:boto:Params: {} 2023-10-12T00:48:30.730 DEBUG:boto:Token: None 2023-10-12T00:48:30.730 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ= 2023-10-12T00:48:30.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ=', 'Content-Length': '0'} 2023-10-12T00:48:30.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff6b5079bf5ec59-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.734 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.734 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.735 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.735 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:30.736 DEBUG:boto:Method: GET 2023-10-12T00:48:30.736 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.736 DEBUG:boto:Data: 2023-10-12T00:48:30.737 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.738 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.738 DEBUG:boto:Params: {} 2023-10-12T00:48:30.739 DEBUG:boto:Token: None 2023-10-12T00:48:30.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:30.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ= 2023-10-12T00:48:30.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BGSuGzPOzZmBtR+ThBHT+bdJolQ=', 'Content-Length': '0'} 2023-10-12T00:48:30.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004451c53997930aa-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.786 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.787 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.787 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.788 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.788 DEBUG:boto:Method: GET 2023-10-12T00:48:30.789 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:30.789 DEBUG:boto:Data: 2023-10-12T00:48:30.790 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.791 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.791 DEBUG:boto:Params: {} 2023-10-12T00:48:30.791 DEBUG:boto:Token: None 2023-10-12T00:48:30.792 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83877af25943162-006527425e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.796 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.797 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.797 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.797 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.798 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.798 DEBUG:boto:Method: GET 2023-10-12T00:48:30.799 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:30.799 DEBUG:boto:Data: 2023-10-12T00:48:30.800 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.800 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.801 DEBUG:boto:Params: {} 2023-10-12T00:48:30.801 DEBUG:boto:Token: None 2023-10-12T00:48:30.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.802 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064a4a63858a863a9-006527425e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.806 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.806 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.807 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.807 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.808 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:30.808 DEBUG:boto:Data: 2023-10-12T00:48:30.808 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.809 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.810 DEBUG:boto:Params: {} 2023-10-12T00:48:30.810 DEBUG:boto:Token: None 2023-10-12T00:48:30.811 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a 2023-10-12T00:48:30.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rSz5wb3/aRcMUGok15D7OxAGPLs= 2023-10-12T00:48:30.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rSz5wb3/aRcMUGok15D7OxAGPLs=', 'Content-Length': '0'} 2023-10-12T00:48:30.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070eb60d9969b0770-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.814 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.814 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.815 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.815 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:30.816 DEBUG:boto:Data: 2023-10-12T00:48:30.816 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.817 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.817 DEBUG:boto:Params: {} 2023-10-12T00:48:30.818 DEBUG:boto:Token: None 2023-10-12T00:48:30.818 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a 2023-10-12T00:48:30.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rSz5wb3/aRcMUGok15D7OxAGPLs= 2023-10-12T00:48:30.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rSz5wb3/aRcMUGok15D7OxAGPLs=', 'Content-Length': '0'} 2023-10-12T00:48:30.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218b735b0e578f02-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:30.822 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.823 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.823 DEBUG:boto:Method: GET 2023-10-12T00:48:30.824 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:30.824 DEBUG:boto:Data: 2023-10-12T00:48:30.824 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.825 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.826 DEBUG:boto:Params: {} 2023-10-12T00:48:30.826 DEBUG:boto:Token: None 2023-10-12T00:48:30.827 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a 2023-10-12T00:48:30.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sGnsMP0mArVdFIC6tqD53dVtHXo= 2023-10-12T00:48:30.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sGnsMP0mArVdFIC6tqD53dVtHXo=', 'Content-Length': '0'} 2023-10-12T00:48:30.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015025487c3ee5e83-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.831 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:30.832 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:30.833 DEBUG:boto:Method: GET 2023-10-12T00:48:30.833 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:30.834 DEBUG:boto:Data: 2023-10-12T00:48:30.834 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.835 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.835 DEBUG:boto:Params: {} 2023-10-12T00:48:30.836 DEBUG:boto:Token: None 2023-10-12T00:48:30.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/a 2023-10-12T00:48:30.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sGnsMP0mArVdFIC6tqD53dVtHXo= 2023-10-12T00:48:30.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sGnsMP0mArVdFIC6tqD53dVtHXo=', 'Content-Length': '0'} 2023-10-12T00:48:30.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531b3440f19bb9a1-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.840 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.840 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.841 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.841 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.842 DEBUG:boto:Method: GET 2023-10-12T00:48:30.842 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:30.843 DEBUG:boto:Data: 2023-10-12T00:48:30.843 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.844 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.844 DEBUG:boto:Params: {} 2023-10-12T00:48:30.845 DEBUG:boto:Token: None 2023-10-12T00:48:30.845 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.846 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a2170294a4b93c-006527425e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.894 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.895 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.896 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.896 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.897 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:30.897 DEBUG:boto:Method: GET 2023-10-12T00:48:30.897 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:30.898 DEBUG:boto:Data: 2023-10-12T00:48:30.898 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.899 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.900 DEBUG:boto:Params: {} 2023-10-12T00:48:30.900 DEBUG:boto:Token: None 2023-10-12T00:48:30.902 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000485c252791a744bc-006527425e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.905 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:30.907 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.907 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.907 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.908 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.908 DEBUG:boto:Method: GET 2023-10-12T00:48:30.909 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.909 DEBUG:boto:Data: 2023-10-12T00:48:30.910 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.910 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.911 DEBUG:boto:Params: {} 2023-10-12T00:48:30.911 DEBUG:boto:Token: None 2023-10-12T00:48:30.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA= 2023-10-12T00:48:30.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA=', 'Content-Length': '0'} 2023-10-12T00:48:30.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d729fb622dc97f2a-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.915 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.916 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.916 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.917 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.917 DEBUG:boto:Method: GET 2023-10-12T00:48:30.918 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.918 DEBUG:boto:Data: 2023-10-12T00:48:30.918 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.919 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.920 DEBUG:boto:Params: {} 2023-10-12T00:48:30.920 DEBUG:boto:Token: None 2023-10-12T00:48:30.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA= 2023-10-12T00:48:30.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA=', 'Content-Length': '0'} 2023-10-12T00:48:30.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051fe2be4c3f0bd3-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.924 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.925 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.925 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.926 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.926 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.927 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.927 DEBUG:boto:Data: 2023-10-12T00:48:30.927 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.928 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.929 DEBUG:boto:Params: {} 2023-10-12T00:48:30.929 DEBUG:boto:Token: None 2023-10-12T00:48:30.929 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wOFoetDb2OgjpRo++yCR5eCy4dY= 2023-10-12T00:48:30.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wOFoetDb2OgjpRo++yCR5eCy4dY=', 'Content-Length': '0'} 2023-10-12T00:48:30.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d07b1aa14256347-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.933 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.933 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.934 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.934 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.934 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.935 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.935 DEBUG:boto:Data: 2023-10-12T00:48:30.936 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.936 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.937 DEBUG:boto:Params: {} 2023-10-12T00:48:30.937 DEBUG:boto:Token: None 2023-10-12T00:48:30.938 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wOFoetDb2OgjpRo++yCR5eCy4dY= 2023-10-12T00:48:30.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wOFoetDb2OgjpRo++yCR5eCy4dY=', 'Content-Length': '0'} 2023-10-12T00:48:30.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915ba92d6c13904b-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:30.941 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.942 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.942 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.943 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.943 DEBUG:boto:Method: GET 2023-10-12T00:48:30.944 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.944 DEBUG:boto:Data: 2023-10-12T00:48:30.945 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.945 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.946 DEBUG:boto:Params: {} 2023-10-12T00:48:30.946 DEBUG:boto:Token: None 2023-10-12T00:48:30.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA= 2023-10-12T00:48:30.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA=', 'Content-Length': '0'} 2023-10-12T00:48:30.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7faf277d8d9daa-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.951 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.951 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.952 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.952 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:30.953 DEBUG:boto:Method: GET 2023-10-12T00:48:30.953 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.953 DEBUG:boto:Data: 2023-10-12T00:48:30.954 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.955 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.955 DEBUG:boto:Params: {} 2023-10-12T00:48:30.956 DEBUG:boto:Token: None 2023-10-12T00:48:30.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:30.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA= 2023-10-12T00:48:30.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dBp7qyVawZICNf0RyyapHrxHzbA=', 'Content-Length': '0'} 2023-10-12T00:48:30.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20628535c6dca8e-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.960 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.960 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.961 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:30.961 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:30.962 DEBUG:boto:Method: GET 2023-10-12T00:48:30.962 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:30.963 DEBUG:boto:Data: 2023-10-12T00:48:30.963 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.964 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.964 DEBUG:boto:Params: {} 2023-10-12T00:48:30.965 DEBUG:boto:Token: None 2023-10-12T00:48:30.965 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f66b896742697d51-006527425e-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.969 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.970 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:30.970 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:30.971 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:30.971 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:30.972 DEBUG:boto:Method: GET 2023-10-12T00:48:30.972 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:30.973 DEBUG:boto:Data: 2023-10-12T00:48:30.973 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.974 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.975 DEBUG:boto:Params: {} 2023-10-12T00:48:30.976 DEBUG:boto:Token: None 2023-10-12T00:48:30.976 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/ 2023-10-12T00:48:30.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0= 2023-10-12T00:48:30.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:anxcH3Lh8u/uIRb+e+QypLp8Le0=', 'Content-Length': '0'} 2023-10-12T00:48:30.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1340a74da67a7c6-006527425e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.980 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:30.981 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.981 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.981 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.982 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:30.982 DEBUG:boto:Data: 2023-10-12T00:48:30.983 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:30.984 DEBUG:boto:Port: 8001 2023-10-12T00:48:30.984 DEBUG:boto:Params: {} 2023-10-12T00:48:30.985 DEBUG:boto:Token: None 2023-10-12T00:48:30.985 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b 2023-10-12T00:48:30.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/ao/R2By/P5lNFRlnNn/i+oD70k= 2023-10-12T00:48:30.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/ao/R2By/P5lNFRlnNn/i+oD70k=', 'Content-Length': '0'} 2023-10-12T00:48:30.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030829d54dc44aea0-006527425e-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.990 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:30.991 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:30.991 DEBUG:boto:Method: HEAD 2023-10-12T00:48:30.992 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:30.992 DEBUG:boto:Data: 2023-10-12T00:48:30.993 DEBUG:boto:Headers: {} 2023-10-12T00:48:30.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:30.994 DEBUG:boto:Port: 8000 2023-10-12T00:48:30.994 DEBUG:boto:Params: {} 2023-10-12T00:48:30.995 DEBUG:boto:Token: None 2023-10-12T00:48:30.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:30 GMT /wafmoa-55/b 2023-10-12T00:48:30.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/ao/R2By/P5lNFRlnNn/i+oD70k= 2023-10-12T00:48:30.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/ao/R2By/P5lNFRlnNn/i+oD70k=', 'Content-Length': '0'} 2023-10-12T00:48:30.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e3cdde32357597-006527425e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:30.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:30.999 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:31.000 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:31.000 DEBUG:boto:Method: GET 2023-10-12T00:48:31.001 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:31.002 DEBUG:boto:Data: 2023-10-12T00:48:31.002 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.003 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.003 DEBUG:boto:Params: {} 2023-10-12T00:48:31.004 DEBUG:boto:Token: None 2023-10-12T00:48:31.004 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/b 2023-10-12T00:48:31.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:37ltGTr40vHKmBO46zbyiKliP9M= 2023-10-12T00:48:31.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:37ltGTr40vHKmBO46zbyiKliP9M=', 'Content-Length': '0'} 2023-10-12T00:48:31.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da90f3195a55a87-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.009 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:31.010 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:31.011 DEBUG:boto:Method: GET 2023-10-12T00:48:31.012 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:31.012 DEBUG:boto:Data: 2023-10-12T00:48:31.012 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.013 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.014 DEBUG:boto:Params: {} 2023-10-12T00:48:31.014 DEBUG:boto:Token: None 2023-10-12T00:48:31.015 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/b 2023-10-12T00:48:31.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:37ltGTr40vHKmBO46zbyiKliP9M= 2023-10-12T00:48:31.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:37ltGTr40vHKmBO46zbyiKliP9M=', 'Content-Length': '0'} 2023-10-12T00:48:31.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004502cd90e201aaa6-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.019 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.019 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.019 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:31.020 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:31.020 DEBUG:boto:Method: GET 2023-10-12T00:48:31.021 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:31.021 DEBUG:boto:Data: 2023-10-12T00:48:31.022 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.022 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.023 DEBUG:boto:Params: {} 2023-10-12T00:48:31.023 DEBUG:boto:Token: None 2023-10-12T00:48:31.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013788d88d1465a39-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.028 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.028 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.029 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.029 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:31.030 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:31.030 DEBUG:boto:Method: GET 2023-10-12T00:48:31.031 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:31.031 DEBUG:boto:Data: 2023-10-12T00:48:31.031 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.032 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.033 DEBUG:boto:Params: {} 2023-10-12T00:48:31.033 DEBUG:boto:Token: None 2023-10-12T00:48:31.034 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.034 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ebd6d36d1e2620-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.037 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:31.039 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.039 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.039 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.040 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.040 DEBUG:boto:Method: GET 2023-10-12T00:48:31.041 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.041 DEBUG:boto:Data: 2023-10-12T00:48:31.042 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.042 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.043 DEBUG:boto:Params: {} 2023-10-12T00:48:31.043 DEBUG:boto:Token: None 2023-10-12T00:48:31.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI= 2023-10-12T00:48:31.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI=', 'Content-Length': '0'} 2023-10-12T00:48:31.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f9fcc5c69cbb6d-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.094 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.096 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.096 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.097 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.097 DEBUG:boto:Method: GET 2023-10-12T00:48:31.098 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.098 DEBUG:boto:Data: 2023-10-12T00:48:31.098 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.099 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.100 DEBUG:boto:Params: {} 2023-10-12T00:48:31.100 DEBUG:boto:Token: None 2023-10-12T00:48:31.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI= 2023-10-12T00:48:31.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI=', 'Content-Length': '0'} 2023-10-12T00:48:31.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1d156f3862ab61-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.104 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.105 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.105 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.106 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.106 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.106 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.107 DEBUG:boto:Data: 2023-10-12T00:48:31.107 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.108 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.108 DEBUG:boto:Params: {} 2023-10-12T00:48:31.109 DEBUG:boto:Token: None 2023-10-12T00:48:31.109 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJcmrGnuKMEuyY8DQYnziH8kh4E= 2023-10-12T00:48:31.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJcmrGnuKMEuyY8DQYnziH8kh4E=', 'Content-Length': '0'} 2023-10-12T00:48:31.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219f738dd039752f-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.113 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.113 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.114 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.114 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.115 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.115 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.115 DEBUG:boto:Data: 2023-10-12T00:48:31.116 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.117 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.117 DEBUG:boto:Params: {} 2023-10-12T00:48:31.118 DEBUG:boto:Token: None 2023-10-12T00:48:31.118 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJcmrGnuKMEuyY8DQYnziH8kh4E= 2023-10-12T00:48:31.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJcmrGnuKMEuyY8DQYnziH8kh4E=', 'Content-Length': '0'} 2023-10-12T00:48:31.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009072942d62f47d5c-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:31.122 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.122 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.123 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.123 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.124 DEBUG:boto:Method: GET 2023-10-12T00:48:31.124 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.125 DEBUG:boto:Data: 2023-10-12T00:48:31.125 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.126 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.126 DEBUG:boto:Params: {} 2023-10-12T00:48:31.127 DEBUG:boto:Token: None 2023-10-12T00:48:31.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI= 2023-10-12T00:48:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI=', 'Content-Length': '0'} 2023-10-12T00:48:31.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008623e9a1a6f18251-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.131 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.131 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.132 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.132 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:31.133 DEBUG:boto:Method: GET 2023-10-12T00:48:31.133 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.134 DEBUG:boto:Data: 2023-10-12T00:48:31.134 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.135 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.135 DEBUG:boto:Params: {} 2023-10-12T00:48:31.136 DEBUG:boto:Token: None 2023-10-12T00:48:31.136 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:31.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI= 2023-10-12T00:48:31.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a3/cJI6U30ExMu0WQ5j+kwJ+4BI=', 'Content-Length': '0'} 2023-10-12T00:48:31.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008faecc29fb02e110-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.140 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.141 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.141 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.142 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.142 DEBUG:boto:Method: GET 2023-10-12T00:48:31.143 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:31.143 DEBUG:boto:Data: 2023-10-12T00:48:31.144 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.144 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.145 DEBUG:boto:Params: {} 2023-10-12T00:48:31.145 DEBUG:boto:Token: None 2023-10-12T00:48:31.146 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab9b82e62f0bea8-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.149 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.150 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.151 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.151 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.152 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.152 DEBUG:boto:Method: GET 2023-10-12T00:48:31.152 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:31.153 DEBUG:boto:Data: 2023-10-12T00:48:31.153 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.155 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.155 DEBUG:boto:Params: {} 2023-10-12T00:48:31.156 DEBUG:boto:Token: None 2023-10-12T00:48:31.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007967c5c33233f9c9-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.161 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.161 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.162 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.162 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.163 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:31.163 DEBUG:boto:Data: 2023-10-12T00:48:31.164 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.164 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.165 DEBUG:boto:Params: {} 2023-10-12T00:48:31.165 DEBUG:boto:Token: None 2023-10-12T00:48:31.166 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c 2023-10-12T00:48:31.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D/3VmL0VrN+WP75H8hkd2DcSNf8= 2023-10-12T00:48:31.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D/3VmL0VrN+WP75H8hkd2DcSNf8=', 'Content-Length': '0'} 2023-10-12T00:48:31.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad8e93fe211e8ef-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.169 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.169 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.170 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.170 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:31.171 DEBUG:boto:Data: 2023-10-12T00:48:31.171 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.172 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.173 DEBUG:boto:Params: {} 2023-10-12T00:48:31.173 DEBUG:boto:Token: None 2023-10-12T00:48:31.174 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c 2023-10-12T00:48:31.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D/3VmL0VrN+WP75H8hkd2DcSNf8= 2023-10-12T00:48:31.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D/3VmL0VrN+WP75H8hkd2DcSNf8=', 'Content-Length': '0'} 2023-10-12T00:48:31.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64424238e346b80-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:31.177 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.178 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.178 DEBUG:boto:Method: GET 2023-10-12T00:48:31.179 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:31.179 DEBUG:boto:Data: 2023-10-12T00:48:31.180 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.180 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.181 DEBUG:boto:Params: {} 2023-10-12T00:48:31.181 DEBUG:boto:Token: None 2023-10-12T00:48:31.182 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c 2023-10-12T00:48:31.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vqrTz0LAjyuqHuvv+hjmz9Rc8o= 2023-10-12T00:48:31.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vqrTz0LAjyuqHuvv+hjmz9Rc8o=', 'Content-Length': '0'} 2023-10-12T00:48:31.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037945c0c7bfdbd6b-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.185 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:31.186 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:31.186 DEBUG:boto:Method: GET 2023-10-12T00:48:31.187 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:31.187 DEBUG:boto:Data: 2023-10-12T00:48:31.188 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.189 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.189 DEBUG:boto:Params: {} 2023-10-12T00:48:31.190 DEBUG:boto:Token: None 2023-10-12T00:48:31.190 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/c 2023-10-12T00:48:31.191 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5vqrTz0LAjyuqHuvv+hjmz9Rc8o= 2023-10-12T00:48:31.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5vqrTz0LAjyuqHuvv+hjmz9Rc8o=', 'Content-Length': '0'} 2023-10-12T00:48:31.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f04f592414863c-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.194 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.194 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.195 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.195 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.196 DEBUG:boto:Method: GET 2023-10-12T00:48:31.196 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:31.197 DEBUG:boto:Data: 2023-10-12T00:48:31.197 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.198 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.198 DEBUG:boto:Params: {} 2023-10-12T00:48:31.199 DEBUG:boto:Token: None 2023-10-12T00:48:31.200 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3012bd1d7c6ec19-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.203 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.204 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.205 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.205 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.205 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:31.206 DEBUG:boto:Method: GET 2023-10-12T00:48:31.206 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:31.207 DEBUG:boto:Data: 2023-10-12T00:48:31.207 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.208 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.209 DEBUG:boto:Params: {} 2023-10-12T00:48:31.209 DEBUG:boto:Token: None 2023-10-12T00:48:31.210 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caea347a0090f9e4-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.214 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:31.215 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.215 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.216 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.216 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.217 DEBUG:boto:Method: GET 2023-10-12T00:48:31.217 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.218 DEBUG:boto:Data: 2023-10-12T00:48:31.218 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.219 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.219 DEBUG:boto:Params: {} 2023-10-12T00:48:31.220 DEBUG:boto:Token: None 2023-10-12T00:48:31.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg= 2023-10-12T00:48:31.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg=', 'Content-Length': '0'} 2023-10-12T00:48:31.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9f564b3b37f148-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.266 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.267 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.267 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.268 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.268 DEBUG:boto:Method: GET 2023-10-12T00:48:31.269 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.269 DEBUG:boto:Data: 2023-10-12T00:48:31.269 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.270 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.271 DEBUG:boto:Params: {} 2023-10-12T00:48:31.271 DEBUG:boto:Token: None 2023-10-12T00:48:31.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg= 2023-10-12T00:48:31.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg=', 'Content-Length': '0'} 2023-10-12T00:48:31.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3297c072478c7b2-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.275 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.276 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.276 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.276 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.277 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.277 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.278 DEBUG:boto:Data: 2023-10-12T00:48:31.278 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.279 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.279 DEBUG:boto:Params: {} 2023-10-12T00:48:31.280 DEBUG:boto:Token: None 2023-10-12T00:48:31.280 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k2IBQUm9Wp+x6xnxC7S3vj0bzGg= 2023-10-12T00:48:31.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k2IBQUm9Wp+x6xnxC7S3vj0bzGg=', 'Content-Length': '0'} 2023-10-12T00:48:31.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3ce8d454c276ba-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.284 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.284 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.285 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.285 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.286 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.286 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.287 DEBUG:boto:Data: 2023-10-12T00:48:31.287 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.288 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.288 DEBUG:boto:Params: {} 2023-10-12T00:48:31.289 DEBUG:boto:Token: None 2023-10-12T00:48:31.290 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k2IBQUm9Wp+x6xnxC7S3vj0bzGg= 2023-10-12T00:48:31.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k2IBQUm9Wp+x6xnxC7S3vj0bzGg=', 'Content-Length': '0'} 2023-10-12T00:48:31.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b1f7391ef76352-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:31.294 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.295 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.295 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.296 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.296 DEBUG:boto:Method: GET 2023-10-12T00:48:31.297 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.297 DEBUG:boto:Data: 2023-10-12T00:48:31.297 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.298 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.299 DEBUG:boto:Params: {} 2023-10-12T00:48:31.299 DEBUG:boto:Token: None 2023-10-12T00:48:31.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg= 2023-10-12T00:48:31.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg=', 'Content-Length': '0'} 2023-10-12T00:48:31.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca28334d01e29b1b-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.304 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.304 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.305 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.306 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:31.306 DEBUG:boto:Method: GET 2023-10-12T00:48:31.307 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.307 DEBUG:boto:Data: 2023-10-12T00:48:31.308 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.309 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.309 DEBUG:boto:Params: {} 2023-10-12T00:48:31.310 DEBUG:boto:Token: None 2023-10-12T00:48:31.310 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:31.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg= 2023-10-12T00:48:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hHETyla/o/eUHyT6Xwevbe9MHZg=', 'Content-Length': '0'} 2023-10-12T00:48:31.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698c91fbf157db5a-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.354 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.355 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.355 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.356 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.356 DEBUG:boto:Method: GET 2023-10-12T00:48:31.357 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:31.357 DEBUG:boto:Data: 2023-10-12T00:48:31.358 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.359 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.359 DEBUG:boto:Params: {} 2023-10-12T00:48:31.360 DEBUG:boto:Token: None 2023-10-12T00:48:31.360 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd6e7d6acc478a5-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.364 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.365 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.365 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.366 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.366 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.367 DEBUG:boto:Method: GET 2023-10-12T00:48:31.367 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:31.368 DEBUG:boto:Data: 2023-10-12T00:48:31.368 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.369 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.370 DEBUG:boto:Params: {} 2023-10-12T00:48:31.370 DEBUG:boto:Token: None 2023-10-12T00:48:31.371 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd72b48e95907fea-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.374 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.375 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.376 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.376 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.376 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:31.377 DEBUG:boto:Data: 2023-10-12T00:48:31.377 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.378 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.379 DEBUG:boto:Params: {} 2023-10-12T00:48:31.379 DEBUG:boto:Token: None 2023-10-12T00:48:31.380 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d 2023-10-12T00:48:31.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IOvbWydl3/TMqUEaxL1OdQm9fMY= 2023-10-12T00:48:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IOvbWydl3/TMqUEaxL1OdQm9fMY=', 'Content-Length': '0'} 2023-10-12T00:48:31.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001639248baa388cea-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.383 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.384 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.384 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.385 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:31.385 DEBUG:boto:Data: 2023-10-12T00:48:31.385 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.386 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.387 DEBUG:boto:Params: {} 2023-10-12T00:48:31.388 DEBUG:boto:Token: None 2023-10-12T00:48:31.388 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d 2023-10-12T00:48:31.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IOvbWydl3/TMqUEaxL1OdQm9fMY= 2023-10-12T00:48:31.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IOvbWydl3/TMqUEaxL1OdQm9fMY=', 'Content-Length': '0'} 2023-10-12T00:48:31.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000105d6c2a895b5638-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:31.392 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.393 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.393 DEBUG:boto:Method: GET 2023-10-12T00:48:31.394 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:31.394 DEBUG:boto:Data: 2023-10-12T00:48:31.395 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.396 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.396 DEBUG:boto:Params: {} 2023-10-12T00:48:31.397 DEBUG:boto:Token: None 2023-10-12T00:48:31.398 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d 2023-10-12T00:48:31.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9z2Rs3gQNEC+qNKe0xCqY+1lf4Q= 2023-10-12T00:48:31.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9z2Rs3gQNEC+qNKe0xCqY+1lf4Q=', 'Content-Length': '0'} 2023-10-12T00:48:31.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5dd991db76c0888-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.442 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:31.443 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:31.443 DEBUG:boto:Method: GET 2023-10-12T00:48:31.444 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:31.444 DEBUG:boto:Data: 2023-10-12T00:48:31.445 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.446 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.446 DEBUG:boto:Params: {} 2023-10-12T00:48:31.447 DEBUG:boto:Token: None 2023-10-12T00:48:31.447 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/d 2023-10-12T00:48:31.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9z2Rs3gQNEC+qNKe0xCqY+1lf4Q= 2023-10-12T00:48:31.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9z2Rs3gQNEC+qNKe0xCqY+1lf4Q=', 'Content-Length': '0'} 2023-10-12T00:48:31.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271fc33190116c3c-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.451 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.452 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.452 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.452 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.453 DEBUG:boto:Method: GET 2023-10-12T00:48:31.453 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:31.454 DEBUG:boto:Data: 2023-10-12T00:48:31.454 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.455 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.456 DEBUG:boto:Params: {} 2023-10-12T00:48:31.456 DEBUG:boto:Token: None 2023-10-12T00:48:31.457 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e1c5db89829d78c-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.460 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.461 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.462 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.462 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.463 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:31.463 DEBUG:boto:Method: GET 2023-10-12T00:48:31.463 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:31.464 DEBUG:boto:Data: 2023-10-12T00:48:31.464 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.465 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.466 DEBUG:boto:Params: {} 2023-10-12T00:48:31.466 DEBUG:boto:Token: None 2023-10-12T00:48:31.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6acca379970994a-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.470 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:31.471 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.472 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.472 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.472 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.473 DEBUG:boto:Method: GET 2023-10-12T00:48:31.473 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.474 DEBUG:boto:Data: 2023-10-12T00:48:31.474 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.475 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.475 DEBUG:boto:Params: {} 2023-10-12T00:48:31.476 DEBUG:boto:Token: None 2023-10-12T00:48:31.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8= 2023-10-12T00:48:31.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8=', 'Content-Length': '0'} 2023-10-12T00:48:31.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f319fe0c9332ed-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.480 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.481 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.481 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.482 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.482 DEBUG:boto:Method: GET 2023-10-12T00:48:31.483 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.483 DEBUG:boto:Data: 2023-10-12T00:48:31.483 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.484 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.485 DEBUG:boto:Params: {} 2023-10-12T00:48:31.485 DEBUG:boto:Token: None 2023-10-12T00:48:31.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8= 2023-10-12T00:48:31.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8=', 'Content-Length': '0'} 2023-10-12T00:48:31.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c28bd1bbc8c0eabd-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.489 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.490 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.490 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.491 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.491 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.492 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.492 DEBUG:boto:Data: 2023-10-12T00:48:31.492 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.493 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.494 DEBUG:boto:Params: {} 2023-10-12T00:48:31.494 DEBUG:boto:Token: None 2023-10-12T00:48:31.495 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+0DcAlOM6itMoBaxniVxKQq7sA= 2023-10-12T00:48:31.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+0DcAlOM6itMoBaxniVxKQq7sA=', 'Content-Length': '0'} 2023-10-12T00:48:31.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f44ada986b83064c-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.498 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.499 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.499 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.499 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.500 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.500 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.501 DEBUG:boto:Data: 2023-10-12T00:48:31.501 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.502 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.502 DEBUG:boto:Params: {} 2023-10-12T00:48:31.503 DEBUG:boto:Token: None 2023-10-12T00:48:31.503 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+0DcAlOM6itMoBaxniVxKQq7sA= 2023-10-12T00:48:31.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+0DcAlOM6itMoBaxniVxKQq7sA=', 'Content-Length': '0'} 2023-10-12T00:48:31.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e04a1026bbd2f70-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:31.507 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.507 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.508 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.508 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.509 DEBUG:boto:Method: GET 2023-10-12T00:48:31.509 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.510 DEBUG:boto:Data: 2023-10-12T00:48:31.510 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.511 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.512 DEBUG:boto:Params: {} 2023-10-12T00:48:31.512 DEBUG:boto:Token: None 2023-10-12T00:48:31.513 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8= 2023-10-12T00:48:31.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8=', 'Content-Length': '0'} 2023-10-12T00:48:31.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b8023b8856d496-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.517 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.518 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.518 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.519 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:31.519 DEBUG:boto:Method: GET 2023-10-12T00:48:31.520 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.520 DEBUG:boto:Data: 2023-10-12T00:48:31.520 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.521 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.522 DEBUG:boto:Params: {} 2023-10-12T00:48:31.522 DEBUG:boto:Token: None 2023-10-12T00:48:31.523 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:31.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8= 2023-10-12T00:48:31.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSBC4Wp6mLfjWN+XvhoAvLmSDB8=', 'Content-Length': '0'} 2023-10-12T00:48:31.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959d5661c2b18356-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.527 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.527 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.528 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.528 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.528 DEBUG:boto:Method: GET 2023-10-12T00:48:31.529 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:31.529 DEBUG:boto:Data: 2023-10-12T00:48:31.530 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.531 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.531 DEBUG:boto:Params: {} 2023-10-12T00:48:31.532 DEBUG:boto:Token: None 2023-10-12T00:48:31.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a280a1424bb616f3-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.536 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.537 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.537 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.538 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.538 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.538 DEBUG:boto:Method: GET 2023-10-12T00:48:31.539 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:31.539 DEBUG:boto:Data: 2023-10-12T00:48:31.540 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.540 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.541 DEBUG:boto:Params: {} 2023-10-12T00:48:31.541 DEBUG:boto:Token: None 2023-10-12T00:48:31.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d50b72327141b8-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.545 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.546 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.547 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.547 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.547 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:31.548 DEBUG:boto:Data: 2023-10-12T00:48:31.548 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.549 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.549 DEBUG:boto:Params: {} 2023-10-12T00:48:31.550 DEBUG:boto:Token: None 2023-10-12T00:48:31.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e 2023-10-12T00:48:31.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UtCOoYCjlmS0GMsYC2LFxQZp8UI= 2023-10-12T00:48:31.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UtCOoYCjlmS0GMsYC2LFxQZp8UI=', 'Content-Length': '0'} 2023-10-12T00:48:31.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9b094bd094ff1f-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.553 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.554 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.554 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.555 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:31.555 DEBUG:boto:Data: 2023-10-12T00:48:31.555 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.556 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.557 DEBUG:boto:Params: {} 2023-10-12T00:48:31.557 DEBUG:boto:Token: None 2023-10-12T00:48:31.558 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e 2023-10-12T00:48:31.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UtCOoYCjlmS0GMsYC2LFxQZp8UI= 2023-10-12T00:48:31.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UtCOoYCjlmS0GMsYC2LFxQZp8UI=', 'Content-Length': '0'} 2023-10-12T00:48:31.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589cea3e693b130e-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:31.561 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.562 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.563 DEBUG:boto:Method: GET 2023-10-12T00:48:31.563 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:31.564 DEBUG:boto:Data: 2023-10-12T00:48:31.564 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.565 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.566 DEBUG:boto:Params: {} 2023-10-12T00:48:31.566 DEBUG:boto:Token: None 2023-10-12T00:48:31.567 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e 2023-10-12T00:48:31.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wGnA7ys7frrB5UHstkDvxq2d0B8= 2023-10-12T00:48:31.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wGnA7ys7frrB5UHstkDvxq2d0B8=', 'Content-Length': '0'} 2023-10-12T00:48:31.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360e155354fc9f61-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.571 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:31.572 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:31.573 DEBUG:boto:Method: GET 2023-10-12T00:48:31.574 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:31.574 DEBUG:boto:Data: 2023-10-12T00:48:31.575 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.576 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.576 DEBUG:boto:Params: {} 2023-10-12T00:48:31.577 DEBUG:boto:Token: None 2023-10-12T00:48:31.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/e 2023-10-12T00:48:31.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wGnA7ys7frrB5UHstkDvxq2d0B8= 2023-10-12T00:48:31.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wGnA7ys7frrB5UHstkDvxq2d0B8=', 'Content-Length': '0'} 2023-10-12T00:48:31.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859fb06ad771d2e3-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.581 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.582 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.582 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.582 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.583 DEBUG:boto:Method: GET 2023-10-12T00:48:31.583 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:31.584 DEBUG:boto:Data: 2023-10-12T00:48:31.584 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.585 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.585 DEBUG:boto:Params: {} 2023-10-12T00:48:31.586 DEBUG:boto:Token: None 2023-10-12T00:48:31.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e94330b1cf5aa935-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.590 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.591 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.591 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.592 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.592 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:31.593 DEBUG:boto:Method: GET 2023-10-12T00:48:31.593 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:31.594 DEBUG:boto:Data: 2023-10-12T00:48:31.594 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.595 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.595 DEBUG:boto:Params: {} 2023-10-12T00:48:31.596 DEBUG:boto:Token: None 2023-10-12T00:48:31.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d8922ba567ce3c-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.600 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:31.601 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.602 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.602 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.602 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.603 DEBUG:boto:Method: GET 2023-10-12T00:48:31.603 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.604 DEBUG:boto:Data: 2023-10-12T00:48:31.604 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.605 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.605 DEBUG:boto:Params: {} 2023-10-12T00:48:31.606 DEBUG:boto:Token: None 2023-10-12T00:48:31.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs= 2023-10-12T00:48:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs=', 'Content-Length': '0'} 2023-10-12T00:48:31.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ad11f7d63c3037-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.610 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.611 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.611 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.611 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.612 DEBUG:boto:Method: GET 2023-10-12T00:48:31.612 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.613 DEBUG:boto:Data: 2023-10-12T00:48:31.613 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.614 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.615 DEBUG:boto:Params: {} 2023-10-12T00:48:31.615 DEBUG:boto:Token: None 2023-10-12T00:48:31.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs= 2023-10-12T00:48:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs=', 'Content-Length': '0'} 2023-10-12T00:48:31.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b5ccbe8e5364f4-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.619 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.619 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.620 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.620 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.621 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.621 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.621 DEBUG:boto:Data: 2023-10-12T00:48:31.622 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.623 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.623 DEBUG:boto:Params: {} 2023-10-12T00:48:31.624 DEBUG:boto:Token: None 2023-10-12T00:48:31.624 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I/NLTk3gyKw0EMKTA1OXGfhCn74= 2023-10-12T00:48:31.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I/NLTk3gyKw0EMKTA1OXGfhCn74=', 'Content-Length': '0'} 2023-10-12T00:48:31.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5462cb22f2bbec3-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.627 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.628 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.628 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.629 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.629 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.630 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.630 DEBUG:boto:Data: 2023-10-12T00:48:31.630 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.631 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.632 DEBUG:boto:Params: {} 2023-10-12T00:48:31.632 DEBUG:boto:Token: None 2023-10-12T00:48:31.633 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I/NLTk3gyKw0EMKTA1OXGfhCn74= 2023-10-12T00:48:31.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I/NLTk3gyKw0EMKTA1OXGfhCn74=', 'Content-Length': '0'} 2023-10-12T00:48:31.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8c5365b93e61fa-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:31.636 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.637 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.637 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.638 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.638 DEBUG:boto:Method: GET 2023-10-12T00:48:31.639 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.639 DEBUG:boto:Data: 2023-10-12T00:48:31.639 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.640 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.641 DEBUG:boto:Params: {} 2023-10-12T00:48:31.641 DEBUG:boto:Token: None 2023-10-12T00:48:31.642 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.642 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs= 2023-10-12T00:48:31.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs=', 'Content-Length': '0'} 2023-10-12T00:48:31.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d3b8b644bb9307-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.690 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.691 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.691 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.692 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:31.692 DEBUG:boto:Method: GET 2023-10-12T00:48:31.693 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.693 DEBUG:boto:Data: 2023-10-12T00:48:31.694 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.695 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.695 DEBUG:boto:Params: {} 2023-10-12T00:48:31.695 DEBUG:boto:Token: None 2023-10-12T00:48:31.696 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:31.696 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs= 2023-10-12T00:48:31.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hYZZXApkEhl7kqpukZZewu9+OGs=', 'Content-Length': '0'} 2023-10-12T00:48:31.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8c61ffd4baa599-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.699 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.700 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.700 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.701 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.701 DEBUG:boto:Method: GET 2023-10-12T00:48:31.702 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:31.702 DEBUG:boto:Data: 2023-10-12T00:48:31.703 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.703 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.704 DEBUG:boto:Params: {} 2023-10-12T00:48:31.704 DEBUG:boto:Token: None 2023-10-12T00:48:31.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.705 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c64525a8b26899e-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.709 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.710 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.710 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.710 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.711 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.711 DEBUG:boto:Method: GET 2023-10-12T00:48:31.712 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:31.712 DEBUG:boto:Data: 2023-10-12T00:48:31.713 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.713 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.714 DEBUG:boto:Params: {} 2023-10-12T00:48:31.714 DEBUG:boto:Token: None 2023-10-12T00:48:31.715 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b1916822bea526-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.718 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.719 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.719 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.720 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.720 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:31.721 DEBUG:boto:Data: 2023-10-12T00:48:31.721 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.722 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.722 DEBUG:boto:Params: {} 2023-10-12T00:48:31.723 DEBUG:boto:Token: None 2023-10-12T00:48:31.723 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f 2023-10-12T00:48:31.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9A/dWg6jpQMgNW01wAsgHn89wI= 2023-10-12T00:48:31.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9A/dWg6jpQMgNW01wAsgHn89wI=', 'Content-Length': '0'} 2023-10-12T00:48:31.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead5384e7d763534-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.727 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.727 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.728 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.728 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:31.728 DEBUG:boto:Data: 2023-10-12T00:48:31.729 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.730 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.730 DEBUG:boto:Params: {} 2023-10-12T00:48:31.730 DEBUG:boto:Token: None 2023-10-12T00:48:31.731 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f 2023-10-12T00:48:31.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9A/dWg6jpQMgNW01wAsgHn89wI= 2023-10-12T00:48:31.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9A/dWg6jpQMgNW01wAsgHn89wI=', 'Content-Length': '0'} 2023-10-12T00:48:31.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e3cbdf0d2f7b72-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:31.735 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.735 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.736 DEBUG:boto:Method: GET 2023-10-12T00:48:31.736 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:31.737 DEBUG:boto:Data: 2023-10-12T00:48:31.737 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.738 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.738 DEBUG:boto:Params: {} 2023-10-12T00:48:31.739 DEBUG:boto:Token: None 2023-10-12T00:48:31.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f 2023-10-12T00:48:31.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PQLty+mTy3C83hAtpNLJoORf8bw= 2023-10-12T00:48:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PQLty+mTy3C83hAtpNLJoORf8bw=', 'Content-Length': '0'} 2023-10-12T00:48:31.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d7129b72b25425-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.743 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:31.743 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:31.744 DEBUG:boto:Method: GET 2023-10-12T00:48:31.744 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:31.745 DEBUG:boto:Data: 2023-10-12T00:48:31.745 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.746 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.746 DEBUG:boto:Params: {} 2023-10-12T00:48:31.747 DEBUG:boto:Token: None 2023-10-12T00:48:31.747 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/f 2023-10-12T00:48:31.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PQLty+mTy3C83hAtpNLJoORf8bw= 2023-10-12T00:48:31.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PQLty+mTy3C83hAtpNLJoORf8bw=', 'Content-Length': '0'} 2023-10-12T00:48:31.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000171cc43066b5e0e0-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.751 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.751 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.752 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.752 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.753 DEBUG:boto:Method: GET 2023-10-12T00:48:31.753 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:31.753 DEBUG:boto:Data: 2023-10-12T00:48:31.754 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.755 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.755 DEBUG:boto:Params: {} 2023-10-12T00:48:31.756 DEBUG:boto:Token: None 2023-10-12T00:48:31.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a07eda1551f8bd-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.760 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.761 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.762 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.762 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.762 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:31.763 DEBUG:boto:Method: GET 2023-10-12T00:48:31.763 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:31.764 DEBUG:boto:Data: 2023-10-12T00:48:31.764 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.765 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.765 DEBUG:boto:Params: {} 2023-10-12T00:48:31.766 DEBUG:boto:Token: None 2023-10-12T00:48:31.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000758797785bb63321-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.770 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:31.771 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.772 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.772 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.773 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.773 DEBUG:boto:Method: GET 2023-10-12T00:48:31.774 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.774 DEBUG:boto:Data: 2023-10-12T00:48:31.774 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.775 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.776 DEBUG:boto:Params: {} 2023-10-12T00:48:31.776 DEBUG:boto:Token: None 2023-10-12T00:48:31.777 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c= 2023-10-12T00:48:31.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c=', 'Content-Length': '0'} 2023-10-12T00:48:31.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045be0fb9eafe07e6-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.781 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.781 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.782 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.782 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.783 DEBUG:boto:Method: GET 2023-10-12T00:48:31.783 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.783 DEBUG:boto:Data: 2023-10-12T00:48:31.784 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.785 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.785 DEBUG:boto:Params: {} 2023-10-12T00:48:31.786 DEBUG:boto:Token: None 2023-10-12T00:48:31.786 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c= 2023-10-12T00:48:31.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c=', 'Content-Length': '0'} 2023-10-12T00:48:31.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7467f6e4d99cebf-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.790 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.790 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.791 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.791 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.792 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.792 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.793 DEBUG:boto:Data: 2023-10-12T00:48:31.793 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.794 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.794 DEBUG:boto:Params: {} 2023-10-12T00:48:31.795 DEBUG:boto:Token: None 2023-10-12T00:48:31.795 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.796 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JwoCo28eVi+oqVf3Wy72oQ9r8r4= 2023-10-12T00:48:31.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JwoCo28eVi+oqVf3Wy72oQ9r8r4=', 'Content-Length': '0'} 2023-10-12T00:48:31.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f9b06a3773f332-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.798 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.799 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.799 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.800 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.800 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.801 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.801 DEBUG:boto:Data: 2023-10-12T00:48:31.801 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.802 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.803 DEBUG:boto:Params: {} 2023-10-12T00:48:31.803 DEBUG:boto:Token: None 2023-10-12T00:48:31.804 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JwoCo28eVi+oqVf3Wy72oQ9r8r4= 2023-10-12T00:48:31.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JwoCo28eVi+oqVf3Wy72oQ9r8r4=', 'Content-Length': '0'} 2023-10-12T00:48:31.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f29024f9e2eb8e-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:31.807 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.808 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.808 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.808 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.809 DEBUG:boto:Method: GET 2023-10-12T00:48:31.809 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.810 DEBUG:boto:Data: 2023-10-12T00:48:31.810 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.811 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.811 DEBUG:boto:Params: {} 2023-10-12T00:48:31.812 DEBUG:boto:Token: None 2023-10-12T00:48:31.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c= 2023-10-12T00:48:31.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c=', 'Content-Length': '0'} 2023-10-12T00:48:31.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9f1586ce00ff17-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.816 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.816 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.817 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.817 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:31.818 DEBUG:boto:Method: GET 2023-10-12T00:48:31.818 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.819 DEBUG:boto:Data: 2023-10-12T00:48:31.819 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.820 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.820 DEBUG:boto:Params: {} 2023-10-12T00:48:31.821 DEBUG:boto:Token: None 2023-10-12T00:48:31.821 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:31.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c= 2023-10-12T00:48:31.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FsBPjFmxToj85SJtsn4JO2sA45c=', 'Content-Length': '0'} 2023-10-12T00:48:31.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005378b8cce777ae79-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.825 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.825 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.826 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.826 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.827 DEBUG:boto:Method: GET 2023-10-12T00:48:31.827 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:31.828 DEBUG:boto:Data: 2023-10-12T00:48:31.828 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.829 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.829 DEBUG:boto:Params: {} 2023-10-12T00:48:31.830 DEBUG:boto:Token: None 2023-10-12T00:48:31.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695c57ca359f72b4-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.834 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.835 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.835 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.836 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.836 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.837 DEBUG:boto:Method: GET 2023-10-12T00:48:31.837 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:31.838 DEBUG:boto:Data: 2023-10-12T00:48:31.838 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.839 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.839 DEBUG:boto:Params: {} 2023-10-12T00:48:31.840 DEBUG:boto:Token: None 2023-10-12T00:48:31.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003e92fc274d89b62-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.844 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.845 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.845 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.845 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.846 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:31.846 DEBUG:boto:Data: 2023-10-12T00:48:31.847 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.848 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.848 DEBUG:boto:Params: {} 2023-10-12T00:48:31.848 DEBUG:boto:Token: None 2023-10-12T00:48:31.849 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g 2023-10-12T00:48:31.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F91obIs7JN7YpYBje1adTH/uB8w= 2023-10-12T00:48:31.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F91obIs7JN7YpYBje1adTH/uB8w=', 'Content-Length': '0'} 2023-10-12T00:48:31.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e5638d384138f8-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.854 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.855 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.855 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.855 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:31.856 DEBUG:boto:Data: 2023-10-12T00:48:31.856 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.857 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.858 DEBUG:boto:Params: {} 2023-10-12T00:48:31.858 DEBUG:boto:Token: None 2023-10-12T00:48:31.858 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g 2023-10-12T00:48:31.859 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F91obIs7JN7YpYBje1adTH/uB8w= 2023-10-12T00:48:31.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F91obIs7JN7YpYBje1adTH/uB8w=', 'Content-Length': '0'} 2023-10-12T00:48:31.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630378ede5e42af4-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:31.862 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.863 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.863 DEBUG:boto:Method: GET 2023-10-12T00:48:31.864 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:31.864 DEBUG:boto:Data: 2023-10-12T00:48:31.865 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.865 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.866 DEBUG:boto:Params: {} 2023-10-12T00:48:31.866 DEBUG:boto:Token: None 2023-10-12T00:48:31.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g 2023-10-12T00:48:31.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3uHEhavJbMW2oUfTxhlMHeyIa5k= 2023-10-12T00:48:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3uHEhavJbMW2oUfTxhlMHeyIa5k=', 'Content-Length': '0'} 2023-10-12T00:48:31.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008773b06d5abb1cfc-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.870 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:31.871 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:31.871 DEBUG:boto:Method: GET 2023-10-12T00:48:31.872 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:48:31.872 DEBUG:boto:Data: 2023-10-12T00:48:31.873 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.874 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.874 DEBUG:boto:Params: {} 2023-10-12T00:48:31.875 DEBUG:boto:Token: None 2023-10-12T00:48:31.875 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/g 2023-10-12T00:48:31.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3uHEhavJbMW2oUfTxhlMHeyIa5k= 2023-10-12T00:48:31.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3uHEhavJbMW2oUfTxhlMHeyIa5k=', 'Content-Length': '0'} 2023-10-12T00:48:31.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089e69bc0de60ed5-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.879 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.879 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.879 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.880 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.880 DEBUG:boto:Method: GET 2023-10-12T00:48:31.881 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:31.881 DEBUG:boto:Data: 2023-10-12T00:48:31.882 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.883 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.883 DEBUG:boto:Params: {} 2023-10-12T00:48:31.884 DEBUG:boto:Token: None 2023-10-12T00:48:31.884 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007577c63bbf11c214-006527425f-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.888 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.889 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:31.889 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:31.889 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.890 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:48:31.890 DEBUG:boto:Method: GET 2023-10-12T00:48:31.891 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:48:31.891 DEBUG:boto:Data: 2023-10-12T00:48:31.892 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.893 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.893 DEBUG:boto:Params: {} 2023-10-12T00:48:31.894 DEBUG:boto:Token: None 2023-10-12T00:48:31.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/ 2023-10-12T00:48:31.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs= 2023-10-12T00:48:31.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v4zR4P1jccSbZDI0j7qUILfDqAs=', 'Content-Length': '0'} 2023-10-12T00:48:31.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001edeb25c61a5458d-006527425f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.897 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:31.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:31.899 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.899 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.900 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.900 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.900 DEBUG:boto:Method: GET 2023-10-12T00:48:31.901 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.901 DEBUG:boto:Data: 2023-10-12T00:48:31.902 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.903 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.903 DEBUG:boto:Params: {} 2023-10-12T00:48:31.904 DEBUG:boto:Token: None 2023-10-12T00:48:31.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4= 2023-10-12T00:48:31.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4=', 'Content-Length': '0'} 2023-10-12T00:48:31.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbdd8bb72c94cf7-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.954 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.955 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.956 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.956 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.956 DEBUG:boto:Method: GET 2023-10-12T00:48:31.957 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.957 DEBUG:boto:Data: 2023-10-12T00:48:31.958 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.959 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.959 DEBUG:boto:Params: {} 2023-10-12T00:48:31.960 DEBUG:boto:Token: None 2023-10-12T00:48:31.960 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4= 2023-10-12T00:48:31.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4=', 'Content-Length': '0'} 2023-10-12T00:48:31.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001749ff3618fcc125-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.964 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.964 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.965 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.965 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.965 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.966 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.966 DEBUG:boto:Data: 2023-10-12T00:48:31.967 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.967 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.968 DEBUG:boto:Params: {} 2023-10-12T00:48:31.968 DEBUG:boto:Token: None 2023-10-12T00:48:31.969 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ubyI2aZxdDaH133i1aoblwN3H0U= 2023-10-12T00:48:31.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ubyI2aZxdDaH133i1aoblwN3H0U=', 'Content-Length': '0'} 2023-10-12T00:48:31.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006904cbcacfad3bff-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.972 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.973 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.973 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.973 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.974 DEBUG:boto:Method: HEAD 2023-10-12T00:48:31.974 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.975 DEBUG:boto:Data: 2023-10-12T00:48:31.975 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.976 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.976 DEBUG:boto:Params: {} 2023-10-12T00:48:31.977 DEBUG:boto:Token: None 2023-10-12T00:48:31.977 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ubyI2aZxdDaH133i1aoblwN3H0U= 2023-10-12T00:48:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ubyI2aZxdDaH133i1aoblwN3H0U=', 'Content-Length': '0'} 2023-10-12T00:48:31.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072c675c23ba24fb2-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:31.981 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.982 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.982 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.983 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.983 DEBUG:boto:Method: GET 2023-10-12T00:48:31.984 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.984 DEBUG:boto:Data: 2023-10-12T00:48:31.985 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:31.985 DEBUG:boto:Port: 8001 2023-10-12T00:48:31.986 DEBUG:boto:Params: {} 2023-10-12T00:48:31.986 DEBUG:boto:Token: None 2023-10-12T00:48:31.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4= 2023-10-12T00:48:31.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4=', 'Content-Length': '0'} 2023-10-12T00:48:31.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7ebca1412b4637-006527425f-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:31.991 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:31.991 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:31.992 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.993 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:48:31.993 DEBUG:boto:Method: GET 2023-10-12T00:48:31.993 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.994 DEBUG:boto:Data: 2023-10-12T00:48:31.994 DEBUG:boto:Headers: {} 2023-10-12T00:48:31.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:31.995 DEBUG:boto:Port: 8000 2023-10-12T00:48:31.996 DEBUG:boto:Params: {} 2023-10-12T00:48:31.996 DEBUG:boto:Token: None 2023-10-12T00:48:31.997 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:31 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:48:31.997 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4= 2023-10-12T00:48:31.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29QjNoLd58BAJLhUQdNhOcRfLv4=', 'Content-Length': '0'} 2023-10-12T00:48:32.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebd481efea186cc-006527425f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.046 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:32.047 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:32.048 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.048 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.049 DEBUG:boto:Method: GET 2023-10-12T00:48:32.049 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:32.050 DEBUG:boto:Data: 2023-10-12T00:48:32.050 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:32.051 DEBUG:boto:Port: 8001 2023-10-12T00:48:32.051 DEBUG:boto:Params: {} 2023-10-12T00:48:32.052 DEBUG:boto:Token: None 2023-10-12T00:48:32.052 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/ 2023-10-12T00:48:32.053 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA= 2023-10-12T00:48:32.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA=', 'Content-Length': '0'} 2023-10-12T00:48:32.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007481ca399801a0de-0065274260-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.058 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:32.058 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:32.059 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:32.059 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.060 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.060 DEBUG:boto:Method: GET 2023-10-12T00:48:32.061 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:32.061 DEBUG:boto:Data: 2023-10-12T00:48:32.062 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:32.062 DEBUG:boto:Port: 8000 2023-10-12T00:48:32.063 DEBUG:boto:Params: {} 2023-10-12T00:48:32.063 DEBUG:boto:Token: None 2023-10-12T00:48:32.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/ 2023-10-12T00:48:32.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA= 2023-10-12T00:48:32.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA=', 'Content-Length': '0'} 2023-10-12T00:48:32.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f803619cebf15eea-0065274260-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.067 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:32.068 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:32.068 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:32.068 DEBUG:boto:Method: HEAD 2023-10-12T00:48:32.069 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:32.069 DEBUG:boto:Data: 2023-10-12T00:48:32.070 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:32.071 DEBUG:boto:Port: 8001 2023-10-12T00:48:32.071 DEBUG:boto:Params: {} 2023-10-12T00:48:32.071 DEBUG:boto:Token: None 2023-10-12T00:48:32.072 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/h 2023-10-12T00:48:32.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JtQ5GX9RYmaTInnjFNEnecSJ3mc= 2023-10-12T00:48:32.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JtQ5GX9RYmaTInnjFNEnecSJ3mc=', 'Content-Length': '0'} 2023-10-12T00:48:32.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d10656dcaa601da-0065274260-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.075 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:32.076 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:32.076 DEBUG:boto:Method: HEAD 2023-10-12T00:48:32.077 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:32.077 DEBUG:boto:Data: 2023-10-12T00:48:32.078 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:32.078 DEBUG:boto:Port: 8000 2023-10-12T00:48:32.079 DEBUG:boto:Params: {} 2023-10-12T00:48:32.079 DEBUG:boto:Token: None 2023-10-12T00:48:32.080 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/h 2023-10-12T00:48:32.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JtQ5GX9RYmaTInnjFNEnecSJ3mc= 2023-10-12T00:48:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JtQ5GX9RYmaTInnjFNEnecSJ3mc=', 'Content-Length': '0'} 2023-10-12T00:48:32.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a3dfb132d7892a-0065274260-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:48:32.083 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:32.084 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:32.084 DEBUG:boto:Method: GET 2023-10-12T00:48:32.085 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:32.085 DEBUG:boto:Data: 2023-10-12T00:48:32.086 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:32.087 DEBUG:boto:Port: 8001 2023-10-12T00:48:32.087 DEBUG:boto:Params: {} 2023-10-12T00:48:32.088 DEBUG:boto:Token: None 2023-10-12T00:48:32.088 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/h 2023-10-12T00:48:32.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ERPWH1uAxHoF1+PYDaIUNiwf29E= 2023-10-12T00:48:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ERPWH1uAxHoF1+PYDaIUNiwf29E=', 'Content-Length': '0'} 2023-10-12T00:48:32.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d360d0dca4e09e8b-0065274260-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.091 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:48:32.092 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:48:32.092 DEBUG:boto:Method: GET 2023-10-12T00:48:32.093 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:48:32.093 DEBUG:boto:Data: 2023-10-12T00:48:32.094 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:32.095 DEBUG:boto:Port: 8000 2023-10-12T00:48:32.095 DEBUG:boto:Params: {} 2023-10-12T00:48:32.096 DEBUG:boto:Token: None 2023-10-12T00:48:32.096 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/h 2023-10-12T00:48:32.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ERPWH1uAxHoF1+PYDaIUNiwf29E= 2023-10-12T00:48:32.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ERPWH1uAxHoF1+PYDaIUNiwf29E=', 'Content-Length': '0'} 2023-10-12T00:48:32.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00ae03f93e8a654-0065274260-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.146 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:32.147 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:32.147 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.148 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.148 DEBUG:boto:Method: GET 2023-10-12T00:48:32.149 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:32.149 DEBUG:boto:Data: 2023-10-12T00:48:32.149 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:32.150 DEBUG:boto:Port: 8001 2023-10-12T00:48:32.151 DEBUG:boto:Params: {} 2023-10-12T00:48:32.151 DEBUG:boto:Token: None 2023-10-12T00:48:32.152 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/ 2023-10-12T00:48:32.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA= 2023-10-12T00:48:32.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA=', 'Content-Length': '0'} 2023-10-12T00:48:32.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8eb81dd37c68e4-0065274260-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.156 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:32.156 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:32.157 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:32.157 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.158 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:48:32.158 DEBUG:boto:Method: GET 2023-10-12T00:48:32.159 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:48:32.159 DEBUG:boto:Data: 2023-10-12T00:48:32.159 DEBUG:boto:Headers: {} 2023-10-12T00:48:32.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:32.160 DEBUG:boto:Port: 8000 2023-10-12T00:48:32.161 DEBUG:boto:Params: {} 2023-10-12T00:48:32.161 DEBUG:boto:Token: None 2023-10-12T00:48:32.162 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:32 GMT /wafmoa-55/ 2023-10-12T00:48:32.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA= 2023-10-12T00:48:32.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Agckx2J7e2LqEisp4H3EzfNhLHA=', 'Content-Length': '0'} 2023-10-12T00:48:32.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966676816844ebbf-0065274260-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:32.165 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:32.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:48:32.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.167 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:32.212 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:32.212 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:32.253 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.252+0000 7fbe5d5d0700 -1 req 14049083980622742894 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-37:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.2,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-37:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.2,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.253 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.252+0000 7fbe5b5cc700 -1 req 16739561261549639154 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-19:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.19,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-19:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.19,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.255 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.254+0000 7fbe39588700 -1 req 14459411490001047737 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-27:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.27,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-27:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.27,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.265 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.264+0000 7fbe58dc7700 -1 req 8302304479675819469 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-23:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.23,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-23:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.23,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.297 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.297+0000 7fbe4edb3700 -1 req 15944286541323352904 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-43:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.8,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-43:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.8,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.299 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.299+0000 7fbe4bdad700 -1 req 1331053811555393499 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-5:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.5,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-5:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.5,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.340 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.339+0000 7fbe27564700 -1 req 15178186611193080088 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-1:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.1,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-1:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.1,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.347 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.346+0000 7fbe575c4700 -1 req 17729748976988355736 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-4:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.4,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-4:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.4,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.409 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:32.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:32.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:32.495 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:48:32.496 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.496 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-10-12T00:48:32.496 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-10-12T00:48:32.496 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.497 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.497 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-10-12T00:48:32.497 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-10-12T00:48:32.497 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:32.498 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:32.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:32.498 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-10-12T00:48:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:48:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:32.500 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:32.500 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-10-12T00:48:32.500 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.500 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:32.500 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:32.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:32.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:32.502 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.502 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.502 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:32.502 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:32.502 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.503 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.503 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:32.504 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:32.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.505 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:48:32.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.505 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:48:32.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.507 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:32.552 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:32.552 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:32.590 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.590+0000 7fbe28d67700 -1 req 14118911866984209335 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-24:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.24,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-24:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.24,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.591 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.590+0000 7fbe0fd35700 -1 req 17525678709128692238 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-21:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.21,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-21:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.21,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.594 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.593+0000 7fbe24d5f700 -1 req 11032240148678619900 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-6:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.6,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-6:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.6,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.596 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.595+0000 7fbe0dd31700 -1 req 10421016735742929270 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-20:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.20,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-20:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.20,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.598 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.598+0000 7fbdfed13700 -1 req 10307952445363394495 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-2:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.2,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-2:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.2,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.638 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.637+0000 7fbeeaeeb700 -1 req 14522711584875136292 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-26:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.26,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-26:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.26,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.688 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.687+0000 7fbeceeb3700 -1 req 1134522356921442981 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-29:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.29,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-29:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.29,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.746 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:32.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:32.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:32.834 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:48:32.834 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.834 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-10-12T00:48:32.834 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-10-12T00:48:32.835 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.835 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.835 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-10-12T00:48:32.835 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-10-12T00:48:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:48:32.837 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-10-12T00:48:32.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:48:32.837 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.837 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.838 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-10-12T00:48:32.838 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-10-12T00:48:32.838 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-10-12T00:48:32.838 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-10-12T00:48:32.839 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-10-12T00:48:32.839 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-10-12T00:48:32.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.840 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:48:32.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:48:32.840 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:48:32.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:48:32.843 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:48:32.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:32.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:32.923 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.922+0000 7fbecf6b4700 -1 req 10306459556735896190 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-36:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.1,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-36:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.1,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.923 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.922+0000 7fbec96a8700 -1 req 9089679134150138661 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-38:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.3,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-38:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.3,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.924 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.923+0000 7fbeac66e700 -1 req 12237518887404939273 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-25:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.25,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-25:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.25,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.924 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.923+0000 7fbebc68e700 -1 req 14029035695934955689 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-28:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.28,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-28:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.28,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.925 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.925+0000 7fbed16b8700 -1 req 14551171723327831803 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-41:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.6,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-41:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.6,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.968 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.967+0000 7fbebe692700 -1 req 14954026590411898761 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-42:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.7,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-42:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.7,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.972 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.972+0000 7fbeb4e7f700 -1 req 14340686216627015796 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-51:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.16,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-51:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.16,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:32.973 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:48:32.973+0000 7fbe94e3f700 -1 req 4231365777695785974 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=wafmoa-22:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.22,z=677f5a1d-3ab4-4b5f-af15-9890c2c79648,az=0},d={b=wafmoa-22:677f5a1d-3ab4-4b5f-af15-9890c2c79648.4375.22,z=c93c7bc5-2183-452b-b2b7-6a812a26f746,az=0}} returned ret=-2 2023-10-12T00:48:33.078 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:33.078 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:33.079 DEBUG:boto:Method: HEAD 2023-10-12T00:48:33.079 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:33.080 DEBUG:boto:Data: 2023-10-12T00:48:33.080 DEBUG:boto:Headers: {} 2023-10-12T00:48:33.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.081 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.081 DEBUG:boto:Params: {} 2023-10-12T00:48:33.082 DEBUG:boto:Token: None 2023-10-12T00:48:33.082 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/ 2023-10-12T00:48:33.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo= 2023-10-12T00:48:33.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo=', 'Content-Length': '0'} 2023-10-12T00:48:33.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c27f29e44b351fe5-0065274261-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.087 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:48:33.087 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:48:33.088 DEBUG:boto:Method: PUT 2023-10-12T00:48:33.088 DEBUG:boto:Path: /wafmoa-55/i 2023-10-12T00:48:33.089 DEBUG:boto:Data: 2023-10-12T00:48:33.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:48:33.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.090 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.090 DEBUG:boto:Params: {} 2023-10-12T00:48:33.091 DEBUG:boto:Token: None 2023-10-12T00:48:33.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/i 2023-10-12T00:48:33.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4IRUb6rOD5NAS6N0kLdmbpIe4P4= 2023-10-12T00:48:33.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4IRUb6rOD5NAS6N0kLdmbpIe4P4='} 2023-10-12T00:48:33.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f36dd9ad9442a89-0065274261-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.097 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:33.098 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:33.098 DEBUG:boto:Method: HEAD 2023-10-12T00:48:33.099 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:33.099 DEBUG:boto:Data: 2023-10-12T00:48:33.100 DEBUG:boto:Headers: {} 2023-10-12T00:48:33.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.101 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.101 DEBUG:boto:Params: {} 2023-10-12T00:48:33.101 DEBUG:boto:Token: None 2023-10-12T00:48:33.102 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/ 2023-10-12T00:48:33.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo= 2023-10-12T00:48:33.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo=', 'Content-Length': '0'} 2023-10-12T00:48:33.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087211df281e815bf-0065274261-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.105 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:48:33.106 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:48:33.106 DEBUG:boto:Method: PUT 2023-10-12T00:48:33.106 DEBUG:boto:Path: /wafmoa-55/j 2023-10-12T00:48:33.107 DEBUG:boto:Data: 2023-10-12T00:48:33.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:48:33.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.108 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.108 DEBUG:boto:Params: {} 2023-10-12T00:48:33.109 DEBUG:boto:Token: None 2023-10-12T00:48:33.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/j 2023-10-12T00:48:33.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aD5qyv/lsAEP0itNf2GuktZPJ4E= 2023-10-12T00:48:33.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aD5qyv/lsAEP0itNf2GuktZPJ4E='} 2023-10-12T00:48:33.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009af2b8337e6e03dd-0065274261-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.155 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:33.156 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:33.156 DEBUG:boto:Method: HEAD 2023-10-12T00:48:33.156 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:33.157 DEBUG:boto:Data: 2023-10-12T00:48:33.157 DEBUG:boto:Headers: {} 2023-10-12T00:48:33.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.158 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.158 DEBUG:boto:Params: {} 2023-10-12T00:48:33.159 DEBUG:boto:Token: None 2023-10-12T00:48:33.160 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/ 2023-10-12T00:48:33.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo= 2023-10-12T00:48:33.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo=', 'Content-Length': '0'} 2023-10-12T00:48:33.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c269fea76ff6692d-0065274261-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.163 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:48:33.163 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:48:33.164 DEBUG:boto:Method: PUT 2023-10-12T00:48:33.164 DEBUG:boto:Path: /wafmoa-55/k 2023-10-12T00:48:33.165 DEBUG:boto:Data: 2023-10-12T00:48:33.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:48:33.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.166 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.167 DEBUG:boto:Params: {} 2023-10-12T00:48:33.167 DEBUG:boto:Token: None 2023-10-12T00:48:33.167 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/k 2023-10-12T00:48:33.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9wZ1D6EXRUVTjCrbacm7UvpyR28= 2023-10-12T00:48:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9wZ1D6EXRUVTjCrbacm7UvpyR28='} 2023-10-12T00:48:33.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000081f2373c824a3dd-0065274261-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.174 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:33.174 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:33.175 DEBUG:boto:Method: HEAD 2023-10-12T00:48:33.175 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:33.176 DEBUG:boto:Data: 2023-10-12T00:48:33.176 DEBUG:boto:Headers: {} 2023-10-12T00:48:33.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.177 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.177 DEBUG:boto:Params: {} 2023-10-12T00:48:33.178 DEBUG:boto:Token: None 2023-10-12T00:48:33.178 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/ 2023-10-12T00:48:33.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo= 2023-10-12T00:48:33.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D8hjamyIiMCBzGQDbExfNPjV9oo=', 'Content-Length': '0'} 2023-10-12T00:48:33.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b02513cfe3f1694-0065274261-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.181 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:48:33.182 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:48:33.182 DEBUG:boto:Method: PUT 2023-10-12T00:48:33.183 DEBUG:boto:Path: /wafmoa-55/l 2023-10-12T00:48:33.183 DEBUG:boto:Data: 2023-10-12T00:48:33.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:48:33.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:33.184 DEBUG:boto:Port: 8000 2023-10-12T00:48:33.185 DEBUG:boto:Params: {} 2023-10-12T00:48:33.185 DEBUG:boto:Token: None 2023-10-12T00:48:33.186 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:48:33 GMT /wafmoa-55/l 2023-10-12T00:48:33.186 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hVOaPCpr653EiMPGtS6N5U4fkwA= 2023-10-12T00:48:33.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:48:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hVOaPCpr653EiMPGtS6N5U4fkwA='} 2023-10-12T00:48:33.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7562362b304feed-0065274261-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:33.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-55 2023-10-12T00:48:33.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:33.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:33.237 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:48:33.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:33.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:33.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:33.326+0000 7f7b19e86780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:33.327 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:48:33.327 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.45.6, 00000000002.54.6, 00000000003.43.6] 2023-10-12T00:48:38.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:38.327+0000 7f7b19e86780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:38.328 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:48:38.328 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.45.6, 00000000002.54.6, 00000000003.43.6] 2023-10-12T00:48:43.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:43.328+0000 7f7b19e86780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:43.330 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:48:43.330 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.45.6, 00000000002.54.6, 00000000003.43.6] 2023-10-12T00:48:48.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:48.329+0000 7f7b19e86780 1 bucket sync caught up with source: 2023-10-12T00:48:48.330 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.45.6, 00000000002.54.6, 00000000003.43.6] 2023-10-12T00:48:48.331 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.45.6, 00000000002.54.6, 00000000003.43.6] 2023-10-12T00:48:48.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:48.329+0000 7f7b19e86780 0 bucket checkpoint complete 2023-10-12T00:48:48.347 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-55 2023-10-12T00:48:48.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:48.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:48:48.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:48:48.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:48:48.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:48:48.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:48.448+0000 7f414f762780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:48.450 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.298.6, , , 00000000001.151.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:48.450 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.171.6, 00000000002.298.6, 00000000001.260.6, , 00000000002.157.6, 00000000001.201.6, 00000000001.195.6, , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:53.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:53.450+0000 7f414f762780 1 waiting for incremental sync to catch up: 2023-10-12T00:48:53.450 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.171.6, 00000000002.298.6, , , 00000000002.157.6, 00000000001.201.6, , , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:53.450 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.171.6, 00000000002.298.6, 00000000001.260.6, , 00000000002.157.6, 00000000001.201.6, 00000000001.195.6, , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:56.983 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:48:56.984+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T00:48:58.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:58.451+0000 7f414f762780 1 bucket sync caught up with source: 2023-10-12T00:48:58.452 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.171.6, 00000000002.298.6, 00000000001.260.6, , 00000000002.157.6, 00000000001.201.6, 00000000001.195.6, , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:58.452 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.171.6, 00000000002.298.6, 00000000001.260.6, , 00000000002.157.6, 00000000001.201.6, 00000000001.195.6, , 00000000002.352.6, , 00000000002.350.6] 2023-10-12T00:48:58.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:48:58.451+0000 7f414f762780 0 bucket checkpoint complete 2023-10-12T00:48:58.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:48:58.469 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.469 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.470 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.470 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:58.471 DEBUG:boto:Data: 2023-10-12T00:48:58.471 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.472 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.473 DEBUG:boto:Params: {} 2023-10-12T00:48:58.473 DEBUG:boto:Token: None 2023-10-12T00:48:58.474 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rsgq+MniyQtFOVSrt20swAz9TK4= 2023-10-12T00:48:58.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rsgq+MniyQtFOVSrt20swAz9TK4=', 'Content-Length': '0'} 2023-10-12T00:48:58.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067182ceb2bfaeeeb-006527427a-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.478 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.479 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.479 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.480 DEBUG:boto:Path: /wafmoa-55/ 2023-10-12T00:48:58.480 DEBUG:boto:Data: 2023-10-12T00:48:58.480 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.481 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.482 DEBUG:boto:Params: {} 2023-10-12T00:48:58.482 DEBUG:boto:Token: None 2023-10-12T00:48:58.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rsgq+MniyQtFOVSrt20swAz9TK4= 2023-10-12T00:48:58.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rsgq+MniyQtFOVSrt20swAz9TK4=', 'Content-Length': '0'} 2023-10-12T00:48:58.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000514cbadd43ceb684-006527427a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.489 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:48:58.489 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.490 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.490 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.490 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.491 DEBUG:boto:Method: GET 2023-10-12T00:48:58.491 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.496 DEBUG:boto:Data: 2023-10-12T00:48:58.496 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.497 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.498 DEBUG:boto:Params: {} 2023-10-12T00:48:58.499 DEBUG:boto:Token: None 2023-10-12T00:48:58.500 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/?versions 2023-10-12T00:48:58.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8= 2023-10-12T00:48:58.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8=', 'Content-Length': '0'} 2023-10-12T00:48:58.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9959765b1bac7a1-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.504 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.506 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:58.506 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:58.508 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:58.508 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:58.509 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:58.509 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:58.509 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:58.510 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:58.510 DEBUG:tasks.rgw_multi.zone_rados:o=i 2023-10-12T00:48:58.510 DEBUG:tasks.rgw_multi.zone_rados:o=j 2023-10-12T00:48:58.511 DEBUG:tasks.rgw_multi.zone_rados:o=k 2023-10-12T00:48:58.511 DEBUG:tasks.rgw_multi.zone_rados:o=l 2023-10-12T00:48:58.512 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:48:58.512 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.512 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.513 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.513 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.514 DEBUG:boto:Method: GET 2023-10-12T00:48:58.514 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.515 DEBUG:boto:Data: 2023-10-12T00:48:58.515 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.516 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.516 DEBUG:boto:Params: {} 2023-10-12T00:48:58.517 DEBUG:boto:Token: None 2023-10-12T00:48:58.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/?versions 2023-10-12T00:48:58.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8= 2023-10-12T00:48:58.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8=', 'Content-Length': '0'} 2023-10-12T00:48:58.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a55f891db0d4f56-006527427a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.521 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.523 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:48:58.523 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:48:58.523 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:48:58.524 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:48:58.524 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:48:58.524 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:48:58.525 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:48:58.525 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:48:58.526 DEBUG:tasks.rgw_multi.zone_rados:o=i 2023-10-12T00:48:58.526 DEBUG:tasks.rgw_multi.zone_rados:o=j 2023-10-12T00:48:58.526 DEBUG:tasks.rgw_multi.zone_rados:o=k 2023-10-12T00:48:58.527 DEBUG:tasks.rgw_multi.zone_rados:o=l 2023-10-12T00:48:58.527 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.528 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.528 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.528 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.529 DEBUG:boto:Method: GET 2023-10-12T00:48:58.529 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.530 DEBUG:boto:Data: 2023-10-12T00:48:58.530 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.531 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.531 DEBUG:boto:Params: {} 2023-10-12T00:48:58.532 DEBUG:boto:Token: None 2023-10-12T00:48:58.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/?versions 2023-10-12T00:48:58.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8= 2023-10-12T00:48:58.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8=', 'Content-Length': '0'} 2023-10-12T00:48:58.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff589436239a7a1-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.537 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.538 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.538 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.539 DEBUG:boto:path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.539 DEBUG:boto:auth_path=/wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.539 DEBUG:boto:Method: GET 2023-10-12T00:48:58.540 DEBUG:boto:Path: /wafmoa-55/?versions&max-keys=999 2023-10-12T00:48:58.540 DEBUG:boto:Data: 2023-10-12T00:48:58.541 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.542 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.542 DEBUG:boto:Params: {} 2023-10-12T00:48:58.542 DEBUG:boto:Token: None 2023-10-12T00:48:58.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/?versions 2023-10-12T00:48:58.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8= 2023-10-12T00:48:58.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YBWzhruOqQYzovahEhx6gTkulT8=', 'Content-Length': '0'} 2023-10-12T00:48:58.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c28c0639e691ee-006527427a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.547 DEBUG:boto:b'wafmoa-55999falseanulltrue2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalinulltrue2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaljnulltrue2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalknulltrue2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormallnulltrue2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:58.548 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.549 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.549 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.550 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.550 DEBUG:boto:Method: GET 2023-10-12T00:48:58.550 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.551 DEBUG:boto:Data: 2023-10-12T00:48:58.551 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.552 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.553 DEBUG:boto:Params: {} 2023-10-12T00:48:58.553 DEBUG:boto:Token: None 2023-10-12T00:48:58.554 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow= 2023-10-12T00:48:58.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow=', 'Content-Length': '0'} 2023-10-12T00:48:58.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ffc6258654a26ba-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.598 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.599 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.599 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.600 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.600 DEBUG:boto:Method: GET 2023-10-12T00:48:58.601 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.601 DEBUG:boto:Data: 2023-10-12T00:48:58.602 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.603 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.603 DEBUG:boto:Params: {} 2023-10-12T00:48:58.603 DEBUG:boto:Token: None 2023-10-12T00:48:58.604 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow= 2023-10-12T00:48:58.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow=', 'Content-Length': '0'} 2023-10-12T00:48:58.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b945cff5c79bca57-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.608 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.608 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.608 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.609 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.609 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.610 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.610 DEBUG:boto:Data: 2023-10-12T00:48:58.611 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.611 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.612 DEBUG:boto:Params: {} 2023-10-12T00:48:58.612 DEBUG:boto:Token: None 2023-10-12T00:48:58.613 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0IB13PGl+vIaJsSSdaZvymkcmfE= 2023-10-12T00:48:58.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0IB13PGl+vIaJsSSdaZvymkcmfE=', 'Content-Length': '0'} 2023-10-12T00:48:58.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b99b1e4407751d-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.617 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.617 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.618 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.618 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.618 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.619 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.619 DEBUG:boto:Data: 2023-10-12T00:48:58.620 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.620 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.621 DEBUG:boto:Params: {} 2023-10-12T00:48:58.621 DEBUG:boto:Token: None 2023-10-12T00:48:58.622 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0IB13PGl+vIaJsSSdaZvymkcmfE= 2023-10-12T00:48:58.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0IB13PGl+vIaJsSSdaZvymkcmfE=', 'Content-Length': '0'} 2023-10-12T00:48:58.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167af696ebbae55b-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:58.626 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.627 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.627 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.627 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.628 DEBUG:boto:Method: GET 2023-10-12T00:48:58.628 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.629 DEBUG:boto:Data: 2023-10-12T00:48:58.629 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.630 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.630 DEBUG:boto:Params: {} 2023-10-12T00:48:58.631 DEBUG:boto:Token: None 2023-10-12T00:48:58.631 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow= 2023-10-12T00:48:58.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow=', 'Content-Length': '0'} 2023-10-12T00:48:58.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ca91b70d228e0c-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.635 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.635 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.636 DEBUG:boto:path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.636 DEBUG:boto:auth_path=/wafmoa-55/a?versionId=null 2023-10-12T00:48:58.637 DEBUG:boto:Method: GET 2023-10-12T00:48:58.637 DEBUG:boto:Path: /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.638 DEBUG:boto:Data: 2023-10-12T00:48:58.638 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.639 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.639 DEBUG:boto:Params: {} 2023-10-12T00:48:58.640 DEBUG:boto:Token: None 2023-10-12T00:48:58.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a?versionId=null 2023-10-12T00:48:58.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow= 2023-10-12T00:48:58.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B2s9oNsq9RheIqq+zxsMGE6INow=', 'Content-Length': '0'} 2023-10-12T00:48:58.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000056f31027838b972-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.644 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.644 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.645 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.645 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.646 DEBUG:boto:Method: GET 2023-10-12T00:48:58.646 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:58.646 DEBUG:boto:Data: 2023-10-12T00:48:58.647 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.648 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.648 DEBUG:boto:Params: {} 2023-10-12T00:48:58.649 DEBUG:boto:Token: None 2023-10-12T00:48:58.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f12d3c89dfc47e-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.653 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.654 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.654 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.655 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.655 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.656 DEBUG:boto:Method: GET 2023-10-12T00:48:58.656 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:58.657 DEBUG:boto:Data: 2023-10-12T00:48:58.657 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.658 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.658 DEBUG:boto:Params: {} 2023-10-12T00:48:58.659 DEBUG:boto:Token: None 2023-10-12T00:48:58.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080775b68b6493505-006527427a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.663 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.663 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.664 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.664 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.665 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:58.665 DEBUG:boto:Data: 2023-10-12T00:48:58.665 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.666 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.667 DEBUG:boto:Params: {} 2023-10-12T00:48:58.667 DEBUG:boto:Token: None 2023-10-12T00:48:58.667 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a 2023-10-12T00:48:58.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aWxtORwIyfdCFN2yIvKBOIXRlGQ= 2023-10-12T00:48:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aWxtORwIyfdCFN2yIvKBOIXRlGQ=', 'Content-Length': '0'} 2023-10-12T00:48:58.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5d93296b5afb2f6-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.671 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.671 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.672 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.672 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:58.673 DEBUG:boto:Data: 2023-10-12T00:48:58.673 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.674 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.674 DEBUG:boto:Params: {} 2023-10-12T00:48:58.675 DEBUG:boto:Token: None 2023-10-12T00:48:58.675 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a 2023-10-12T00:48:58.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aWxtORwIyfdCFN2yIvKBOIXRlGQ= 2023-10-12T00:48:58.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aWxtORwIyfdCFN2yIvKBOIXRlGQ=', 'Content-Length': '0'} 2023-10-12T00:48:58.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc4c1fde027695b-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:48:58.679 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.679 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.680 DEBUG:boto:Method: GET 2023-10-12T00:48:58.680 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:58.681 DEBUG:boto:Data: 2023-10-12T00:48:58.681 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.682 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.682 DEBUG:boto:Params: {} 2023-10-12T00:48:58.683 DEBUG:boto:Token: None 2023-10-12T00:48:58.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a 2023-10-12T00:48:58.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IQoDLmfF6eRE6X0+8mUDAJAYofc= 2023-10-12T00:48:58.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IQoDLmfF6eRE6X0+8mUDAJAYofc=', 'Content-Length': '0'} 2023-10-12T00:48:58.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c827bc38345dc3-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.734 DEBUG:boto:path=/wafmoa-55/a 2023-10-12T00:48:58.735 DEBUG:boto:auth_path=/wafmoa-55/a 2023-10-12T00:48:58.735 DEBUG:boto:Method: GET 2023-10-12T00:48:58.736 DEBUG:boto:Path: /wafmoa-55/a 2023-10-12T00:48:58.736 DEBUG:boto:Data: 2023-10-12T00:48:58.737 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.738 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.738 DEBUG:boto:Params: {} 2023-10-12T00:48:58.739 DEBUG:boto:Token: None 2023-10-12T00:48:58.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/a 2023-10-12T00:48:58.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IQoDLmfF6eRE6X0+8mUDAJAYofc= 2023-10-12T00:48:58.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IQoDLmfF6eRE6X0+8mUDAJAYofc=', 'Content-Length': '0'} 2023-10-12T00:48:58.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006edda0d5755f782d-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.743 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.743 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.744 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.744 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.744 DEBUG:boto:Method: GET 2023-10-12T00:48:58.745 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:58.745 DEBUG:boto:Data: 2023-10-12T00:48:58.746 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.747 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.747 DEBUG:boto:Params: {} 2023-10-12T00:48:58.748 DEBUG:boto:Token: None 2023-10-12T00:48:58.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af37649cfe822d4c-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.752 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.753 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.753 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.753 DEBUG:boto:path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.754 DEBUG:boto:auth_path=/wafmoa-55/?prefix=a 2023-10-12T00:48:58.754 DEBUG:boto:Method: GET 2023-10-12T00:48:58.755 DEBUG:boto:Path: /wafmoa-55/?prefix=a 2023-10-12T00:48:58.755 DEBUG:boto:Data: 2023-10-12T00:48:58.756 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.756 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.757 DEBUG:boto:Params: {} 2023-10-12T00:48:58.757 DEBUG:boto:Token: None 2023-10-12T00:48:58.758 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca294ebd9341ed4-006527427a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.761 DEBUG:boto:b'wafmoa-55a1000falsea2023-10-12T00:44:35.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:58.763 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.763 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.763 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.764 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.764 DEBUG:boto:Method: GET 2023-10-12T00:48:58.765 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.765 DEBUG:boto:Data: 2023-10-12T00:48:58.766 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.766 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.767 DEBUG:boto:Params: {} 2023-10-12T00:48:58.767 DEBUG:boto:Token: None 2023-10-12T00:48:58.768 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw= 2023-10-12T00:48:58.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw=', 'Content-Length': '0'} 2023-10-12T00:48:58.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ff0658c3f9261b-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.771 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.772 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.772 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.773 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.773 DEBUG:boto:Method: GET 2023-10-12T00:48:58.774 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.774 DEBUG:boto:Data: 2023-10-12T00:48:58.774 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.775 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.776 DEBUG:boto:Params: {} 2023-10-12T00:48:58.776 DEBUG:boto:Token: None 2023-10-12T00:48:58.777 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw= 2023-10-12T00:48:58.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw=', 'Content-Length': '0'} 2023-10-12T00:48:58.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04786066e4587eb-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.781 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.781 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.781 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.782 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.782 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.783 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.783 DEBUG:boto:Data: 2023-10-12T00:48:58.784 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.784 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.785 DEBUG:boto:Params: {} 2023-10-12T00:48:58.785 DEBUG:boto:Token: None 2023-10-12T00:48:58.786 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rtr87QrU0TOnGRV7hzRmtLKVCL8= 2023-10-12T00:48:58.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rtr87QrU0TOnGRV7hzRmtLKVCL8=', 'Content-Length': '0'} 2023-10-12T00:48:58.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e46c79cd89ce49-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.789 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.790 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.790 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.791 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.791 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.791 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.792 DEBUG:boto:Data: 2023-10-12T00:48:58.792 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.793 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.793 DEBUG:boto:Params: {} 2023-10-12T00:48:58.794 DEBUG:boto:Token: None 2023-10-12T00:48:58.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rtr87QrU0TOnGRV7hzRmtLKVCL8= 2023-10-12T00:48:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rtr87QrU0TOnGRV7hzRmtLKVCL8=', 'Content-Length': '0'} 2023-10-12T00:48:58.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ccf5506ec9e270-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:58.798 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.799 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.799 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.800 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.800 DEBUG:boto:Method: GET 2023-10-12T00:48:58.801 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.801 DEBUG:boto:Data: 2023-10-12T00:48:58.802 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.802 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.803 DEBUG:boto:Params: {} 2023-10-12T00:48:58.803 DEBUG:boto:Token: None 2023-10-12T00:48:58.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw= 2023-10-12T00:48:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw=', 'Content-Length': '0'} 2023-10-12T00:48:58.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197f4cad83094e74-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.850 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.851 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.852 DEBUG:boto:path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.852 DEBUG:boto:auth_path=/wafmoa-55/b?versionId=null 2023-10-12T00:48:58.852 DEBUG:boto:Method: GET 2023-10-12T00:48:58.853 DEBUG:boto:Path: /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.853 DEBUG:boto:Data: 2023-10-12T00:48:58.854 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.855 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.855 DEBUG:boto:Params: {} 2023-10-12T00:48:58.856 DEBUG:boto:Token: None 2023-10-12T00:48:58.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b?versionId=null 2023-10-12T00:48:58.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw= 2023-10-12T00:48:58.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xX3ec+lsMNaUEVUvm8l68Ss2uhw=', 'Content-Length': '0'} 2023-10-12T00:48:58.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcd6029f597209c-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.860 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.860 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.861 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.861 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.862 DEBUG:boto:Method: GET 2023-10-12T00:48:58.862 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:58.863 DEBUG:boto:Data: 2023-10-12T00:48:58.863 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.864 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.864 DEBUG:boto:Params: {} 2023-10-12T00:48:58.865 DEBUG:boto:Token: None 2023-10-12T00:48:58.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067386c53b5bfbcc1-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.869 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.870 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.871 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.871 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.872 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.872 DEBUG:boto:Method: GET 2023-10-12T00:48:58.873 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:58.873 DEBUG:boto:Data: 2023-10-12T00:48:58.873 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.874 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.875 DEBUG:boto:Params: {} 2023-10-12T00:48:58.875 DEBUG:boto:Token: None 2023-10-12T00:48:58.876 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d27075dc6245d0-006527427a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.922 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:58.923 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.924 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.924 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.925 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:58.925 DEBUG:boto:Data: 2023-10-12T00:48:58.925 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.926 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.927 DEBUG:boto:Params: {} 2023-10-12T00:48:58.927 DEBUG:boto:Token: None 2023-10-12T00:48:58.928 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b 2023-10-12T00:48:58.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zbFmjQOLKE/TJSTUiOrb4nlgUFU= 2023-10-12T00:48:58.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zbFmjQOLKE/TJSTUiOrb4nlgUFU=', 'Content-Length': '0'} 2023-10-12T00:48:58.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006caf56efa42d6774-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.931 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.932 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.932 DEBUG:boto:Method: HEAD 2023-10-12T00:48:58.933 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:58.933 DEBUG:boto:Data: 2023-10-12T00:48:58.934 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.934 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.935 DEBUG:boto:Params: {} 2023-10-12T00:48:58.935 DEBUG:boto:Token: None 2023-10-12T00:48:58.936 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b 2023-10-12T00:48:58.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zbFmjQOLKE/TJSTUiOrb4nlgUFU= 2023-10-12T00:48:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zbFmjQOLKE/TJSTUiOrb4nlgUFU=', 'Content-Length': '0'} 2023-10-12T00:48:58.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078699e245a7627ce-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:48:58.940 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.940 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.941 DEBUG:boto:Method: GET 2023-10-12T00:48:58.941 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:58.941 DEBUG:boto:Data: 2023-10-12T00:48:58.942 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.943 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.943 DEBUG:boto:Params: {} 2023-10-12T00:48:58.944 DEBUG:boto:Token: None 2023-10-12T00:48:58.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b 2023-10-12T00:48:58.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zS5io83DopKxQ+75Wl/3yXafHew= 2023-10-12T00:48:58.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zS5io83DopKxQ+75Wl/3yXafHew=', 'Content-Length': '0'} 2023-10-12T00:48:58.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003630c35f1e12d157-006527427a-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.948 DEBUG:boto:path=/wafmoa-55/b 2023-10-12T00:48:58.948 DEBUG:boto:auth_path=/wafmoa-55/b 2023-10-12T00:48:58.949 DEBUG:boto:Method: GET 2023-10-12T00:48:58.949 DEBUG:boto:Path: /wafmoa-55/b 2023-10-12T00:48:58.950 DEBUG:boto:Data: 2023-10-12T00:48:58.950 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:58.951 DEBUG:boto:Port: 8000 2023-10-12T00:48:58.951 DEBUG:boto:Params: {} 2023-10-12T00:48:58.952 DEBUG:boto:Token: None 2023-10-12T00:48:58.952 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/b 2023-10-12T00:48:58.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zS5io83DopKxQ+75Wl/3yXafHew= 2023-10-12T00:48:58.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zS5io83DopKxQ+75Wl/3yXafHew=', 'Content-Length': '0'} 2023-10-12T00:48:58.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019117e4cb2b0449f-006527427a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:58.956 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:58.956 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:58.957 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.957 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:58.958 DEBUG:boto:Method: GET 2023-10-12T00:48:58.958 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:58.959 DEBUG:boto:Data: 2023-10-12T00:48:58.959 DEBUG:boto:Headers: {} 2023-10-12T00:48:58.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:58.960 DEBUG:boto:Port: 8001 2023-10-12T00:48:58.960 DEBUG:boto:Params: {} 2023-10-12T00:48:58.961 DEBUG:boto:Token: None 2023-10-12T00:48:58.961 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:58 GMT /wafmoa-55/ 2023-10-12T00:48:58.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc= 2023-10-12T00:48:58.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:euv//bMMkkBciK8MPe670oQ+vuc=', 'Content-Length': '0'} 2023-10-12T00:48:58.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032148564fa31f4dc-006527427a-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.010 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.011 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.012 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.012 DEBUG:boto:path=/wafmoa-55/?prefix=b 2023-10-12T00:48:59.013 DEBUG:boto:auth_path=/wafmoa-55/?prefix=b 2023-10-12T00:48:59.013 DEBUG:boto:Method: GET 2023-10-12T00:48:59.013 DEBUG:boto:Path: /wafmoa-55/?prefix=b 2023-10-12T00:48:59.014 DEBUG:boto:Data: 2023-10-12T00:48:59.014 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.015 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.016 DEBUG:boto:Params: {} 2023-10-12T00:48:59.016 DEBUG:boto:Token: None 2023-10-12T00:48:59.017 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5b29f2acd23c80-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.020 DEBUG:boto:b'wafmoa-55b1000falseb2023-10-12T00:44:36.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:59.022 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.022 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.022 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.023 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.023 DEBUG:boto:Method: GET 2023-10-12T00:48:59.024 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.024 DEBUG:boto:Data: 2023-10-12T00:48:59.025 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.026 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.026 DEBUG:boto:Params: {} 2023-10-12T00:48:59.027 DEBUG:boto:Token: None 2023-10-12T00:48:59.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk= 2023-10-12T00:48:59.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk=', 'Content-Length': '0'} 2023-10-12T00:48:59.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfde0787140b55aa-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.031 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.031 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.031 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.032 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.032 DEBUG:boto:Method: GET 2023-10-12T00:48:59.033 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.033 DEBUG:boto:Data: 2023-10-12T00:48:59.034 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.035 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.035 DEBUG:boto:Params: {} 2023-10-12T00:48:59.035 DEBUG:boto:Token: None 2023-10-12T00:48:59.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk= 2023-10-12T00:48:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk=', 'Content-Length': '0'} 2023-10-12T00:48:59.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b902c35de646f9-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.040 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.040 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.041 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.041 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.041 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.042 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.042 DEBUG:boto:Data: 2023-10-12T00:48:59.043 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.043 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.044 DEBUG:boto:Params: {} 2023-10-12T00:48:59.044 DEBUG:boto:Token: None 2023-10-12T00:48:59.045 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gYRq9Q9lckORnhLbPczntPWNSDo= 2023-10-12T00:48:59.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gYRq9Q9lckORnhLbPczntPWNSDo=', 'Content-Length': '0'} 2023-10-12T00:48:59.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4443a4be5903201-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.048 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.049 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.049 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.050 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.050 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.051 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.051 DEBUG:boto:Data: 2023-10-12T00:48:59.051 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.052 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.053 DEBUG:boto:Params: {} 2023-10-12T00:48:59.053 DEBUG:boto:Token: None 2023-10-12T00:48:59.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gYRq9Q9lckORnhLbPczntPWNSDo= 2023-10-12T00:48:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gYRq9Q9lckORnhLbPczntPWNSDo=', 'Content-Length': '0'} 2023-10-12T00:48:59.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86b16c8fba131a3-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:59.058 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.058 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.059 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.059 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.060 DEBUG:boto:Method: GET 2023-10-12T00:48:59.060 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.060 DEBUG:boto:Data: 2023-10-12T00:48:59.061 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.062 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.062 DEBUG:boto:Params: {} 2023-10-12T00:48:59.063 DEBUG:boto:Token: None 2023-10-12T00:48:59.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk= 2023-10-12T00:48:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk=', 'Content-Length': '0'} 2023-10-12T00:48:59.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078472d4230bde057-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.067 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.067 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.068 DEBUG:boto:path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.068 DEBUG:boto:auth_path=/wafmoa-55/c?versionId=null 2023-10-12T00:48:59.069 DEBUG:boto:Method: GET 2023-10-12T00:48:59.069 DEBUG:boto:Path: /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.069 DEBUG:boto:Data: 2023-10-12T00:48:59.070 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.071 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.071 DEBUG:boto:Params: {} 2023-10-12T00:48:59.072 DEBUG:boto:Token: None 2023-10-12T00:48:59.072 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c?versionId=null 2023-10-12T00:48:59.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk= 2023-10-12T00:48:59.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iqvYy5+z1vheJLS+4yNEyNZXDkk=', 'Content-Length': '0'} 2023-10-12T00:48:59.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a8161dfbea22b2-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.118 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.119 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.119 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.120 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.120 DEBUG:boto:Method: GET 2023-10-12T00:48:59.121 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:59.121 DEBUG:boto:Data: 2023-10-12T00:48:59.122 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.123 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.123 DEBUG:boto:Params: {} 2023-10-12T00:48:59.124 DEBUG:boto:Token: None 2023-10-12T00:48:59.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f2e6d5f258fe8d-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.128 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.129 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.130 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.130 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.130 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.131 DEBUG:boto:Method: GET 2023-10-12T00:48:59.131 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:59.132 DEBUG:boto:Data: 2023-10-12T00:48:59.132 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.133 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.134 DEBUG:boto:Params: {} 2023-10-12T00:48:59.134 DEBUG:boto:Token: None 2023-10-12T00:48:59.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d497298bcdf81105-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.138 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.139 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.139 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.140 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.140 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:59.141 DEBUG:boto:Data: 2023-10-12T00:48:59.141 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.142 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.142 DEBUG:boto:Params: {} 2023-10-12T00:48:59.143 DEBUG:boto:Token: None 2023-10-12T00:48:59.143 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c 2023-10-12T00:48:59.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5N7WSWBQ8sdq/ST9IFj69NXwfZM= 2023-10-12T00:48:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5N7WSWBQ8sdq/ST9IFj69NXwfZM=', 'Content-Length': '0'} 2023-10-12T00:48:59.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bff18bf41689b3a-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.147 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.148 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.148 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.148 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:59.149 DEBUG:boto:Data: 2023-10-12T00:48:59.149 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.150 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.151 DEBUG:boto:Params: {} 2023-10-12T00:48:59.151 DEBUG:boto:Token: None 2023-10-12T00:48:59.151 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c 2023-10-12T00:48:59.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5N7WSWBQ8sdq/ST9IFj69NXwfZM= 2023-10-12T00:48:59.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5N7WSWBQ8sdq/ST9IFj69NXwfZM=', 'Content-Length': '0'} 2023-10-12T00:48:59.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3ba0efaa98ce11-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:48:59.155 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.156 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.156 DEBUG:boto:Method: GET 2023-10-12T00:48:59.157 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:59.157 DEBUG:boto:Data: 2023-10-12T00:48:59.158 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.159 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.159 DEBUG:boto:Params: {} 2023-10-12T00:48:59.160 DEBUG:boto:Token: None 2023-10-12T00:48:59.160 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c 2023-10-12T00:48:59.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TMVXlFjzL2CYps2lASKmcOLWaiw= 2023-10-12T00:48:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TMVXlFjzL2CYps2lASKmcOLWaiw=', 'Content-Length': '0'} 2023-10-12T00:48:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aebecfe2a78bbf67-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.206 DEBUG:boto:path=/wafmoa-55/c 2023-10-12T00:48:59.207 DEBUG:boto:auth_path=/wafmoa-55/c 2023-10-12T00:48:59.208 DEBUG:boto:Method: GET 2023-10-12T00:48:59.208 DEBUG:boto:Path: /wafmoa-55/c 2023-10-12T00:48:59.208 DEBUG:boto:Data: 2023-10-12T00:48:59.209 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.210 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.210 DEBUG:boto:Params: {} 2023-10-12T00:48:59.211 DEBUG:boto:Token: None 2023-10-12T00:48:59.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/c 2023-10-12T00:48:59.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TMVXlFjzL2CYps2lASKmcOLWaiw= 2023-10-12T00:48:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TMVXlFjzL2CYps2lASKmcOLWaiw=', 'Content-Length': '0'} 2023-10-12T00:48:59.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b6cd6e89f730d9-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.215 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.216 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.216 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.217 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.217 DEBUG:boto:Method: GET 2023-10-12T00:48:59.218 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:59.218 DEBUG:boto:Data: 2023-10-12T00:48:59.219 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.220 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.220 DEBUG:boto:Params: {} 2023-10-12T00:48:59.221 DEBUG:boto:Token: None 2023-10-12T00:48:59.221 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231b1d5705daa774-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.225 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.226 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.227 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.227 DEBUG:boto:path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.228 DEBUG:boto:auth_path=/wafmoa-55/?prefix=c 2023-10-12T00:48:59.228 DEBUG:boto:Method: GET 2023-10-12T00:48:59.229 DEBUG:boto:Path: /wafmoa-55/?prefix=c 2023-10-12T00:48:59.229 DEBUG:boto:Data: 2023-10-12T00:48:59.229 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.230 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.231 DEBUG:boto:Params: {} 2023-10-12T00:48:59.231 DEBUG:boto:Token: None 2023-10-12T00:48:59.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6bdd8a758fac20-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.236 DEBUG:boto:b'wafmoa-55c1000falsec2023-10-12T00:44:36.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:59.237 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.237 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.238 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.238 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.239 DEBUG:boto:Method: GET 2023-10-12T00:48:59.239 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.240 DEBUG:boto:Data: 2023-10-12T00:48:59.240 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.241 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.242 DEBUG:boto:Params: {} 2023-10-12T00:48:59.242 DEBUG:boto:Token: None 2023-10-12T00:48:59.243 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.243 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY= 2023-10-12T00:48:59.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY=', 'Content-Length': '0'} 2023-10-12T00:48:59.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8386d36206f759-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.247 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.247 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.248 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.248 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.248 DEBUG:boto:Method: GET 2023-10-12T00:48:59.249 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.249 DEBUG:boto:Data: 2023-10-12T00:48:59.250 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.251 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.251 DEBUG:boto:Params: {} 2023-10-12T00:48:59.252 DEBUG:boto:Token: None 2023-10-12T00:48:59.252 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY= 2023-10-12T00:48:59.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY=', 'Content-Length': '0'} 2023-10-12T00:48:59.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf851d507a7f47f-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.302 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.305 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.306 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.306 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.307 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.307 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.307 DEBUG:boto:Data: 2023-10-12T00:48:59.308 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.309 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.309 DEBUG:boto:Params: {} 2023-10-12T00:48:59.310 DEBUG:boto:Token: None 2023-10-12T00:48:59.310 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1li19HVDoDDeaPe8rr1ZQy+Eh98= 2023-10-12T00:48:59.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1li19HVDoDDeaPe8rr1ZQy+Eh98=', 'Content-Length': '0'} 2023-10-12T00:48:59.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c13b52ad36a1d2-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.315 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.315 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.316 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.316 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.317 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.317 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.318 DEBUG:boto:Data: 2023-10-12T00:48:59.319 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.320 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.320 DEBUG:boto:Params: {} 2023-10-12T00:48:59.321 DEBUG:boto:Token: None 2023-10-12T00:48:59.322 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1li19HVDoDDeaPe8rr1ZQy+Eh98= 2023-10-12T00:48:59.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1li19HVDoDDeaPe8rr1ZQy+Eh98=', 'Content-Length': '0'} 2023-10-12T00:48:59.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f35ca5bcc1b5d52-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:59.326 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.326 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.327 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.327 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.328 DEBUG:boto:Method: GET 2023-10-12T00:48:59.328 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.329 DEBUG:boto:Data: 2023-10-12T00:48:59.329 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.331 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.333 DEBUG:boto:Params: {} 2023-10-12T00:48:59.333 DEBUG:boto:Token: None 2023-10-12T00:48:59.334 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY= 2023-10-12T00:48:59.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY=', 'Content-Length': '0'} 2023-10-12T00:48:59.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbac1d83d2dde1d9-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.338 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.338 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.339 DEBUG:boto:path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.339 DEBUG:boto:auth_path=/wafmoa-55/d?versionId=null 2023-10-12T00:48:59.340 DEBUG:boto:Method: GET 2023-10-12T00:48:59.340 DEBUG:boto:Path: /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.341 DEBUG:boto:Data: 2023-10-12T00:48:59.341 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.342 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.342 DEBUG:boto:Params: {} 2023-10-12T00:48:59.343 DEBUG:boto:Token: None 2023-10-12T00:48:59.344 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d?versionId=null 2023-10-12T00:48:59.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY= 2023-10-12T00:48:59.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6W2gAANK2kHEI9xMQlmmYnqtkY=', 'Content-Length': '0'} 2023-10-12T00:48:59.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05a19af7532ab40-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.347 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.348 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.348 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.348 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.349 DEBUG:boto:Method: GET 2023-10-12T00:48:59.349 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:59.350 DEBUG:boto:Data: 2023-10-12T00:48:59.350 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.351 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.352 DEBUG:boto:Params: {} 2023-10-12T00:48:59.352 DEBUG:boto:Token: None 2023-10-12T00:48:59.353 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb0c1e4cc504f64f-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.357 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.358 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.358 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.359 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.359 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.359 DEBUG:boto:Method: GET 2023-10-12T00:48:59.360 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:59.360 DEBUG:boto:Data: 2023-10-12T00:48:59.361 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.362 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.362 DEBUG:boto:Params: {} 2023-10-12T00:48:59.363 DEBUG:boto:Token: None 2023-10-12T00:48:59.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f444fb8668aa5512-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.367 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.367 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.368 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.368 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.369 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:59.369 DEBUG:boto:Data: 2023-10-12T00:48:59.370 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.370 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.371 DEBUG:boto:Params: {} 2023-10-12T00:48:59.371 DEBUG:boto:Token: None 2023-10-12T00:48:59.372 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d 2023-10-12T00:48:59.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:csRlyO1+I1bKc8833ik9d5KIqrk= 2023-10-12T00:48:59.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:csRlyO1+I1bKc8833ik9d5KIqrk=', 'Content-Length': '0'} 2023-10-12T00:48:59.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1bc903f5e7f85f-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.375 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.376 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.376 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.376 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:59.377 DEBUG:boto:Data: 2023-10-12T00:48:59.377 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.378 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.379 DEBUG:boto:Params: {} 2023-10-12T00:48:59.379 DEBUG:boto:Token: None 2023-10-12T00:48:59.379 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d 2023-10-12T00:48:59.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:csRlyO1+I1bKc8833ik9d5KIqrk= 2023-10-12T00:48:59.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:csRlyO1+I1bKc8833ik9d5KIqrk=', 'Content-Length': '0'} 2023-10-12T00:48:59.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3859c44839dfec-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:48:59.383 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.384 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.384 DEBUG:boto:Method: GET 2023-10-12T00:48:59.385 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:59.385 DEBUG:boto:Data: 2023-10-12T00:48:59.386 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.387 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.387 DEBUG:boto:Params: {} 2023-10-12T00:48:59.388 DEBUG:boto:Token: None 2023-10-12T00:48:59.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d 2023-10-12T00:48:59.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BdZiVHUdcoFSrQSGKlZJx08pmC8= 2023-10-12T00:48:59.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BdZiVHUdcoFSrQSGKlZJx08pmC8=', 'Content-Length': '0'} 2023-10-12T00:48:59.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd142220354bf227-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.392 DEBUG:boto:path=/wafmoa-55/d 2023-10-12T00:48:59.392 DEBUG:boto:auth_path=/wafmoa-55/d 2023-10-12T00:48:59.393 DEBUG:boto:Method: GET 2023-10-12T00:48:59.393 DEBUG:boto:Path: /wafmoa-55/d 2023-10-12T00:48:59.393 DEBUG:boto:Data: 2023-10-12T00:48:59.394 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.395 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.395 DEBUG:boto:Params: {} 2023-10-12T00:48:59.396 DEBUG:boto:Token: None 2023-10-12T00:48:59.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/d 2023-10-12T00:48:59.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BdZiVHUdcoFSrQSGKlZJx08pmC8= 2023-10-12T00:48:59.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BdZiVHUdcoFSrQSGKlZJx08pmC8=', 'Content-Length': '0'} 2023-10-12T00:48:59.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:44:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004161acdee1b31d79-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.442 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.443 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.443 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.444 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.444 DEBUG:boto:Method: GET 2023-10-12T00:48:59.445 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:59.445 DEBUG:boto:Data: 2023-10-12T00:48:59.446 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.447 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.447 DEBUG:boto:Params: {} 2023-10-12T00:48:59.448 DEBUG:boto:Token: None 2023-10-12T00:48:59.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8c010f0307153e-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.452 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.453 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.454 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.454 DEBUG:boto:path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.455 DEBUG:boto:auth_path=/wafmoa-55/?prefix=d 2023-10-12T00:48:59.455 DEBUG:boto:Method: GET 2023-10-12T00:48:59.456 DEBUG:boto:Path: /wafmoa-55/?prefix=d 2023-10-12T00:48:59.456 DEBUG:boto:Data: 2023-10-12T00:48:59.456 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.457 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.458 DEBUG:boto:Params: {} 2023-10-12T00:48:59.458 DEBUG:boto:Token: None 2023-10-12T00:48:59.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea71a118ea4c1ed7-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.463 DEBUG:boto:b'wafmoa-55d1000falsed2023-10-12T00:44:36.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:59.464 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.465 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.465 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.466 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.466 DEBUG:boto:Method: GET 2023-10-12T00:48:59.467 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.467 DEBUG:boto:Data: 2023-10-12T00:48:59.467 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.468 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.469 DEBUG:boto:Params: {} 2023-10-12T00:48:59.469 DEBUG:boto:Token: None 2023-10-12T00:48:59.470 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0= 2023-10-12T00:48:59.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0=', 'Content-Length': '0'} 2023-10-12T00:48:59.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f899faf127efef72-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.514 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.515 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.515 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.516 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.516 DEBUG:boto:Method: GET 2023-10-12T00:48:59.517 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.517 DEBUG:boto:Data: 2023-10-12T00:48:59.518 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.518 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.519 DEBUG:boto:Params: {} 2023-10-12T00:48:59.519 DEBUG:boto:Token: None 2023-10-12T00:48:59.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0= 2023-10-12T00:48:59.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0=', 'Content-Length': '0'} 2023-10-12T00:48:59.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db1b63e1da0af690-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.523 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.524 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.524 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.525 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.525 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.525 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.526 DEBUG:boto:Data: 2023-10-12T00:48:59.526 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.527 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.528 DEBUG:boto:Params: {} 2023-10-12T00:48:59.528 DEBUG:boto:Token: None 2023-10-12T00:48:59.529 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C61iHSMa8gUte59Am06BUyVHgM4= 2023-10-12T00:48:59.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C61iHSMa8gUte59Am06BUyVHgM4=', 'Content-Length': '0'} 2023-10-12T00:48:59.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad7f15422eb11a5-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.532 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.533 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.533 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.533 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.534 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.534 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.535 DEBUG:boto:Data: 2023-10-12T00:48:59.535 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.536 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.536 DEBUG:boto:Params: {} 2023-10-12T00:48:59.537 DEBUG:boto:Token: None 2023-10-12T00:48:59.537 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C61iHSMa8gUte59Am06BUyVHgM4= 2023-10-12T00:48:59.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C61iHSMa8gUte59Am06BUyVHgM4=', 'Content-Length': '0'} 2023-10-12T00:48:59.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffa2a3bc1f05f93-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:59.541 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.542 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.542 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.543 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.543 DEBUG:boto:Method: GET 2023-10-12T00:48:59.544 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.544 DEBUG:boto:Data: 2023-10-12T00:48:59.544 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.545 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.546 DEBUG:boto:Params: {} 2023-10-12T00:48:59.546 DEBUG:boto:Token: None 2023-10-12T00:48:59.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0= 2023-10-12T00:48:59.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0=', 'Content-Length': '0'} 2023-10-12T00:48:59.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a99a0b0ada51e3a-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.550 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.551 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.551 DEBUG:boto:path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.552 DEBUG:boto:auth_path=/wafmoa-55/e?versionId=null 2023-10-12T00:48:59.552 DEBUG:boto:Method: GET 2023-10-12T00:48:59.553 DEBUG:boto:Path: /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.553 DEBUG:boto:Data: 2023-10-12T00:48:59.554 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.555 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.555 DEBUG:boto:Params: {} 2023-10-12T00:48:59.555 DEBUG:boto:Token: None 2023-10-12T00:48:59.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e?versionId=null 2023-10-12T00:48:59.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0= 2023-10-12T00:48:59.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cgx980AHjgSh+xmQ72CGQKSi5U0=', 'Content-Length': '0'} 2023-10-12T00:48:59.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e5d1212274095a-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.560 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.560 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.561 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.561 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.562 DEBUG:boto:Method: GET 2023-10-12T00:48:59.562 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:59.563 DEBUG:boto:Data: 2023-10-12T00:48:59.563 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.564 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.564 DEBUG:boto:Params: {} 2023-10-12T00:48:59.565 DEBUG:boto:Token: None 2023-10-12T00:48:59.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b6f2185ac6448b-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.570 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.570 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.571 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.571 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.572 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.572 DEBUG:boto:Method: GET 2023-10-12T00:48:59.573 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:59.573 DEBUG:boto:Data: 2023-10-12T00:48:59.574 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.575 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.575 DEBUG:boto:Params: {} 2023-10-12T00:48:59.576 DEBUG:boto:Token: None 2023-10-12T00:48:59.576 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e110c01ea12df11-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.622 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.623 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.624 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.624 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.625 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:59.625 DEBUG:boto:Data: 2023-10-12T00:48:59.625 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.626 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.627 DEBUG:boto:Params: {} 2023-10-12T00:48:59.627 DEBUG:boto:Token: None 2023-10-12T00:48:59.628 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e 2023-10-12T00:48:59.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QvoMk2A9POWJZIjkrGF4XQAqWFQ= 2023-10-12T00:48:59.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QvoMk2A9POWJZIjkrGF4XQAqWFQ=', 'Content-Length': '0'} 2023-10-12T00:48:59.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883a842dd28fd4ed-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.631 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.632 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.632 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.633 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:59.633 DEBUG:boto:Data: 2023-10-12T00:48:59.633 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.634 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.635 DEBUG:boto:Params: {} 2023-10-12T00:48:59.635 DEBUG:boto:Token: None 2023-10-12T00:48:59.635 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e 2023-10-12T00:48:59.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QvoMk2A9POWJZIjkrGF4XQAqWFQ= 2023-10-12T00:48:59.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QvoMk2A9POWJZIjkrGF4XQAqWFQ=', 'Content-Length': '0'} 2023-10-12T00:48:59.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd35416191f483a-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:48:59.640 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.641 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.642 DEBUG:boto:Method: GET 2023-10-12T00:48:59.642 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:59.643 DEBUG:boto:Data: 2023-10-12T00:48:59.643 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.644 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.645 DEBUG:boto:Params: {} 2023-10-12T00:48:59.645 DEBUG:boto:Token: None 2023-10-12T00:48:59.646 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e 2023-10-12T00:48:59.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j02F8PpNyrPDtnU84bGzzxKO5ko= 2023-10-12T00:48:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j02F8PpNyrPDtnU84bGzzxKO5ko=', 'Content-Length': '0'} 2023-10-12T00:48:59.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004feb3b4479554cd-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.649 DEBUG:boto:path=/wafmoa-55/e 2023-10-12T00:48:59.650 DEBUG:boto:auth_path=/wafmoa-55/e 2023-10-12T00:48:59.651 DEBUG:boto:Method: GET 2023-10-12T00:48:59.651 DEBUG:boto:Path: /wafmoa-55/e 2023-10-12T00:48:59.652 DEBUG:boto:Data: 2023-10-12T00:48:59.653 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.654 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.655 DEBUG:boto:Params: {} 2023-10-12T00:48:59.656 DEBUG:boto:Token: None 2023-10-12T00:48:59.657 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/e 2023-10-12T00:48:59.658 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j02F8PpNyrPDtnU84bGzzxKO5ko= 2023-10-12T00:48:59.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j02F8PpNyrPDtnU84bGzzxKO5ko=', 'Content-Length': '0'} 2023-10-12T00:48:59.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e50314acc90c78-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.662 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.663 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.663 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.663 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.664 DEBUG:boto:Method: GET 2023-10-12T00:48:59.664 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:59.665 DEBUG:boto:Data: 2023-10-12T00:48:59.665 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.666 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.667 DEBUG:boto:Params: {} 2023-10-12T00:48:59.667 DEBUG:boto:Token: None 2023-10-12T00:48:59.668 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306dd607b44669a3-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.672 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.673 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.673 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.673 DEBUG:boto:path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.674 DEBUG:boto:auth_path=/wafmoa-55/?prefix=e 2023-10-12T00:48:59.674 DEBUG:boto:Method: GET 2023-10-12T00:48:59.675 DEBUG:boto:Path: /wafmoa-55/?prefix=e 2023-10-12T00:48:59.675 DEBUG:boto:Data: 2023-10-12T00:48:59.676 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.676 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.677 DEBUG:boto:Params: {} 2023-10-12T00:48:59.677 DEBUG:boto:Token: None 2023-10-12T00:48:59.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338ad72b723e2d3d-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.682 DEBUG:boto:b'wafmoa-55e1000falsee2023-10-12T00:45:58.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:59.683 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.683 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.685 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.685 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.686 DEBUG:boto:Method: GET 2023-10-12T00:48:59.686 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.687 DEBUG:boto:Data: 2023-10-12T00:48:59.688 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.689 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.689 DEBUG:boto:Params: {} 2023-10-12T00:48:59.689 DEBUG:boto:Token: None 2023-10-12T00:48:59.690 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.690 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk= 2023-10-12T00:48:59.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk=', 'Content-Length': '0'} 2023-10-12T00:48:59.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed0b0baf2c49160-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.734 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.735 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.735 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.736 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.736 DEBUG:boto:Method: GET 2023-10-12T00:48:59.737 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.737 DEBUG:boto:Data: 2023-10-12T00:48:59.738 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.739 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.739 DEBUG:boto:Params: {} 2023-10-12T00:48:59.740 DEBUG:boto:Token: None 2023-10-12T00:48:59.740 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk= 2023-10-12T00:48:59.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk=', 'Content-Length': '0'} 2023-10-12T00:48:59.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bd12ba173ef7c5-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.744 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.744 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.744 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.745 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.745 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.746 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.746 DEBUG:boto:Data: 2023-10-12T00:48:59.747 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.748 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.748 DEBUG:boto:Params: {} 2023-10-12T00:48:59.748 DEBUG:boto:Token: None 2023-10-12T00:48:59.749 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CX76o3MU+hzK2+gSKWswhty7RsM= 2023-10-12T00:48:59.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CX76o3MU+hzK2+gSKWswhty7RsM=', 'Content-Length': '0'} 2023-10-12T00:48:59.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e0739f480c09747-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.752 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.753 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.753 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.754 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.754 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.755 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.755 DEBUG:boto:Data: 2023-10-12T00:48:59.755 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.756 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.757 DEBUG:boto:Params: {} 2023-10-12T00:48:59.757 DEBUG:boto:Token: None 2023-10-12T00:48:59.758 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CX76o3MU+hzK2+gSKWswhty7RsM= 2023-10-12T00:48:59.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CX76o3MU+hzK2+gSKWswhty7RsM=', 'Content-Length': '0'} 2023-10-12T00:48:59.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a81a6d581dadda5-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:59.761 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.762 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.762 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.763 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.763 DEBUG:boto:Method: GET 2023-10-12T00:48:59.764 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.764 DEBUG:boto:Data: 2023-10-12T00:48:59.764 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.765 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.766 DEBUG:boto:Params: {} 2023-10-12T00:48:59.766 DEBUG:boto:Token: None 2023-10-12T00:48:59.767 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk= 2023-10-12T00:48:59.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk=', 'Content-Length': '0'} 2023-10-12T00:48:59.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e37a83ef72fa253-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.770 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.771 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.771 DEBUG:boto:path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.772 DEBUG:boto:auth_path=/wafmoa-55/f?versionId=null 2023-10-12T00:48:59.772 DEBUG:boto:Method: GET 2023-10-12T00:48:59.772 DEBUG:boto:Path: /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.773 DEBUG:boto:Data: 2023-10-12T00:48:59.773 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.774 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.775 DEBUG:boto:Params: {} 2023-10-12T00:48:59.775 DEBUG:boto:Token: None 2023-10-12T00:48:59.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f?versionId=null 2023-10-12T00:48:59.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk= 2023-10-12T00:48:59.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uxGDgPY1u0HaTet1aqXGAfKwqsk=', 'Content-Length': '0'} 2023-10-12T00:48:59.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db5613d67330196-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.822 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.823 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.823 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.824 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.824 DEBUG:boto:Method: GET 2023-10-12T00:48:59.825 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:59.825 DEBUG:boto:Data: 2023-10-12T00:48:59.826 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.827 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.828 DEBUG:boto:Params: {} 2023-10-12T00:48:59.828 DEBUG:boto:Token: None 2023-10-12T00:48:59.829 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9decb613c842bf-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.833 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.834 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.834 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.835 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.835 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.836 DEBUG:boto:Method: GET 2023-10-12T00:48:59.836 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:59.837 DEBUG:boto:Data: 2023-10-12T00:48:59.837 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.838 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.839 DEBUG:boto:Params: {} 2023-10-12T00:48:59.839 DEBUG:boto:Token: None 2023-10-12T00:48:59.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552ee3811dfa2442-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.843 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.844 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.844 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.845 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.845 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:59.846 DEBUG:boto:Data: 2023-10-12T00:48:59.846 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.847 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.847 DEBUG:boto:Params: {} 2023-10-12T00:48:59.848 DEBUG:boto:Token: None 2023-10-12T00:48:59.848 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f 2023-10-12T00:48:59.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7X2gLdm+rPtygifbf9I/daU6DCs= 2023-10-12T00:48:59.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7X2gLdm+rPtygifbf9I/daU6DCs=', 'Content-Length': '0'} 2023-10-12T00:48:59.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ab5380f943eed9-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.852 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.852 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.852 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.853 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:59.853 DEBUG:boto:Data: 2023-10-12T00:48:59.854 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.854 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.855 DEBUG:boto:Params: {} 2023-10-12T00:48:59.856 DEBUG:boto:Token: None 2023-10-12T00:48:59.856 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f 2023-10-12T00:48:59.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7X2gLdm+rPtygifbf9I/daU6DCs= 2023-10-12T00:48:59.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7X2gLdm+rPtygifbf9I/daU6DCs=', 'Content-Length': '0'} 2023-10-12T00:48:59.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f85a86891d06cf7b-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:48:59.860 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.860 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.861 DEBUG:boto:Method: GET 2023-10-12T00:48:59.861 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:59.862 DEBUG:boto:Data: 2023-10-12T00:48:59.862 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.863 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.863 DEBUG:boto:Params: {} 2023-10-12T00:48:59.864 DEBUG:boto:Token: None 2023-10-12T00:48:59.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f 2023-10-12T00:48:59.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4GvGWz97VfoFWjHnuEVh9kWFu8Q= 2023-10-12T00:48:59.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4GvGWz97VfoFWjHnuEVh9kWFu8Q=', 'Content-Length': '0'} 2023-10-12T00:48:59.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7cbdb05121ee74-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.868 DEBUG:boto:path=/wafmoa-55/f 2023-10-12T00:48:59.869 DEBUG:boto:auth_path=/wafmoa-55/f 2023-10-12T00:48:59.869 DEBUG:boto:Method: GET 2023-10-12T00:48:59.870 DEBUG:boto:Path: /wafmoa-55/f 2023-10-12T00:48:59.870 DEBUG:boto:Data: 2023-10-12T00:48:59.871 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.872 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.872 DEBUG:boto:Params: {} 2023-10-12T00:48:59.873 DEBUG:boto:Token: None 2023-10-12T00:48:59.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/f 2023-10-12T00:48:59.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4GvGWz97VfoFWjHnuEVh9kWFu8Q= 2023-10-12T00:48:59.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4GvGWz97VfoFWjHnuEVh9kWFu8Q=', 'Content-Length': '0'} 2023-10-12T00:48:59.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3392dbef0f2692d-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.918 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.919 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.919 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.920 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.920 DEBUG:boto:Method: GET 2023-10-12T00:48:59.921 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:59.921 DEBUG:boto:Data: 2023-10-12T00:48:59.922 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.923 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.923 DEBUG:boto:Params: {} 2023-10-12T00:48:59.924 DEBUG:boto:Token: None 2023-10-12T00:48:59.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c4396cc21ec723-006527427b-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.928 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.929 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:48:59.930 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:48:59.930 DEBUG:boto:path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.931 DEBUG:boto:auth_path=/wafmoa-55/?prefix=f 2023-10-12T00:48:59.931 DEBUG:boto:Method: GET 2023-10-12T00:48:59.932 DEBUG:boto:Path: /wafmoa-55/?prefix=f 2023-10-12T00:48:59.932 DEBUG:boto:Data: 2023-10-12T00:48:59.932 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.933 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.934 DEBUG:boto:Params: {} 2023-10-12T00:48:59.934 DEBUG:boto:Token: None 2023-10-12T00:48:59.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/ 2023-10-12T00:48:59.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4= 2023-10-12T00:48:59.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+ROW1aCFdW+LV+HqCaM2R0ySfT4=', 'Content-Length': '0'} 2023-10-12T00:48:59.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4c39ca4b9b62b3e-006527427b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.939 DEBUG:boto:b'wafmoa-55f1000falsef2023-10-12T00:45:58.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:48:59.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:59.940 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:59.940 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:59.941 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.941 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.942 DEBUG:boto:Method: GET 2023-10-12T00:48:59.942 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.943 DEBUG:boto:Data: 2023-10-12T00:48:59.943 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.944 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.944 DEBUG:boto:Params: {} 2023-10-12T00:48:59.945 DEBUG:boto:Token: None 2023-10-12T00:48:59.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w= 2023-10-12T00:48:59.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w=', 'Content-Length': '0'} 2023-10-12T00:48:59.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a90d8078ca6867-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.949 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:59.950 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:59.950 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.951 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.951 DEBUG:boto:Method: GET 2023-10-12T00:48:59.951 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.952 DEBUG:boto:Data: 2023-10-12T00:48:59.952 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.953 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.954 DEBUG:boto:Params: {} 2023-10-12T00:48:59.954 DEBUG:boto:Token: None 2023-10-12T00:48:59.955 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w= 2023-10-12T00:48:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w=', 'Content-Length': '0'} 2023-10-12T00:48:59.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f387269f8dd89489-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.959 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:59.959 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:59.960 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.960 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.960 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.961 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.961 DEBUG:boto:Data: 2023-10-12T00:48:59.962 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.962 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.963 DEBUG:boto:Params: {} 2023-10-12T00:48:59.963 DEBUG:boto:Token: None 2023-10-12T00:48:59.964 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QDXuk6FqTitVay/aP00VWp8cH0Q= 2023-10-12T00:48:59.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QDXuk6FqTitVay/aP00VWp8cH0Q=', 'Content-Length': '0'} 2023-10-12T00:48:59.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcc056e43c4f3e8-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.968 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:59.968 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:59.968 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.969 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.969 DEBUG:boto:Method: HEAD 2023-10-12T00:48:59.970 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.970 DEBUG:boto:Data: 2023-10-12T00:48:59.971 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:48:59.971 DEBUG:boto:Port: 8000 2023-10-12T00:48:59.972 DEBUG:boto:Params: {} 2023-10-12T00:48:59.972 DEBUG:boto:Token: None 2023-10-12T00:48:59.973 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QDXuk6FqTitVay/aP00VWp8cH0Q= 2023-10-12T00:48:59.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QDXuk6FqTitVay/aP00VWp8cH0Q=', 'Content-Length': '0'} 2023-10-12T00:48:59.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426e6f24b6cd18c9-006527427b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:48:59.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:48:59.977 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:48:59.978 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:48:59.978 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.978 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:48:59.979 DEBUG:boto:Method: GET 2023-10-12T00:48:59.979 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.980 DEBUG:boto:Data: 2023-10-12T00:48:59.980 DEBUG:boto:Headers: {} 2023-10-12T00:48:59.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:48:59.981 DEBUG:boto:Port: 8001 2023-10-12T00:48:59.982 DEBUG:boto:Params: {} 2023-10-12T00:48:59.982 DEBUG:boto:Token: None 2023-10-12T00:48:59.983 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:48:59 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:48:59.983 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w= 2023-10-12T00:48:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:48:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:57ehxJ3Mv4MYkOzSYPUtffS513w=', 'Content-Length': '0'} 2023-10-12T00:48:59.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c9ef72dfe86180-006527427b-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:48:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.026 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:49:00.027 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:49:00.027 DEBUG:boto:path=/wafmoa-55/g?versionId=null 2023-10-12T00:49:00.028 DEBUG:boto:auth_path=/wafmoa-55/g?versionId=null 2023-10-12T00:49:00.028 DEBUG:boto:Method: GET 2023-10-12T00:49:00.029 DEBUG:boto:Path: /wafmoa-55/g?versionId=null 2023-10-12T00:49:00.029 DEBUG:boto:Data: 2023-10-12T00:49:00.030 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.031 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.031 DEBUG:boto:Params: {} 2023-10-12T00:49:00.032 DEBUG:boto:Token: None 2023-10-12T00:49:00.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/g?versionId=null 2023-10-12T00:49:00.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PX2r2atEq4Q2s+ynEC3QbofIwIo= 2023-10-12T00:49:00.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PX2r2atEq4Q2s+ynEC3QbofIwIo=', 'Content-Length': '0'} 2023-10-12T00:49:00.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ae32cdfe5d73c7-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.037 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.038 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.038 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.039 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.039 DEBUG:boto:Method: GET 2023-10-12T00:49:00.040 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:49:00.040 DEBUG:boto:Data: 2023-10-12T00:49:00.040 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.041 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.042 DEBUG:boto:Params: {} 2023-10-12T00:49:00.042 DEBUG:boto:Token: None 2023-10-12T00:49:00.043 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.043 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e1346a06eb6dbfb-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.048 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.049 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.049 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.050 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.050 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.051 DEBUG:boto:Method: GET 2023-10-12T00:49:00.051 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:49:00.052 DEBUG:boto:Data: 2023-10-12T00:49:00.052 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.053 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.054 DEBUG:boto:Params: {} 2023-10-12T00:49:00.054 DEBUG:boto:Token: None 2023-10-12T00:49:00.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097618ede2e518ea3-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.058 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.059 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:49:00.060 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:49:00.060 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.061 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:49:00.061 DEBUG:boto:Data: 2023-10-12T00:49:00.061 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.062 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.063 DEBUG:boto:Params: {} 2023-10-12T00:49:00.063 DEBUG:boto:Token: None 2023-10-12T00:49:00.064 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/g 2023-10-12T00:49:00.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DBN2YmrPqGHoPM2rAW/ScE+K3Yk= 2023-10-12T00:49:00.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DBN2YmrPqGHoPM2rAW/ScE+K3Yk=', 'Content-Length': '0'} 2023-10-12T00:49:00.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fedadf2aab7b5afb-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.067 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:49:00.067 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:49:00.068 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.068 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:49:00.069 DEBUG:boto:Data: 2023-10-12T00:49:00.069 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.070 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.070 DEBUG:boto:Params: {} 2023-10-12T00:49:00.071 DEBUG:boto:Token: None 2023-10-12T00:49:00.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/g 2023-10-12T00:49:00.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DBN2YmrPqGHoPM2rAW/ScE+K3Yk= 2023-10-12T00:49:00.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DBN2YmrPqGHoPM2rAW/ScE+K3Yk=', 'Content-Length': '0'} 2023-10-12T00:49:00.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e131b6d9890af4d-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:49:00.075 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:49:00.075 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:49:00.076 DEBUG:boto:Method: GET 2023-10-12T00:49:00.076 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:49:00.077 DEBUG:boto:Data: 2023-10-12T00:49:00.077 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.078 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.078 DEBUG:boto:Params: {} 2023-10-12T00:49:00.079 DEBUG:boto:Token: None 2023-10-12T00:49:00.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/g 2023-10-12T00:49:00.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WN2HSQPxR89tlqNXJ8VE/AJvbAQ= 2023-10-12T00:49:00.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WN2HSQPxR89tlqNXJ8VE/AJvbAQ=', 'Content-Length': '0'} 2023-10-12T00:49:00.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9e698813ce7ec3-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.126 DEBUG:boto:path=/wafmoa-55/g 2023-10-12T00:49:00.127 DEBUG:boto:auth_path=/wafmoa-55/g 2023-10-12T00:49:00.127 DEBUG:boto:Method: GET 2023-10-12T00:49:00.128 DEBUG:boto:Path: /wafmoa-55/g 2023-10-12T00:49:00.128 DEBUG:boto:Data: 2023-10-12T00:49:00.128 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.129 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.130 DEBUG:boto:Params: {} 2023-10-12T00:49:00.130 DEBUG:boto:Token: None 2023-10-12T00:49:00.131 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/g 2023-10-12T00:49:00.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WN2HSQPxR89tlqNXJ8VE/AJvbAQ= 2023-10-12T00:49:00.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WN2HSQPxR89tlqNXJ8VE/AJvbAQ=', 'Content-Length': '0'} 2023-10-12T00:49:00.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047904d9f07701cf6-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.135 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.135 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.136 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.136 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.136 DEBUG:boto:Method: GET 2023-10-12T00:49:00.137 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:49:00.137 DEBUG:boto:Data: 2023-10-12T00:49:00.138 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.139 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.139 DEBUG:boto:Params: {} 2023-10-12T00:49:00.140 DEBUG:boto:Token: None 2023-10-12T00:49:00.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076babedce1a804e9-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.144 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.145 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.145 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.146 DEBUG:boto:path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.146 DEBUG:boto:auth_path=/wafmoa-55/?prefix=g 2023-10-12T00:49:00.147 DEBUG:boto:Method: GET 2023-10-12T00:49:00.147 DEBUG:boto:Path: /wafmoa-55/?prefix=g 2023-10-12T00:49:00.148 DEBUG:boto:Data: 2023-10-12T00:49:00.148 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.149 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.149 DEBUG:boto:Params: {} 2023-10-12T00:49:00.150 DEBUG:boto:Token: None 2023-10-12T00:49:00.150 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.151 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000993f23739b670444-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.155 DEBUG:boto:b'wafmoa-55g1000falseg2023-10-12T00:45:58.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:49:00.156 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.157 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.157 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.158 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.158 DEBUG:boto:Method: GET 2023-10-12T00:49:00.159 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.159 DEBUG:boto:Data: 2023-10-12T00:49:00.160 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.161 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.161 DEBUG:boto:Params: {} 2023-10-12T00:49:00.162 DEBUG:boto:Token: None 2023-10-12T00:49:00.162 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.163 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho= 2023-10-12T00:49:00.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho=', 'Content-Length': '0'} 2023-10-12T00:49:00.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000360b2d27575ad76-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.166 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.166 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.167 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.167 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.167 DEBUG:boto:Method: GET 2023-10-12T00:49:00.168 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.168 DEBUG:boto:Data: 2023-10-12T00:49:00.169 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.170 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.170 DEBUG:boto:Params: {} 2023-10-12T00:49:00.171 DEBUG:boto:Token: None 2023-10-12T00:49:00.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho= 2023-10-12T00:49:00.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho=', 'Content-Length': '0'} 2023-10-12T00:49:00.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7778d4515f0202-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.175 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.175 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.176 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.176 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.177 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.177 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.178 DEBUG:boto:Data: 2023-10-12T00:49:00.178 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.179 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.179 DEBUG:boto:Params: {} 2023-10-12T00:49:00.180 DEBUG:boto:Token: None 2023-10-12T00:49:00.180 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UGXzj2Dp/v/HjmXS1lpMawaNXq0= 2023-10-12T00:49:00.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UGXzj2Dp/v/HjmXS1lpMawaNXq0=', 'Content-Length': '0'} 2023-10-12T00:49:00.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe145f1fde94b601-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.184 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.184 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.185 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.185 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.186 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.186 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.187 DEBUG:boto:Data: 2023-10-12T00:49:00.187 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.188 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.188 DEBUG:boto:Params: {} 2023-10-12T00:49:00.189 DEBUG:boto:Token: None 2023-10-12T00:49:00.190 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.190 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UGXzj2Dp/v/HjmXS1lpMawaNXq0= 2023-10-12T00:49:00.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UGXzj2Dp/v/HjmXS1lpMawaNXq0=', 'Content-Length': '0'} 2023-10-12T00:49:00.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005caa108fcfa3df45-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:49:00.194 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.194 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.194 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.195 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.195 DEBUG:boto:Method: GET 2023-10-12T00:49:00.196 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.196 DEBUG:boto:Data: 2023-10-12T00:49:00.197 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.198 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.198 DEBUG:boto:Params: {} 2023-10-12T00:49:00.199 DEBUG:boto:Token: None 2023-10-12T00:49:00.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.199 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho= 2023-10-12T00:49:00.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho=', 'Content-Length': '0'} 2023-10-12T00:49:00.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375e671e500f2668-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.246 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.247 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.247 DEBUG:boto:path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.248 DEBUG:boto:auth_path=/wafmoa-55/h?versionId=null 2023-10-12T00:49:00.248 DEBUG:boto:Method: GET 2023-10-12T00:49:00.249 DEBUG:boto:Path: /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.249 DEBUG:boto:Data: 2023-10-12T00:49:00.250 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.251 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.252 DEBUG:boto:Params: {} 2023-10-12T00:49:00.252 DEBUG:boto:Token: None 2023-10-12T00:49:00.253 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h?versionId=null 2023-10-12T00:49:00.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho= 2023-10-12T00:49:00.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pfw6IbSkh62b8vFkSZd7vYcc6ho=', 'Content-Length': '0'} 2023-10-12T00:49:00.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000313ef47973505f9-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.257 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.258 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.258 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.259 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.259 DEBUG:boto:Method: GET 2023-10-12T00:49:00.260 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:49:00.260 DEBUG:boto:Data: 2023-10-12T00:49:00.261 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.262 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.262 DEBUG:boto:Params: {} 2023-10-12T00:49:00.263 DEBUG:boto:Token: None 2023-10-12T00:49:00.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f6de7db136c22e7-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.268 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.268 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.269 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.269 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.270 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.270 DEBUG:boto:Method: GET 2023-10-12T00:49:00.271 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:49:00.272 DEBUG:boto:Data: 2023-10-12T00:49:00.272 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.273 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.274 DEBUG:boto:Params: {} 2023-10-12T00:49:00.274 DEBUG:boto:Token: None 2023-10-12T00:49:00.275 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9186246db93ae5-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.278 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.279 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.280 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.280 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.280 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:49:00.281 DEBUG:boto:Data: 2023-10-12T00:49:00.281 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.283 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.283 DEBUG:boto:Params: {} 2023-10-12T00:49:00.283 DEBUG:boto:Token: None 2023-10-12T00:49:00.284 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h 2023-10-12T00:49:00.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PT0Zb8Gfol54es2v1i1P82koCcg= 2023-10-12T00:49:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PT0Zb8Gfol54es2v1i1P82koCcg=', 'Content-Length': '0'} 2023-10-12T00:49:00.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6aa0801af42d3ab-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.287 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.288 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.288 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.289 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:49:00.289 DEBUG:boto:Data: 2023-10-12T00:49:00.290 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.290 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.291 DEBUG:boto:Params: {} 2023-10-12T00:49:00.291 DEBUG:boto:Token: None 2023-10-12T00:49:00.292 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h 2023-10-12T00:49:00.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PT0Zb8Gfol54es2v1i1P82koCcg= 2023-10-12T00:49:00.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PT0Zb8Gfol54es2v1i1P82koCcg=', 'Content-Length': '0'} 2023-10-12T00:49:00.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e7922356ba78ee-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:49:00.296 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.296 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.297 DEBUG:boto:Method: GET 2023-10-12T00:49:00.297 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:49:00.298 DEBUG:boto:Data: 2023-10-12T00:49:00.298 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.299 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.300 DEBUG:boto:Params: {} 2023-10-12T00:49:00.300 DEBUG:boto:Token: None 2023-10-12T00:49:00.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h 2023-10-12T00:49:00.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a/+w/dzzwDsH2HyVjryJgyIEbEk= 2023-10-12T00:49:00.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a/+w/dzzwDsH2HyVjryJgyIEbEk=', 'Content-Length': '0'} 2023-10-12T00:49:00.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006826726347a76cb0-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.350 DEBUG:boto:path=/wafmoa-55/h 2023-10-12T00:49:00.351 DEBUG:boto:auth_path=/wafmoa-55/h 2023-10-12T00:49:00.351 DEBUG:boto:Method: GET 2023-10-12T00:49:00.352 DEBUG:boto:Path: /wafmoa-55/h 2023-10-12T00:49:00.352 DEBUG:boto:Data: 2023-10-12T00:49:00.353 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.354 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.354 DEBUG:boto:Params: {} 2023-10-12T00:49:00.355 DEBUG:boto:Token: None 2023-10-12T00:49:00.355 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/h 2023-10-12T00:49:00.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a/+w/dzzwDsH2HyVjryJgyIEbEk= 2023-10-12T00:49:00.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a/+w/dzzwDsH2HyVjryJgyIEbEk=', 'Content-Length': '0'} 2023-10-12T00:49:00.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:45:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701bc6491e806756-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.359 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.360 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.360 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.360 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.361 DEBUG:boto:Method: GET 2023-10-12T00:49:00.361 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:49:00.362 DEBUG:boto:Data: 2023-10-12T00:49:00.362 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.363 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.363 DEBUG:boto:Params: {} 2023-10-12T00:49:00.364 DEBUG:boto:Token: None 2023-10-12T00:49:00.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007052bb70cdd6bd1f-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.369 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.369 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.370 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.370 DEBUG:boto:path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.371 DEBUG:boto:auth_path=/wafmoa-55/?prefix=h 2023-10-12T00:49:00.371 DEBUG:boto:Method: GET 2023-10-12T00:49:00.372 DEBUG:boto:Path: /wafmoa-55/?prefix=h 2023-10-12T00:49:00.372 DEBUG:boto:Data: 2023-10-12T00:49:00.372 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.373 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.374 DEBUG:boto:Params: {} 2023-10-12T00:49:00.374 DEBUG:boto:Token: None 2023-10-12T00:49:00.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d3b424bc6107aca-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.379 DEBUG:boto:b'wafmoa-55h1000falseh2023-10-12T00:45:58.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-10-12T00:49:00.380 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.380 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.381 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.381 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.382 DEBUG:boto:Method: GET 2023-10-12T00:49:00.382 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.383 DEBUG:boto:Data: 2023-10-12T00:49:00.383 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.384 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.384 DEBUG:boto:Params: {} 2023-10-12T00:49:00.385 DEBUG:boto:Token: None 2023-10-12T00:49:00.385 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.386 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8= 2023-10-12T00:49:00.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8=', 'Content-Length': '0'} 2023-10-12T00:49:00.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14325b179b52ac3-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.389 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.390 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.390 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.391 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.391 DEBUG:boto:Method: GET 2023-10-12T00:49:00.391 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.392 DEBUG:boto:Data: 2023-10-12T00:49:00.392 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.393 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.394 DEBUG:boto:Params: {} 2023-10-12T00:49:00.394 DEBUG:boto:Token: None 2023-10-12T00:49:00.395 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.395 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8= 2023-10-12T00:49:00.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8=', 'Content-Length': '0'} 2023-10-12T00:49:00.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb66c569c2faa7a-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.398 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.399 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.399 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.400 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.400 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.401 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.401 DEBUG:boto:Data: 2023-10-12T00:49:00.402 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.402 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.403 DEBUG:boto:Params: {} 2023-10-12T00:49:00.403 DEBUG:boto:Token: None 2023-10-12T00:49:00.404 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t5xRM23TG1X5ScLeSHvhwmxBVKc= 2023-10-12T00:49:00.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t5xRM23TG1X5ScLeSHvhwmxBVKc=', 'Content-Length': '0'} 2023-10-12T00:49:00.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54b7fca6a3a0520-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.407 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.408 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.408 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.409 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.409 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.410 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.410 DEBUG:boto:Data: 2023-10-12T00:49:00.411 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.412 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.412 DEBUG:boto:Params: {} 2023-10-12T00:49:00.412 DEBUG:boto:Token: None 2023-10-12T00:49:00.413 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t5xRM23TG1X5ScLeSHvhwmxBVKc= 2023-10-12T00:49:00.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t5xRM23TG1X5ScLeSHvhwmxBVKc=', 'Content-Length': '0'} 2023-10-12T00:49:00.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009eceebb1ca6fee2-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-10-12T00:49:00.417 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.418 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.418 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.419 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.419 DEBUG:boto:Method: GET 2023-10-12T00:49:00.420 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.420 DEBUG:boto:Data: 2023-10-12T00:49:00.421 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.422 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.422 DEBUG:boto:Params: {} 2023-10-12T00:49:00.422 DEBUG:boto:Token: None 2023-10-12T00:49:00.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8= 2023-10-12T00:49:00.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8=', 'Content-Length': '0'} 2023-10-12T00:49:00.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6935b1e2e3bbac8-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.466 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.467 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.467 DEBUG:boto:path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.468 DEBUG:boto:auth_path=/wafmoa-55/i?versionId=null 2023-10-12T00:49:00.468 DEBUG:boto:Method: GET 2023-10-12T00:49:00.469 DEBUG:boto:Path: /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.469 DEBUG:boto:Data: 2023-10-12T00:49:00.470 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.470 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.471 DEBUG:boto:Params: {} 2023-10-12T00:49:00.471 DEBUG:boto:Token: None 2023-10-12T00:49:00.472 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i?versionId=null 2023-10-12T00:49:00.472 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8= 2023-10-12T00:49:00.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KklSh6AG2VbEM5mfqJ57y7Iz6z8=', 'Content-Length': '0'} 2023-10-12T00:49:00.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ddedb43acc74d7-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.476 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.476 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.477 DEBUG:boto:path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.477 DEBUG:boto:auth_path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.478 DEBUG:boto:Method: GET 2023-10-12T00:49:00.478 DEBUG:boto:Path: /wafmoa-55/?prefix=i 2023-10-12T00:49:00.479 DEBUG:boto:Data: 2023-10-12T00:49:00.479 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.480 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.480 DEBUG:boto:Params: {} 2023-10-12T00:49:00.481 DEBUG:boto:Token: None 2023-10-12T00:49:00.481 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f264371f5953354-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.485 DEBUG:boto:b'wafmoa-55i1000falsei2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.486 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.487 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.487 DEBUG:boto:path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.487 DEBUG:boto:auth_path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.488 DEBUG:boto:Method: GET 2023-10-12T00:49:00.488 DEBUG:boto:Path: /wafmoa-55/?prefix=i 2023-10-12T00:49:00.489 DEBUG:boto:Data: 2023-10-12T00:49:00.489 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.490 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.491 DEBUG:boto:Params: {} 2023-10-12T00:49:00.491 DEBUG:boto:Token: None 2023-10-12T00:49:00.492 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061ffeb694972741d-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.538 DEBUG:boto:b'wafmoa-55i1000falsei2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.542 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.542 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.543 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.543 DEBUG:boto:Path: /wafmoa-55/i 2023-10-12T00:49:00.544 DEBUG:boto:Data: 2023-10-12T00:49:00.544 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.545 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.546 DEBUG:boto:Params: {} 2023-10-12T00:49:00.546 DEBUG:boto:Token: None 2023-10-12T00:49:00.547 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i 2023-10-12T00:49:00.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBkItzixULJQpSoxu/PBd5Z9Tlk= 2023-10-12T00:49:00.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBkItzixULJQpSoxu/PBd5Z9Tlk=', 'Content-Length': '0'} 2023-10-12T00:49:00.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092425f3a3dedde9-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.550 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.551 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.551 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.551 DEBUG:boto:Path: /wafmoa-55/i 2023-10-12T00:49:00.552 DEBUG:boto:Data: 2023-10-12T00:49:00.552 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.553 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.553 DEBUG:boto:Params: {} 2023-10-12T00:49:00.554 DEBUG:boto:Token: None 2023-10-12T00:49:00.554 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i 2023-10-12T00:49:00.555 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rBkItzixULJQpSoxu/PBd5Z9Tlk= 2023-10-12T00:49:00.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rBkItzixULJQpSoxu/PBd5Z9Tlk=', 'Content-Length': '0'} 2023-10-12T00:49:00.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f66b5ffa2d50b0c-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=i 2023-10-12T00:49:00.559 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.559 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.559 DEBUG:boto:Method: GET 2023-10-12T00:49:00.560 DEBUG:boto:Path: /wafmoa-55/i 2023-10-12T00:49:00.560 DEBUG:boto:Data: 2023-10-12T00:49:00.561 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.562 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.562 DEBUG:boto:Params: {} 2023-10-12T00:49:00.562 DEBUG:boto:Token: None 2023-10-12T00:49:00.563 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i 2023-10-12T00:49:00.563 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WcumaTvD1s/TBNk6uVTuYZqj50o= 2023-10-12T00:49:00.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WcumaTvD1s/TBNk6uVTuYZqj50o=', 'Content-Length': '0'} 2023-10-12T00:49:00.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c4460b50973a7f-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.567 DEBUG:boto:path=/wafmoa-55/i 2023-10-12T00:49:00.567 DEBUG:boto:auth_path=/wafmoa-55/i 2023-10-12T00:49:00.567 DEBUG:boto:Method: GET 2023-10-12T00:49:00.568 DEBUG:boto:Path: /wafmoa-55/i 2023-10-12T00:49:00.568 DEBUG:boto:Data: 2023-10-12T00:49:00.569 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.570 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.570 DEBUG:boto:Params: {} 2023-10-12T00:49:00.571 DEBUG:boto:Token: None 2023-10-12T00:49:00.571 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/i 2023-10-12T00:49:00.572 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WcumaTvD1s/TBNk6uVTuYZqj50o= 2023-10-12T00:49:00.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WcumaTvD1s/TBNk6uVTuYZqj50o=', 'Content-Length': '0'} 2023-10-12T00:49:00.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1fa74707c375e9-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.575 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.575 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.576 DEBUG:boto:path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.576 DEBUG:boto:auth_path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.576 DEBUG:boto:Method: GET 2023-10-12T00:49:00.577 DEBUG:boto:Path: /wafmoa-55/?prefix=i 2023-10-12T00:49:00.577 DEBUG:boto:Data: 2023-10-12T00:49:00.578 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.579 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.579 DEBUG:boto:Params: {} 2023-10-12T00:49:00.579 DEBUG:boto:Token: None 2023-10-12T00:49:00.580 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.580 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005119192bc9242480-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.584 DEBUG:boto:b'wafmoa-55i1000falsei2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.585 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.585 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.586 DEBUG:boto:path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.586 DEBUG:boto:auth_path=/wafmoa-55/?prefix=i 2023-10-12T00:49:00.587 DEBUG:boto:Method: GET 2023-10-12T00:49:00.587 DEBUG:boto:Path: /wafmoa-55/?prefix=i 2023-10-12T00:49:00.588 DEBUG:boto:Data: 2023-10-12T00:49:00.588 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.589 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.589 DEBUG:boto:Params: {} 2023-10-12T00:49:00.590 DEBUG:boto:Token: None 2023-10-12T00:49:00.590 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0c2020cff59ebd-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.594 DEBUG:boto:b'wafmoa-55i1000falsei2023-10-12T00:48:33.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-10-12T00:49:00.595 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.595 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.596 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.596 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.597 DEBUG:boto:Method: GET 2023-10-12T00:49:00.597 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.598 DEBUG:boto:Data: 2023-10-12T00:49:00.598 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.599 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.599 DEBUG:boto:Params: {} 2023-10-12T00:49:00.600 DEBUG:boto:Token: None 2023-10-12T00:49:00.600 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8= 2023-10-12T00:49:00.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8=', 'Content-Length': '0'} 2023-10-12T00:49:00.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0fc47bf81ba5e5-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.604 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.605 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.605 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.606 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.606 DEBUG:boto:Method: GET 2023-10-12T00:49:00.607 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.607 DEBUG:boto:Data: 2023-10-12T00:49:00.607 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.608 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.609 DEBUG:boto:Params: {} 2023-10-12T00:49:00.610 DEBUG:boto:Token: None 2023-10-12T00:49:00.610 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8= 2023-10-12T00:49:00.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8=', 'Content-Length': '0'} 2023-10-12T00:49:00.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d7b2f484f0a888-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.613 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.614 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.614 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.615 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.615 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.616 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.616 DEBUG:boto:Data: 2023-10-12T00:49:00.617 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.617 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.618 DEBUG:boto:Params: {} 2023-10-12T00:49:00.618 DEBUG:boto:Token: None 2023-10-12T00:49:00.619 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FrkkyR8LrG+gFG4EKM8JFwGeV74= 2023-10-12T00:49:00.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FrkkyR8LrG+gFG4EKM8JFwGeV74=', 'Content-Length': '0'} 2023-10-12T00:49:00.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015479b864b491fdb-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.622 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.622 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.623 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.623 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.624 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.624 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.624 DEBUG:boto:Data: 2023-10-12T00:49:00.625 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.626 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.626 DEBUG:boto:Params: {} 2023-10-12T00:49:00.627 DEBUG:boto:Token: None 2023-10-12T00:49:00.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FrkkyR8LrG+gFG4EKM8JFwGeV74= 2023-10-12T00:49:00.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FrkkyR8LrG+gFG4EKM8JFwGeV74=', 'Content-Length': '0'} 2023-10-12T00:49:00.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4305c5fb5434e4d-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-10-12T00:49:00.631 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.632 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.632 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.632 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.633 DEBUG:boto:Method: GET 2023-10-12T00:49:00.633 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.634 DEBUG:boto:Data: 2023-10-12T00:49:00.634 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.635 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.635 DEBUG:boto:Params: {} 2023-10-12T00:49:00.636 DEBUG:boto:Token: None 2023-10-12T00:49:00.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8= 2023-10-12T00:49:00.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8=', 'Content-Length': '0'} 2023-10-12T00:49:00.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253c75153364408f-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.640 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.640 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.641 DEBUG:boto:path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.641 DEBUG:boto:auth_path=/wafmoa-55/j?versionId=null 2023-10-12T00:49:00.642 DEBUG:boto:Method: GET 2023-10-12T00:49:00.642 DEBUG:boto:Path: /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.643 DEBUG:boto:Data: 2023-10-12T00:49:00.643 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.644 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.644 DEBUG:boto:Params: {} 2023-10-12T00:49:00.645 DEBUG:boto:Token: None 2023-10-12T00:49:00.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j?versionId=null 2023-10-12T00:49:00.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8= 2023-10-12T00:49:00.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R2NiAlmiqzT7dxee1WdwxNyo7n8=', 'Content-Length': '0'} 2023-10-12T00:49:00.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054031b125fc1a79e-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.649 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.649 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.650 DEBUG:boto:path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.650 DEBUG:boto:auth_path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.651 DEBUG:boto:Method: GET 2023-10-12T00:49:00.651 DEBUG:boto:Path: /wafmoa-55/?prefix=j 2023-10-12T00:49:00.651 DEBUG:boto:Data: 2023-10-12T00:49:00.652 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.653 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.653 DEBUG:boto:Params: {} 2023-10-12T00:49:00.653 DEBUG:boto:Token: None 2023-10-12T00:49:00.654 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000eca2715b870a42-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.698 DEBUG:boto:b'wafmoa-55j1000falsej2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.699 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.700 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.700 DEBUG:boto:path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.701 DEBUG:boto:auth_path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.701 DEBUG:boto:Method: GET 2023-10-12T00:49:00.701 DEBUG:boto:Path: /wafmoa-55/?prefix=j 2023-10-12T00:49:00.702 DEBUG:boto:Data: 2023-10-12T00:49:00.702 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.703 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.704 DEBUG:boto:Params: {} 2023-10-12T00:49:00.704 DEBUG:boto:Token: None 2023-10-12T00:49:00.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.705 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756b8afc7937232c-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.709 DEBUG:boto:b'wafmoa-55j1000falsej2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.709 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.710 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.710 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.711 DEBUG:boto:Path: /wafmoa-55/j 2023-10-12T00:49:00.711 DEBUG:boto:Data: 2023-10-12T00:49:00.711 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.712 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.713 DEBUG:boto:Params: {} 2023-10-12T00:49:00.713 DEBUG:boto:Token: None 2023-10-12T00:49:00.714 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j 2023-10-12T00:49:00.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AQAPqNfalUB5OBcKH4lfADLcPbQ= 2023-10-12T00:49:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AQAPqNfalUB5OBcKH4lfADLcPbQ=', 'Content-Length': '0'} 2023-10-12T00:49:00.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28905b5198affd2-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.718 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.718 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.719 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.719 DEBUG:boto:Path: /wafmoa-55/j 2023-10-12T00:49:00.719 DEBUG:boto:Data: 2023-10-12T00:49:00.720 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.721 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.721 DEBUG:boto:Params: {} 2023-10-12T00:49:00.721 DEBUG:boto:Token: None 2023-10-12T00:49:00.722 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j 2023-10-12T00:49:00.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AQAPqNfalUB5OBcKH4lfADLcPbQ= 2023-10-12T00:49:00.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AQAPqNfalUB5OBcKH4lfADLcPbQ=', 'Content-Length': '0'} 2023-10-12T00:49:00.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001442ef6d9bde9495-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=j 2023-10-12T00:49:00.726 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.726 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.727 DEBUG:boto:Method: GET 2023-10-12T00:49:00.727 DEBUG:boto:Path: /wafmoa-55/j 2023-10-12T00:49:00.728 DEBUG:boto:Data: 2023-10-12T00:49:00.728 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.729 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.729 DEBUG:boto:Params: {} 2023-10-12T00:49:00.730 DEBUG:boto:Token: None 2023-10-12T00:49:00.730 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j 2023-10-12T00:49:00.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uc5hvdaRjuzVa5SBBHvkU/09Pf0= 2023-10-12T00:49:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uc5hvdaRjuzVa5SBBHvkU/09Pf0=', 'Content-Length': '0'} 2023-10-12T00:49:00.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411061dab0d061fe-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.734 DEBUG:boto:path=/wafmoa-55/j 2023-10-12T00:49:00.734 DEBUG:boto:auth_path=/wafmoa-55/j 2023-10-12T00:49:00.735 DEBUG:boto:Method: GET 2023-10-12T00:49:00.735 DEBUG:boto:Path: /wafmoa-55/j 2023-10-12T00:49:00.736 DEBUG:boto:Data: 2023-10-12T00:49:00.736 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.737 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.737 DEBUG:boto:Params: {} 2023-10-12T00:49:00.738 DEBUG:boto:Token: None 2023-10-12T00:49:00.738 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/j 2023-10-12T00:49:00.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uc5hvdaRjuzVa5SBBHvkU/09Pf0= 2023-10-12T00:49:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uc5hvdaRjuzVa5SBBHvkU/09Pf0=', 'Content-Length': '0'} 2023-10-12T00:49:00.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241f439af0a28427-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.742 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.743 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.743 DEBUG:boto:path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.744 DEBUG:boto:auth_path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.744 DEBUG:boto:Method: GET 2023-10-12T00:49:00.744 DEBUG:boto:Path: /wafmoa-55/?prefix=j 2023-10-12T00:49:00.745 DEBUG:boto:Data: 2023-10-12T00:49:00.745 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.746 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.747 DEBUG:boto:Params: {} 2023-10-12T00:49:00.747 DEBUG:boto:Token: None 2023-10-12T00:49:00.747 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85dc233657ddb66-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.751 DEBUG:boto:b'wafmoa-55j1000falsej2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.752 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.752 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.753 DEBUG:boto:path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.753 DEBUG:boto:auth_path=/wafmoa-55/?prefix=j 2023-10-12T00:49:00.754 DEBUG:boto:Method: GET 2023-10-12T00:49:00.754 DEBUG:boto:Path: /wafmoa-55/?prefix=j 2023-10-12T00:49:00.755 DEBUG:boto:Data: 2023-10-12T00:49:00.755 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.756 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.756 DEBUG:boto:Params: {} 2023-10-12T00:49:00.757 DEBUG:boto:Token: None 2023-10-12T00:49:00.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d4c2cdef2fc7bc1-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.761 DEBUG:boto:b'wafmoa-55j1000falsej2023-10-12T00:48:33.112Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-10-12T00:49:00.762 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.762 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.763 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.763 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.764 DEBUG:boto:Method: GET 2023-10-12T00:49:00.764 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.765 DEBUG:boto:Data: 2023-10-12T00:49:00.765 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.766 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.766 DEBUG:boto:Params: {} 2023-10-12T00:49:00.767 DEBUG:boto:Token: None 2023-10-12T00:49:00.767 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE= 2023-10-12T00:49:00.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE=', 'Content-Length': '0'} 2023-10-12T00:49:00.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222539d2eb5a14aa-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.771 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.772 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.772 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.773 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.773 DEBUG:boto:Method: GET 2023-10-12T00:49:00.773 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.774 DEBUG:boto:Data: 2023-10-12T00:49:00.774 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.775 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.776 DEBUG:boto:Params: {} 2023-10-12T00:49:00.776 DEBUG:boto:Token: None 2023-10-12T00:49:00.777 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE= 2023-10-12T00:49:00.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE=', 'Content-Length': '0'} 2023-10-12T00:49:00.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912046adf781d8b3-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.780 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.781 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.781 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.781 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.782 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.782 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.783 DEBUG:boto:Data: 2023-10-12T00:49:00.783 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.784 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.784 DEBUG:boto:Params: {} 2023-10-12T00:49:00.785 DEBUG:boto:Token: None 2023-10-12T00:49:00.785 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qPt9cnR+duG0xru8rR1bVImH/mo= 2023-10-12T00:49:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qPt9cnR+duG0xru8rR1bVImH/mo=', 'Content-Length': '0'} 2023-10-12T00:49:00.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972f4dcf68801319-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.789 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.789 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.790 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.790 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.791 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.791 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.791 DEBUG:boto:Data: 2023-10-12T00:49:00.792 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.793 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.793 DEBUG:boto:Params: {} 2023-10-12T00:49:00.794 DEBUG:boto:Token: None 2023-10-12T00:49:00.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qPt9cnR+duG0xru8rR1bVImH/mo= 2023-10-12T00:49:00.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qPt9cnR+duG0xru8rR1bVImH/mo=', 'Content-Length': '0'} 2023-10-12T00:49:00.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c04b50ce2024abc-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-10-12T00:49:00.837 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.837 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.838 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.838 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.839 DEBUG:boto:Method: GET 2023-10-12T00:49:00.839 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.840 DEBUG:boto:Data: 2023-10-12T00:49:00.840 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.841 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.841 DEBUG:boto:Params: {} 2023-10-12T00:49:00.842 DEBUG:boto:Token: None 2023-10-12T00:49:00.842 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.843 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE= 2023-10-12T00:49:00.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE=', 'Content-Length': '0'} 2023-10-12T00:49:00.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417aae47b4a640ed-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.846 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.847 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.847 DEBUG:boto:path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.848 DEBUG:boto:auth_path=/wafmoa-55/k?versionId=null 2023-10-12T00:49:00.848 DEBUG:boto:Method: GET 2023-10-12T00:49:00.849 DEBUG:boto:Path: /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.849 DEBUG:boto:Data: 2023-10-12T00:49:00.850 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.851 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.851 DEBUG:boto:Params: {} 2023-10-12T00:49:00.851 DEBUG:boto:Token: None 2023-10-12T00:49:00.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k?versionId=null 2023-10-12T00:49:00.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE= 2023-10-12T00:49:00.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NEuDDEhoZSDDRY9qUFB96cLjdcE=', 'Content-Length': '0'} 2023-10-12T00:49:00.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d8c5b747d8d649-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.856 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.856 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.857 DEBUG:boto:path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.857 DEBUG:boto:auth_path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.858 DEBUG:boto:Method: GET 2023-10-12T00:49:00.858 DEBUG:boto:Path: /wafmoa-55/?prefix=k 2023-10-12T00:49:00.859 DEBUG:boto:Data: 2023-10-12T00:49:00.859 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.860 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.861 DEBUG:boto:Params: {} 2023-10-12T00:49:00.861 DEBUG:boto:Token: None 2023-10-12T00:49:00.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a6ae29174e0e34-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.865 DEBUG:boto:b'wafmoa-55k1000falsek2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.866 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.867 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.867 DEBUG:boto:path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.868 DEBUG:boto:auth_path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.868 DEBUG:boto:Method: GET 2023-10-12T00:49:00.869 DEBUG:boto:Path: /wafmoa-55/?prefix=k 2023-10-12T00:49:00.869 DEBUG:boto:Data: 2023-10-12T00:49:00.870 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.871 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.871 DEBUG:boto:Params: {} 2023-10-12T00:49:00.872 DEBUG:boto:Token: None 2023-10-12T00:49:00.872 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b6427458b395a0-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.876 DEBUG:boto:b'wafmoa-55k1000falsek2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.876 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.877 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.877 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.878 DEBUG:boto:Path: /wafmoa-55/k 2023-10-12T00:49:00.878 DEBUG:boto:Data: 2023-10-12T00:49:00.879 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.879 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.880 DEBUG:boto:Params: {} 2023-10-12T00:49:00.881 DEBUG:boto:Token: None 2023-10-12T00:49:00.881 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k 2023-10-12T00:49:00.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lkuAvT2U2uX7SrCFBCfMR/4Ka0k= 2023-10-12T00:49:00.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lkuAvT2U2uX7SrCFBCfMR/4Ka0k=', 'Content-Length': '0'} 2023-10-12T00:49:00.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178398c9a339e229-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.885 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.885 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.886 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.886 DEBUG:boto:Path: /wafmoa-55/k 2023-10-12T00:49:00.887 DEBUG:boto:Data: 2023-10-12T00:49:00.887 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.888 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.888 DEBUG:boto:Params: {} 2023-10-12T00:49:00.889 DEBUG:boto:Token: None 2023-10-12T00:49:00.889 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k 2023-10-12T00:49:00.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lkuAvT2U2uX7SrCFBCfMR/4Ka0k= 2023-10-12T00:49:00.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lkuAvT2U2uX7SrCFBCfMR/4Ka0k=', 'Content-Length': '0'} 2023-10-12T00:49:00.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f24d55799d37e19-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=k 2023-10-12T00:49:00.893 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.894 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.894 DEBUG:boto:Method: GET 2023-10-12T00:49:00.895 DEBUG:boto:Path: /wafmoa-55/k 2023-10-12T00:49:00.895 DEBUG:boto:Data: 2023-10-12T00:49:00.895 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.896 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.897 DEBUG:boto:Params: {} 2023-10-12T00:49:00.897 DEBUG:boto:Token: None 2023-10-12T00:49:00.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k 2023-10-12T00:49:00.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G/KXxMvj6WRRJI2OWCP0bgTCUq0= 2023-10-12T00:49:00.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G/KXxMvj6WRRJI2OWCP0bgTCUq0=', 'Content-Length': '0'} 2023-10-12T00:49:00.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce34e5882cc25550-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.901 DEBUG:boto:path=/wafmoa-55/k 2023-10-12T00:49:00.902 DEBUG:boto:auth_path=/wafmoa-55/k 2023-10-12T00:49:00.902 DEBUG:boto:Method: GET 2023-10-12T00:49:00.903 DEBUG:boto:Path: /wafmoa-55/k 2023-10-12T00:49:00.903 DEBUG:boto:Data: 2023-10-12T00:49:00.904 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.905 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.905 DEBUG:boto:Params: {} 2023-10-12T00:49:00.905 DEBUG:boto:Token: None 2023-10-12T00:49:00.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/k 2023-10-12T00:49:00.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G/KXxMvj6WRRJI2OWCP0bgTCUq0= 2023-10-12T00:49:00.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G/KXxMvj6WRRJI2OWCP0bgTCUq0=', 'Content-Length': '0'} 2023-10-12T00:49:00.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b145cdb9f7a70f-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.953 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.954 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.954 DEBUG:boto:path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.955 DEBUG:boto:auth_path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.955 DEBUG:boto:Method: GET 2023-10-12T00:49:00.955 DEBUG:boto:Path: /wafmoa-55/?prefix=k 2023-10-12T00:49:00.956 DEBUG:boto:Data: 2023-10-12T00:49:00.956 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.957 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.958 DEBUG:boto:Params: {} 2023-10-12T00:49:00.958 DEBUG:boto:Token: None 2023-10-12T00:49:00.958 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b56a5059d7f5b4-006527427c-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.963 DEBUG:boto:b'wafmoa-55k1000falsek2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.963 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:00.964 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:00.964 DEBUG:boto:path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.965 DEBUG:boto:auth_path=/wafmoa-55/?prefix=k 2023-10-12T00:49:00.965 DEBUG:boto:Method: GET 2023-10-12T00:49:00.966 DEBUG:boto:Path: /wafmoa-55/?prefix=k 2023-10-12T00:49:00.966 DEBUG:boto:Data: 2023-10-12T00:49:00.966 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.967 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.968 DEBUG:boto:Params: {} 2023-10-12T00:49:00.968 DEBUG:boto:Token: None 2023-10-12T00:49:00.969 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/ 2023-10-12T00:49:00.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M= 2023-10-12T00:49:00.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6COQx17bBLsZ7OZBXoazQF6CH9M=', 'Content-Length': '0'} 2023-10-12T00:49:00.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8ff636b29b58a86-006527427c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.972 DEBUG:boto:b'wafmoa-55k1000falsek2023-10-12T00:48:33.171Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:00.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-10-12T00:49:00.973 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:00.973 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:00.974 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.974 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.975 DEBUG:boto:Method: GET 2023-10-12T00:49:00.975 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.976 DEBUG:boto:Data: 2023-10-12T00:49:00.976 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.977 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.977 DEBUG:boto:Params: {} 2023-10-12T00:49:00.978 DEBUG:boto:Token: None 2023-10-12T00:49:00.978 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YDqOFY3nn8exhXKbMUAPIayDNec= 2023-10-12T00:49:00.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YDqOFY3nn8exhXKbMUAPIayDNec=', 'Content-Length': '0'} 2023-10-12T00:49:00.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f54aa7361b2c1ac-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.982 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:00.982 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:00.982 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.983 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.983 DEBUG:boto:Method: GET 2023-10-12T00:49:00.984 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.984 DEBUG:boto:Data: 2023-10-12T00:49:00.985 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:00.985 DEBUG:boto:Port: 8000 2023-10-12T00:49:00.986 DEBUG:boto:Params: {} 2023-10-12T00:49:00.986 DEBUG:boto:Token: None 2023-10-12T00:49:00.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YDqOFY3nn8exhXKbMUAPIayDNec= 2023-10-12T00:49:00.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YDqOFY3nn8exhXKbMUAPIayDNec=', 'Content-Length': '0'} 2023-10-12T00:49:00.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ed593019432123-006527427c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.991 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:00.991 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:00.991 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.992 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:00.992 DEBUG:boto:Method: HEAD 2023-10-12T00:49:00.992 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.993 DEBUG:boto:Data: 2023-10-12T00:49:00.993 DEBUG:boto:Headers: {} 2023-10-12T00:49:00.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:00.994 DEBUG:boto:Port: 8001 2023-10-12T00:49:00.994 DEBUG:boto:Params: {} 2023-10-12T00:49:00.995 DEBUG:boto:Token: None 2023-10-12T00:49:00.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:00 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:00.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c2AW5slgeTsrZafHSj+5wZLxITU= 2023-10-12T00:49:00.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c2AW5slgeTsrZafHSj+5wZLxITU=', 'Content-Length': '0'} 2023-10-12T00:49:00.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9632f1357c5617c-006527427c-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:00.999 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:00.999 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:00.999 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.000 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.000 DEBUG:boto:Method: HEAD 2023-10-12T00:49:01.000 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.001 DEBUG:boto:Data: 2023-10-12T00:49:01.001 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.002 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.002 DEBUG:boto:Params: {} 2023-10-12T00:49:01.003 DEBUG:boto:Token: None 2023-10-12T00:49:01.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zKfTUmlw7jjOpKO1e6HVw7cUmuI= 2023-10-12T00:49:01.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zKfTUmlw7jjOpKO1e6HVw7cUmuI=', 'Content-Length': '0'} 2023-10-12T00:49:01.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4d94a00b7e3953-006527427d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-10-12T00:49:01.007 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.007 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.008 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.008 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.009 DEBUG:boto:Method: GET 2023-10-12T00:49:01.009 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.009 DEBUG:boto:Data: 2023-10-12T00:49:01.010 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:01.011 DEBUG:boto:Port: 8001 2023-10-12T00:49:01.011 DEBUG:boto:Params: {} 2023-10-12T00:49:01.012 DEBUG:boto:Token: None 2023-10-12T00:49:01.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8e0ZVRx8UQVIoE3sASR91o+6TUM= 2023-10-12T00:49:01.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8e0ZVRx8UQVIoE3sASR91o+6TUM=', 'Content-Length': '0'} 2023-10-12T00:49:01.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244582e18e11e22f-006527427d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.016 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.016 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.017 DEBUG:boto:path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.017 DEBUG:boto:auth_path=/wafmoa-55/l?versionId=null 2023-10-12T00:49:01.018 DEBUG:boto:Method: GET 2023-10-12T00:49:01.018 DEBUG:boto:Path: /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.018 DEBUG:boto:Data: 2023-10-12T00:49:01.019 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.020 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.020 DEBUG:boto:Params: {} 2023-10-12T00:49:01.020 DEBUG:boto:Token: None 2023-10-12T00:49:01.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l?versionId=null 2023-10-12T00:49:01.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8e0ZVRx8UQVIoE3sASR91o+6TUM= 2023-10-12T00:49:01.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8e0ZVRx8UQVIoE3sASR91o+6TUM=', 'Content-Length': '0'} 2023-10-12T00:49:01.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a39a2b4d309609-006527427d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.024 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:01.025 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:01.025 DEBUG:boto:path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.026 DEBUG:boto:auth_path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.026 DEBUG:boto:Method: GET 2023-10-12T00:49:01.027 DEBUG:boto:Path: /wafmoa-55/?prefix=l 2023-10-12T00:49:01.027 DEBUG:boto:Data: 2023-10-12T00:49:01.027 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:01.028 DEBUG:boto:Port: 8001 2023-10-12T00:49:01.029 DEBUG:boto:Params: {} 2023-10-12T00:49:01.029 DEBUG:boto:Token: None 2023-10-12T00:49:01.030 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/ 2023-10-12T00:49:01.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc= 2023-10-12T00:49:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc=', 'Content-Length': '0'} 2023-10-12T00:49:01.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736ea3c17bc91914-006527427d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.034 DEBUG:boto:b'wafmoa-55l1000falsel2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:01.034 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:01.035 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:01.035 DEBUG:boto:path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.036 DEBUG:boto:auth_path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.036 DEBUG:boto:Method: GET 2023-10-12T00:49:01.037 DEBUG:boto:Path: /wafmoa-55/?prefix=l 2023-10-12T00:49:01.037 DEBUG:boto:Data: 2023-10-12T00:49:01.038 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.038 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.039 DEBUG:boto:Params: {} 2023-10-12T00:49:01.039 DEBUG:boto:Token: None 2023-10-12T00:49:01.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/ 2023-10-12T00:49:01.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc= 2023-10-12T00:49:01.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc=', 'Content-Length': '0'} 2023-10-12T00:49:01.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec8335ad827d11f-006527427d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.044 DEBUG:boto:b'wafmoa-55l1000falsel2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:01.044 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.045 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.045 DEBUG:boto:Method: HEAD 2023-10-12T00:49:01.045 DEBUG:boto:Path: /wafmoa-55/l 2023-10-12T00:49:01.046 DEBUG:boto:Data: 2023-10-12T00:49:01.046 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:01.047 DEBUG:boto:Port: 8001 2023-10-12T00:49:01.047 DEBUG:boto:Params: {} 2023-10-12T00:49:01.048 DEBUG:boto:Token: None 2023-10-12T00:49:01.048 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l 2023-10-12T00:49:01.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U2vB1PuOgxvndca/p+8+LBs8ahE= 2023-10-12T00:49:01.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U2vB1PuOgxvndca/p+8+LBs8ahE=', 'Content-Length': '0'} 2023-10-12T00:49:01.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef346cf878ee70d4-006527427d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.052 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.052 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.053 DEBUG:boto:Method: HEAD 2023-10-12T00:49:01.053 DEBUG:boto:Path: /wafmoa-55/l 2023-10-12T00:49:01.053 DEBUG:boto:Data: 2023-10-12T00:49:01.054 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.055 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.055 DEBUG:boto:Params: {} 2023-10-12T00:49:01.055 DEBUG:boto:Token: None 2023-10-12T00:49:01.056 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l 2023-10-12T00:49:01.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U2vB1PuOgxvndca/p+8+LBs8ahE= 2023-10-12T00:49:01.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U2vB1PuOgxvndca/p+8+LBs8ahE=', 'Content-Length': '0'} 2023-10-12T00:49:01.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbaf7f818efdf43b-006527427d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=l 2023-10-12T00:49:01.060 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.060 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.061 DEBUG:boto:Method: GET 2023-10-12T00:49:01.061 DEBUG:boto:Path: /wafmoa-55/l 2023-10-12T00:49:01.062 DEBUG:boto:Data: 2023-10-12T00:49:01.062 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:01.063 DEBUG:boto:Port: 8001 2023-10-12T00:49:01.063 DEBUG:boto:Params: {} 2023-10-12T00:49:01.064 DEBUG:boto:Token: None 2023-10-12T00:49:01.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l 2023-10-12T00:49:01.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m8/8faxzUUG4un/lrj4s4LNQbBU= 2023-10-12T00:49:01.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m8/8faxzUUG4un/lrj4s4LNQbBU=', 'Content-Length': '0'} 2023-10-12T00:49:01.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe0c639dc6cc2f4-006527427d-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.114 DEBUG:boto:path=/wafmoa-55/l 2023-10-12T00:49:01.115 DEBUG:boto:auth_path=/wafmoa-55/l 2023-10-12T00:49:01.116 DEBUG:boto:Method: GET 2023-10-12T00:49:01.116 DEBUG:boto:Path: /wafmoa-55/l 2023-10-12T00:49:01.116 DEBUG:boto:Data: 2023-10-12T00:49:01.117 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.118 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.118 DEBUG:boto:Params: {} 2023-10-12T00:49:01.119 DEBUG:boto:Token: None 2023-10-12T00:49:01.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/l 2023-10-12T00:49:01.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m8/8faxzUUG4un/lrj4s4LNQbBU= 2023-10-12T00:49:01.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m8/8faxzUUG4un/lrj4s4LNQbBU=', 'Content-Length': '0'} 2023-10-12T00:49:01.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:48:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ec91f1511220fd-006527427d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.123 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:01.123 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:01.124 DEBUG:boto:path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.124 DEBUG:boto:auth_path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.125 DEBUG:boto:Method: GET 2023-10-12T00:49:01.125 DEBUG:boto:Path: /wafmoa-55/?prefix=l 2023-10-12T00:49:01.125 DEBUG:boto:Data: 2023-10-12T00:49:01.126 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:01.127 DEBUG:boto:Port: 8001 2023-10-12T00:49:01.127 DEBUG:boto:Params: {} 2023-10-12T00:49:01.128 DEBUG:boto:Token: None 2023-10-12T00:49:01.128 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/ 2023-10-12T00:49:01.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc= 2023-10-12T00:49:01.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc=', 'Content-Length': '0'} 2023-10-12T00:49:01.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b460b675a52357-006527427d-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.133 DEBUG:boto:b'wafmoa-55l1000falsel2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:01.134 DEBUG:boto:path=/wafmoa-55/ 2023-10-12T00:49:01.134 DEBUG:boto:auth_path=/wafmoa-55/ 2023-10-12T00:49:01.135 DEBUG:boto:path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.135 DEBUG:boto:auth_path=/wafmoa-55/?prefix=l 2023-10-12T00:49:01.136 DEBUG:boto:Method: GET 2023-10-12T00:49:01.136 DEBUG:boto:Path: /wafmoa-55/?prefix=l 2023-10-12T00:49:01.137 DEBUG:boto:Data: 2023-10-12T00:49:01.137 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.138 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.138 DEBUG:boto:Params: {} 2023-10-12T00:49:01.139 DEBUG:boto:Token: None 2023-10-12T00:49:01.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-55/ 2023-10-12T00:49:01.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc= 2023-10-12T00:49:01.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CZYmagwrlporjnH8MWxe5yNrzyc=', 'Content-Length': '0'} 2023-10-12T00:49:01.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ff655076ad2ca9-006527427d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.190 DEBUG:boto:b'wafmoa-55l1000falsel2023-10-12T00:48:33.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:01.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-55 zones={a2, a1} 2023-10-12T00:49:01.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'wafmoa-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.193 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket wafmoa-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:49:01.246 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:01.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:01.291 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:49:01.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.292 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "0#00000000002.45.6", 2023-10-12T00:49:01.292 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.585225456767509469", 2023-10-12T00:49:01.292 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:49:01.293 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "k", 2023-10-12T00:49:01.293 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:49:01.293 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:49:01.293 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 2, 2023-10-12T00:49:01.293 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:48:33.171231759Z", 2023-10-12T00:49:01.294 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:49:01.294 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:49:01.294 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:49:01.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.295 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:49:01.295 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:49:01.295 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:49:01.295 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:49:01.295 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:49:01.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.296 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:49:01.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.296 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:01.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.297 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "1#00000000002.54.6", 2023-10-12T00:49:01.297 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11165188957508862941", 2023-10-12T00:49:01.297 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:49:01.298 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "j", 2023-10-12T00:49:01.298 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:49:01.298 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:49:01.299 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 2, 2023-10-12T00:49:01.299 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:48:33.112957625Z", 2023-10-12T00:49:01.299 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:49:01.299 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:49:01.300 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:49:01.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.300 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:49:01.300 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:49:01.300 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:49:01.301 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:49:01.301 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:49:01.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.301 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:49:01.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.302 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:01.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.302 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "2#00000000002.41.6", 2023-10-12T00:49:01.303 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.9166757748699048585", 2023-10-12T00:49:01.303 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:49:01.303 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "i", 2023-10-12T00:49:01.303 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:49:01.303 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:49:01.304 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 2, 2023-10-12T00:49:01.304 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:48:33.095022543Z", 2023-10-12T00:49:01.304 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:49:01.304 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:49:01.304 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:49:01.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.305 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:49:01.305 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:49:01.305 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:49:01.306 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:49:01.306 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:49:01.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.306 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:49:01.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.307 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:01.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.307 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "2#00000000003.43.6", 2023-10-12T00:49:01.308 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.14363706968361991917", 2023-10-12T00:49:01.308 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-10-12T00:49:01.308 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "l", 2023-10-12T00:49:01.308 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:49:01.308 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-10-12T00:49:01.309 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 3, 2023-10-12T00:49:01.309 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:48:33.189488431Z", 2023-10-12T00:49:01.309 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:49:01.309 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:49:01.309 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:49:01.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.310 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-10-12T00:49:01.310 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-10-12T00:49:01.310 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-10-12T00:49:01.310 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-10-12T00:49:01.311 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-10-12T00:49:01.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.311 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-55:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.20" 2023-10-12T00:49:01.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:01.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.312 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:49:01.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ok 2023-10-12T00:49:01.315 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:01.315 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:01.316 DEBUG:boto:Method: PUT 2023-10-12T00:49:01.316 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:01.317 DEBUG:boto:Data: 2023-10-12T00:49:01.318 DEBUG:boto:Headers: {} 2023-10-12T00:49:01.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:01.319 DEBUG:boto:Port: 8000 2023-10-12T00:49:01.320 DEBUG:boto:Params: {} 2023-10-12T00:49:01.320 DEBUG:boto:Token: None 2023-10-12T00:49:01.321 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:49:01 GMT /wafmoa-56/ 2023-10-12T00:49:01.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oH30fOX7mTGy8mVMh01UeGBzEFI= 2023-10-12T00:49:01.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:01 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oH30fOX7mTGy8mVMh01UeGBzEFI=', 'Content-Length': '0'} 2023-10-12T00:49:01.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e09a9c470aad3a0b-006527427d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:01.338 DEBUG:rgw_multi.tests:created bucket=wafmoa-56 2023-10-12T00:49:01.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.339 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:49:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:01.420 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:49:01.432 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:49:01.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.432 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:49:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:49:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:49:01.434 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:49:01.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071741.336393_553.1", 2023-10-12T00:49:01.434 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:49:01.336393Z" 2023-10-12T00:49:01.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.435 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:49:01.435 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:49:01.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.436 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:49:01.437 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071741.336393_553.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:49:01.437 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:49:01.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:01.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:49:01.487 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:01.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:01.516+0000 7f6e34807780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:49:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:01.516+0000 7f6e34807780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:49:01.529 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:49:01.529 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:49:01.529 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:49:01.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:49:01.530 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:49:01.530 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:49:01.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:01.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.531 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:49:01.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:49:01.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:01.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:01.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:49:01.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:01.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:49:01.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:01.533 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:49:01.533 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:01.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:49:01.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:01.534 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:01.534 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:49:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:49:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:01.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:49:01.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:01.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:01.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071741.332103_551.1", 2023-10-12T00:49:01.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:01.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:01.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:01.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:49:01.332103Z", 2023-10-12T00:49:01.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:01.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:01.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:49:01.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:01.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:01.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:49:01.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:01.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:01.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:01.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:49:01.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:01.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:01.541 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:01.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.542 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:49:01.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:49:01.542 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:49:01.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:01.543 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:49:01Z" 2023-10-12T00:49:01.543 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:49:01.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071741.332103_551.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:49:01.332103Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:49:01Z" } 2023-10-12T00:49:01.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071741.332103_551.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:49:01.332103Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:49:01.545 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071741.336393_553.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:49:01.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071741.332103_551.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:49:01.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071741.336393_553.1 target=1_1697071741.332103_551.1 2023-10-12T00:49:06.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:06.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:49:06.548 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:49:06.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:06.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:06.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:06.636+0000 7fc31f308780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:49:06.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:06.637+0000 7fc31f308780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:49:06.651 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:49:06.651 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:49:06.651 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:49:06.652 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:49:06.652 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:49:06.652 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:49:06.652 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:06.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.653 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:49:06.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:06.653 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:49:06.653 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:06.654 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:06.654 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:49:06.654 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:06.654 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:49:06.654 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:06.655 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:49:06.655 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:06.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:06.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:06.656 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:49:06.656 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:06.656 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:06.656 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:49:06.657 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:06.657 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:06.657 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:06.657 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:49:06.657 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:06.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:06.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:06.658 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:49:06.658 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:06.659 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:06.659 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071741.336393_553.1", 2023-10-12T00:49:06.659 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:06.659 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:06.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:06.660 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:49:01.336393Z", 2023-10-12T00:49:06.660 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:06.660 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:06.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:49:06.661 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:49:06.661 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:49:06.661 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:49:06.661 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:49:06.662 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:49:06.662 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:49:06.662 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:49:06.662 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:49:06.662 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:49:06.663 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:06.663 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:49:06.663 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:49:06.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:49:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:49:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:49:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:49:06.664 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:49:06Z" 2023-10-12T00:49:06.665 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:49:06.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071741.336393_553.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:49:01.336393Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:49:06Z" } 2023-10-12T00:49:06.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071741.336393_553.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:49:01.336393Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:49:06.667 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071741.336393_553.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:49:06.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071741.336393_553.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:49:06.667 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:49:06.668 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:06.668 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:06.669 DEBUG:boto:Method: HEAD 2023-10-12T00:49:06.669 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:06.670 DEBUG:boto:Data: 2023-10-12T00:49:06.670 DEBUG:boto:Headers: {} 2023-10-12T00:49:06.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.671 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.671 DEBUG:boto:Params: {} 2023-10-12T00:49:06.672 DEBUG:boto:Token: None 2023-10-12T00:49:06.672 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/ 2023-10-12T00:49:06.673 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM= 2023-10-12T00:49:06.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM=', 'Content-Length': '0'} 2023-10-12T00:49:06.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009dd889bb6f15b37-0065274282-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.677 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:06.677 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:06.678 DEBUG:boto:Method: PUT 2023-10-12T00:49:06.678 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:49:06.678 DEBUG:boto:Data: 2023-10-12T00:49:06.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:06.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.680 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.680 DEBUG:boto:Params: {} 2023-10-12T00:49:06.680 DEBUG:boto:Token: None 2023-10-12T00:49:06.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/a 2023-10-12T00:49:06.681 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hym8mgjhvupuzrqqIhgOQdp33Js= 2023-10-12T00:49:06.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hym8mgjhvupuzrqqIhgOQdp33Js='} 2023-10-12T00:49:06.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e8e4cb42663f143-0065274282-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.689 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:06.690 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:06.690 DEBUG:boto:Method: HEAD 2023-10-12T00:49:06.691 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:06.691 DEBUG:boto:Data: 2023-10-12T00:49:06.691 DEBUG:boto:Headers: {} 2023-10-12T00:49:06.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.692 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.693 DEBUG:boto:Params: {} 2023-10-12T00:49:06.693 DEBUG:boto:Token: None 2023-10-12T00:49:06.693 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/ 2023-10-12T00:49:06.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM= 2023-10-12T00:49:06.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM=', 'Content-Length': '0'} 2023-10-12T00:49:06.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2e3af0f58689fba-0065274282-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.697 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:06.698 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:06.698 DEBUG:boto:Method: PUT 2023-10-12T00:49:06.698 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:49:06.699 DEBUG:boto:Data: 2023-10-12T00:49:06.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:06.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.700 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.700 DEBUG:boto:Params: {} 2023-10-12T00:49:06.701 DEBUG:boto:Token: None 2023-10-12T00:49:06.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/b 2023-10-12T00:49:06.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4vVERUmBG4G7/nmHokBehR9Bh6Y= 2023-10-12T00:49:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4vVERUmBG4G7/nmHokBehR9Bh6Y='} 2023-10-12T00:49:06.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da37b1d0941a8817-0065274282-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.708 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:06.708 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:06.709 DEBUG:boto:Method: HEAD 2023-10-12T00:49:06.709 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:06.709 DEBUG:boto:Data: 2023-10-12T00:49:06.710 DEBUG:boto:Headers: {} 2023-10-12T00:49:06.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.711 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.711 DEBUG:boto:Params: {} 2023-10-12T00:49:06.712 DEBUG:boto:Token: None 2023-10-12T00:49:06.712 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/ 2023-10-12T00:49:06.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM= 2023-10-12T00:49:06.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM=', 'Content-Length': '0'} 2023-10-12T00:49:06.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b0ae221e2dd2137-0065274282-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.716 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:06.717 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:06.717 DEBUG:boto:Method: PUT 2023-10-12T00:49:06.717 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:49:06.718 DEBUG:boto:Data: 2023-10-12T00:49:06.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:06.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.719 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.720 DEBUG:boto:Params: {} 2023-10-12T00:49:06.720 DEBUG:boto:Token: None 2023-10-12T00:49:06.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/c 2023-10-12T00:49:06.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OOZcQ4A+JxI4/mEEJGG2TXOWMNY= 2023-10-12T00:49:06.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OOZcQ4A+JxI4/mEEJGG2TXOWMNY='} 2023-10-12T00:49:06.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009dbf178e62503bd-0065274282-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.767 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:06.768 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:06.768 DEBUG:boto:Method: HEAD 2023-10-12T00:49:06.769 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:06.769 DEBUG:boto:Data: 2023-10-12T00:49:06.769 DEBUG:boto:Headers: {} 2023-10-12T00:49:06.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.770 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.771 DEBUG:boto:Params: {} 2023-10-12T00:49:06.771 DEBUG:boto:Token: None 2023-10-12T00:49:06.772 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/ 2023-10-12T00:49:06.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM= 2023-10-12T00:49:06.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Tb1KbH6x1fJ6r7Lg9w2FNY24gM=', 'Content-Length': '0'} 2023-10-12T00:49:06.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008896e2183151fa95-0065274282-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.777 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:06.778 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:06.778 DEBUG:boto:Method: PUT 2023-10-12T00:49:06.779 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:49:06.779 DEBUG:boto:Data: 2023-10-12T00:49:06.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:06.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:06.780 DEBUG:boto:Port: 8000 2023-10-12T00:49:06.781 DEBUG:boto:Params: {} 2023-10-12T00:49:06.781 DEBUG:boto:Token: None 2023-10-12T00:49:06.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:06 GMT /wafmoa-56/d 2023-10-12T00:49:06.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PMruR225REmsApK5ltggbth1e2U= 2023-10-12T00:49:06.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PMruR225REmsApK5ltggbth1e2U='} 2023-10-12T00:49:06.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d87d51bcb510c62f-0065274282-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:06.788 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-56 2023-10-12T00:49:06.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:06.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:06.790 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:49:06.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:06.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:06.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:06.879+0000 7fa190a31780 1 waiting to reach incremental sync.. 2023-10-12T00:49:11.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:11.881+0000 7fa190a31780 1 bucket sync caught up with source: 2023-10-12T00:49:11.882 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.46.6, , , , 00000000001.19.6, , , 00000000001.31.6, , 00000000001.54.6] 2023-10-12T00:49:11.883 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.46.6, , , , 00000000001.19.6, , , 00000000001.31.6, , 00000000001.54.6] 2023-10-12T00:49:11.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:11.881+0000 7fa190a31780 0 bucket checkpoint complete 2023-10-12T00:49:11.899 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-56 2023-10-12T00:49:11.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:11.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:11.901 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:49:11.964 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:11.964 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:12.000+0000 7fcaed2dc780 1 waiting to reach incremental sync.. 2023-10-12T00:49:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:17.001+0000 7fcaed2dc780 1 waiting to reach incremental sync.. 2023-10-12T00:49:22.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:22.003+0000 7fcaed2dc780 1 bucket sync caught up with source: 2023-10-12T00:49:22.004 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.316.6, , , , 00000000001.172.6, , , 00000000001.206.6, , 00000000001.174.6] 2023-10-12T00:49:22.005 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.316.6, , , , 00000000001.172.6, , , 00000000001.206.6, , 00000000001.174.6] 2023-10-12T00:49:22.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:22.003+0000 7fcaed2dc780 0 bucket checkpoint complete 2023-10-12T00:49:22.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-56 zones={a2, a1} 2023-10-12T00:49:22.021 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.022 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.023 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.023 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:22.024 DEBUG:boto:Data: 2023-10-12T00:49:22.024 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.025 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.026 DEBUG:boto:Params: {} 2023-10-12T00:49:22.026 DEBUG:boto:Token: None 2023-10-12T00:49:22.027 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hL0lDjcdtow8mzQadVXoRuoQAvg= 2023-10-12T00:49:22.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hL0lDjcdtow8mzQadVXoRuoQAvg=', 'Content-Length': '0'} 2023-10-12T00:49:22.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000669842db123b15c8-0065274292-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.032 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.033 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.033 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.034 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:22.034 DEBUG:boto:Data: 2023-10-12T00:49:22.035 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.036 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.036 DEBUG:boto:Params: {} 2023-10-12T00:49:22.037 DEBUG:boto:Token: None 2023-10-12T00:49:22.038 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hL0lDjcdtow8mzQadVXoRuoQAvg= 2023-10-12T00:49:22.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hL0lDjcdtow8mzQadVXoRuoQAvg=', 'Content-Length': '0'} 2023-10-12T00:49:22.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4097d03c5baa98d-0065274292-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:49:22.043 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.043 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.044 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.044 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.045 DEBUG:boto:Method: GET 2023-10-12T00:49:22.046 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.046 DEBUG:boto:Data: 2023-10-12T00:49:22.047 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.048 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.048 DEBUG:boto:Params: {} 2023-10-12T00:49:22.049 DEBUG:boto:Token: None 2023-10-12T00:49:22.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/?versions 2023-10-12T00:49:22.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc= 2023-10-12T00:49:22.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc=', 'Content-Length': '0'} 2023-10-12T00:49:22.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046121365012798aa-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.054 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.055 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:49:22.055 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:49:22.056 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:49:22.056 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:49:22.057 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:49:22.057 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.058 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.058 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.059 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.059 DEBUG:boto:Method: GET 2023-10-12T00:49:22.060 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.061 DEBUG:boto:Data: 2023-10-12T00:49:22.061 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.062 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.062 DEBUG:boto:Params: {} 2023-10-12T00:49:22.065 DEBUG:boto:Token: None 2023-10-12T00:49:22.066 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/?versions 2023-10-12T00:49:22.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc= 2023-10-12T00:49:22.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc=', 'Content-Length': '0'} 2023-10-12T00:49:22.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e63230603b14b2-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.071 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.072 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:49:22.072 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:49:22.073 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:49:22.073 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:49:22.074 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.074 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.075 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.075 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.076 DEBUG:boto:Method: GET 2023-10-12T00:49:22.076 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.077 DEBUG:boto:Data: 2023-10-12T00:49:22.077 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.078 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.079 DEBUG:boto:Params: {} 2023-10-12T00:49:22.079 DEBUG:boto:Token: None 2023-10-12T00:49:22.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/?versions 2023-10-12T00:49:22.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc= 2023-10-12T00:49:22.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc=', 'Content-Length': '0'} 2023-10-12T00:49:22.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d68d0fa2360de8-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.084 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.085 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.086 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.086 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.087 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.087 DEBUG:boto:Method: GET 2023-10-12T00:49:22.087 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:49:22.088 DEBUG:boto:Data: 2023-10-12T00:49:22.088 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.089 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.090 DEBUG:boto:Params: {} 2023-10-12T00:49:22.090 DEBUG:boto:Token: None 2023-10-12T00:49:22.091 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/?versions 2023-10-12T00:49:22.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc= 2023-10-12T00:49:22.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:krr0tFvg8lmI9wMXhVbYlzZ0oEc=', 'Content-Length': '0'} 2023-10-12T00:49:22.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec3eccd86d0f34a-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.095 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:49:22.097 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.097 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.098 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.098 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.099 DEBUG:boto:Method: GET 2023-10-12T00:49:22.099 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.100 DEBUG:boto:Data: 2023-10-12T00:49:22.100 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.102 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.102 DEBUG:boto:Params: {} 2023-10-12T00:49:22.103 DEBUG:boto:Token: None 2023-10-12T00:49:22.103 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ= 2023-10-12T00:49:22.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ=', 'Content-Length': '0'} 2023-10-12T00:49:22.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a00e4cea1580ab01-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.150 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.151 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.152 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.152 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.153 DEBUG:boto:Method: GET 2023-10-12T00:49:22.153 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.154 DEBUG:boto:Data: 2023-10-12T00:49:22.154 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.155 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.156 DEBUG:boto:Params: {} 2023-10-12T00:49:22.156 DEBUG:boto:Token: None 2023-10-12T00:49:22.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ= 2023-10-12T00:49:22.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ=', 'Content-Length': '0'} 2023-10-12T00:49:22.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038eb13d23f193007-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.161 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.161 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.161 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.162 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.162 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.163 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.163 DEBUG:boto:Data: 2023-10-12T00:49:22.164 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.164 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.165 DEBUG:boto:Params: {} 2023-10-12T00:49:22.165 DEBUG:boto:Token: None 2023-10-12T00:49:22.166 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BeKiRXTuf0ABd7gxQh/81x7nCFY= 2023-10-12T00:49:22.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BeKiRXTuf0ABd7gxQh/81x7nCFY=', 'Content-Length': '0'} 2023-10-12T00:49:22.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dbd7c8dcf847444-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.170 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.170 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.171 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.171 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.172 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.172 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.173 DEBUG:boto:Data: 2023-10-12T00:49:22.173 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.174 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.175 DEBUG:boto:Params: {} 2023-10-12T00:49:22.175 DEBUG:boto:Token: None 2023-10-12T00:49:22.176 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BeKiRXTuf0ABd7gxQh/81x7nCFY= 2023-10-12T00:49:22.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BeKiRXTuf0ABd7gxQh/81x7nCFY=', 'Content-Length': '0'} 2023-10-12T00:49:22.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f2f8586f5385c4-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:49:22.180 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.180 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.181 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.181 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.182 DEBUG:boto:Method: GET 2023-10-12T00:49:22.182 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.183 DEBUG:boto:Data: 2023-10-12T00:49:22.184 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.185 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.185 DEBUG:boto:Params: {} 2023-10-12T00:49:22.186 DEBUG:boto:Token: None 2023-10-12T00:49:22.186 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ= 2023-10-12T00:49:22.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ=', 'Content-Length': '0'} 2023-10-12T00:49:22.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cdb71a6829e48f-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.190 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.190 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.191 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.191 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:49:22.191 DEBUG:boto:Method: GET 2023-10-12T00:49:22.192 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.192 DEBUG:boto:Data: 2023-10-12T00:49:22.193 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.193 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.194 DEBUG:boto:Params: {} 2023-10-12T00:49:22.194 DEBUG:boto:Token: None 2023-10-12T00:49:22.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:49:22.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ= 2023-10-12T00:49:22.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u2pR3UN0C7h9+W+IV8TSi8sQjRQ=', 'Content-Length': '0'} 2023-10-12T00:49:22.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074bca46046a6a561-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.239 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.240 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.240 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.240 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.241 DEBUG:boto:Method: GET 2023-10-12T00:49:22.241 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:49:22.242 DEBUG:boto:Data: 2023-10-12T00:49:22.242 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.243 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.244 DEBUG:boto:Params: {} 2023-10-12T00:49:22.244 DEBUG:boto:Token: None 2023-10-12T00:49:22.245 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044cd446bed5a5863-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.249 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.250 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.250 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.251 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.251 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.252 DEBUG:boto:Method: GET 2023-10-12T00:49:22.252 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:49:22.253 DEBUG:boto:Data: 2023-10-12T00:49:22.253 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.254 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.254 DEBUG:boto:Params: {} 2023-10-12T00:49:22.255 DEBUG:boto:Token: None 2023-10-12T00:49:22.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000323efd54c1d9770-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.260 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.261 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.261 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.261 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.262 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:49:22.262 DEBUG:boto:Data: 2023-10-12T00:49:22.263 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.263 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.264 DEBUG:boto:Params: {} 2023-10-12T00:49:22.264 DEBUG:boto:Token: None 2023-10-12T00:49:22.265 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a 2023-10-12T00:49:22.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJhOCgm3F2MUHvjDfzf1/9VJn88= 2023-10-12T00:49:22.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJhOCgm3F2MUHvjDfzf1/9VJn88=', 'Content-Length': '0'} 2023-10-12T00:49:22.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000572ef14db90e0a92-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.272 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.272 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.272 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.273 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:49:22.273 DEBUG:boto:Data: 2023-10-12T00:49:22.274 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.275 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.275 DEBUG:boto:Params: {} 2023-10-12T00:49:22.275 DEBUG:boto:Token: None 2023-10-12T00:49:22.276 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a 2023-10-12T00:49:22.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJhOCgm3F2MUHvjDfzf1/9VJn88= 2023-10-12T00:49:22.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJhOCgm3F2MUHvjDfzf1/9VJn88=', 'Content-Length': '0'} 2023-10-12T00:49:22.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcc842258bd4255-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:49:22.280 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.280 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.281 DEBUG:boto:Method: GET 2023-10-12T00:49:22.281 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:49:22.282 DEBUG:boto:Data: 2023-10-12T00:49:22.282 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.283 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.283 DEBUG:boto:Params: {} 2023-10-12T00:49:22.284 DEBUG:boto:Token: None 2023-10-12T00:49:22.284 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a 2023-10-12T00:49:22.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5qQ1uOqpxJ+sqOk8iDe1N4afwFc= 2023-10-12T00:49:22.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5qQ1uOqpxJ+sqOk8iDe1N4afwFc=', 'Content-Length': '0'} 2023-10-12T00:49:22.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82ab32096053995-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.288 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:49:22.288 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:49:22.289 DEBUG:boto:Method: GET 2023-10-12T00:49:22.289 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:49:22.290 DEBUG:boto:Data: 2023-10-12T00:49:22.290 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.291 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.292 DEBUG:boto:Params: {} 2023-10-12T00:49:22.292 DEBUG:boto:Token: None 2023-10-12T00:49:22.293 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/a 2023-10-12T00:49:22.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5qQ1uOqpxJ+sqOk8iDe1N4afwFc= 2023-10-12T00:49:22.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5qQ1uOqpxJ+sqOk8iDe1N4afwFc=', 'Content-Length': '0'} 2023-10-12T00:49:22.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009417617545a3afe9-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.338 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.339 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.339 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.340 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.340 DEBUG:boto:Method: GET 2023-10-12T00:49:22.341 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:49:22.341 DEBUG:boto:Data: 2023-10-12T00:49:22.342 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.343 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.343 DEBUG:boto:Params: {} 2023-10-12T00:49:22.344 DEBUG:boto:Token: None 2023-10-12T00:49:22.344 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.345 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b06e4f6172a1931-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.348 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.349 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.349 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.350 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.350 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:49:22.351 DEBUG:boto:Method: GET 2023-10-12T00:49:22.351 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:49:22.352 DEBUG:boto:Data: 2023-10-12T00:49:22.352 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.353 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.353 DEBUG:boto:Params: {} 2023-10-12T00:49:22.354 DEBUG:boto:Token: None 2023-10-12T00:49:22.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c108b01d414808-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.359 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:49:22.360 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.360 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.361 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.361 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.362 DEBUG:boto:Method: GET 2023-10-12T00:49:22.362 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.363 DEBUG:boto:Data: 2023-10-12T00:49:22.363 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.364 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.364 DEBUG:boto:Params: {} 2023-10-12T00:49:22.365 DEBUG:boto:Token: None 2023-10-12T00:49:22.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg= 2023-10-12T00:49:22.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg=', 'Content-Length': '0'} 2023-10-12T00:49:22.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081878baecdaafc58-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.369 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.370 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.370 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.371 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.371 DEBUG:boto:Method: GET 2023-10-12T00:49:22.372 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.372 DEBUG:boto:Data: 2023-10-12T00:49:22.372 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.373 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.374 DEBUG:boto:Params: {} 2023-10-12T00:49:22.374 DEBUG:boto:Token: None 2023-10-12T00:49:22.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg= 2023-10-12T00:49:22.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg=', 'Content-Length': '0'} 2023-10-12T00:49:22.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780f98d4afb8eff7-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.379 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.379 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.380 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.380 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.380 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.381 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.381 DEBUG:boto:Data: 2023-10-12T00:49:22.382 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.382 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.383 DEBUG:boto:Params: {} 2023-10-12T00:49:22.384 DEBUG:boto:Token: None 2023-10-12T00:49:22.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sOw72sx6KUELYgBCG+1PAEyak2U= 2023-10-12T00:49:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sOw72sx6KUELYgBCG+1PAEyak2U=', 'Content-Length': '0'} 2023-10-12T00:49:22.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c253866169ebd2da-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.388 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.388 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.389 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.389 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.389 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.390 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.390 DEBUG:boto:Data: 2023-10-12T00:49:22.391 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.392 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.392 DEBUG:boto:Params: {} 2023-10-12T00:49:22.392 DEBUG:boto:Token: None 2023-10-12T00:49:22.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sOw72sx6KUELYgBCG+1PAEyak2U= 2023-10-12T00:49:22.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sOw72sx6KUELYgBCG+1PAEyak2U=', 'Content-Length': '0'} 2023-10-12T00:49:22.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fbcd90eafaf616-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:49:22.397 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.397 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.398 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.398 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.399 DEBUG:boto:Method: GET 2023-10-12T00:49:22.399 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.400 DEBUG:boto:Data: 2023-10-12T00:49:22.400 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.401 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.401 DEBUG:boto:Params: {} 2023-10-12T00:49:22.402 DEBUG:boto:Token: None 2023-10-12T00:49:22.403 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg= 2023-10-12T00:49:22.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg=', 'Content-Length': '0'} 2023-10-12T00:49:22.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c0ee282fa61521-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.446 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.447 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.447 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.448 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:49:22.448 DEBUG:boto:Method: GET 2023-10-12T00:49:22.449 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.449 DEBUG:boto:Data: 2023-10-12T00:49:22.450 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.451 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.451 DEBUG:boto:Params: {} 2023-10-12T00:49:22.452 DEBUG:boto:Token: None 2023-10-12T00:49:22.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:49:22.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg= 2023-10-12T00:49:22.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVAFOvj9Nne1i/KUBQ37xwauTVg=', 'Content-Length': '0'} 2023-10-12T00:49:22.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f9ba9984afd86e-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.456 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.457 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.457 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.458 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.458 DEBUG:boto:Method: GET 2023-10-12T00:49:22.459 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:49:22.459 DEBUG:boto:Data: 2023-10-12T00:49:22.460 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.461 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.462 DEBUG:boto:Params: {} 2023-10-12T00:49:22.462 DEBUG:boto:Token: None 2023-10-12T00:49:22.463 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009581bad89a1f30fa-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.467 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.468 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.468 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.469 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.469 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.470 DEBUG:boto:Method: GET 2023-10-12T00:49:22.470 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:49:22.471 DEBUG:boto:Data: 2023-10-12T00:49:22.471 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.472 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.472 DEBUG:boto:Params: {} 2023-10-12T00:49:22.473 DEBUG:boto:Token: None 2023-10-12T00:49:22.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92493933c19ee3c-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.518 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.519 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.520 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.520 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.521 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:49:22.521 DEBUG:boto:Data: 2023-10-12T00:49:22.521 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.522 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.523 DEBUG:boto:Params: {} 2023-10-12T00:49:22.523 DEBUG:boto:Token: None 2023-10-12T00:49:22.524 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b 2023-10-12T00:49:22.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l+FlpCV1tG9dtkdByciEKrjr7HA= 2023-10-12T00:49:22.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l+FlpCV1tG9dtkdByciEKrjr7HA=', 'Content-Length': '0'} 2023-10-12T00:49:22.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26e0f8f65cab8c9-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.527 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.528 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.528 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.529 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:49:22.529 DEBUG:boto:Data: 2023-10-12T00:49:22.530 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.531 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.531 DEBUG:boto:Params: {} 2023-10-12T00:49:22.531 DEBUG:boto:Token: None 2023-10-12T00:49:22.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b 2023-10-12T00:49:22.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l+FlpCV1tG9dtkdByciEKrjr7HA= 2023-10-12T00:49:22.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l+FlpCV1tG9dtkdByciEKrjr7HA=', 'Content-Length': '0'} 2023-10-12T00:49:22.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9af4b844059849-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:49:22.536 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.536 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.537 DEBUG:boto:Method: GET 2023-10-12T00:49:22.537 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:49:22.538 DEBUG:boto:Data: 2023-10-12T00:49:22.538 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.539 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.539 DEBUG:boto:Params: {} 2023-10-12T00:49:22.540 DEBUG:boto:Token: None 2023-10-12T00:49:22.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b 2023-10-12T00:49:22.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8mAlp5opjPaWAoIKCACPiHmZqbs= 2023-10-12T00:49:22.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8mAlp5opjPaWAoIKCACPiHmZqbs=', 'Content-Length': '0'} 2023-10-12T00:49:22.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068052120b173785-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.544 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:49:22.544 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:49:22.545 DEBUG:boto:Method: GET 2023-10-12T00:49:22.545 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:49:22.546 DEBUG:boto:Data: 2023-10-12T00:49:22.546 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.547 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.547 DEBUG:boto:Params: {} 2023-10-12T00:49:22.548 DEBUG:boto:Token: None 2023-10-12T00:49:22.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/b 2023-10-12T00:49:22.549 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8mAlp5opjPaWAoIKCACPiHmZqbs= 2023-10-12T00:49:22.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8mAlp5opjPaWAoIKCACPiHmZqbs=', 'Content-Length': '0'} 2023-10-12T00:49:22.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c21cb7b2cfd5cd-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.552 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.552 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.553 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.553 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.554 DEBUG:boto:Method: GET 2023-10-12T00:49:22.554 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:49:22.555 DEBUG:boto:Data: 2023-10-12T00:49:22.555 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.556 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.556 DEBUG:boto:Params: {} 2023-10-12T00:49:22.557 DEBUG:boto:Token: None 2023-10-12T00:49:22.557 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25119db4bb0342b-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.602 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.603 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.604 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.604 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.604 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:49:22.605 DEBUG:boto:Method: GET 2023-10-12T00:49:22.605 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:49:22.606 DEBUG:boto:Data: 2023-10-12T00:49:22.606 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.607 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.607 DEBUG:boto:Params: {} 2023-10-12T00:49:22.608 DEBUG:boto:Token: None 2023-10-12T00:49:22.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf059b2f960b3d01-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.613 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:49:22.614 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.614 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.615 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.615 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.615 DEBUG:boto:Method: GET 2023-10-12T00:49:22.616 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.616 DEBUG:boto:Data: 2023-10-12T00:49:22.617 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.618 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.618 DEBUG:boto:Params: {} 2023-10-12T00:49:22.619 DEBUG:boto:Token: None 2023-10-12T00:49:22.619 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY= 2023-10-12T00:49:22.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY=', 'Content-Length': '0'} 2023-10-12T00:49:22.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9a65dc1cc4c965-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.623 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.623 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.624 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.624 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.625 DEBUG:boto:Method: GET 2023-10-12T00:49:22.625 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.625 DEBUG:boto:Data: 2023-10-12T00:49:22.626 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.627 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.627 DEBUG:boto:Params: {} 2023-10-12T00:49:22.628 DEBUG:boto:Token: None 2023-10-12T00:49:22.628 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY= 2023-10-12T00:49:22.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY=', 'Content-Length': '0'} 2023-10-12T00:49:22.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd887422dcfea0d-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.632 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.632 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.633 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.633 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.633 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.634 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.634 DEBUG:boto:Data: 2023-10-12T00:49:22.635 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.636 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.636 DEBUG:boto:Params: {} 2023-10-12T00:49:22.637 DEBUG:boto:Token: None 2023-10-12T00:49:22.637 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yt8GxMp9EZSgLmN2DalQSP1Vq3k= 2023-10-12T00:49:22.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yt8GxMp9EZSgLmN2DalQSP1Vq3k=', 'Content-Length': '0'} 2023-10-12T00:49:22.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bcd5a7bfec287d-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.640 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.641 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.641 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.642 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.642 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.643 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.643 DEBUG:boto:Data: 2023-10-12T00:49:22.643 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.644 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.645 DEBUG:boto:Params: {} 2023-10-12T00:49:22.645 DEBUG:boto:Token: None 2023-10-12T00:49:22.645 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yt8GxMp9EZSgLmN2DalQSP1Vq3k= 2023-10-12T00:49:22.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yt8GxMp9EZSgLmN2DalQSP1Vq3k=', 'Content-Length': '0'} 2023-10-12T00:49:22.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81295d6180e8b72-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:49:22.649 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.650 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.650 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.651 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.651 DEBUG:boto:Method: GET 2023-10-12T00:49:22.652 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.652 DEBUG:boto:Data: 2023-10-12T00:49:22.652 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.653 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.654 DEBUG:boto:Params: {} 2023-10-12T00:49:22.654 DEBUG:boto:Token: None 2023-10-12T00:49:22.655 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY= 2023-10-12T00:49:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY=', 'Content-Length': '0'} 2023-10-12T00:49:22.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c603d65733cdd5bf-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.658 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.659 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.659 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.660 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:49:22.660 DEBUG:boto:Method: GET 2023-10-12T00:49:22.660 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.661 DEBUG:boto:Data: 2023-10-12T00:49:22.661 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.662 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.663 DEBUG:boto:Params: {} 2023-10-12T00:49:22.663 DEBUG:boto:Token: None 2023-10-12T00:49:22.664 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:49:22.664 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY= 2023-10-12T00:49:22.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4aiumaEhK5J6kGCRb4N+1++nOY=', 'Content-Length': '0'} 2023-10-12T00:49:22.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b8d2c87b4c3ea2-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.710 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.711 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.711 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.712 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.712 DEBUG:boto:Method: GET 2023-10-12T00:49:22.713 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:49:22.713 DEBUG:boto:Data: 2023-10-12T00:49:22.714 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.714 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.715 DEBUG:boto:Params: {} 2023-10-12T00:49:22.715 DEBUG:boto:Token: None 2023-10-12T00:49:22.716 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.716 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000461207d17b37c587-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.720 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.721 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.721 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.722 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.722 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.723 DEBUG:boto:Method: GET 2023-10-12T00:49:22.723 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:49:22.723 DEBUG:boto:Data: 2023-10-12T00:49:22.724 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.725 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.725 DEBUG:boto:Params: {} 2023-10-12T00:49:22.726 DEBUG:boto:Token: None 2023-10-12T00:49:22.726 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003796c1c5b6bd118d-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.731 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.732 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.732 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.732 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.733 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:49:22.733 DEBUG:boto:Data: 2023-10-12T00:49:22.734 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.735 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.735 DEBUG:boto:Params: {} 2023-10-12T00:49:22.735 DEBUG:boto:Token: None 2023-10-12T00:49:22.736 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c 2023-10-12T00:49:22.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3KmmBYpMB8ePAfKo+bb2aIRJroI= 2023-10-12T00:49:22.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3KmmBYpMB8ePAfKo+bb2aIRJroI=', 'Content-Length': '0'} 2023-10-12T00:49:22.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3466476d7c5bb12-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.739 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.740 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.740 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.741 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:49:22.741 DEBUG:boto:Data: 2023-10-12T00:49:22.742 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.742 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.743 DEBUG:boto:Params: {} 2023-10-12T00:49:22.744 DEBUG:boto:Token: None 2023-10-12T00:49:22.744 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c 2023-10-12T00:49:22.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3KmmBYpMB8ePAfKo+bb2aIRJroI= 2023-10-12T00:49:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3KmmBYpMB8ePAfKo+bb2aIRJroI=', 'Content-Length': '0'} 2023-10-12T00:49:22.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003086d6d686ea9293-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:49:22.749 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.750 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.750 DEBUG:boto:Method: GET 2023-10-12T00:49:22.751 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:49:22.751 DEBUG:boto:Data: 2023-10-12T00:49:22.752 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.752 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.753 DEBUG:boto:Params: {} 2023-10-12T00:49:22.753 DEBUG:boto:Token: None 2023-10-12T00:49:22.754 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c 2023-10-12T00:49:22.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4LpgX8fd5NXXyjMh+zxFWdGGtlY= 2023-10-12T00:49:22.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4LpgX8fd5NXXyjMh+zxFWdGGtlY=', 'Content-Length': '0'} 2023-10-12T00:49:22.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d318ad7425e7e88c-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.802 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:49:22.803 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:49:22.803 DEBUG:boto:Method: GET 2023-10-12T00:49:22.804 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:49:22.804 DEBUG:boto:Data: 2023-10-12T00:49:22.805 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.806 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.806 DEBUG:boto:Params: {} 2023-10-12T00:49:22.807 DEBUG:boto:Token: None 2023-10-12T00:49:22.807 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/c 2023-10-12T00:49:22.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4LpgX8fd5NXXyjMh+zxFWdGGtlY= 2023-10-12T00:49:22.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4LpgX8fd5NXXyjMh+zxFWdGGtlY=', 'Content-Length': '0'} 2023-10-12T00:49:22.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005471fc8c3d304354-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.811 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.811 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.812 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.812 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.813 DEBUG:boto:Method: GET 2023-10-12T00:49:22.813 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:49:22.813 DEBUG:boto:Data: 2023-10-12T00:49:22.814 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.815 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.815 DEBUG:boto:Params: {} 2023-10-12T00:49:22.816 DEBUG:boto:Token: None 2023-10-12T00:49:22.816 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000241d085fcbadf271-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.820 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.821 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.821 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.822 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.822 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:49:22.823 DEBUG:boto:Method: GET 2023-10-12T00:49:22.823 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:49:22.824 DEBUG:boto:Data: 2023-10-12T00:49:22.824 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.825 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.825 DEBUG:boto:Params: {} 2023-10-12T00:49:22.826 DEBUG:boto:Token: None 2023-10-12T00:49:22.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d341d1003efd14d4-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.870 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:49:22.872 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.872 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.873 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.873 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.873 DEBUG:boto:Method: GET 2023-10-12T00:49:22.874 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.874 DEBUG:boto:Data: 2023-10-12T00:49:22.875 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.876 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.876 DEBUG:boto:Params: {} 2023-10-12T00:49:22.877 DEBUG:boto:Token: None 2023-10-12T00:49:22.877 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o= 2023-10-12T00:49:22.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o=', 'Content-Length': '0'} 2023-10-12T00:49:22.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009228d6a725a7e2bc-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.881 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.882 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.882 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.882 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.883 DEBUG:boto:Method: GET 2023-10-12T00:49:22.883 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.884 DEBUG:boto:Data: 2023-10-12T00:49:22.884 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.885 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.885 DEBUG:boto:Params: {} 2023-10-12T00:49:22.886 DEBUG:boto:Token: None 2023-10-12T00:49:22.889 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o= 2023-10-12T00:49:22.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o=', 'Content-Length': '0'} 2023-10-12T00:49:22.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b5bdce46acf4e4-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.893 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.893 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.894 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.894 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.895 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.895 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.895 DEBUG:boto:Data: 2023-10-12T00:49:22.896 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.897 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.898 DEBUG:boto:Params: {} 2023-10-12T00:49:22.898 DEBUG:boto:Token: None 2023-10-12T00:49:22.899 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xayp7ScJmfkb7Ov1cNW6EkkDlnw= 2023-10-12T00:49:22.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xayp7ScJmfkb7Ov1cNW6EkkDlnw=', 'Content-Length': '0'} 2023-10-12T00:49:22.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9613a6cf73f87f-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.902 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.903 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.903 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.904 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.904 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.904 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.905 DEBUG:boto:Data: 2023-10-12T00:49:22.905 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.906 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.906 DEBUG:boto:Params: {} 2023-10-12T00:49:22.907 DEBUG:boto:Token: None 2023-10-12T00:49:22.907 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xayp7ScJmfkb7Ov1cNW6EkkDlnw= 2023-10-12T00:49:22.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xayp7ScJmfkb7Ov1cNW6EkkDlnw=', 'Content-Length': '0'} 2023-10-12T00:49:22.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a0dbf27f0efb14-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:49:22.912 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.912 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.913 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.913 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.913 DEBUG:boto:Method: GET 2023-10-12T00:49:22.914 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.914 DEBUG:boto:Data: 2023-10-12T00:49:22.915 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.916 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.916 DEBUG:boto:Params: {} 2023-10-12T00:49:22.916 DEBUG:boto:Token: None 2023-10-12T00:49:22.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o= 2023-10-12T00:49:22.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o=', 'Content-Length': '0'} 2023-10-12T00:49:22.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5a31ca798ebb6b-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.921 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.921 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.921 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.922 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:49:22.922 DEBUG:boto:Method: GET 2023-10-12T00:49:22.923 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.923 DEBUG:boto:Data: 2023-10-12T00:49:22.924 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.924 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.925 DEBUG:boto:Params: {} 2023-10-12T00:49:22.925 DEBUG:boto:Token: None 2023-10-12T00:49:22.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:49:22.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o= 2023-10-12T00:49:22.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QBi0iQExU5QlXN2+4MVotsPeJ/o=', 'Content-Length': '0'} 2023-10-12T00:49:22.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006639a0da09f9be1d-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.930 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.930 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.931 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:49:22.931 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:49:22.932 DEBUG:boto:Method: GET 2023-10-12T00:49:22.932 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:49:22.933 DEBUG:boto:Data: 2023-10-12T00:49:22.933 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.934 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.935 DEBUG:boto:Params: {} 2023-10-12T00:49:22.935 DEBUG:boto:Token: None 2023-10-12T00:49:22.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd16a5227f0fd6bd-0065274292-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.940 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.941 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:22.942 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:22.942 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:49:22.943 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:49:22.943 DEBUG:boto:Method: GET 2023-10-12T00:49:22.944 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:49:22.944 DEBUG:boto:Data: 2023-10-12T00:49:22.945 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.946 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.946 DEBUG:boto:Params: {} 2023-10-12T00:49:22.947 DEBUG:boto:Token: None 2023-10-12T00:49:22.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/ 2023-10-12T00:49:22.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0= 2023-10-12T00:49:22.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kOn7b0pF5K0JmQt1FNoNcldnMA0=', 'Content-Length': '0'} 2023-10-12T00:49:22.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b817674f3176de-0065274292-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.951 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:22.952 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.952 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.953 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.953 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:49:22.954 DEBUG:boto:Data: 2023-10-12T00:49:22.954 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.955 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.956 DEBUG:boto:Params: {} 2023-10-12T00:49:22.956 DEBUG:boto:Token: None 2023-10-12T00:49:22.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d 2023-10-12T00:49:22.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DdKOpmH9LR4dQ/3wmM0G15YiUYg= 2023-10-12T00:49:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DdKOpmH9LR4dQ/3wmM0G15YiUYg=', 'Content-Length': '0'} 2023-10-12T00:49:22.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003caeea930c99c908-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.960 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.961 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.961 DEBUG:boto:Method: HEAD 2023-10-12T00:49:22.961 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:49:22.962 DEBUG:boto:Data: 2023-10-12T00:49:22.962 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.963 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.964 DEBUG:boto:Params: {} 2023-10-12T00:49:22.964 DEBUG:boto:Token: None 2023-10-12T00:49:22.965 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d 2023-10-12T00:49:22.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DdKOpmH9LR4dQ/3wmM0G15YiUYg= 2023-10-12T00:49:22.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DdKOpmH9LR4dQ/3wmM0G15YiUYg=', 'Content-Length': '0'} 2023-10-12T00:49:22.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002daca8740e1cbfad-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:49:22.969 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.969 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.970 DEBUG:boto:Method: GET 2023-10-12T00:49:22.970 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:49:22.971 DEBUG:boto:Data: 2023-10-12T00:49:22.971 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:22.972 DEBUG:boto:Port: 8001 2023-10-12T00:49:22.972 DEBUG:boto:Params: {} 2023-10-12T00:49:22.973 DEBUG:boto:Token: None 2023-10-12T00:49:22.973 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d 2023-10-12T00:49:22.974 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N4LU6blaB1Uyl+ugTGSfrQaKC7Y= 2023-10-12T00:49:22.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N4LU6blaB1Uyl+ugTGSfrQaKC7Y=', 'Content-Length': '0'} 2023-10-12T00:49:22.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a558691e895bb51b-0065274292-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:22.977 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:49:22.977 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:49:22.978 DEBUG:boto:Method: GET 2023-10-12T00:49:22.978 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:49:22.979 DEBUG:boto:Data: 2023-10-12T00:49:22.979 DEBUG:boto:Headers: {} 2023-10-12T00:49:22.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:22.980 DEBUG:boto:Port: 8000 2023-10-12T00:49:22.981 DEBUG:boto:Params: {} 2023-10-12T00:49:22.981 DEBUG:boto:Token: None 2023-10-12T00:49:22.982 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:22 GMT /wafmoa-56/d 2023-10-12T00:49:22.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N4LU6blaB1Uyl+ugTGSfrQaKC7Y= 2023-10-12T00:49:22.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N4LU6blaB1Uyl+ugTGSfrQaKC7Y=', 'Content-Length': '0'} 2023-10-12T00:49:22.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6d20a9af427be3b-0065274292-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:49:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.030 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.031 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.031 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:49:23.032 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:49:23.032 DEBUG:boto:Method: GET 2023-10-12T00:49:23.033 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:49:23.033 DEBUG:boto:Data: 2023-10-12T00:49:23.034 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:49:23.035 DEBUG:boto:Port: 8001 2023-10-12T00:49:23.035 DEBUG:boto:Params: {} 2023-10-12T00:49:23.036 DEBUG:boto:Token: None 2023-10-12T00:49:23.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eUP/8IPMZ0TVWjQdvkv4/Tx9H8g= 2023-10-12T00:49:23.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eUP/8IPMZ0TVWjQdvkv4/Tx9H8g=', 'Content-Length': '0'} 2023-10-12T00:49:23.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a146e846e75dea7e-0065274293-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.040 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:23.041 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.041 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.042 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:49:23.042 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:49:23.043 DEBUG:boto:Method: GET 2023-10-12T00:49:23.043 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:49:23.044 DEBUG:boto:Data: 2023-10-12T00:49:23.044 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.045 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.045 DEBUG:boto:Params: {} 2023-10-12T00:49:23.046 DEBUG:boto:Token: None 2023-10-12T00:49:23.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eUP/8IPMZ0TVWjQdvkv4/Tx9H8g= 2023-10-12T00:49:23.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eUP/8IPMZ0TVWjQdvkv4/Tx9H8g=', 'Content-Length': '0'} 2023-10-12T00:49:23.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e4007d0e2b81e3-0065274293-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.051 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:49:23.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-56 zones={a2, a1} 2023-10-12T00:49:23.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'wafmoa-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:49:23.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'wafmoa-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:49:23.053 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket wafmoa-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:49:23.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:23.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:23.149 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:49:23.149 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-56 2023-10-12T00:49:23.153 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-10-12T00:49:23.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'wafmoa-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:49:23.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'wafmoa-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:49:23.190 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket wafmoa-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:49:23.239 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:23.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:23.277 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:49:23.277 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-56 2023-10-12T00:49:23.278 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-10-12T00:49:23.304 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.304 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.305 DEBUG:boto:Method: HEAD 2023-10-12T00:49:23.305 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:23.306 DEBUG:boto:Data: 2023-10-12T00:49:23.306 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.307 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.308 DEBUG:boto:Params: {} 2023-10-12T00:49:23.308 DEBUG:boto:Token: None 2023-10-12T00:49:23.309 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI= 2023-10-12T00:49:23.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI=', 'Content-Length': '0'} 2023-10-12T00:49:23.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000435c2a19a230838f-0065274293-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.313 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:49:23.313 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:49:23.314 DEBUG:boto:Method: PUT 2023-10-12T00:49:23.314 DEBUG:boto:Path: /wafmoa-56/e 2023-10-12T00:49:23.314 DEBUG:boto:Data: 2023-10-12T00:49:23.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:23.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.316 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.317 DEBUG:boto:Params: {} 2023-10-12T00:49:23.318 DEBUG:boto:Token: None 2023-10-12T00:49:23.318 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/e 2023-10-12T00:49:23.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AoHn0Uzi9HcaIJNez430D3pHxXI= 2023-10-12T00:49:23.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AoHn0Uzi9HcaIJNez430D3pHxXI='} 2023-10-12T00:49:23.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3fafd30561f7543-0065274293-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.327 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.327 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.328 DEBUG:boto:Method: HEAD 2023-10-12T00:49:23.328 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:23.328 DEBUG:boto:Data: 2023-10-12T00:49:23.329 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.330 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.331 DEBUG:boto:Params: {} 2023-10-12T00:49:23.331 DEBUG:boto:Token: None 2023-10-12T00:49:23.332 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI= 2023-10-12T00:49:23.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI=', 'Content-Length': '0'} 2023-10-12T00:49:23.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008da53bd87f6572ea-0065274293-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.335 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:49:23.336 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:49:23.336 DEBUG:boto:Method: PUT 2023-10-12T00:49:23.337 DEBUG:boto:Path: /wafmoa-56/f 2023-10-12T00:49:23.337 DEBUG:boto:Data: 2023-10-12T00:49:23.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:23.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.338 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.339 DEBUG:boto:Params: {} 2023-10-12T00:49:23.339 DEBUG:boto:Token: None 2023-10-12T00:49:23.340 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/f 2023-10-12T00:49:23.341 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SeuIktiKzNa1BpTawDL6Czx6zQw= 2023-10-12T00:49:23.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SeuIktiKzNa1BpTawDL6Czx6zQw='} 2023-10-12T00:49:23.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df6ae916ca319b4e-0065274293-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.387 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.388 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.388 DEBUG:boto:Method: HEAD 2023-10-12T00:49:23.388 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:23.389 DEBUG:boto:Data: 2023-10-12T00:49:23.389 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.390 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.390 DEBUG:boto:Params: {} 2023-10-12T00:49:23.391 DEBUG:boto:Token: None 2023-10-12T00:49:23.391 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI= 2023-10-12T00:49:23.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI=', 'Content-Length': '0'} 2023-10-12T00:49:23.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000714739ba31229a33-0065274293-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.395 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:49:23.395 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:49:23.396 DEBUG:boto:Method: PUT 2023-10-12T00:49:23.396 DEBUG:boto:Path: /wafmoa-56/g 2023-10-12T00:49:23.396 DEBUG:boto:Data: 2023-10-12T00:49:23.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:23.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.398 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.398 DEBUG:boto:Params: {} 2023-10-12T00:49:23.399 DEBUG:boto:Token: None 2023-10-12T00:49:23.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/g 2023-10-12T00:49:23.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:najIH6yl91Gv7Cmno98UEiRxjDU= 2023-10-12T00:49:23.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:najIH6yl91Gv7Cmno98UEiRxjDU='} 2023-10-12T00:49:23.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c1b5a7bdf376363-0065274293-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.406 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:49:23.406 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:49:23.407 DEBUG:boto:Method: HEAD 2023-10-12T00:49:23.407 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:49:23.408 DEBUG:boto:Data: 2023-10-12T00:49:23.408 DEBUG:boto:Headers: {} 2023-10-12T00:49:23.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.409 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.409 DEBUG:boto:Params: {} 2023-10-12T00:49:23.410 DEBUG:boto:Token: None 2023-10-12T00:49:23.410 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/ 2023-10-12T00:49:23.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI= 2023-10-12T00:49:23.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e2OxUgYqkNrIG1UHvrboshN9kuI=', 'Content-Length': '0'} 2023-10-12T00:49:23.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2c95185d2ec3960-0065274293-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.414 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:49:23.415 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:49:23.415 DEBUG:boto:Method: PUT 2023-10-12T00:49:23.416 DEBUG:boto:Path: /wafmoa-56/h 2023-10-12T00:49:23.416 DEBUG:boto:Data: 2023-10-12T00:49:23.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:49:23.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:49:23.417 DEBUG:boto:Port: 8000 2023-10-12T00:49:23.418 DEBUG:boto:Params: {} 2023-10-12T00:49:23.418 DEBUG:boto:Token: None 2023-10-12T00:49:23.419 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:49:23 GMT /wafmoa-56/h 2023-10-12T00:49:23.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w6JHDRVV6ioQL17uJpZ4iqalKiw= 2023-10-12T00:49:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:49:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w6JHDRVV6ioQL17uJpZ4iqalKiw='} 2023-10-12T00:49:23.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1c1fab6c561337c-0065274293-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:49:23.425 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-56 2023-10-12T00:49:23.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:23.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:49:23.427 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:49:23.483 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:49:23.483 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:49:23.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:23.515+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 0.. 2023-10-12T00:49:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:28.517+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:33.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:33.518+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:38.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:38.519+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:43.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:43.518+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:48.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:48.519+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:53.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:53.520+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:49:58.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:49:58.520+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:03.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:03.521+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:08.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:08.522+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:13.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:13.522+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:18.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:18.523+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:23.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:23.523+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:28.523+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:33.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:33.524+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:38.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:38.525+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:43.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:43.526+0000 7fbd51f54780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T00:50:48.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:48.527+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:50:48.527 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:50:48.528 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:50:53.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:53.528+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:50:53.528 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:50:53.528 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:50:58.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:50:58.527+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:50:58.529 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:50:58.529 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:03.528+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:03.529 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:51:03.530 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:08.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:08.529+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:08.530 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:51:08.530 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:13.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:13.530+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:13.531 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:51:13.531 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:18.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:18.531+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:18.532 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:51:18.532 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:23.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:23.533+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:23.533 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-10-12T00:51:23.533 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:28.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:28.533+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:28.533 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:28.534 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:33.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:33.534+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:33.534 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:33.534 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:38.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:38.534+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:38.535 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:38.535 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:43.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:43.535+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:43.536 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:43.536 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:48.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:48.535+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:48.537 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:48.537 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:53.536+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:53.537 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:53.538 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:51:58.538+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:51:58.538 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:51:58.538 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:03.539+0000 7fbd51f54780 1 waiting for incremental sync to catch up: 2023-10-12T00:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:52:08.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:08.540+0000 7fbd51f54780 1 bucket sync caught up with source: 2023-10-12T00:52:08.540 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:52:08.540 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.26.6, 00000000002.36.6, 00000000002.26.6] 2023-10-12T00:52:08.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:08.540+0000 7fbd51f54780 0 bucket checkpoint complete 2023-10-12T00:52:08.557 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-56 2023-10-12T00:52:08.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:52:08.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:52:08.559 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:52:08.621 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:08.622 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:08.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:08.659+0000 7f393a93f780 1 waiting for incremental sync to catch up: 2023-10-12T00:52:08.659 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.339.6, , , , 00000000001.172.6, , , 00000000002.229.6, , 00000000001.174.6] 2023-10-12T00:52:08.659 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.339.6, , , 00000000001.214.6, 00000000001.172.6, , , 00000000002.229.6, , 00000000002.197.6] 2023-10-12T00:52:13.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:13.659+0000 7f393a93f780 1 waiting for incremental sync to catch up: 2023-10-12T00:52:13.661 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.339.6, , , 00000000001.214.6, 00000000001.172.6, , , 00000000002.229.6, , 00000000001.174.6] 2023-10-12T00:52:13.661 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.339.6, , , 00000000001.214.6, 00000000001.172.6, , , 00000000002.229.6, , 00000000002.197.6] 2023-10-12T00:52:18.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:18.661+0000 7f393a93f780 1 bucket sync caught up with source: 2023-10-12T00:52:18.662 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.339.6, , , 00000000001.214.6, 00000000001.172.6, , , 00000000002.229.6, , 00000000002.197.6] 2023-10-12T00:52:18.662 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.339.6, , , 00000000001.214.6, 00000000001.172.6, , , 00000000002.229.6, , 00000000002.197.6] 2023-10-12T00:52:18.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:18.661+0000 7f393a93f780 0 bucket checkpoint complete 2023-10-12T00:52:18.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-56 zones={a2, a1} 2023-10-12T00:52:18.679 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.680 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.680 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.680 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:52:18.681 DEBUG:boto:Data: 2023-10-12T00:52:18.681 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.682 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.683 DEBUG:boto:Params: {} 2023-10-12T00:52:18.683 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:52:18.684 DEBUG:boto:Token: None 2023-10-12T00:52:18.684 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.685 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PWF+6uDkIVXhw4sjf8VqB3gB7ck= 2023-10-12T00:52:18.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PWF+6uDkIVXhw4sjf8VqB3gB7ck=', 'Content-Length': '0'} 2023-10-12T00:52:18.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e8090377af76da9-0065274342-1689-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.692 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.693 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.694 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.695 DEBUG:boto:Path: /wafmoa-56/ 2023-10-12T00:52:18.695 DEBUG:boto:Data: 2023-10-12T00:52:18.696 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.697 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.698 DEBUG:boto:Params: {} 2023-10-12T00:52:18.699 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:52:18.699 DEBUG:boto:Token: None 2023-10-12T00:52:18.699 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PWF+6uDkIVXhw4sjf8VqB3gB7ck= 2023-10-12T00:52:18.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PWF+6uDkIVXhw4sjf8VqB3gB7ck=', 'Content-Length': '0'} 2023-10-12T00:52:18.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044293e274ebb38c8-0065274342-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:52:18.706 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.707 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.707 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.708 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.708 DEBUG:boto:Method: GET 2023-10-12T00:52:18.708 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.709 DEBUG:boto:Data: 2023-10-12T00:52:18.709 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.710 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.711 DEBUG:boto:Params: {} 2023-10-12T00:52:18.712 DEBUG:boto:Token: None 2023-10-12T00:52:18.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/?versions 2023-10-12T00:52:18.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c= 2023-10-12T00:52:18.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c=', 'Content-Length': '0'} 2023-10-12T00:52:18.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091becdf9c490d94e-0065274342-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.716 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:52:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:52:18.718 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:52:18.719 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:52:18.719 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:52:18.720 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:52:18.720 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:52:18.720 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:52:18.721 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:52:18.721 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.722 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.722 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.722 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.723 DEBUG:boto:Method: GET 2023-10-12T00:52:18.723 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.724 DEBUG:boto:Data: 2023-10-12T00:52:18.724 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.725 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.726 DEBUG:boto:Params: {} 2023-10-12T00:52:18.726 DEBUG:boto:Token: None 2023-10-12T00:52:18.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/?versions 2023-10-12T00:52:18.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c= 2023-10-12T00:52:18.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c=', 'Content-Length': '0'} 2023-10-12T00:52:18.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70456427f68b5d2-0065274342-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.730 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.731 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:52:18.732 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:52:18.732 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:52:18.732 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:52:18.733 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:52:18.733 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:52:18.734 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:52:18.734 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:52:18.734 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.735 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.735 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.736 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.736 DEBUG:boto:Method: GET 2023-10-12T00:52:18.736 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.737 DEBUG:boto:Data: 2023-10-12T00:52:18.737 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.738 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.738 DEBUG:boto:Params: {} 2023-10-12T00:52:18.739 DEBUG:boto:Token: None 2023-10-12T00:52:18.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/?versions 2023-10-12T00:52:18.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c= 2023-10-12T00:52:18.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c=', 'Content-Length': '0'} 2023-10-12T00:52:18.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf6eb4b8fca63e7-0065274342-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.744 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.745 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.745 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.746 DEBUG:boto:path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.746 DEBUG:boto:auth_path=/wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.746 DEBUG:boto:Method: GET 2023-10-12T00:52:18.747 DEBUG:boto:Path: /wafmoa-56/?versions&max-keys=999 2023-10-12T00:52:18.747 DEBUG:boto:Data: 2023-10-12T00:52:18.748 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.749 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.749 DEBUG:boto:Params: {} 2023-10-12T00:52:18.749 DEBUG:boto:Token: None 2023-10-12T00:52:18.750 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/?versions 2023-10-12T00:52:18.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c= 2023-10-12T00:52:18.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fe+JJ2IF2fhqwhFXlmzTEe0WU/c=', 'Content-Length': '0'} 2023-10-12T00:52:18.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bf3dac71da72a3-0065274342-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.753 DEBUG:boto:b'wafmoa-56999falseanulltrue2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:52:18.755 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.755 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.756 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.756 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.757 DEBUG:boto:Method: GET 2023-10-12T00:52:18.757 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.757 DEBUG:boto:Data: 2023-10-12T00:52:18.758 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.759 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.759 DEBUG:boto:Params: {} 2023-10-12T00:52:18.760 DEBUG:boto:Token: None 2023-10-12T00:52:18.760 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ= 2023-10-12T00:52:18.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ=', 'Content-Length': '0'} 2023-10-12T00:52:18.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d75c8218976c71d-0065274342-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.806 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.807 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.807 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.808 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.808 DEBUG:boto:Method: GET 2023-10-12T00:52:18.809 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.809 DEBUG:boto:Data: 2023-10-12T00:52:18.810 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.810 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.811 DEBUG:boto:Params: {} 2023-10-12T00:52:18.811 DEBUG:boto:Token: None 2023-10-12T00:52:18.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ= 2023-10-12T00:52:18.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ=', 'Content-Length': '0'} 2023-10-12T00:52:18.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3db598d5c3e6324-0065274342-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.816 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.816 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.816 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.817 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.817 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.818 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.818 DEBUG:boto:Data: 2023-10-12T00:52:18.818 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.819 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.820 DEBUG:boto:Params: {} 2023-10-12T00:52:18.820 DEBUG:boto:Token: None 2023-10-12T00:52:18.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TzjwLRreKJbZI7Vezqj4ZlRQRA= 2023-10-12T00:52:18.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TzjwLRreKJbZI7Vezqj4ZlRQRA=', 'Content-Length': '0'} 2023-10-12T00:52:18.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700a36491cbcc744-0065274342-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.824 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.825 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.825 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.825 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.826 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.826 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.827 DEBUG:boto:Data: 2023-10-12T00:52:18.827 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.828 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.828 DEBUG:boto:Params: {} 2023-10-12T00:52:18.829 DEBUG:boto:Token: None 2023-10-12T00:52:18.829 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TzjwLRreKJbZI7Vezqj4ZlRQRA= 2023-10-12T00:52:18.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TzjwLRreKJbZI7Vezqj4ZlRQRA=', 'Content-Length': '0'} 2023-10-12T00:52:18.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456cab342d0666d6-0065274342-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:52:18.833 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.833 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.834 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.834 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.835 DEBUG:boto:Method: GET 2023-10-12T00:52:18.835 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.835 DEBUG:boto:Data: 2023-10-12T00:52:18.836 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.837 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.837 DEBUG:boto:Params: {} 2023-10-12T00:52:18.838 DEBUG:boto:Token: None 2023-10-12T00:52:18.838 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ= 2023-10-12T00:52:18.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ=', 'Content-Length': '0'} 2023-10-12T00:52:18.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305b9d69dc09738a-0065274342-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.842 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.842 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.843 DEBUG:boto:path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.843 DEBUG:boto:auth_path=/wafmoa-56/a?versionId=null 2023-10-12T00:52:18.844 DEBUG:boto:Method: GET 2023-10-12T00:52:18.844 DEBUG:boto:Path: /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.844 DEBUG:boto:Data: 2023-10-12T00:52:18.845 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.846 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.846 DEBUG:boto:Params: {} 2023-10-12T00:52:18.847 DEBUG:boto:Token: None 2023-10-12T00:52:18.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a?versionId=null 2023-10-12T00:52:18.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ= 2023-10-12T00:52:18.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kza1qqujWcotHlKIC2mE1b4+huQ=', 'Content-Length': '0'} 2023-10-12T00:52:18.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550e2e4fd7cdb0f7-0065274342-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.851 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.851 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.852 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.852 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.852 DEBUG:boto:Method: GET 2023-10-12T00:52:18.853 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:52:18.853 DEBUG:boto:Data: 2023-10-12T00:52:18.854 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.855 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.855 DEBUG:boto:Params: {} 2023-10-12T00:52:18.855 DEBUG:boto:Token: None 2023-10-12T00:52:18.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4= 2023-10-12T00:52:18.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4=', 'Content-Length': '0'} 2023-10-12T00:52:18.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e12e7a6de01cbc5-0065274342-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.861 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.862 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.862 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.863 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.863 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.863 DEBUG:boto:Method: GET 2023-10-12T00:52:18.864 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:52:18.864 DEBUG:boto:Data: 2023-10-12T00:52:18.865 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.866 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.866 DEBUG:boto:Params: {} 2023-10-12T00:52:18.867 DEBUG:boto:Token: None 2023-10-12T00:52:18.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4= 2023-10-12T00:52:18.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4=', 'Content-Length': '0'} 2023-10-12T00:52:18.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc4ad6ebf9b55dd-0065274342-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.871 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.871 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.872 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.872 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.873 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:52:18.873 DEBUG:boto:Data: 2023-10-12T00:52:18.874 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.874 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.875 DEBUG:boto:Params: {} 2023-10-12T00:52:18.875 DEBUG:boto:Token: None 2023-10-12T00:52:18.876 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a 2023-10-12T00:52:18.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cf2lysoTQ1ATb5oeXdtWcqsaczI= 2023-10-12T00:52:18.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cf2lysoTQ1ATb5oeXdtWcqsaczI=', 'Content-Length': '0'} 2023-10-12T00:52:18.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c9cd283e5377ad-0065274342-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.879 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.880 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.880 DEBUG:boto:Method: HEAD 2023-10-12T00:52:18.881 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:52:18.881 DEBUG:boto:Data: 2023-10-12T00:52:18.882 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.883 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.883 DEBUG:boto:Params: {} 2023-10-12T00:52:18.883 DEBUG:boto:Token: None 2023-10-12T00:52:18.884 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a 2023-10-12T00:52:18.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cf2lysoTQ1ATb5oeXdtWcqsaczI= 2023-10-12T00:52:18.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cf2lysoTQ1ATb5oeXdtWcqsaczI=', 'Content-Length': '0'} 2023-10-12T00:52:18.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e0dbf78f90fb25-0065274342-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:52:18.888 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.888 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.889 DEBUG:boto:Method: GET 2023-10-12T00:52:18.889 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:52:18.889 DEBUG:boto:Data: 2023-10-12T00:52:18.890 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.891 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.891 DEBUG:boto:Params: {} 2023-10-12T00:52:18.892 DEBUG:boto:Token: None 2023-10-12T00:52:18.892 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a 2023-10-12T00:52:18.892 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:36j9lbApgIuq0ZsUplDUtF8Prxc= 2023-10-12T00:52:18.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:36j9lbApgIuq0ZsUplDUtF8Prxc=', 'Content-Length': '0'} 2023-10-12T00:52:18.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62037bca1d96095-0065274342-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.938 DEBUG:boto:path=/wafmoa-56/a 2023-10-12T00:52:18.939 DEBUG:boto:auth_path=/wafmoa-56/a 2023-10-12T00:52:18.939 DEBUG:boto:Method: GET 2023-10-12T00:52:18.940 DEBUG:boto:Path: /wafmoa-56/a 2023-10-12T00:52:18.940 DEBUG:boto:Data: 2023-10-12T00:52:18.941 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.941 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.942 DEBUG:boto:Params: {} 2023-10-12T00:52:18.942 DEBUG:boto:Token: None 2023-10-12T00:52:18.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/a 2023-10-12T00:52:18.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:36j9lbApgIuq0ZsUplDUtF8Prxc= 2023-10-12T00:52:18.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:36j9lbApgIuq0ZsUplDUtF8Prxc=', 'Content-Length': '0'} 2023-10-12T00:52:18.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d914b2ce5792894-0065274342-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.947 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.948 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.948 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.949 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.949 DEBUG:boto:Method: GET 2023-10-12T00:52:18.950 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:52:18.951 DEBUG:boto:Data: 2023-10-12T00:52:18.952 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:18.953 DEBUG:boto:Port: 8001 2023-10-12T00:52:18.954 DEBUG:boto:Params: {} 2023-10-12T00:52:18.955 DEBUG:boto:Token: None 2023-10-12T00:52:18.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4= 2023-10-12T00:52:18.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4=', 'Content-Length': '0'} 2023-10-12T00:52:18.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a85f369be47a56fd-0065274342-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:18.961 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:18.962 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:18.962 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:18.963 DEBUG:boto:path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.963 DEBUG:boto:auth_path=/wafmoa-56/?prefix=a 2023-10-12T00:52:18.963 DEBUG:boto:Method: GET 2023-10-12T00:52:18.964 DEBUG:boto:Path: /wafmoa-56/?prefix=a 2023-10-12T00:52:18.964 DEBUG:boto:Data: 2023-10-12T00:52:18.965 DEBUG:boto:Headers: {} 2023-10-12T00:52:18.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:18.966 DEBUG:boto:Port: 8000 2023-10-12T00:52:18.966 DEBUG:boto:Params: {} 2023-10-12T00:52:18.966 DEBUG:boto:Token: None 2023-10-12T00:52:18.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:18 GMT /wafmoa-56/ 2023-10-12T00:52:18.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4= 2023-10-12T00:52:18.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D2qcGsmcnl7ddRQfrEI9wK+Dnc4=', 'Content-Length': '0'} 2023-10-12T00:52:18.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de58584a7abb1997-0065274342-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.014 DEBUG:boto:b'wafmoa-56a1000falsea2023-10-12T00:49:06.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:52:19.016 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.016 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.017 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.017 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.018 DEBUG:boto:Method: GET 2023-10-12T00:52:19.018 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.019 DEBUG:boto:Data: 2023-10-12T00:52:19.019 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.020 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.020 DEBUG:boto:Params: {} 2023-10-12T00:52:19.021 DEBUG:boto:Token: None 2023-10-12T00:52:19.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw= 2023-10-12T00:52:19.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw=', 'Content-Length': '0'} 2023-10-12T00:52:19.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1b97969e8f36da-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.026 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.026 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.027 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.027 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.027 DEBUG:boto:Method: GET 2023-10-12T00:52:19.028 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.028 DEBUG:boto:Data: 2023-10-12T00:52:19.029 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.030 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.030 DEBUG:boto:Params: {} 2023-10-12T00:52:19.031 DEBUG:boto:Token: None 2023-10-12T00:52:19.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw= 2023-10-12T00:52:19.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw=', 'Content-Length': '0'} 2023-10-12T00:52:19.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc1ddefe49f59e4-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.035 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.035 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.036 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.037 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.037 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.038 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.038 DEBUG:boto:Data: 2023-10-12T00:52:19.039 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.039 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.040 DEBUG:boto:Params: {} 2023-10-12T00:52:19.040 DEBUG:boto:Token: None 2023-10-12T00:52:19.041 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UfX3AISLYjoC8TZbeWcBuFHmNJo= 2023-10-12T00:52:19.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UfX3AISLYjoC8TZbeWcBuFHmNJo=', 'Content-Length': '0'} 2023-10-12T00:52:19.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4b78fa36b99805-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.044 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.045 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.045 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.046 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.046 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.046 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.047 DEBUG:boto:Data: 2023-10-12T00:52:19.047 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.048 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.048 DEBUG:boto:Params: {} 2023-10-12T00:52:19.049 DEBUG:boto:Token: None 2023-10-12T00:52:19.049 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UfX3AISLYjoC8TZbeWcBuFHmNJo= 2023-10-12T00:52:19.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UfX3AISLYjoC8TZbeWcBuFHmNJo=', 'Content-Length': '0'} 2023-10-12T00:52:19.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efc41fdb2b020e60-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:52:19.053 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.054 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.054 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.055 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.055 DEBUG:boto:Method: GET 2023-10-12T00:52:19.056 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.056 DEBUG:boto:Data: 2023-10-12T00:52:19.057 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.058 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.058 DEBUG:boto:Params: {} 2023-10-12T00:52:19.059 DEBUG:boto:Token: None 2023-10-12T00:52:19.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw= 2023-10-12T00:52:19.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw=', 'Content-Length': '0'} 2023-10-12T00:52:19.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004800d7cf3eae9f49-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.063 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.063 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.064 DEBUG:boto:path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.064 DEBUG:boto:auth_path=/wafmoa-56/b?versionId=null 2023-10-12T00:52:19.065 DEBUG:boto:Method: GET 2023-10-12T00:52:19.065 DEBUG:boto:Path: /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.065 DEBUG:boto:Data: 2023-10-12T00:52:19.066 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.067 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.067 DEBUG:boto:Params: {} 2023-10-12T00:52:19.068 DEBUG:boto:Token: None 2023-10-12T00:52:19.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b?versionId=null 2023-10-12T00:52:19.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw= 2023-10-12T00:52:19.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2tmcld4KNpfbvNsX0OE2U9u0bvw=', 'Content-Length': '0'} 2023-10-12T00:52:19.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c515cef8213a1655-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.072 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.072 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.073 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.073 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.074 DEBUG:boto:Method: GET 2023-10-12T00:52:19.074 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:52:19.074 DEBUG:boto:Data: 2023-10-12T00:52:19.075 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.076 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.076 DEBUG:boto:Params: {} 2023-10-12T00:52:19.077 DEBUG:boto:Token: None 2023-10-12T00:52:19.077 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.078 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5989630b2f996b0-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.081 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.082 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.083 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.083 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.084 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.084 DEBUG:boto:Method: GET 2023-10-12T00:52:19.085 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:52:19.085 DEBUG:boto:Data: 2023-10-12T00:52:19.086 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.087 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.087 DEBUG:boto:Params: {} 2023-10-12T00:52:19.088 DEBUG:boto:Token: None 2023-10-12T00:52:19.088 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.089 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342a481db720eb7a-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.138 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.139 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.140 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.140 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.140 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:52:19.141 DEBUG:boto:Data: 2023-10-12T00:52:19.141 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.142 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.143 DEBUG:boto:Params: {} 2023-10-12T00:52:19.143 DEBUG:boto:Token: None 2023-10-12T00:52:19.144 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b 2023-10-12T00:52:19.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:03WlOIhbSr8quwKyF37Gyk27WPo= 2023-10-12T00:52:19.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:03WlOIhbSr8quwKyF37Gyk27WPo=', 'Content-Length': '0'} 2023-10-12T00:52:19.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000640faf04731993ae-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.147 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.148 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.148 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.149 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:52:19.149 DEBUG:boto:Data: 2023-10-12T00:52:19.149 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.150 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.151 DEBUG:boto:Params: {} 2023-10-12T00:52:19.151 DEBUG:boto:Token: None 2023-10-12T00:52:19.152 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b 2023-10-12T00:52:19.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:03WlOIhbSr8quwKyF37Gyk27WPo= 2023-10-12T00:52:19.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:03WlOIhbSr8quwKyF37Gyk27WPo=', 'Content-Length': '0'} 2023-10-12T00:52:19.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec86f79f6fe15ad-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:52:19.156 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.156 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.156 DEBUG:boto:Method: GET 2023-10-12T00:52:19.157 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:52:19.157 DEBUG:boto:Data: 2023-10-12T00:52:19.158 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.159 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.159 DEBUG:boto:Params: {} 2023-10-12T00:52:19.160 DEBUG:boto:Token: None 2023-10-12T00:52:19.160 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b 2023-10-12T00:52:19.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OgEs+15Chuc42kO/vz4jNkEHGuE= 2023-10-12T00:52:19.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OgEs+15Chuc42kO/vz4jNkEHGuE=', 'Content-Length': '0'} 2023-10-12T00:52:19.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ab1842dee2b2e5-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.210 DEBUG:boto:path=/wafmoa-56/b 2023-10-12T00:52:19.211 DEBUG:boto:auth_path=/wafmoa-56/b 2023-10-12T00:52:19.212 DEBUG:boto:Method: GET 2023-10-12T00:52:19.212 DEBUG:boto:Path: /wafmoa-56/b 2023-10-12T00:52:19.212 DEBUG:boto:Data: 2023-10-12T00:52:19.213 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.214 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.214 DEBUG:boto:Params: {} 2023-10-12T00:52:19.215 DEBUG:boto:Token: None 2023-10-12T00:52:19.215 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/b 2023-10-12T00:52:19.216 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OgEs+15Chuc42kO/vz4jNkEHGuE= 2023-10-12T00:52:19.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OgEs+15Chuc42kO/vz4jNkEHGuE=', 'Content-Length': '0'} 2023-10-12T00:52:19.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb0e315b9596aa8-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.219 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.219 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.220 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.220 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.221 DEBUG:boto:Method: GET 2023-10-12T00:52:19.221 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:52:19.222 DEBUG:boto:Data: 2023-10-12T00:52:19.222 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.223 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.224 DEBUG:boto:Params: {} 2023-10-12T00:52:19.224 DEBUG:boto:Token: None 2023-10-12T00:52:19.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fbe942fd2ec1a25-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.229 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.230 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.230 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.230 DEBUG:boto:path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.232 DEBUG:boto:auth_path=/wafmoa-56/?prefix=b 2023-10-12T00:52:19.232 DEBUG:boto:Method: GET 2023-10-12T00:52:19.233 DEBUG:boto:Path: /wafmoa-56/?prefix=b 2023-10-12T00:52:19.233 DEBUG:boto:Data: 2023-10-12T00:52:19.234 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.234 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.235 DEBUG:boto:Params: {} 2023-10-12T00:52:19.235 DEBUG:boto:Token: None 2023-10-12T00:52:19.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1505e25a16bce87-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.240 DEBUG:boto:b'wafmoa-56b1000falseb2023-10-12T00:49:06.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:52:19.241 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.241 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.242 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.242 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.243 DEBUG:boto:Method: GET 2023-10-12T00:52:19.243 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.244 DEBUG:boto:Data: 2023-10-12T00:52:19.244 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.245 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.245 DEBUG:boto:Params: {} 2023-10-12T00:52:19.246 DEBUG:boto:Token: None 2023-10-12T00:52:19.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0= 2023-10-12T00:52:19.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0=', 'Content-Length': '0'} 2023-10-12T00:52:19.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add96a1d022842e5-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.250 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.250 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.251 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.251 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.251 DEBUG:boto:Method: GET 2023-10-12T00:52:19.252 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.252 DEBUG:boto:Data: 2023-10-12T00:52:19.253 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.254 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.254 DEBUG:boto:Params: {} 2023-10-12T00:52:19.254 DEBUG:boto:Token: None 2023-10-12T00:52:19.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0= 2023-10-12T00:52:19.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0=', 'Content-Length': '0'} 2023-10-12T00:52:19.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562c4f84958128c9-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.302 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.303 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.303 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.304 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.304 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.304 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.305 DEBUG:boto:Data: 2023-10-12T00:52:19.305 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.306 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.307 DEBUG:boto:Params: {} 2023-10-12T00:52:19.307 DEBUG:boto:Token: None 2023-10-12T00:52:19.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AFMvLKOw5hXfwnw+JFfX/nlFaXY= 2023-10-12T00:52:19.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AFMvLKOw5hXfwnw+JFfX/nlFaXY=', 'Content-Length': '0'} 2023-10-12T00:52:19.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003178564fe0f4992a-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.311 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.311 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.312 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.312 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.313 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.313 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.314 DEBUG:boto:Data: 2023-10-12T00:52:19.314 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.317 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.317 DEBUG:boto:Params: {} 2023-10-12T00:52:19.318 DEBUG:boto:Token: None 2023-10-12T00:52:19.318 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AFMvLKOw5hXfwnw+JFfX/nlFaXY= 2023-10-12T00:52:19.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AFMvLKOw5hXfwnw+JFfX/nlFaXY=', 'Content-Length': '0'} 2023-10-12T00:52:19.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01145491b75137a-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:52:19.322 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.323 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.323 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.324 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.324 DEBUG:boto:Method: GET 2023-10-12T00:52:19.324 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.325 DEBUG:boto:Data: 2023-10-12T00:52:19.325 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.326 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.326 DEBUG:boto:Params: {} 2023-10-12T00:52:19.327 DEBUG:boto:Token: None 2023-10-12T00:52:19.327 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0= 2023-10-12T00:52:19.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0=', 'Content-Length': '0'} 2023-10-12T00:52:19.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000fa8fb6fff8728d-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.331 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.332 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.332 DEBUG:boto:path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.333 DEBUG:boto:auth_path=/wafmoa-56/c?versionId=null 2023-10-12T00:52:19.333 DEBUG:boto:Method: GET 2023-10-12T00:52:19.333 DEBUG:boto:Path: /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.334 DEBUG:boto:Data: 2023-10-12T00:52:19.334 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.335 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.336 DEBUG:boto:Params: {} 2023-10-12T00:52:19.336 DEBUG:boto:Token: None 2023-10-12T00:52:19.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c?versionId=null 2023-10-12T00:52:19.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0= 2023-10-12T00:52:19.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rbDn0W75dUGOvnwE7nErHjwwvO0=', 'Content-Length': '0'} 2023-10-12T00:52:19.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df55aba8659e4dc7-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.341 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.341 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.342 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.342 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.342 DEBUG:boto:Method: GET 2023-10-12T00:52:19.343 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:52:19.343 DEBUG:boto:Data: 2023-10-12T00:52:19.344 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.345 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.345 DEBUG:boto:Params: {} 2023-10-12T00:52:19.345 DEBUG:boto:Token: None 2023-10-12T00:52:19.346 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.347 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cac9205d8acf61-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.390 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.391 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.392 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.392 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.392 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.393 DEBUG:boto:Method: GET 2023-10-12T00:52:19.393 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:52:19.394 DEBUG:boto:Data: 2023-10-12T00:52:19.394 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.395 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.396 DEBUG:boto:Params: {} 2023-10-12T00:52:19.396 DEBUG:boto:Token: None 2023-10-12T00:52:19.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50fe2655774dd49-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.401 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.401 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.402 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.402 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.403 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:52:19.403 DEBUG:boto:Data: 2023-10-12T00:52:19.404 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.404 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.405 DEBUG:boto:Params: {} 2023-10-12T00:52:19.405 DEBUG:boto:Token: None 2023-10-12T00:52:19.406 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c 2023-10-12T00:52:19.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BTW74jPhmhs+Yk/gMLglwD0y/gM= 2023-10-12T00:52:19.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BTW74jPhmhs+Yk/gMLglwD0y/gM=', 'Content-Length': '0'} 2023-10-12T00:52:19.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa916793b641c66-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.409 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.410 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.410 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.411 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:52:19.411 DEBUG:boto:Data: 2023-10-12T00:52:19.411 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.412 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.413 DEBUG:boto:Params: {} 2023-10-12T00:52:19.413 DEBUG:boto:Token: None 2023-10-12T00:52:19.414 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c 2023-10-12T00:52:19.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BTW74jPhmhs+Yk/gMLglwD0y/gM= 2023-10-12T00:52:19.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BTW74jPhmhs+Yk/gMLglwD0y/gM=', 'Content-Length': '0'} 2023-10-12T00:52:19.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da30785b5f9d0d3e-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:52:19.418 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.419 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.419 DEBUG:boto:Method: GET 2023-10-12T00:52:19.420 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:52:19.420 DEBUG:boto:Data: 2023-10-12T00:52:19.421 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.422 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.422 DEBUG:boto:Params: {} 2023-10-12T00:52:19.423 DEBUG:boto:Token: None 2023-10-12T00:52:19.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c 2023-10-12T00:52:19.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PNkc5UwywrW1o+iwGLlUUm0GO84= 2023-10-12T00:52:19.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PNkc5UwywrW1o+iwGLlUUm0GO84=', 'Content-Length': '0'} 2023-10-12T00:52:19.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7fd96b693ab0b5-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.427 DEBUG:boto:path=/wafmoa-56/c 2023-10-12T00:52:19.427 DEBUG:boto:auth_path=/wafmoa-56/c 2023-10-12T00:52:19.428 DEBUG:boto:Method: GET 2023-10-12T00:52:19.428 DEBUG:boto:Path: /wafmoa-56/c 2023-10-12T00:52:19.428 DEBUG:boto:Data: 2023-10-12T00:52:19.429 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.430 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.430 DEBUG:boto:Params: {} 2023-10-12T00:52:19.431 DEBUG:boto:Token: None 2023-10-12T00:52:19.431 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/c 2023-10-12T00:52:19.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PNkc5UwywrW1o+iwGLlUUm0GO84= 2023-10-12T00:52:19.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PNkc5UwywrW1o+iwGLlUUm0GO84=', 'Content-Length': '0'} 2023-10-12T00:52:19.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001805a9d7d78cfaba-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.435 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.435 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.436 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.436 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.437 DEBUG:boto:Method: GET 2023-10-12T00:52:19.437 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:52:19.438 DEBUG:boto:Data: 2023-10-12T00:52:19.438 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.439 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.439 DEBUG:boto:Params: {} 2023-10-12T00:52:19.440 DEBUG:boto:Token: None 2023-10-12T00:52:19.440 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eab7ca11e58b6219-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.444 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.445 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.446 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.446 DEBUG:boto:path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.447 DEBUG:boto:auth_path=/wafmoa-56/?prefix=c 2023-10-12T00:52:19.447 DEBUG:boto:Method: GET 2023-10-12T00:52:19.448 DEBUG:boto:Path: /wafmoa-56/?prefix=c 2023-10-12T00:52:19.448 DEBUG:boto:Data: 2023-10-12T00:52:19.449 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.449 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.450 DEBUG:boto:Params: {} 2023-10-12T00:52:19.450 DEBUG:boto:Token: None 2023-10-12T00:52:19.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1781aac02612111-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.455 DEBUG:boto:b'wafmoa-56c1000falsec2023-10-12T00:49:06.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:52:19.456 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.457 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.457 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.458 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.458 DEBUG:boto:Method: GET 2023-10-12T00:52:19.459 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.459 DEBUG:boto:Data: 2023-10-12T00:52:19.460 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.460 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.461 DEBUG:boto:Params: {} 2023-10-12T00:52:19.461 DEBUG:boto:Token: None 2023-10-12T00:52:19.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I= 2023-10-12T00:52:19.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I=', 'Content-Length': '0'} 2023-10-12T00:52:19.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e0abf5bef3b2f9-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.466 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.466 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.467 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.467 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.468 DEBUG:boto:Method: GET 2023-10-12T00:52:19.468 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.469 DEBUG:boto:Data: 2023-10-12T00:52:19.469 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.470 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.471 DEBUG:boto:Params: {} 2023-10-12T00:52:19.471 DEBUG:boto:Token: None 2023-10-12T00:52:19.472 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.472 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I= 2023-10-12T00:52:19.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I=', 'Content-Length': '0'} 2023-10-12T00:52:19.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061210038f6a8a5b3-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.476 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.476 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.477 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.477 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.477 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.478 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.478 DEBUG:boto:Data: 2023-10-12T00:52:19.479 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.480 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.480 DEBUG:boto:Params: {} 2023-10-12T00:52:19.481 DEBUG:boto:Token: None 2023-10-12T00:52:19.481 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HL64TFXLnjyvqA4xRS/fiF6l5qk= 2023-10-12T00:52:19.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HL64TFXLnjyvqA4xRS/fiF6l5qk=', 'Content-Length': '0'} 2023-10-12T00:52:19.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342645af28b96a5f-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.487 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.488 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.488 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.488 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.489 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.489 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.489 DEBUG:boto:Data: 2023-10-12T00:52:19.490 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.491 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.491 DEBUG:boto:Params: {} 2023-10-12T00:52:19.491 DEBUG:boto:Token: None 2023-10-12T00:52:19.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HL64TFXLnjyvqA4xRS/fiF6l5qk= 2023-10-12T00:52:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HL64TFXLnjyvqA4xRS/fiF6l5qk=', 'Content-Length': '0'} 2023-10-12T00:52:19.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b0e9adf8ba5bcd-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:52:19.496 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.496 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.496 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.497 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.497 DEBUG:boto:Method: GET 2023-10-12T00:52:19.498 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.498 DEBUG:boto:Data: 2023-10-12T00:52:19.499 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.500 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.500 DEBUG:boto:Params: {} 2023-10-12T00:52:19.500 DEBUG:boto:Token: None 2023-10-12T00:52:19.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I= 2023-10-12T00:52:19.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I=', 'Content-Length': '0'} 2023-10-12T00:52:19.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29da3e04c7004cc-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.505 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.505 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.506 DEBUG:boto:path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.506 DEBUG:boto:auth_path=/wafmoa-56/d?versionId=null 2023-10-12T00:52:19.506 DEBUG:boto:Method: GET 2023-10-12T00:52:19.507 DEBUG:boto:Path: /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.507 DEBUG:boto:Data: 2023-10-12T00:52:19.508 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.509 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.509 DEBUG:boto:Params: {} 2023-10-12T00:52:19.509 DEBUG:boto:Token: None 2023-10-12T00:52:19.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d?versionId=null 2023-10-12T00:52:19.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I= 2023-10-12T00:52:19.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6mh9l05MQJZCOaRRlNjKrmmN1I=', 'Content-Length': '0'} 2023-10-12T00:52:19.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d2a71fc46abcd8-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.558 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.559 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.560 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.560 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.561 DEBUG:boto:Method: GET 2023-10-12T00:52:19.561 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:52:19.562 DEBUG:boto:Data: 2023-10-12T00:52:19.562 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.563 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.563 DEBUG:boto:Params: {} 2023-10-12T00:52:19.564 DEBUG:boto:Token: None 2023-10-12T00:52:19.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0995fd6c87cbdd-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.569 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.570 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.570 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.571 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.571 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.571 DEBUG:boto:Method: GET 2023-10-12T00:52:19.572 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:52:19.572 DEBUG:boto:Data: 2023-10-12T00:52:19.573 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.574 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.574 DEBUG:boto:Params: {} 2023-10-12T00:52:19.575 DEBUG:boto:Token: None 2023-10-12T00:52:19.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f1064e1fe5d82e6-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.579 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.580 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.580 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.581 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.581 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:52:19.582 DEBUG:boto:Data: 2023-10-12T00:52:19.582 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.583 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.583 DEBUG:boto:Params: {} 2023-10-12T00:52:19.584 DEBUG:boto:Token: None 2023-10-12T00:52:19.584 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d 2023-10-12T00:52:19.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xXJplST6kMnX99oGPMJtpNNd0kw= 2023-10-12T00:52:19.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xXJplST6kMnX99oGPMJtpNNd0kw=', 'Content-Length': '0'} 2023-10-12T00:52:19.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785d70c19bb76023-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.588 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.588 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.589 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.589 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:52:19.589 DEBUG:boto:Data: 2023-10-12T00:52:19.590 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.591 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.591 DEBUG:boto:Params: {} 2023-10-12T00:52:19.592 DEBUG:boto:Token: None 2023-10-12T00:52:19.592 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d 2023-10-12T00:52:19.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xXJplST6kMnX99oGPMJtpNNd0kw= 2023-10-12T00:52:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xXJplST6kMnX99oGPMJtpNNd0kw=', 'Content-Length': '0'} 2023-10-12T00:52:19.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c500109dfa92c0d-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:52:19.596 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.597 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.597 DEBUG:boto:Method: GET 2023-10-12T00:52:19.598 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:52:19.598 DEBUG:boto:Data: 2023-10-12T00:52:19.598 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.599 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.600 DEBUG:boto:Params: {} 2023-10-12T00:52:19.600 DEBUG:boto:Token: None 2023-10-12T00:52:19.601 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d 2023-10-12T00:52:19.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbIIJTFfOsMICyceY3xaZ57fZ4U= 2023-10-12T00:52:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbIIJTFfOsMICyceY3xaZ57fZ4U=', 'Content-Length': '0'} 2023-10-12T00:52:19.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04f31a45d4efd9f-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.646 DEBUG:boto:path=/wafmoa-56/d 2023-10-12T00:52:19.647 DEBUG:boto:auth_path=/wafmoa-56/d 2023-10-12T00:52:19.647 DEBUG:boto:Method: GET 2023-10-12T00:52:19.648 DEBUG:boto:Path: /wafmoa-56/d 2023-10-12T00:52:19.648 DEBUG:boto:Data: 2023-10-12T00:52:19.649 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.650 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.650 DEBUG:boto:Params: {} 2023-10-12T00:52:19.651 DEBUG:boto:Token: None 2023-10-12T00:52:19.651 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/d 2023-10-12T00:52:19.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KbIIJTFfOsMICyceY3xaZ57fZ4U= 2023-10-12T00:52:19.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KbIIJTFfOsMICyceY3xaZ57fZ4U=', 'Content-Length': '0'} 2023-10-12T00:52:19.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc1d385df460c10-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.655 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.656 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.656 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.656 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.657 DEBUG:boto:Method: GET 2023-10-12T00:52:19.657 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:52:19.658 DEBUG:boto:Data: 2023-10-12T00:52:19.658 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.659 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.659 DEBUG:boto:Params: {} 2023-10-12T00:52:19.660 DEBUG:boto:Token: None 2023-10-12T00:52:19.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1ce95329c13c243-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.664 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.665 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.666 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.666 DEBUG:boto:path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.666 DEBUG:boto:auth_path=/wafmoa-56/?prefix=d 2023-10-12T00:52:19.667 DEBUG:boto:Method: GET 2023-10-12T00:52:19.667 DEBUG:boto:Path: /wafmoa-56/?prefix=d 2023-10-12T00:52:19.668 DEBUG:boto:Data: 2023-10-12T00:52:19.668 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.669 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.669 DEBUG:boto:Params: {} 2023-10-12T00:52:19.670 DEBUG:boto:Token: None 2023-10-12T00:52:19.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b6cb06debb97d1-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.718 DEBUG:boto:b'wafmoa-56d1000falsed2023-10-12T00:49:06.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:52:19.720 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.720 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.721 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.721 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.721 DEBUG:boto:Method: GET 2023-10-12T00:52:19.722 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.722 DEBUG:boto:Data: 2023-10-12T00:52:19.723 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.724 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.724 DEBUG:boto:Params: {} 2023-10-12T00:52:19.725 DEBUG:boto:Token: None 2023-10-12T00:52:19.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM= 2023-10-12T00:52:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM=', 'Content-Length': '0'} 2023-10-12T00:52:19.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa77fd7aada4fae-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.732 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.732 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.733 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.733 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.734 DEBUG:boto:Method: GET 2023-10-12T00:52:19.734 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.735 DEBUG:boto:Data: 2023-10-12T00:52:19.735 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.736 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.736 DEBUG:boto:Params: {} 2023-10-12T00:52:19.737 DEBUG:boto:Token: None 2023-10-12T00:52:19.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM= 2023-10-12T00:52:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM=', 'Content-Length': '0'} 2023-10-12T00:52:19.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a01a1bd1a3e7eab-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.741 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.742 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.742 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.743 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.743 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.743 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.744 DEBUG:boto:Data: 2023-10-12T00:52:19.744 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.745 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.745 DEBUG:boto:Params: {} 2023-10-12T00:52:19.746 DEBUG:boto:Token: None 2023-10-12T00:52:19.746 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PV3Cjhr/YOylem98CarLl5yR6u8= 2023-10-12T00:52:19.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PV3Cjhr/YOylem98CarLl5yR6u8=', 'Content-Length': '0'} 2023-10-12T00:52:19.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f9ddb82cb03a78-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.750 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.750 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.751 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.751 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.752 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.752 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.753 DEBUG:boto:Data: 2023-10-12T00:52:19.753 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.754 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.754 DEBUG:boto:Params: {} 2023-10-12T00:52:19.755 DEBUG:boto:Token: None 2023-10-12T00:52:19.755 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PV3Cjhr/YOylem98CarLl5yR6u8= 2023-10-12T00:52:19.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PV3Cjhr/YOylem98CarLl5yR6u8=', 'Content-Length': '0'} 2023-10-12T00:52:19.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c24500c17769c2a-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:52:19.759 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.759 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.760 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.760 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.761 DEBUG:boto:Method: GET 2023-10-12T00:52:19.761 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.761 DEBUG:boto:Data: 2023-10-12T00:52:19.762 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.763 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.763 DEBUG:boto:Params: {} 2023-10-12T00:52:19.764 DEBUG:boto:Token: None 2023-10-12T00:52:19.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM= 2023-10-12T00:52:19.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM=', 'Content-Length': '0'} 2023-10-12T00:52:19.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7d549b08871be2-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.768 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.769 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.769 DEBUG:boto:path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.769 DEBUG:boto:auth_path=/wafmoa-56/e?versionId=null 2023-10-12T00:52:19.770 DEBUG:boto:Method: GET 2023-10-12T00:52:19.770 DEBUG:boto:Path: /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.771 DEBUG:boto:Data: 2023-10-12T00:52:19.771 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.772 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.773 DEBUG:boto:Params: {} 2023-10-12T00:52:19.773 DEBUG:boto:Token: None 2023-10-12T00:52:19.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e?versionId=null 2023-10-12T00:52:19.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM= 2023-10-12T00:52:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rLH3L2tHhf1DtHrbD8OKqSezUyM=', 'Content-Length': '0'} 2023-10-12T00:52:19.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b8ee3e176149f8-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.777 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.778 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.778 DEBUG:boto:path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.779 DEBUG:boto:auth_path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.779 DEBUG:boto:Method: GET 2023-10-12T00:52:19.779 DEBUG:boto:Path: /wafmoa-56/?prefix=e 2023-10-12T00:52:19.780 DEBUG:boto:Data: 2023-10-12T00:52:19.780 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.781 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.782 DEBUG:boto:Params: {} 2023-10-12T00:52:19.782 DEBUG:boto:Token: None 2023-10-12T00:52:19.783 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009287e4a5e54b6bd4-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.826 DEBUG:boto:b'wafmoa-56e1000falsee2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.827 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.828 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.828 DEBUG:boto:path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.829 DEBUG:boto:auth_path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.829 DEBUG:boto:Method: GET 2023-10-12T00:52:19.830 DEBUG:boto:Path: /wafmoa-56/?prefix=e 2023-10-12T00:52:19.830 DEBUG:boto:Data: 2023-10-12T00:52:19.830 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.831 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.832 DEBUG:boto:Params: {} 2023-10-12T00:52:19.832 DEBUG:boto:Token: None 2023-10-12T00:52:19.833 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac375ca29c6aac4f-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.837 DEBUG:boto:b'wafmoa-56e1000falsee2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.837 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.838 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.838 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.839 DEBUG:boto:Path: /wafmoa-56/e 2023-10-12T00:52:19.839 DEBUG:boto:Data: 2023-10-12T00:52:19.840 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.840 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.841 DEBUG:boto:Params: {} 2023-10-12T00:52:19.841 DEBUG:boto:Token: None 2023-10-12T00:52:19.842 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e 2023-10-12T00:52:19.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/83UT7VJ2f0Sw1bRlRyZAxUfu0= 2023-10-12T00:52:19.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/83UT7VJ2f0Sw1bRlRyZAxUfu0=', 'Content-Length': '0'} 2023-10-12T00:52:19.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb62cd8a39c016ef-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.845 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.846 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.846 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.847 DEBUG:boto:Path: /wafmoa-56/e 2023-10-12T00:52:19.847 DEBUG:boto:Data: 2023-10-12T00:52:19.848 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.848 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.849 DEBUG:boto:Params: {} 2023-10-12T00:52:19.849 DEBUG:boto:Token: None 2023-10-12T00:52:19.850 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e 2023-10-12T00:52:19.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/83UT7VJ2f0Sw1bRlRyZAxUfu0= 2023-10-12T00:52:19.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/83UT7VJ2f0Sw1bRlRyZAxUfu0=', 'Content-Length': '0'} 2023-10-12T00:52:19.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf8b09fce47bb39-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:52:19.854 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.854 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.855 DEBUG:boto:Method: GET 2023-10-12T00:52:19.855 DEBUG:boto:Path: /wafmoa-56/e 2023-10-12T00:52:19.856 DEBUG:boto:Data: 2023-10-12T00:52:19.856 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.857 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.857 DEBUG:boto:Params: {} 2023-10-12T00:52:19.858 DEBUG:boto:Token: None 2023-10-12T00:52:19.858 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e 2023-10-12T00:52:19.859 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4CbFmTUjeOdOmE69UwWdMByaCrg= 2023-10-12T00:52:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4CbFmTUjeOdOmE69UwWdMByaCrg=', 'Content-Length': '0'} 2023-10-12T00:52:19.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e2e17ee4fca3e6-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.862 DEBUG:boto:path=/wafmoa-56/e 2023-10-12T00:52:19.863 DEBUG:boto:auth_path=/wafmoa-56/e 2023-10-12T00:52:19.863 DEBUG:boto:Method: GET 2023-10-12T00:52:19.863 DEBUG:boto:Path: /wafmoa-56/e 2023-10-12T00:52:19.864 DEBUG:boto:Data: 2023-10-12T00:52:19.864 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.865 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.866 DEBUG:boto:Params: {} 2023-10-12T00:52:19.866 DEBUG:boto:Token: None 2023-10-12T00:52:19.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/e 2023-10-12T00:52:19.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4CbFmTUjeOdOmE69UwWdMByaCrg= 2023-10-12T00:52:19.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4CbFmTUjeOdOmE69UwWdMByaCrg=', 'Content-Length': '0'} 2023-10-12T00:52:19.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285e88678b21111f-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.870 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.871 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.871 DEBUG:boto:path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.872 DEBUG:boto:auth_path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.872 DEBUG:boto:Method: GET 2023-10-12T00:52:19.872 DEBUG:boto:Path: /wafmoa-56/?prefix=e 2023-10-12T00:52:19.873 DEBUG:boto:Data: 2023-10-12T00:52:19.873 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.874 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.875 DEBUG:boto:Params: {} 2023-10-12T00:52:19.875 DEBUG:boto:Token: None 2023-10-12T00:52:19.876 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004151ca07bc078643-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.880 DEBUG:boto:b'wafmoa-56e1000falsee2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.880 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.881 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.881 DEBUG:boto:path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.882 DEBUG:boto:auth_path=/wafmoa-56/?prefix=e 2023-10-12T00:52:19.882 DEBUG:boto:Method: GET 2023-10-12T00:52:19.882 DEBUG:boto:Path: /wafmoa-56/?prefix=e 2023-10-12T00:52:19.883 DEBUG:boto:Data: 2023-10-12T00:52:19.883 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.884 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.885 DEBUG:boto:Params: {} 2023-10-12T00:52:19.885 DEBUG:boto:Token: None 2023-10-12T00:52:19.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.886 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:19.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8207c7f3956ae7-0065274343-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.889 DEBUG:boto:b'wafmoa-56e1000falsee2023-10-12T00:49:23.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:19.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:52:19.890 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.891 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.891 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.892 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.892 DEBUG:boto:Method: GET 2023-10-12T00:52:19.893 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.893 DEBUG:boto:Data: 2023-10-12T00:52:19.893 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.894 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.895 DEBUG:boto:Params: {} 2023-10-12T00:52:19.895 DEBUG:boto:Token: None 2023-10-12T00:52:19.896 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8= 2023-10-12T00:52:19.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8=', 'Content-Length': '0'} 2023-10-12T00:52:19.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cee20c8f79e1d7d-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.900 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.900 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.901 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.901 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.902 DEBUG:boto:Method: GET 2023-10-12T00:52:19.902 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.902 DEBUG:boto:Data: 2023-10-12T00:52:19.903 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.904 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.904 DEBUG:boto:Params: {} 2023-10-12T00:52:19.905 DEBUG:boto:Token: None 2023-10-12T00:52:19.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8= 2023-10-12T00:52:19.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8=', 'Content-Length': '0'} 2023-10-12T00:52:19.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cbe8d7091777ddd-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.909 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.909 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.910 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.910 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.911 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.911 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.911 DEBUG:boto:Data: 2023-10-12T00:52:19.912 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.913 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.913 DEBUG:boto:Params: {} 2023-10-12T00:52:19.913 DEBUG:boto:Token: None 2023-10-12T00:52:19.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rydi3CapwDEX6S71n/MEyWT5PhI= 2023-10-12T00:52:19.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rydi3CapwDEX6S71n/MEyWT5PhI=', 'Content-Length': '0'} 2023-10-12T00:52:19.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c33b6b11d942ad-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.917 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.918 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.918 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.919 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.919 DEBUG:boto:Method: HEAD 2023-10-12T00:52:19.919 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.920 DEBUG:boto:Data: 2023-10-12T00:52:19.920 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.921 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.922 DEBUG:boto:Params: {} 2023-10-12T00:52:19.922 DEBUG:boto:Token: None 2023-10-12T00:52:19.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rydi3CapwDEX6S71n/MEyWT5PhI= 2023-10-12T00:52:19.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rydi3CapwDEX6S71n/MEyWT5PhI=', 'Content-Length': '0'} 2023-10-12T00:52:19.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b24d849f35629f2-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:52:19.926 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.927 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.927 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.928 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.928 DEBUG:boto:Method: GET 2023-10-12T00:52:19.928 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.929 DEBUG:boto:Data: 2023-10-12T00:52:19.929 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.930 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.931 DEBUG:boto:Params: {} 2023-10-12T00:52:19.931 DEBUG:boto:Token: None 2023-10-12T00:52:19.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8= 2023-10-12T00:52:19.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8=', 'Content-Length': '0'} 2023-10-12T00:52:19.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafb9cff0e154fd1-0065274343-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.982 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:19.983 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:19.983 DEBUG:boto:path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.984 DEBUG:boto:auth_path=/wafmoa-56/f?versionId=null 2023-10-12T00:52:19.984 DEBUG:boto:Method: GET 2023-10-12T00:52:19.985 DEBUG:boto:Path: /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.985 DEBUG:boto:Data: 2023-10-12T00:52:19.986 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:19.987 DEBUG:boto:Port: 8000 2023-10-12T00:52:19.987 DEBUG:boto:Params: {} 2023-10-12T00:52:19.987 DEBUG:boto:Token: None 2023-10-12T00:52:19.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/f?versionId=null 2023-10-12T00:52:19.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8= 2023-10-12T00:52:19.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XmerB+szZDyqZ2ft4WCEe6ia7q8=', 'Content-Length': '0'} 2023-10-12T00:52:19.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff0af2fc53885e0-0065274343-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:19.992 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:19.992 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:19.993 DEBUG:boto:path=/wafmoa-56/?prefix=f 2023-10-12T00:52:19.993 DEBUG:boto:auth_path=/wafmoa-56/?prefix=f 2023-10-12T00:52:19.994 DEBUG:boto:Method: GET 2023-10-12T00:52:19.994 DEBUG:boto:Path: /wafmoa-56/?prefix=f 2023-10-12T00:52:19.995 DEBUG:boto:Data: 2023-10-12T00:52:19.995 DEBUG:boto:Headers: {} 2023-10-12T00:52:19.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:19.996 DEBUG:boto:Port: 8001 2023-10-12T00:52:19.996 DEBUG:boto:Params: {} 2023-10-12T00:52:19.997 DEBUG:boto:Token: None 2023-10-12T00:52:19.997 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:19 GMT /wafmoa-56/ 2023-10-12T00:52:19.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo= 2023-10-12T00:52:19.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:19 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tSoeSsACYYYC65qsQg+yIuRuUDo=', 'Content-Length': '0'} 2023-10-12T00:52:20.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0032ee881ef4a70-0065274343-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.001 DEBUG:boto:b'wafmoa-56f1000falsef2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.002 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.002 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.003 DEBUG:boto:path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.003 DEBUG:boto:auth_path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.004 DEBUG:boto:Method: GET 2023-10-12T00:52:20.004 DEBUG:boto:Path: /wafmoa-56/?prefix=f 2023-10-12T00:52:20.005 DEBUG:boto:Data: 2023-10-12T00:52:20.005 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.006 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.006 DEBUG:boto:Params: {} 2023-10-12T00:52:20.007 DEBUG:boto:Token: None 2023-10-12T00:52:20.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de168ea93d896dca-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.011 DEBUG:boto:b'wafmoa-56f1000falsef2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.012 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:20.012 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:20.013 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.013 DEBUG:boto:Path: /wafmoa-56/f 2023-10-12T00:52:20.013 DEBUG:boto:Data: 2023-10-12T00:52:20.014 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.015 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.015 DEBUG:boto:Params: {} 2023-10-12T00:52:20.016 DEBUG:boto:Token: None 2023-10-12T00:52:20.016 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/f 2023-10-12T00:52:20.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3ZOddZFgZqbGISojzMtNiX+xttI= 2023-10-12T00:52:20.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3ZOddZFgZqbGISojzMtNiX+xttI=', 'Content-Length': '0'} 2023-10-12T00:52:20.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049abdd883e0bf379-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.020 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:20.020 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:20.021 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.021 DEBUG:boto:Path: /wafmoa-56/f 2023-10-12T00:52:20.021 DEBUG:boto:Data: 2023-10-12T00:52:20.022 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.023 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.023 DEBUG:boto:Params: {} 2023-10-12T00:52:20.024 DEBUG:boto:Token: None 2023-10-12T00:52:20.024 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/f 2023-10-12T00:52:20.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3ZOddZFgZqbGISojzMtNiX+xttI= 2023-10-12T00:52:20.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3ZOddZFgZqbGISojzMtNiX+xttI=', 'Content-Length': '0'} 2023-10-12T00:52:20.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144173ce592e7b9e-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:52:20.028 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:20.028 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:20.029 DEBUG:boto:Method: GET 2023-10-12T00:52:20.029 DEBUG:boto:Path: /wafmoa-56/f 2023-10-12T00:52:20.030 DEBUG:boto:Data: 2023-10-12T00:52:20.030 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.031 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.031 DEBUG:boto:Params: {} 2023-10-12T00:52:20.032 DEBUG:boto:Token: None 2023-10-12T00:52:20.032 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/f 2023-10-12T00:52:20.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZXJEhm5XrnjJnq4utZx6fZayCYg= 2023-10-12T00:52:20.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZXJEhm5XrnjJnq4utZx6fZayCYg=', 'Content-Length': '0'} 2023-10-12T00:52:20.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec0fed266302403-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.036 DEBUG:boto:path=/wafmoa-56/f 2023-10-12T00:52:20.037 DEBUG:boto:auth_path=/wafmoa-56/f 2023-10-12T00:52:20.037 DEBUG:boto:Method: GET 2023-10-12T00:52:20.038 DEBUG:boto:Path: /wafmoa-56/f 2023-10-12T00:52:20.038 DEBUG:boto:Data: 2023-10-12T00:52:20.038 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.039 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.040 DEBUG:boto:Params: {} 2023-10-12T00:52:20.040 DEBUG:boto:Token: None 2023-10-12T00:52:20.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/f 2023-10-12T00:52:20.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZXJEhm5XrnjJnq4utZx6fZayCYg= 2023-10-12T00:52:20.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZXJEhm5XrnjJnq4utZx6fZayCYg=', 'Content-Length': '0'} 2023-10-12T00:52:20.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000834d01a45c1bb7a0-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.090 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.091 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.092 DEBUG:boto:path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.092 DEBUG:boto:auth_path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.092 DEBUG:boto:Method: GET 2023-10-12T00:52:20.093 DEBUG:boto:Path: /wafmoa-56/?prefix=f 2023-10-12T00:52:20.093 DEBUG:boto:Data: 2023-10-12T00:52:20.094 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.095 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.095 DEBUG:boto:Params: {} 2023-10-12T00:52:20.095 DEBUG:boto:Token: None 2023-10-12T00:52:20.096 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000613d6d6e8ac6708-0065274344-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.100 DEBUG:boto:b'wafmoa-56f1000falsef2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.101 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.101 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.102 DEBUG:boto:path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.102 DEBUG:boto:auth_path=/wafmoa-56/?prefix=f 2023-10-12T00:52:20.103 DEBUG:boto:Method: GET 2023-10-12T00:52:20.103 DEBUG:boto:Path: /wafmoa-56/?prefix=f 2023-10-12T00:52:20.103 DEBUG:boto:Data: 2023-10-12T00:52:20.104 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.105 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.105 DEBUG:boto:Params: {} 2023-10-12T00:52:20.105 DEBUG:boto:Token: None 2023-10-12T00:52:20.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283ed5bdc98a6dda-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.110 DEBUG:boto:b'wafmoa-56f1000falsef2023-10-12T00:49:23.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:52:20.111 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.111 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.112 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.112 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.112 DEBUG:boto:Method: GET 2023-10-12T00:52:20.113 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.113 DEBUG:boto:Data: 2023-10-12T00:52:20.114 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.115 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.115 DEBUG:boto:Params: {} 2023-10-12T00:52:20.116 DEBUG:boto:Token: None 2023-10-12T00:52:20.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU= 2023-10-12T00:52:20.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU=', 'Content-Length': '0'} 2023-10-12T00:52:20.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702a7ed9ab33b2aa-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.162 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.163 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.163 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.164 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.164 DEBUG:boto:Method: GET 2023-10-12T00:52:20.165 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.165 DEBUG:boto:Data: 2023-10-12T00:52:20.166 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.167 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.167 DEBUG:boto:Params: {} 2023-10-12T00:52:20.167 DEBUG:boto:Token: None 2023-10-12T00:52:20.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU= 2023-10-12T00:52:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU=', 'Content-Length': '0'} 2023-10-12T00:52:20.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050cfab47a0ae72a7-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.172 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.173 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.173 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.173 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.174 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.174 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.175 DEBUG:boto:Data: 2023-10-12T00:52:20.175 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.176 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.176 DEBUG:boto:Params: {} 2023-10-12T00:52:20.177 DEBUG:boto:Token: None 2023-10-12T00:52:20.177 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ekb8EqQpRIolt12r8nodA3cu6PU= 2023-10-12T00:52:20.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ekb8EqQpRIolt12r8nodA3cu6PU=', 'Content-Length': '0'} 2023-10-12T00:52:20.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ac0c9d22320106-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.180 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.181 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.181 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.181 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.182 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.182 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.183 DEBUG:boto:Data: 2023-10-12T00:52:20.183 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.184 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.184 DEBUG:boto:Params: {} 2023-10-12T00:52:20.185 DEBUG:boto:Token: None 2023-10-12T00:52:20.185 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.186 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ekb8EqQpRIolt12r8nodA3cu6PU= 2023-10-12T00:52:20.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ekb8EqQpRIolt12r8nodA3cu6PU=', 'Content-Length': '0'} 2023-10-12T00:52:20.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf41869cd0d611c0-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:52:20.189 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.190 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.190 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.191 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.191 DEBUG:boto:Method: GET 2023-10-12T00:52:20.191 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.192 DEBUG:boto:Data: 2023-10-12T00:52:20.192 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.193 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.194 DEBUG:boto:Params: {} 2023-10-12T00:52:20.194 DEBUG:boto:Token: None 2023-10-12T00:52:20.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU= 2023-10-12T00:52:20.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU=', 'Content-Length': '0'} 2023-10-12T00:52:20.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d8c11a7d786c494-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.199 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.199 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.199 DEBUG:boto:path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.200 DEBUG:boto:auth_path=/wafmoa-56/g?versionId=null 2023-10-12T00:52:20.200 DEBUG:boto:Method: GET 2023-10-12T00:52:20.201 DEBUG:boto:Path: /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.201 DEBUG:boto:Data: 2023-10-12T00:52:20.201 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.202 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.203 DEBUG:boto:Params: {} 2023-10-12T00:52:20.203 DEBUG:boto:Token: None 2023-10-12T00:52:20.204 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g?versionId=null 2023-10-12T00:52:20.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU= 2023-10-12T00:52:20.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zTO1MyrBkC/krY1zwKhl7+b99uU=', 'Content-Length': '0'} 2023-10-12T00:52:20.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002afd6dbc9c69dd61-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.207 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.208 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.208 DEBUG:boto:path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.209 DEBUG:boto:auth_path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.209 DEBUG:boto:Method: GET 2023-10-12T00:52:20.210 DEBUG:boto:Path: /wafmoa-56/?prefix=g 2023-10-12T00:52:20.210 DEBUG:boto:Data: 2023-10-12T00:52:20.210 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.211 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.212 DEBUG:boto:Params: {} 2023-10-12T00:52:20.212 DEBUG:boto:Token: None 2023-10-12T00:52:20.212 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b8a2e84ebd7d97-0065274344-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.217 DEBUG:boto:b'wafmoa-56g1000falseg2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.217 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.218 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.218 DEBUG:boto:path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.219 DEBUG:boto:auth_path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.219 DEBUG:boto:Method: GET 2023-10-12T00:52:20.220 DEBUG:boto:Path: /wafmoa-56/?prefix=g 2023-10-12T00:52:20.220 DEBUG:boto:Data: 2023-10-12T00:52:20.220 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.221 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.222 DEBUG:boto:Params: {} 2023-10-12T00:52:20.222 DEBUG:boto:Token: None 2023-10-12T00:52:20.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec236f6264729ad-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.266 DEBUG:boto:b'wafmoa-56g1000falseg2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.267 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.268 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.268 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.269 DEBUG:boto:Path: /wafmoa-56/g 2023-10-12T00:52:20.269 DEBUG:boto:Data: 2023-10-12T00:52:20.269 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.270 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.271 DEBUG:boto:Params: {} 2023-10-12T00:52:20.271 DEBUG:boto:Token: None 2023-10-12T00:52:20.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g 2023-10-12T00:52:20.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wWaycDzPj/pE7BaBJ57tBix+NTk= 2023-10-12T00:52:20.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wWaycDzPj/pE7BaBJ57tBix+NTk=', 'Content-Length': '0'} 2023-10-12T00:52:20.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b484949207ace1-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.275 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.276 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.276 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.276 DEBUG:boto:Path: /wafmoa-56/g 2023-10-12T00:52:20.277 DEBUG:boto:Data: 2023-10-12T00:52:20.277 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.278 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.278 DEBUG:boto:Params: {} 2023-10-12T00:52:20.279 DEBUG:boto:Token: None 2023-10-12T00:52:20.279 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g 2023-10-12T00:52:20.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wWaycDzPj/pE7BaBJ57tBix+NTk= 2023-10-12T00:52:20.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wWaycDzPj/pE7BaBJ57tBix+NTk=', 'Content-Length': '0'} 2023-10-12T00:52:20.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3bfcf939cad40b3-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:52:20.283 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.283 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.284 DEBUG:boto:Method: GET 2023-10-12T00:52:20.284 DEBUG:boto:Path: /wafmoa-56/g 2023-10-12T00:52:20.285 DEBUG:boto:Data: 2023-10-12T00:52:20.285 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.286 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.286 DEBUG:boto:Params: {} 2023-10-12T00:52:20.287 DEBUG:boto:Token: None 2023-10-12T00:52:20.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g 2023-10-12T00:52:20.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mN6XHVKBbchyEAzJkkTxRUp7j3Q= 2023-10-12T00:52:20.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mN6XHVKBbchyEAzJkkTxRUp7j3Q=', 'Content-Length': '0'} 2023-10-12T00:52:20.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012bfb23c839ecfde-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.291 DEBUG:boto:path=/wafmoa-56/g 2023-10-12T00:52:20.291 DEBUG:boto:auth_path=/wafmoa-56/g 2023-10-12T00:52:20.292 DEBUG:boto:Method: GET 2023-10-12T00:52:20.292 DEBUG:boto:Path: /wafmoa-56/g 2023-10-12T00:52:20.293 DEBUG:boto:Data: 2023-10-12T00:52:20.293 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.294 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.294 DEBUG:boto:Params: {} 2023-10-12T00:52:20.295 DEBUG:boto:Token: None 2023-10-12T00:52:20.295 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/g 2023-10-12T00:52:20.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mN6XHVKBbchyEAzJkkTxRUp7j3Q= 2023-10-12T00:52:20.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mN6XHVKBbchyEAzJkkTxRUp7j3Q=', 'Content-Length': '0'} 2023-10-12T00:52:20.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e8649914aa290e-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.299 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.299 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.300 DEBUG:boto:path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.300 DEBUG:boto:auth_path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.300 DEBUG:boto:Method: GET 2023-10-12T00:52:20.301 DEBUG:boto:Path: /wafmoa-56/?prefix=g 2023-10-12T00:52:20.301 DEBUG:boto:Data: 2023-10-12T00:52:20.302 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.303 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.303 DEBUG:boto:Params: {} 2023-10-12T00:52:20.304 DEBUG:boto:Token: None 2023-10-12T00:52:20.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.304 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d6cc4cfaa53cf7-0065274344-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.350 DEBUG:boto:b'wafmoa-56g1000falseg2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.351 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.352 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.352 DEBUG:boto:path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.353 DEBUG:boto:auth_path=/wafmoa-56/?prefix=g 2023-10-12T00:52:20.353 DEBUG:boto:Method: GET 2023-10-12T00:52:20.353 DEBUG:boto:Path: /wafmoa-56/?prefix=g 2023-10-12T00:52:20.354 DEBUG:boto:Data: 2023-10-12T00:52:20.354 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.355 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.356 DEBUG:boto:Params: {} 2023-10-12T00:52:20.356 DEBUG:boto:Token: None 2023-10-12T00:52:20.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3e2a493b9beb89c-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.361 DEBUG:boto:b'wafmoa-56g1000falseg2023-10-12T00:49:23.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:52:20.362 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.363 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.364 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.364 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.365 DEBUG:boto:Method: GET 2023-10-12T00:52:20.365 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.366 DEBUG:boto:Data: 2023-10-12T00:52:20.366 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.367 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.367 DEBUG:boto:Params: {} 2023-10-12T00:52:20.368 DEBUG:boto:Token: None 2023-10-12T00:52:20.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs= 2023-10-12T00:52:20.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs=', 'Content-Length': '0'} 2023-10-12T00:52:20.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9db813c4c98c0f1-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.372 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.373 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.373 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.374 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.374 DEBUG:boto:Method: GET 2023-10-12T00:52:20.375 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.375 DEBUG:boto:Data: 2023-10-12T00:52:20.375 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.376 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.377 DEBUG:boto:Params: {} 2023-10-12T00:52:20.377 DEBUG:boto:Token: None 2023-10-12T00:52:20.378 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs= 2023-10-12T00:52:20.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs=', 'Content-Length': '0'} 2023-10-12T00:52:20.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c652467feddf3cea-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.382 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.382 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.383 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.383 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.383 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.384 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.384 DEBUG:boto:Data: 2023-10-12T00:52:20.385 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.386 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.386 DEBUG:boto:Params: {} 2023-10-12T00:52:20.386 DEBUG:boto:Token: None 2023-10-12T00:52:20.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JmLUWUJgG/6fGfeoijl1UIDSVsc= 2023-10-12T00:52:20.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JmLUWUJgG/6fGfeoijl1UIDSVsc=', 'Content-Length': '0'} 2023-10-12T00:52:20.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78f00d8dcc86543-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.391 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.392 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.392 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.392 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.393 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.393 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.394 DEBUG:boto:Data: 2023-10-12T00:52:20.394 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.395 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.395 DEBUG:boto:Params: {} 2023-10-12T00:52:20.396 DEBUG:boto:Token: None 2023-10-12T00:52:20.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JmLUWUJgG/6fGfeoijl1UIDSVsc= 2023-10-12T00:52:20.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JmLUWUJgG/6fGfeoijl1UIDSVsc=', 'Content-Length': '0'} 2023-10-12T00:52:20.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae18136ca4e8e2a0-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:52:20.400 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.401 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.402 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.402 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.403 DEBUG:boto:Method: GET 2023-10-12T00:52:20.403 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.403 DEBUG:boto:Data: 2023-10-12T00:52:20.404 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.405 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.406 DEBUG:boto:Params: {} 2023-10-12T00:52:20.406 DEBUG:boto:Token: None 2023-10-12T00:52:20.407 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs= 2023-10-12T00:52:20.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs=', 'Content-Length': '0'} 2023-10-12T00:52:20.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b86b0f1897aeb0ef-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.411 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.412 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.412 DEBUG:boto:path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.413 DEBUG:boto:auth_path=/wafmoa-56/h?versionId=null 2023-10-12T00:52:20.413 DEBUG:boto:Method: GET 2023-10-12T00:52:20.413 DEBUG:boto:Path: /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.414 DEBUG:boto:Data: 2023-10-12T00:52:20.414 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.415 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.416 DEBUG:boto:Params: {} 2023-10-12T00:52:20.416 DEBUG:boto:Token: None 2023-10-12T00:52:20.417 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h?versionId=null 2023-10-12T00:52:20.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs= 2023-10-12T00:52:20.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4WqOCgJzPa0POnmFLLaq7UNybrs=', 'Content-Length': '0'} 2023-10-12T00:52:20.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703efa95ac70ab17-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.421 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.421 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.422 DEBUG:boto:path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.422 DEBUG:boto:auth_path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.423 DEBUG:boto:Method: GET 2023-10-12T00:52:20.423 DEBUG:boto:Path: /wafmoa-56/?prefix=h 2023-10-12T00:52:20.423 DEBUG:boto:Data: 2023-10-12T00:52:20.424 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.425 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.425 DEBUG:boto:Params: {} 2023-10-12T00:52:20.426 DEBUG:boto:Token: None 2023-10-12T00:52:20.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.427 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611403b24daed345-0065274344-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.431 DEBUG:boto:b'wafmoa-56h1000falseh2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.431 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.432 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.432 DEBUG:boto:path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.433 DEBUG:boto:auth_path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.433 DEBUG:boto:Method: GET 2023-10-12T00:52:20.434 DEBUG:boto:Path: /wafmoa-56/?prefix=h 2023-10-12T00:52:20.434 DEBUG:boto:Data: 2023-10-12T00:52:20.435 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.435 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.436 DEBUG:boto:Params: {} 2023-10-12T00:52:20.436 DEBUG:boto:Token: None 2023-10-12T00:52:20.437 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d170af273450a26-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.441 DEBUG:boto:b'wafmoa-56h1000falseh2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.441 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.442 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.442 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.443 DEBUG:boto:Path: /wafmoa-56/h 2023-10-12T00:52:20.443 DEBUG:boto:Data: 2023-10-12T00:52:20.443 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.444 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.445 DEBUG:boto:Params: {} 2023-10-12T00:52:20.445 DEBUG:boto:Token: None 2023-10-12T00:52:20.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h 2023-10-12T00:52:20.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nhyf3rqKH+Mz9tzgnIorsgtEBWc= 2023-10-12T00:52:20.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nhyf3rqKH+Mz9tzgnIorsgtEBWc=', 'Content-Length': '0'} 2023-10-12T00:52:20.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0651858d8b1a8ea-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.449 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.449 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.450 DEBUG:boto:Method: HEAD 2023-10-12T00:52:20.450 DEBUG:boto:Path: /wafmoa-56/h 2023-10-12T00:52:20.451 DEBUG:boto:Data: 2023-10-12T00:52:20.451 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.452 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.452 DEBUG:boto:Params: {} 2023-10-12T00:52:20.453 DEBUG:boto:Token: None 2023-10-12T00:52:20.453 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h 2023-10-12T00:52:20.454 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nhyf3rqKH+Mz9tzgnIorsgtEBWc= 2023-10-12T00:52:20.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nhyf3rqKH+Mz9tzgnIorsgtEBWc=', 'Content-Length': '0'} 2023-10-12T00:52:20.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628ee4b01c55d836-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:52:20.457 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.458 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.458 DEBUG:boto:Method: GET 2023-10-12T00:52:20.459 DEBUG:boto:Path: /wafmoa-56/h 2023-10-12T00:52:20.459 DEBUG:boto:Data: 2023-10-12T00:52:20.460 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.460 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.461 DEBUG:boto:Params: {} 2023-10-12T00:52:20.461 DEBUG:boto:Token: None 2023-10-12T00:52:20.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h 2023-10-12T00:52:20.462 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LUZUNIQuw0+CPYXCFTNgbso9C20= 2023-10-12T00:52:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LUZUNIQuw0+CPYXCFTNgbso9C20=', 'Content-Length': '0'} 2023-10-12T00:52:20.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-56:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6997cd33372c78-0065274344-1689-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.465 DEBUG:boto:path=/wafmoa-56/h 2023-10-12T00:52:20.466 DEBUG:boto:auth_path=/wafmoa-56/h 2023-10-12T00:52:20.466 DEBUG:boto:Method: GET 2023-10-12T00:52:20.467 DEBUG:boto:Path: /wafmoa-56/h 2023-10-12T00:52:20.467 DEBUG:boto:Data: 2023-10-12T00:52:20.468 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.468 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.469 DEBUG:boto:Params: {} 2023-10-12T00:52:20.469 DEBUG:boto:Token: None 2023-10-12T00:52:20.470 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/h 2023-10-12T00:52:20.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LUZUNIQuw0+CPYXCFTNgbso9C20= 2023-10-12T00:52:20.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LUZUNIQuw0+CPYXCFTNgbso9C20=', 'Content-Length': '0'} 2023-10-12T00:52:20.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:49:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5427f2ccc5045a8-0065274344-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.474 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.474 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.475 DEBUG:boto:path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.475 DEBUG:boto:auth_path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.476 DEBUG:boto:Method: GET 2023-10-12T00:52:20.476 DEBUG:boto:Path: /wafmoa-56/?prefix=h 2023-10-12T00:52:20.477 DEBUG:boto:Data: 2023-10-12T00:52:20.477 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:20.478 DEBUG:boto:Port: 8001 2023-10-12T00:52:20.478 DEBUG:boto:Params: {} 2023-10-12T00:52:20.479 DEBUG:boto:Token: None 2023-10-12T00:52:20.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.480 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40a640ddc36dbe9-0065274344-1689-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.483 DEBUG:boto:b'wafmoa-56h1000falseh2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.484 DEBUG:boto:path=/wafmoa-56/ 2023-10-12T00:52:20.484 DEBUG:boto:auth_path=/wafmoa-56/ 2023-10-12T00:52:20.485 DEBUG:boto:path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.485 DEBUG:boto:auth_path=/wafmoa-56/?prefix=h 2023-10-12T00:52:20.486 DEBUG:boto:Method: GET 2023-10-12T00:52:20.486 DEBUG:boto:Path: /wafmoa-56/?prefix=h 2023-10-12T00:52:20.487 DEBUG:boto:Data: 2023-10-12T00:52:20.487 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.488 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.488 DEBUG:boto:Params: {} 2023-10-12T00:52:20.489 DEBUG:boto:Token: None 2023-10-12T00:52:20.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-56/ 2023-10-12T00:52:20.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ= 2023-10-12T00:52:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b+4LI9CqZ+7+ueokBuj8EKkkNyQ=', 'Content-Length': '0'} 2023-10-12T00:52:20.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aefadac572790e43-0065274344-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.493 DEBUG:boto:b'wafmoa-56h1000falseh2023-10-12T00:49:23.423Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:52:20.494 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-56 zones={a2, a1} 2023-10-12T00:52:20.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-10-12T00:52:20.495 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:20.496 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:20.496 DEBUG:boto:Method: PUT 2023-10-12T00:52:20.497 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:20.497 DEBUG:boto:Data: 2023-10-12T00:52:20.498 DEBUG:boto:Headers: {} 2023-10-12T00:52:20.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:20.498 DEBUG:boto:Port: 8000 2023-10-12T00:52:20.499 DEBUG:boto:Params: {} 2023-10-12T00:52:20.499 DEBUG:boto:Token: None 2023-10-12T00:52:20.500 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:52:20 GMT /wafmoa-57/ 2023-10-12T00:52:20.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LP2hjojwaHA6uwEVbadWSydVPR0= 2023-10-12T00:52:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LP2hjojwaHA6uwEVbadWSydVPR0=', 'Content-Length': '0'} 2023-10-12T00:52:20.518 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c7f26206d4457d5-0065274344-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:20.518 DEBUG:rgw_multi.tests:created bucket=wafmoa-57 2023-10-12T00:52:20.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:20.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:20.520 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:52:20.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:20.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:20.604 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:52:20.617 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:52:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:52:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:52:20.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:52:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:52:20.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071940.516981_565.1", 2023-10-12T00:52:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:52:20.516981Z" 2023-10-12T00:52:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:52:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:52:20.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.621 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:52:20.622 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071940.516981_565.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:52:20.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:52:20.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:20.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:20.624 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:52:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:20.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:20.703+0000 7f3ce3dc3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:52:20.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:20.703+0000 7f3ce3dc3780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:52:20.716 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:52:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:52:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:52:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:52:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:52:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:52:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:52:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:52:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:52:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:52:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:52:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:52:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:52:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:52:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:52:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071940.510037_562.1", 2023-10-12T00:52:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:52:20.510037Z", 2023-10-12T00:52:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:52:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:52:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:52:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:52:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:52:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:52:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:52:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:52:20Z" 2023-10-12T00:52:20.731 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:52:20.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071940.510037_562.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:52:20.510037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:52:20Z" } 2023-10-12T00:52:20.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071940.510037_562.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:52:20.510037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:52:20.733 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071940.516981_565.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:52:20.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071940.510037_562.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:52:20.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697071940.516981_565.1 target=1_1697071940.510037_562.1 2023-10-12T00:52:25.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:25.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:52:25.736 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:52:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:25.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:25.824+0000 7ff62e4fb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:52:25.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:25.825+0000 7ff62e4fb780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:52:25.839 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:52:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:52:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:52:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:52:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:52:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:52:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:52:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:52:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:52:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:52:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:52:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:52:25.846 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:25.846 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:25.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:52:25.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:25.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:25.847 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:25.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:52:25.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:25.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:25.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:25.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:52:25.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:25.849 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:25.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071940.516981_565.1", 2023-10-12T00:52:25.850 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:25.850 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:25.850 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:25.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:52:20.516981Z", 2023-10-12T00:52:25.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:25.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:25.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:52:25.852 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:52:25.853 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:52:25.853 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:52:25.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:52:25.854 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:52:25.882 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:52:25.883 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:52:25.883 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:52:25.883 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:52:25.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:25.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:52:25.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:52:25.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.885 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:52:25.885 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:52:25.885 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:52:25.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:52:25.886 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:52:25Z" 2023-10-12T00:52:25.887 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:52:25.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071940.516981_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:52:20.516981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:52:25Z" } 2023-10-12T00:52:25.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071940.516981_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:52:20.516981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:52:25.890 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071940.516981_565.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:52:25.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071940.516981_565.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:52:25.892 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:52:25.893 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T00:52:25.894 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:52:25.895 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:52:25.893+0000 7fbf24213700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 123030) UID: 0 2023-10-12T00:52:25.896 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:52:25.893+0000 7fbf2847c7c0 -1 shutting down 2023-10-12T00:52:28.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T00:52:31.995 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T00:52:31.996 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:31.997 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:31.997 DEBUG:boto:Method: HEAD 2023-10-12T00:52:31.997 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:31.998 DEBUG:boto:Data: 2023-10-12T00:52:31.998 DEBUG:boto:Headers: {} 2023-10-12T00:52:31.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:31.999 DEBUG:boto:Port: 8000 2023-10-12T00:52:31.999 DEBUG:boto:Params: {} 2023-10-12T00:52:32.000 DEBUG:boto:Token: None 2023-10-12T00:52:32.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc12f481309c121a-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.005 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:52:32.005 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:52:32.006 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.006 DEBUG:boto:Path: /wafmoa-57/a 2023-10-12T00:52:32.007 DEBUG:boto:Data: 2023-10-12T00:52:32.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.008 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.008 DEBUG:boto:Params: {} 2023-10-12T00:52:32.009 DEBUG:boto:Token: None 2023-10-12T00:52:32.009 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/a 2023-10-12T00:52:32.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yNUei06HDV+/5A+PJC8ZmgiPKHA= 2023-10-12T00:52:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yNUei06HDV+/5A+PJC8ZmgiPKHA='} 2023-10-12T00:52:32.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9eeaaf6c07d50d7-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.055 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.056 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.056 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.057 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.057 DEBUG:boto:Data: 2023-10-12T00:52:32.057 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.058 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.059 DEBUG:boto:Params: {} 2023-10-12T00:52:32.059 DEBUG:boto:Token: None 2023-10-12T00:52:32.060 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c8b6e6c99d21ea0-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.064 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:52:32.065 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:52:32.065 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.065 DEBUG:boto:Path: /wafmoa-57/b 2023-10-12T00:52:32.066 DEBUG:boto:Data: 2023-10-12T00:52:32.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.067 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.068 DEBUG:boto:Params: {} 2023-10-12T00:52:32.068 DEBUG:boto:Token: None 2023-10-12T00:52:32.069 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/b 2023-10-12T00:52:32.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:flUZlCFCJ/+ObfBOgOTy+mSEcRg= 2023-10-12T00:52:32.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:flUZlCFCJ/+ObfBOgOTy+mSEcRg='} 2023-10-12T00:52:32.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8071af9b4de3229-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.077 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.077 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.077 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.078 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.078 DEBUG:boto:Data: 2023-10-12T00:52:32.079 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.080 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.080 DEBUG:boto:Params: {} 2023-10-12T00:52:32.081 DEBUG:boto:Token: None 2023-10-12T00:52:32.081 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc67ab2b8292921c-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.086 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:52:32.087 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:52:32.087 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.088 DEBUG:boto:Path: /wafmoa-57/c 2023-10-12T00:52:32.089 DEBUG:boto:Data: 2023-10-12T00:52:32.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.091 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.091 DEBUG:boto:Params: {} 2023-10-12T00:52:32.091 DEBUG:boto:Token: None 2023-10-12T00:52:32.092 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/c 2023-10-12T00:52:32.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/MN5m5zJJjHA/P3DIssNbSv/jPI= 2023-10-12T00:52:32.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/MN5m5zJJjHA/P3DIssNbSv/jPI='} 2023-10-12T00:52:32.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daf9c9839e563d42-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.135 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.136 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.136 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.136 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.137 DEBUG:boto:Data: 2023-10-12T00:52:32.137 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.138 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.139 DEBUG:boto:Params: {} 2023-10-12T00:52:32.139 DEBUG:boto:Token: None 2023-10-12T00:52:32.140 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a57cdb2f3a431f85-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.144 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:52:32.145 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:52:32.145 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.146 DEBUG:boto:Path: /wafmoa-57/d 2023-10-12T00:52:32.146 DEBUG:boto:Data: 2023-10-12T00:52:32.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.147 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.147 DEBUG:boto:Params: {} 2023-10-12T00:52:32.148 DEBUG:boto:Token: None 2023-10-12T00:52:32.148 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/d 2023-10-12T00:52:32.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xrz6SixvkXDTyXg76EVU+nIYD9E= 2023-10-12T00:52:32.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xrz6SixvkXDTyXg76EVU+nIYD9E='} 2023-10-12T00:52:32.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a699cdc0dcc37ec-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'wafmoa-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:52:32.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'wafmoa-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:52:32.157 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket wafmoa-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:52:32.212 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:32.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:32.255 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:52:32.256 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-57 2023-10-12T00:52:32.259 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-10-12T00:52:32.296 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.297 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.297 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.297 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.298 DEBUG:boto:Data: 2023-10-12T00:52:32.298 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.299 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.300 DEBUG:boto:Params: {} 2023-10-12T00:52:32.300 DEBUG:boto:Token: None 2023-10-12T00:52:32.301 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005bd460a038da46d-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.304 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:52:32.305 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:52:32.305 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.306 DEBUG:boto:Path: /wafmoa-57/e 2023-10-12T00:52:32.306 DEBUG:boto:Data: 2023-10-12T00:52:32.307 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.308 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.308 DEBUG:boto:Params: {} 2023-10-12T00:52:32.308 DEBUG:boto:Token: None 2023-10-12T00:52:32.309 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/e 2023-10-12T00:52:32.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DLD9efpuGSY6p2OqPdP4D0FRVFY= 2023-10-12T00:52:32.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DLD9efpuGSY6p2OqPdP4D0FRVFY='} 2023-10-12T00:52:32.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c79d5ec5baffd2d4-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.316 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.316 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.316 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.317 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.317 DEBUG:boto:Data: 2023-10-12T00:52:32.318 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.319 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.319 DEBUG:boto:Params: {} 2023-10-12T00:52:32.320 DEBUG:boto:Token: None 2023-10-12T00:52:32.321 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000746937ef7e0b7961-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.326 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:52:32.326 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:52:32.327 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.327 DEBUG:boto:Path: /wafmoa-57/f 2023-10-12T00:52:32.328 DEBUG:boto:Data: 2023-10-12T00:52:32.328 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.329 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.329 DEBUG:boto:Params: {} 2023-10-12T00:52:32.330 DEBUG:boto:Token: None 2023-10-12T00:52:32.330 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/f 2023-10-12T00:52:32.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AzMfTxKD72Pl4vzq5swQHQ+VLdo= 2023-10-12T00:52:32.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AzMfTxKD72Pl4vzq5swQHQ+VLdo='} 2023-10-12T00:52:32.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adbbda21d8ef3339-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.379 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.380 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.380 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.381 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.381 DEBUG:boto:Data: 2023-10-12T00:52:32.381 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.382 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.383 DEBUG:boto:Params: {} 2023-10-12T00:52:32.383 DEBUG:boto:Token: None 2023-10-12T00:52:32.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028007164ca27e0e0-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.388 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:52:32.388 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:52:32.388 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.389 DEBUG:boto:Path: /wafmoa-57/g 2023-10-12T00:52:32.389 DEBUG:boto:Data: 2023-10-12T00:52:32.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.390 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.391 DEBUG:boto:Params: {} 2023-10-12T00:52:32.391 DEBUG:boto:Token: None 2023-10-12T00:52:32.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/g 2023-10-12T00:52:32.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lshb+SYPkHG3wxChCMLJh/xCAKQ= 2023-10-12T00:52:32.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lshb+SYPkHG3wxChCMLJh/xCAKQ='} 2023-10-12T00:52:32.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000240edf63889d258b-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.398 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:52:32.399 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:52:32.399 DEBUG:boto:Method: HEAD 2023-10-12T00:52:32.399 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:52:32.400 DEBUG:boto:Data: 2023-10-12T00:52:32.400 DEBUG:boto:Headers: {} 2023-10-12T00:52:32.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.401 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.401 DEBUG:boto:Params: {} 2023-10-12T00:52:32.402 DEBUG:boto:Token: None 2023-10-12T00:52:32.402 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/ 2023-10-12T00:52:32.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U= 2023-10-12T00:52:32.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lw9BeMKDMRhD8XRKkHs2g0Kkf5U=', 'Content-Length': '0'} 2023-10-12T00:52:32.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd8c60755ff37634-0065274350-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.406 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:52:32.406 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:52:32.407 DEBUG:boto:Method: PUT 2023-10-12T00:52:32.407 DEBUG:boto:Path: /wafmoa-57/h 2023-10-12T00:52:32.407 DEBUG:boto:Data: 2023-10-12T00:52:32.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:52:32.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:52:32.409 DEBUG:boto:Port: 8000 2023-10-12T00:52:32.409 DEBUG:boto:Params: {} 2023-10-12T00:52:32.409 DEBUG:boto:Token: None 2023-10-12T00:52:32.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:52:32 GMT /wafmoa-57/h 2023-10-12T00:52:32.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zOVa6ZHIVrhm8s1zsPFp7/lyOIw= 2023-10-12T00:52:32.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:52:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zOVa6ZHIVrhm8s1zsPFp7/lyOIw='} 2023-10-12T00:52:32.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000397ad09f7642aadc-0065274350-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:52:32.455 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T00:52:32.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T00:52:32.459 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T00:52:32.459 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:32.479 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:52:32.479 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:52:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:52:32.481 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T00:52:33.483 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:52:33.505 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:52:33.506 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:52:33.507 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:52:33.508 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:52:33.508 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-57 2023-10-12T00:52:33.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:52:33.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:52:33.510 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:52:33.604 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:52:33.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:52:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:33.638+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:52:34.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T00:52:38.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:38.639+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:52:43.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:43.640+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:52:48.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:48.640+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:52:53.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:53.640+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:52:58.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:52:58.640+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:03.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:03.641+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:08.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:08.641+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:13.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:13.643+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:18.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:18.643+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:23.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:23.644+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:28.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:28.644+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:33.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:33.646+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:38.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:38.647+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:43.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:43.648+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:48.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:48.649+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:53.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:53.650+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:53:56.986 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:53:56.987+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T00:53:58.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:53:58.650+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:03.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:03.651+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:08.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:08.650+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:13.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:13.651+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:18.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:18.651+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:23.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:23.652+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:28.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:28.653+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:33.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:33.655+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:38.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:38.655+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:43.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:43.656+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:48.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:48.655+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:53.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:53.656+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:54:58.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:54:58.657+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:03.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:03.657+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:08.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:08.659+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:13.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:13.660+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:18.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:18.660+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:23.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:23.660+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:28.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:28.660+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:33.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:33.661+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:38.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:38.662+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:43.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:43.662+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:48.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:48.664+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:53.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:53.664+0000 7feff1640780 1 waiting to reach incremental sync.. 2023-10-12T00:55:58.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:58.665+0000 7feff1640780 1 bucket sync caught up with source: 2023-10-12T00:55:58.665 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.75.6, 00000000002.47.6, 00000000002.31.6] 2023-10-12T00:55:58.666 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.75.6, 00000000002.47.6, 00000000002.31.6] 2023-10-12T00:55:58.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:58.665+0000 7feff1640780 0 bucket checkpoint complete 2023-10-12T00:55:58.682 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-57 2023-10-12T00:55:58.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:55:58.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:55:58.684 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:55:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:55:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:55:58.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:58.784+0000 7fd5892b4780 1 bucket sync caught up with source: 2023-10-12T00:55:58.784 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.232.6, , , 00000000001.184.6, 00000000001.177.6, , , 00000000002.214.6, , 00000000002.270.6] 2023-10-12T00:55:58.784 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.232.6, , , 00000000001.184.6, 00000000001.177.6, , , 00000000002.214.6, , 00000000002.270.6] 2023-10-12T00:55:58.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:55:58.784+0000 7fd5892b4780 0 bucket checkpoint complete 2023-10-12T00:55:58.797 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-57 zones={a2, a1} 2023-10-12T00:55:58.798 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.798 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.799 DEBUG:boto:Method: HEAD 2023-10-12T00:55:58.799 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:55:58.800 DEBUG:boto:Data: 2023-10-12T00:55:58.801 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.802 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.803 DEBUG:boto:Params: {} 2023-10-12T00:55:58.804 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:55:58.805 DEBUG:boto:Token: None 2023-10-12T00:55:58.808 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/ 2023-10-12T00:55:58.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BuWhgc1o8uPSE9JU9d1CEa3TPms= 2023-10-12T00:55:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BuWhgc1o8uPSE9JU9d1CEa3TPms=', 'Content-Length': '0'} 2023-10-12T00:55:58.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8e6761f4113c2fd-006527441e-1aea-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.818 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.819 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.819 DEBUG:boto:Method: HEAD 2023-10-12T00:55:58.820 DEBUG:boto:Path: /wafmoa-57/ 2023-10-12T00:55:58.820 DEBUG:boto:Data: 2023-10-12T00:55:58.820 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.821 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.822 DEBUG:boto:Params: {} 2023-10-12T00:55:58.822 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:55:58.823 DEBUG:boto:Token: None 2023-10-12T00:55:58.823 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/ 2023-10-12T00:55:58.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BuWhgc1o8uPSE9JU9d1CEa3TPms= 2023-10-12T00:55:58.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BuWhgc1o8uPSE9JU9d1CEa3TPms=', 'Content-Length': '0'} 2023-10-12T00:55:58.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084d38f7e27974835-006527441e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.828 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:55:58.829 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.829 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.830 DEBUG:boto:path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.830 DEBUG:boto:auth_path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.831 DEBUG:boto:Method: GET 2023-10-12T00:55:58.831 DEBUG:boto:Path: /wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.831 DEBUG:boto:Data: 2023-10-12T00:55:58.832 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.834 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.834 DEBUG:boto:Params: {} 2023-10-12T00:55:58.835 DEBUG:boto:Token: None 2023-10-12T00:55:58.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/?versions 2023-10-12T00:55:58.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI= 2023-10-12T00:55:58.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI=', 'Content-Length': '0'} 2023-10-12T00:55:58.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008436a7902f8d72c8-006527441e-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.840 DEBUG:boto:b'wafmoa-57999falseanulltrue2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.841 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:55:58.842 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:55:58.842 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:55:58.842 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:55:58.843 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:55:58.843 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:55:58.844 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:55:58.844 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:55:58.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:55:58.845 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.845 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.846 DEBUG:boto:path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.846 DEBUG:boto:auth_path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.846 DEBUG:boto:Method: GET 2023-10-12T00:55:58.847 DEBUG:boto:Path: /wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.847 DEBUG:boto:Data: 2023-10-12T00:55:58.848 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.849 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.849 DEBUG:boto:Params: {} 2023-10-12T00:55:58.850 DEBUG:boto:Token: None 2023-10-12T00:55:58.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/?versions 2023-10-12T00:55:58.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI= 2023-10-12T00:55:58.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI=', 'Content-Length': '0'} 2023-10-12T00:55:58.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da23a56ecbf90f54-006527441e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.854 DEBUG:boto:b'wafmoa-57999falseanulltrue2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.855 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-12T00:55:58.856 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-12T00:55:58.856 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-12T00:55:58.857 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-12T00:55:58.857 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-12T00:55:58.857 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-12T00:55:58.858 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-12T00:55:58.858 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-12T00:55:58.859 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.859 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.860 DEBUG:boto:path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.860 DEBUG:boto:auth_path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.861 DEBUG:boto:Method: GET 2023-10-12T00:55:58.861 DEBUG:boto:Path: /wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.861 DEBUG:boto:Data: 2023-10-12T00:55:58.862 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.863 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.863 DEBUG:boto:Params: {} 2023-10-12T00:55:58.864 DEBUG:boto:Token: None 2023-10-12T00:55:58.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/?versions 2023-10-12T00:55:58.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI= 2023-10-12T00:55:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI=', 'Content-Length': '0'} 2023-10-12T00:55:58.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750f7ceac3681d3f-006527441e-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.869 DEBUG:boto:b'wafmoa-57999falseanulltrue2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.870 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.870 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.871 DEBUG:boto:path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.871 DEBUG:boto:auth_path=/wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.872 DEBUG:boto:Method: GET 2023-10-12T00:55:58.872 DEBUG:boto:Path: /wafmoa-57/?versions&max-keys=999 2023-10-12T00:55:58.873 DEBUG:boto:Data: 2023-10-12T00:55:58.873 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.874 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.875 DEBUG:boto:Params: {} 2023-10-12T00:55:58.875 DEBUG:boto:Token: None 2023-10-12T00:55:58.876 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/?versions 2023-10-12T00:55:58.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI= 2023-10-12T00:55:58.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eFdOYaOJHHnKxCVCxfhlD2dmJTI=', 'Content-Length': '0'} 2023-10-12T00:55:58.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc74dd2b46f9062a-006527441e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.879 DEBUG:boto:b'wafmoa-57999falseanulltrue2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:55:58.881 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.881 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.882 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.882 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.883 DEBUG:boto:Method: GET 2023-10-12T00:55:58.883 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.883 DEBUG:boto:Data: 2023-10-12T00:55:58.884 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.885 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.885 DEBUG:boto:Params: {} 2023-10-12T00:55:58.886 DEBUG:boto:Token: None 2023-10-12T00:55:58.887 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs= 2023-10-12T00:55:58.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs=', 'Content-Length': '0'} 2023-10-12T00:55:58.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee1decb824ceaa6-006527441e-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.890 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.891 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.891 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.891 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.892 DEBUG:boto:Method: GET 2023-10-12T00:55:58.892 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.893 DEBUG:boto:Data: 2023-10-12T00:55:58.893 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.894 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.895 DEBUG:boto:Params: {} 2023-10-12T00:55:58.895 DEBUG:boto:Token: None 2023-10-12T00:55:58.896 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs= 2023-10-12T00:55:58.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs=', 'Content-Length': '0'} 2023-10-12T00:55:58.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072a5c7a67917a778-006527441e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.899 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.900 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.900 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.901 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.901 DEBUG:boto:Method: HEAD 2023-10-12T00:55:58.901 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.902 DEBUG:boto:Data: 2023-10-12T00:55:58.902 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.903 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.904 DEBUG:boto:Params: {} 2023-10-12T00:55:58.904 DEBUG:boto:Token: None 2023-10-12T00:55:58.904 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r36UUElv04X5B41aiSV6cXyeLcY= 2023-10-12T00:55:58.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r36UUElv04X5B41aiSV6cXyeLcY=', 'Content-Length': '0'} 2023-10-12T00:55:58.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526ad850ff95558c-006527441e-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.908 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.908 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.909 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.909 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.909 DEBUG:boto:Method: HEAD 2023-10-12T00:55:58.910 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.910 DEBUG:boto:Data: 2023-10-12T00:55:58.911 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.912 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.912 DEBUG:boto:Params: {} 2023-10-12T00:55:58.913 DEBUG:boto:Token: None 2023-10-12T00:55:58.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r36UUElv04X5B41aiSV6cXyeLcY= 2023-10-12T00:55:58.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r36UUElv04X5B41aiSV6cXyeLcY=', 'Content-Length': '0'} 2023-10-12T00:55:58.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007176f818ad59fd35-006527441e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:55:58.917 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.918 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.918 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.919 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.919 DEBUG:boto:Method: GET 2023-10-12T00:55:58.919 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.920 DEBUG:boto:Data: 2023-10-12T00:55:58.921 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.921 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.922 DEBUG:boto:Params: {} 2023-10-12T00:55:58.922 DEBUG:boto:Token: None 2023-10-12T00:55:58.923 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs= 2023-10-12T00:55:58.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs=', 'Content-Length': '0'} 2023-10-12T00:55:58.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbffd052dfc776c-006527441e-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.926 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.927 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:58.927 DEBUG:boto:path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.928 DEBUG:boto:auth_path=/wafmoa-57/a?versionId=null 2023-10-12T00:55:58.928 DEBUG:boto:Method: GET 2023-10-12T00:55:58.929 DEBUG:boto:Path: /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.929 DEBUG:boto:Data: 2023-10-12T00:55:58.930 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.930 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.931 DEBUG:boto:Params: {} 2023-10-12T00:55:58.931 DEBUG:boto:Token: None 2023-10-12T00:55:58.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/a?versionId=null 2023-10-12T00:55:58.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs= 2023-10-12T00:55:58.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2vvWfQPr2LwMtrOZe5F2rco+qvs=', 'Content-Length': '0'} 2023-10-12T00:55:58.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041574ff591290715-006527441e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.935 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.936 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.936 DEBUG:boto:path=/wafmoa-57/?prefix=a 2023-10-12T00:55:58.937 DEBUG:boto:auth_path=/wafmoa-57/?prefix=a 2023-10-12T00:55:58.937 DEBUG:boto:Method: GET 2023-10-12T00:55:58.937 DEBUG:boto:Path: /wafmoa-57/?prefix=a 2023-10-12T00:55:58.938 DEBUG:boto:Data: 2023-10-12T00:55:58.938 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:58.939 DEBUG:boto:Port: 8001 2023-10-12T00:55:58.940 DEBUG:boto:Params: {} 2023-10-12T00:55:58.940 DEBUG:boto:Token: None 2023-10-12T00:55:58.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/ 2023-10-12T00:55:58.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VCABqy/ULHb5RaB9Zrce4X01y0s= 2023-10-12T00:55:58.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VCABqy/ULHb5RaB9Zrce4X01y0s=', 'Content-Length': '0'} 2023-10-12T00:55:58.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003857bb332e28347a-006527441e-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.945 DEBUG:boto:b'wafmoa-57a1000falsea2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.946 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:58.946 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:58.947 DEBUG:boto:path=/wafmoa-57/?prefix=a 2023-10-12T00:55:58.947 DEBUG:boto:auth_path=/wafmoa-57/?prefix=a 2023-10-12T00:55:58.948 DEBUG:boto:Method: GET 2023-10-12T00:55:58.948 DEBUG:boto:Path: /wafmoa-57/?prefix=a 2023-10-12T00:55:58.948 DEBUG:boto:Data: 2023-10-12T00:55:58.949 DEBUG:boto:Headers: {} 2023-10-12T00:55:58.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:58.950 DEBUG:boto:Port: 8000 2023-10-12T00:55:58.950 DEBUG:boto:Params: {} 2023-10-12T00:55:58.951 DEBUG:boto:Token: None 2023-10-12T00:55:58.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:58 GMT /wafmoa-57/ 2023-10-12T00:55:58.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VCABqy/ULHb5RaB9Zrce4X01y0s= 2023-10-12T00:55:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:58 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VCABqy/ULHb5RaB9Zrce4X01y0s=', 'Content-Length': '0'} 2023-10-12T00:55:58.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862fe0f449f637b6-006527441e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:58.998 DEBUG:boto:b'wafmoa-57a1000falsea2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:58.999 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:58.999 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:59.000 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.000 DEBUG:boto:Path: /wafmoa-57/a 2023-10-12T00:55:59.001 DEBUG:boto:Data: 2023-10-12T00:55:59.001 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.002 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.002 DEBUG:boto:Params: {} 2023-10-12T00:55:59.003 DEBUG:boto:Token: None 2023-10-12T00:55:59.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/a 2023-10-12T00:55:59.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IxBpiwrWtCQK+TwW7rS+Dj4W2Ug= 2023-10-12T00:55:59.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IxBpiwrWtCQK+TwW7rS+Dj4W2Ug=', 'Content-Length': '0'} 2023-10-12T00:55:59.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d98406fd732df1-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.006 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:59.007 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:59.007 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.008 DEBUG:boto:Path: /wafmoa-57/a 2023-10-12T00:55:59.008 DEBUG:boto:Data: 2023-10-12T00:55:59.008 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.009 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.009 DEBUG:boto:Params: {} 2023-10-12T00:55:59.010 DEBUG:boto:Token: None 2023-10-12T00:55:59.010 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/a 2023-10-12T00:55:59.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IxBpiwrWtCQK+TwW7rS+Dj4W2Ug= 2023-10-12T00:55:59.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IxBpiwrWtCQK+TwW7rS+Dj4W2Ug=', 'Content-Length': '0'} 2023-10-12T00:55:59.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe551d4d12d7f30-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-12T00:55:59.016 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:59.017 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:59.017 DEBUG:boto:Method: GET 2023-10-12T00:55:59.018 DEBUG:boto:Path: /wafmoa-57/a 2023-10-12T00:55:59.018 DEBUG:boto:Data: 2023-10-12T00:55:59.019 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.020 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.020 DEBUG:boto:Params: {} 2023-10-12T00:55:59.021 DEBUG:boto:Token: None 2023-10-12T00:55:59.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/a 2023-10-12T00:55:59.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rIGoqX67m3nQKLEgd+m+tZryJus= 2023-10-12T00:55:59.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rIGoqX67m3nQKLEgd+m+tZryJus=', 'Content-Length': '0'} 2023-10-12T00:55:59.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006850683ecc97d0ce-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.025 DEBUG:boto:path=/wafmoa-57/a 2023-10-12T00:55:59.025 DEBUG:boto:auth_path=/wafmoa-57/a 2023-10-12T00:55:59.026 DEBUG:boto:Method: GET 2023-10-12T00:55:59.026 DEBUG:boto:Path: /wafmoa-57/a 2023-10-12T00:55:59.027 DEBUG:boto:Data: 2023-10-12T00:55:59.027 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.028 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.029 DEBUG:boto:Params: {} 2023-10-12T00:55:59.029 DEBUG:boto:Token: None 2023-10-12T00:55:59.030 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/a 2023-10-12T00:55:59.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rIGoqX67m3nQKLEgd+m+tZryJus= 2023-10-12T00:55:59.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rIGoqX67m3nQKLEgd+m+tZryJus=', 'Content-Length': '0'} 2023-10-12T00:55:59.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003271cf3ed1f3dc40-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.033 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.033 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.034 DEBUG:boto:path=/wafmoa-57/?prefix=a 2023-10-12T00:55:59.034 DEBUG:boto:auth_path=/wafmoa-57/?prefix=a 2023-10-12T00:55:59.035 DEBUG:boto:Method: GET 2023-10-12T00:55:59.035 DEBUG:boto:Path: /wafmoa-57/?prefix=a 2023-10-12T00:55:59.036 DEBUG:boto:Data: 2023-10-12T00:55:59.036 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.037 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.038 DEBUG:boto:Params: {} 2023-10-12T00:55:59.038 DEBUG:boto:Token: None 2023-10-12T00:55:59.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34c4dc43fdc7b58-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.042 DEBUG:boto:b'wafmoa-57a1000falsea2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.043 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.043 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.044 DEBUG:boto:path=/wafmoa-57/?prefix=a 2023-10-12T00:55:59.044 DEBUG:boto:auth_path=/wafmoa-57/?prefix=a 2023-10-12T00:55:59.045 DEBUG:boto:Method: GET 2023-10-12T00:55:59.045 DEBUG:boto:Path: /wafmoa-57/?prefix=a 2023-10-12T00:55:59.046 DEBUG:boto:Data: 2023-10-12T00:55:59.046 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.047 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.047 DEBUG:boto:Params: {} 2023-10-12T00:55:59.048 DEBUG:boto:Token: None 2023-10-12T00:55:59.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc8fa45e82b82ab-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.052 DEBUG:boto:b'wafmoa-57a1000falsea2023-10-12T00:52:32.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:55:59.053 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.053 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.054 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.054 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.055 DEBUG:boto:Method: GET 2023-10-12T00:55:59.055 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.055 DEBUG:boto:Data: 2023-10-12T00:55:59.056 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.057 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.057 DEBUG:boto:Params: {} 2023-10-12T00:55:59.057 DEBUG:boto:Token: None 2023-10-12T00:55:59.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE= 2023-10-12T00:55:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE=', 'Content-Length': '0'} 2023-10-12T00:55:59.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa4d30d0ba505bc-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.061 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.062 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.062 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.063 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.063 DEBUG:boto:Method: GET 2023-10-12T00:55:59.064 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.064 DEBUG:boto:Data: 2023-10-12T00:55:59.064 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.065 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.066 DEBUG:boto:Params: {} 2023-10-12T00:55:59.066 DEBUG:boto:Token: None 2023-10-12T00:55:59.067 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE= 2023-10-12T00:55:59.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE=', 'Content-Length': '0'} 2023-10-12T00:55:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045af6c39e9180331-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.110 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.111 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.111 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.112 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.112 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.113 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.113 DEBUG:boto:Data: 2023-10-12T00:55:59.113 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.114 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.115 DEBUG:boto:Params: {} 2023-10-12T00:55:59.115 DEBUG:boto:Token: None 2023-10-12T00:55:59.115 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBnYE6+Qn4Z0Ifj7omWqSFopEjY= 2023-10-12T00:55:59.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBnYE6+Qn4Z0Ifj7omWqSFopEjY=', 'Content-Length': '0'} 2023-10-12T00:55:59.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbc2379ae89bd5b5-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.119 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.120 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.120 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.121 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.121 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.122 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.122 DEBUG:boto:Data: 2023-10-12T00:55:59.122 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.123 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.124 DEBUG:boto:Params: {} 2023-10-12T00:55:59.124 DEBUG:boto:Token: None 2023-10-12T00:55:59.124 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBnYE6+Qn4Z0Ifj7omWqSFopEjY= 2023-10-12T00:55:59.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBnYE6+Qn4Z0Ifj7omWqSFopEjY=', 'Content-Length': '0'} 2023-10-12T00:55:59.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656381f83a4729ed-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:55:59.128 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.129 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.130 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.130 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.131 DEBUG:boto:Method: GET 2023-10-12T00:55:59.131 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.132 DEBUG:boto:Data: 2023-10-12T00:55:59.132 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.133 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.133 DEBUG:boto:Params: {} 2023-10-12T00:55:59.134 DEBUG:boto:Token: None 2023-10-12T00:55:59.134 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE= 2023-10-12T00:55:59.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE=', 'Content-Length': '0'} 2023-10-12T00:55:59.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff75dca8a8001c06-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.138 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.138 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.139 DEBUG:boto:path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.139 DEBUG:boto:auth_path=/wafmoa-57/b?versionId=null 2023-10-12T00:55:59.139 DEBUG:boto:Method: GET 2023-10-12T00:55:59.140 DEBUG:boto:Path: /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.140 DEBUG:boto:Data: 2023-10-12T00:55:59.141 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.142 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.142 DEBUG:boto:Params: {} 2023-10-12T00:55:59.143 DEBUG:boto:Token: None 2023-10-12T00:55:59.143 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b?versionId=null 2023-10-12T00:55:59.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE= 2023-10-12T00:55:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LxmQxmdLTQtBaiCKz6fETRVrMoE=', 'Content-Length': '0'} 2023-10-12T00:55:59.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec72019dafcbd4b1-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.147 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.147 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.148 DEBUG:boto:path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.148 DEBUG:boto:auth_path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.148 DEBUG:boto:Method: GET 2023-10-12T00:55:59.149 DEBUG:boto:Path: /wafmoa-57/?prefix=b 2023-10-12T00:55:59.149 DEBUG:boto:Data: 2023-10-12T00:55:59.150 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.151 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.151 DEBUG:boto:Params: {} 2023-10-12T00:55:59.152 DEBUG:boto:Token: None 2023-10-12T00:55:59.152 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038fda8b3c6a79dc9-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.156 DEBUG:boto:b'wafmoa-57b1000falseb2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.157 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.157 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.158 DEBUG:boto:path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.158 DEBUG:boto:auth_path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.159 DEBUG:boto:Method: GET 2023-10-12T00:55:59.159 DEBUG:boto:Path: /wafmoa-57/?prefix=b 2023-10-12T00:55:59.160 DEBUG:boto:Data: 2023-10-12T00:55:59.160 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.161 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.161 DEBUG:boto:Params: {} 2023-10-12T00:55:59.162 DEBUG:boto:Token: None 2023-10-12T00:55:59.162 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.163 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428d696c99d449b9-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.166 DEBUG:boto:b'wafmoa-57b1000falseb2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.167 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.167 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.168 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.168 DEBUG:boto:Path: /wafmoa-57/b 2023-10-12T00:55:59.168 DEBUG:boto:Data: 2023-10-12T00:55:59.169 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.170 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.170 DEBUG:boto:Params: {} 2023-10-12T00:55:59.171 DEBUG:boto:Token: None 2023-10-12T00:55:59.171 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b 2023-10-12T00:55:59.171 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5dCJLzCQZbVynUBt3/sWNwoFKcU= 2023-10-12T00:55:59.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5dCJLzCQZbVynUBt3/sWNwoFKcU=', 'Content-Length': '0'} 2023-10-12T00:55:59.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b359b784c0dae09a-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.174 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.175 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.175 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.176 DEBUG:boto:Path: /wafmoa-57/b 2023-10-12T00:55:59.176 DEBUG:boto:Data: 2023-10-12T00:55:59.176 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.177 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.178 DEBUG:boto:Params: {} 2023-10-12T00:55:59.178 DEBUG:boto:Token: None 2023-10-12T00:55:59.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b 2023-10-12T00:55:59.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5dCJLzCQZbVynUBt3/sWNwoFKcU= 2023-10-12T00:55:59.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5dCJLzCQZbVynUBt3/sWNwoFKcU=', 'Content-Length': '0'} 2023-10-12T00:55:59.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed937e3fa640c2ae-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-12T00:55:59.182 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.183 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.183 DEBUG:boto:Method: GET 2023-10-12T00:55:59.184 DEBUG:boto:Path: /wafmoa-57/b 2023-10-12T00:55:59.184 DEBUG:boto:Data: 2023-10-12T00:55:59.184 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.185 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.186 DEBUG:boto:Params: {} 2023-10-12T00:55:59.186 DEBUG:boto:Token: None 2023-10-12T00:55:59.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b 2023-10-12T00:55:59.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nrCatrJfuqv+1EETORFiCGQRcqQ= 2023-10-12T00:55:59.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nrCatrJfuqv+1EETORFiCGQRcqQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa45136e91261b7-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.234 DEBUG:boto:path=/wafmoa-57/b 2023-10-12T00:55:59.235 DEBUG:boto:auth_path=/wafmoa-57/b 2023-10-12T00:55:59.235 DEBUG:boto:Method: GET 2023-10-12T00:55:59.236 DEBUG:boto:Path: /wafmoa-57/b 2023-10-12T00:55:59.236 DEBUG:boto:Data: 2023-10-12T00:55:59.237 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.238 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.238 DEBUG:boto:Params: {} 2023-10-12T00:55:59.239 DEBUG:boto:Token: None 2023-10-12T00:55:59.239 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/b 2023-10-12T00:55:59.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nrCatrJfuqv+1EETORFiCGQRcqQ= 2023-10-12T00:55:59.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nrCatrJfuqv+1EETORFiCGQRcqQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7c4e76f819581b-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.243 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.243 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.243 DEBUG:boto:path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.244 DEBUG:boto:auth_path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.244 DEBUG:boto:Method: GET 2023-10-12T00:55:59.245 DEBUG:boto:Path: /wafmoa-57/?prefix=b 2023-10-12T00:55:59.245 DEBUG:boto:Data: 2023-10-12T00:55:59.246 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.246 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.247 DEBUG:boto:Params: {} 2023-10-12T00:55:59.247 DEBUG:boto:Token: None 2023-10-12T00:55:59.248 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a977bbd341bbff02-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.252 DEBUG:boto:b'wafmoa-57b1000falseb2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.253 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.253 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.253 DEBUG:boto:path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.254 DEBUG:boto:auth_path=/wafmoa-57/?prefix=b 2023-10-12T00:55:59.254 DEBUG:boto:Method: GET 2023-10-12T00:55:59.255 DEBUG:boto:Path: /wafmoa-57/?prefix=b 2023-10-12T00:55:59.255 DEBUG:boto:Data: 2023-10-12T00:55:59.256 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.257 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.257 DEBUG:boto:Params: {} 2023-10-12T00:55:59.258 DEBUG:boto:Token: None 2023-10-12T00:55:59.258 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d0a9dd173f829b-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.261 DEBUG:boto:b'wafmoa-57b1000falseb2023-10-12T00:52:32.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:55:59.262 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.263 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.263 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.264 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.264 DEBUG:boto:Method: GET 2023-10-12T00:55:59.265 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.265 DEBUG:boto:Data: 2023-10-12T00:55:59.265 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.266 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.267 DEBUG:boto:Params: {} 2023-10-12T00:55:59.267 DEBUG:boto:Token: None 2023-10-12T00:55:59.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks= 2023-10-12T00:55:59.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks=', 'Content-Length': '0'} 2023-10-12T00:55:59.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f0018eaa2fb722-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.271 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.272 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.272 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.273 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.273 DEBUG:boto:Method: GET 2023-10-12T00:55:59.274 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.274 DEBUG:boto:Data: 2023-10-12T00:55:59.274 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.275 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.276 DEBUG:boto:Params: {} 2023-10-12T00:55:59.276 DEBUG:boto:Token: None 2023-10-12T00:55:59.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks= 2023-10-12T00:55:59.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks=', 'Content-Length': '0'} 2023-10-12T00:55:59.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce1ebe975091a75-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.280 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.281 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.281 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.282 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.282 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.282 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.283 DEBUG:boto:Data: 2023-10-12T00:55:59.283 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.284 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.285 DEBUG:boto:Params: {} 2023-10-12T00:55:59.285 DEBUG:boto:Token: None 2023-10-12T00:55:59.285 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FoKD/okmJ2zkIzYr5buCHV9hqfA= 2023-10-12T00:55:59.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FoKD/okmJ2zkIzYr5buCHV9hqfA=', 'Content-Length': '0'} 2023-10-12T00:55:59.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041412048b059b7fc-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.289 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.289 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.290 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.290 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.290 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.291 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.291 DEBUG:boto:Data: 2023-10-12T00:55:59.292 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.292 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.293 DEBUG:boto:Params: {} 2023-10-12T00:55:59.293 DEBUG:boto:Token: None 2023-10-12T00:55:59.294 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.294 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FoKD/okmJ2zkIzYr5buCHV9hqfA= 2023-10-12T00:55:59.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FoKD/okmJ2zkIzYr5buCHV9hqfA=', 'Content-Length': '0'} 2023-10-12T00:55:59.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342dbd69fd26b775-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:55:59.298 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.299 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.300 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.300 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.300 DEBUG:boto:Method: GET 2023-10-12T00:55:59.301 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.301 DEBUG:boto:Data: 2023-10-12T00:55:59.302 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.303 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.303 DEBUG:boto:Params: {} 2023-10-12T00:55:59.304 DEBUG:boto:Token: None 2023-10-12T00:55:59.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks= 2023-10-12T00:55:59.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks=', 'Content-Length': '0'} 2023-10-12T00:55:59.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005acee5f59f7cbeda-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.350 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.351 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.351 DEBUG:boto:path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.352 DEBUG:boto:auth_path=/wafmoa-57/c?versionId=null 2023-10-12T00:55:59.352 DEBUG:boto:Method: GET 2023-10-12T00:55:59.353 DEBUG:boto:Path: /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.353 DEBUG:boto:Data: 2023-10-12T00:55:59.354 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.354 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.355 DEBUG:boto:Params: {} 2023-10-12T00:55:59.355 DEBUG:boto:Token: None 2023-10-12T00:55:59.356 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c?versionId=null 2023-10-12T00:55:59.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks= 2023-10-12T00:55:59.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kVND2GFtjdu5al7Lo6sqMUBeWks=', 'Content-Length': '0'} 2023-10-12T00:55:59.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da9d531fb32a3b7-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.359 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.360 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.360 DEBUG:boto:path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.361 DEBUG:boto:auth_path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.361 DEBUG:boto:Method: GET 2023-10-12T00:55:59.362 DEBUG:boto:Path: /wafmoa-57/?prefix=c 2023-10-12T00:55:59.362 DEBUG:boto:Data: 2023-10-12T00:55:59.363 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.364 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.364 DEBUG:boto:Params: {} 2023-10-12T00:55:59.365 DEBUG:boto:Token: None 2023-10-12T00:55:59.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b90c79a4dff8926-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.370 DEBUG:boto:b'wafmoa-57c1000falsec2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.371 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.371 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.372 DEBUG:boto:path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.372 DEBUG:boto:auth_path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.372 DEBUG:boto:Method: GET 2023-10-12T00:55:59.373 DEBUG:boto:Path: /wafmoa-57/?prefix=c 2023-10-12T00:55:59.373 DEBUG:boto:Data: 2023-10-12T00:55:59.374 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.375 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.375 DEBUG:boto:Params: {} 2023-10-12T00:55:59.375 DEBUG:boto:Token: None 2023-10-12T00:55:59.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7c13bf43fa7cd0-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.380 DEBUG:boto:b'wafmoa-57c1000falsec2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.380 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.381 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.381 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.382 DEBUG:boto:Path: /wafmoa-57/c 2023-10-12T00:55:59.382 DEBUG:boto:Data: 2023-10-12T00:55:59.382 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.383 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.384 DEBUG:boto:Params: {} 2023-10-12T00:55:59.384 DEBUG:boto:Token: None 2023-10-12T00:55:59.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c 2023-10-12T00:55:59.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qcGG33XBkyQHAKblR7crqw1LaY= 2023-10-12T00:55:59.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qcGG33XBkyQHAKblR7crqw1LaY=', 'Content-Length': '0'} 2023-10-12T00:55:59.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a556324ae6ba8e4-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.388 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.389 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.389 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.390 DEBUG:boto:Path: /wafmoa-57/c 2023-10-12T00:55:59.390 DEBUG:boto:Data: 2023-10-12T00:55:59.390 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.391 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.392 DEBUG:boto:Params: {} 2023-10-12T00:55:59.392 DEBUG:boto:Token: None 2023-10-12T00:55:59.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c 2023-10-12T00:55:59.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qcGG33XBkyQHAKblR7crqw1LaY= 2023-10-12T00:55:59.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qcGG33XBkyQHAKblR7crqw1LaY=', 'Content-Length': '0'} 2023-10-12T00:55:59.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095647e169b945ac8-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-12T00:55:59.396 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.397 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.397 DEBUG:boto:Method: GET 2023-10-12T00:55:59.398 DEBUG:boto:Path: /wafmoa-57/c 2023-10-12T00:55:59.398 DEBUG:boto:Data: 2023-10-12T00:55:59.399 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.399 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.400 DEBUG:boto:Params: {} 2023-10-12T00:55:59.400 DEBUG:boto:Token: None 2023-10-12T00:55:59.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c 2023-10-12T00:55:59.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HIQ4suySHDFxLJmbQqfAEd+0ujM= 2023-10-12T00:55:59.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HIQ4suySHDFxLJmbQqfAEd+0ujM=', 'Content-Length': '0'} 2023-10-12T00:55:59.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003efe89ad76085890-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.404 DEBUG:boto:path=/wafmoa-57/c 2023-10-12T00:55:59.405 DEBUG:boto:auth_path=/wafmoa-57/c 2023-10-12T00:55:59.405 DEBUG:boto:Method: GET 2023-10-12T00:55:59.406 DEBUG:boto:Path: /wafmoa-57/c 2023-10-12T00:55:59.406 DEBUG:boto:Data: 2023-10-12T00:55:59.406 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.407 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.408 DEBUG:boto:Params: {} 2023-10-12T00:55:59.408 DEBUG:boto:Token: None 2023-10-12T00:55:59.409 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/c 2023-10-12T00:55:59.409 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HIQ4suySHDFxLJmbQqfAEd+0ujM= 2023-10-12T00:55:59.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HIQ4suySHDFxLJmbQqfAEd+0ujM=', 'Content-Length': '0'} 2023-10-12T00:55:59.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e79c37cfe3cd039-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.458 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.459 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.460 DEBUG:boto:path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.460 DEBUG:boto:auth_path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.460 DEBUG:boto:Method: GET 2023-10-12T00:55:59.461 DEBUG:boto:Path: /wafmoa-57/?prefix=c 2023-10-12T00:55:59.461 DEBUG:boto:Data: 2023-10-12T00:55:59.462 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.463 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.463 DEBUG:boto:Params: {} 2023-10-12T00:55:59.464 DEBUG:boto:Token: None 2023-10-12T00:55:59.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.465 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f89112da1a71cd-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.468 DEBUG:boto:b'wafmoa-57c1000falsec2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.469 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.470 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.470 DEBUG:boto:path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.471 DEBUG:boto:auth_path=/wafmoa-57/?prefix=c 2023-10-12T00:55:59.471 DEBUG:boto:Method: GET 2023-10-12T00:55:59.471 DEBUG:boto:Path: /wafmoa-57/?prefix=c 2023-10-12T00:55:59.472 DEBUG:boto:Data: 2023-10-12T00:55:59.473 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.473 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.476 DEBUG:boto:Params: {} 2023-10-12T00:55:59.476 DEBUG:boto:Token: None 2023-10-12T00:55:59.477 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8687be7859eb5e2-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.480 DEBUG:boto:b'wafmoa-57c1000falsec2023-10-12T00:52:32.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:55:59.481 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.482 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.482 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.482 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.483 DEBUG:boto:Method: GET 2023-10-12T00:55:59.483 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.484 DEBUG:boto:Data: 2023-10-12T00:55:59.484 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.485 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.485 DEBUG:boto:Params: {} 2023-10-12T00:55:59.486 DEBUG:boto:Token: None 2023-10-12T00:55:59.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.487 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE= 2023-10-12T00:55:59.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE=', 'Content-Length': '0'} 2023-10-12T00:55:59.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300e4d6d35c1dd18-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.491 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.491 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.492 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.492 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.493 DEBUG:boto:Method: GET 2023-10-12T00:55:59.493 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.494 DEBUG:boto:Data: 2023-10-12T00:55:59.494 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.495 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.496 DEBUG:boto:Params: {} 2023-10-12T00:55:59.496 DEBUG:boto:Token: None 2023-10-12T00:55:59.497 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE= 2023-10-12T00:55:59.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE=', 'Content-Length': '0'} 2023-10-12T00:55:59.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdf6a642207f5db-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.501 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.501 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.501 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.502 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.502 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.503 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.503 DEBUG:boto:Data: 2023-10-12T00:55:59.504 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.505 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.505 DEBUG:boto:Params: {} 2023-10-12T00:55:59.505 DEBUG:boto:Token: None 2023-10-12T00:55:59.506 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hUNE9gVA5hw33wKPHLtJ5Z1QsoU= 2023-10-12T00:55:59.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hUNE9gVA5hw33wKPHLtJ5Z1QsoU=', 'Content-Length': '0'} 2023-10-12T00:55:59.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfaebe2813d52723-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.509 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.510 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.510 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.510 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.511 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.511 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.512 DEBUG:boto:Data: 2023-10-12T00:55:59.512 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.513 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.513 DEBUG:boto:Params: {} 2023-10-12T00:55:59.514 DEBUG:boto:Token: None 2023-10-12T00:55:59.514 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hUNE9gVA5hw33wKPHLtJ5Z1QsoU= 2023-10-12T00:55:59.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hUNE9gVA5hw33wKPHLtJ5Z1QsoU=', 'Content-Length': '0'} 2023-10-12T00:55:59.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c69868c365c2f80-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:55:59.518 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.519 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.519 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.519 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.520 DEBUG:boto:Method: GET 2023-10-12T00:55:59.520 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.521 DEBUG:boto:Data: 2023-10-12T00:55:59.521 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.522 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.522 DEBUG:boto:Params: {} 2023-10-12T00:55:59.523 DEBUG:boto:Token: None 2023-10-12T00:55:59.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE= 2023-10-12T00:55:59.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE=', 'Content-Length': '0'} 2023-10-12T00:55:59.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066b7998da4a77d0-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.528 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.529 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.529 DEBUG:boto:path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.529 DEBUG:boto:auth_path=/wafmoa-57/d?versionId=null 2023-10-12T00:55:59.530 DEBUG:boto:Method: GET 2023-10-12T00:55:59.530 DEBUG:boto:Path: /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.531 DEBUG:boto:Data: 2023-10-12T00:55:59.531 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.532 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.533 DEBUG:boto:Params: {} 2023-10-12T00:55:59.533 DEBUG:boto:Token: None 2023-10-12T00:55:59.534 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d?versionId=null 2023-10-12T00:55:59.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE= 2023-10-12T00:55:59.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZmZg5xMuQpbiyshFBFUiAHU3tiE=', 'Content-Length': '0'} 2023-10-12T00:55:59.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8e4170ee9bec64c-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.537 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.538 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.538 DEBUG:boto:path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.539 DEBUG:boto:auth_path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.539 DEBUG:boto:Method: GET 2023-10-12T00:55:59.540 DEBUG:boto:Path: /wafmoa-57/?prefix=d 2023-10-12T00:55:59.540 DEBUG:boto:Data: 2023-10-12T00:55:59.541 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.541 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.542 DEBUG:boto:Params: {} 2023-10-12T00:55:59.542 DEBUG:boto:Token: None 2023-10-12T00:55:59.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008253fa65a56aa310-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.590 DEBUG:boto:b'wafmoa-57d1000falsed2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.591 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.592 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.592 DEBUG:boto:path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.593 DEBUG:boto:auth_path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.593 DEBUG:boto:Method: GET 2023-10-12T00:55:59.593 DEBUG:boto:Path: /wafmoa-57/?prefix=d 2023-10-12T00:55:59.594 DEBUG:boto:Data: 2023-10-12T00:55:59.594 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.595 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.596 DEBUG:boto:Params: {} 2023-10-12T00:55:59.596 DEBUG:boto:Token: None 2023-10-12T00:55:59.597 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af775632c9b9fa3-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.600 DEBUG:boto:b'wafmoa-57d1000falsed2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.601 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.601 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.602 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.602 DEBUG:boto:Path: /wafmoa-57/d 2023-10-12T00:55:59.603 DEBUG:boto:Data: 2023-10-12T00:55:59.603 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.604 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.604 DEBUG:boto:Params: {} 2023-10-12T00:55:59.605 DEBUG:boto:Token: None 2023-10-12T00:55:59.605 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d 2023-10-12T00:55:59.606 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbQndrMMCY8077vt4oxUIOTY3hQ= 2023-10-12T00:55:59.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbQndrMMCY8077vt4oxUIOTY3hQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2cd6cedae22bfed-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.609 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.609 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.610 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.610 DEBUG:boto:Path: /wafmoa-57/d 2023-10-12T00:55:59.610 DEBUG:boto:Data: 2023-10-12T00:55:59.611 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.612 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.612 DEBUG:boto:Params: {} 2023-10-12T00:55:59.613 DEBUG:boto:Token: None 2023-10-12T00:55:59.613 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d 2023-10-12T00:55:59.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbQndrMMCY8077vt4oxUIOTY3hQ= 2023-10-12T00:55:59.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbQndrMMCY8077vt4oxUIOTY3hQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052890a65fbf08ebd-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-12T00:55:59.617 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.617 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.618 DEBUG:boto:Method: GET 2023-10-12T00:55:59.618 DEBUG:boto:Path: /wafmoa-57/d 2023-10-12T00:55:59.619 DEBUG:boto:Data: 2023-10-12T00:55:59.619 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.620 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.620 DEBUG:boto:Params: {} 2023-10-12T00:55:59.621 DEBUG:boto:Token: None 2023-10-12T00:55:59.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d 2023-10-12T00:55:59.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J3K9ZoyRRdKocuRdz+i6/dED4AQ= 2023-10-12T00:55:59.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J3K9ZoyRRdKocuRdz+i6/dED4AQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013e0ec11d7d49b6-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.625 DEBUG:boto:path=/wafmoa-57/d 2023-10-12T00:55:59.625 DEBUG:boto:auth_path=/wafmoa-57/d 2023-10-12T00:55:59.626 DEBUG:boto:Method: GET 2023-10-12T00:55:59.626 DEBUG:boto:Path: /wafmoa-57/d 2023-10-12T00:55:59.627 DEBUG:boto:Data: 2023-10-12T00:55:59.627 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.629 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.629 DEBUG:boto:Params: {} 2023-10-12T00:55:59.630 DEBUG:boto:Token: None 2023-10-12T00:55:59.630 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/d 2023-10-12T00:55:59.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J3K9ZoyRRdKocuRdz+i6/dED4AQ= 2023-10-12T00:55:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J3K9ZoyRRdKocuRdz+i6/dED4AQ=', 'Content-Length': '0'} 2023-10-12T00:55:59.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a628ab76d795270-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.634 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.634 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.635 DEBUG:boto:path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.635 DEBUG:boto:auth_path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.636 DEBUG:boto:Method: GET 2023-10-12T00:55:59.636 DEBUG:boto:Path: /wafmoa-57/?prefix=d 2023-10-12T00:55:59.636 DEBUG:boto:Data: 2023-10-12T00:55:59.637 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.638 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.638 DEBUG:boto:Params: {} 2023-10-12T00:55:59.639 DEBUG:boto:Token: None 2023-10-12T00:55:59.639 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c985740d8b4473-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.643 DEBUG:boto:b'wafmoa-57d1000falsed2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.644 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.644 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.644 DEBUG:boto:path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.645 DEBUG:boto:auth_path=/wafmoa-57/?prefix=d 2023-10-12T00:55:59.645 DEBUG:boto:Method: GET 2023-10-12T00:55:59.646 DEBUG:boto:Path: /wafmoa-57/?prefix=d 2023-10-12T00:55:59.646 DEBUG:boto:Data: 2023-10-12T00:55:59.647 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.647 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.648 DEBUG:boto:Params: {} 2023-10-12T00:55:59.648 DEBUG:boto:Token: None 2023-10-12T00:55:59.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2a711afc284097-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.652 DEBUG:boto:b'wafmoa-57d1000falsed2023-10-12T00:52:32.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:55:59.653 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.654 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.654 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.655 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.655 DEBUG:boto:Method: GET 2023-10-12T00:55:59.655 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.656 DEBUG:boto:Data: 2023-10-12T00:55:59.656 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.657 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.658 DEBUG:boto:Params: {} 2023-10-12T00:55:59.658 DEBUG:boto:Token: None 2023-10-12T00:55:59.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc= 2023-10-12T00:55:59.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc=', 'Content-Length': '0'} 2023-10-12T00:55:59.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea58a35299362fdc-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.662 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.663 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.663 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.663 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.664 DEBUG:boto:Method: GET 2023-10-12T00:55:59.664 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.665 DEBUG:boto:Data: 2023-10-12T00:55:59.665 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.666 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.666 DEBUG:boto:Params: {} 2023-10-12T00:55:59.667 DEBUG:boto:Token: None 2023-10-12T00:55:59.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc= 2023-10-12T00:55:59.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc=', 'Content-Length': '0'} 2023-10-12T00:55:59.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e339770e16cb23-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.671 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.671 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.672 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.672 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.672 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.673 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.673 DEBUG:boto:Data: 2023-10-12T00:55:59.674 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.674 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.675 DEBUG:boto:Params: {} 2023-10-12T00:55:59.675 DEBUG:boto:Token: None 2023-10-12T00:55:59.676 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+qI5C9AtYbQwXvijDxRLanVXDP8= 2023-10-12T00:55:59.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+qI5C9AtYbQwXvijDxRLanVXDP8=', 'Content-Length': '0'} 2023-10-12T00:55:59.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9f4e851f3ccd37-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.679 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.680 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.680 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.680 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.681 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.681 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.682 DEBUG:boto:Data: 2023-10-12T00:55:59.682 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.683 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.683 DEBUG:boto:Params: {} 2023-10-12T00:55:59.684 DEBUG:boto:Token: None 2023-10-12T00:55:59.684 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.685 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+qI5C9AtYbQwXvijDxRLanVXDP8= 2023-10-12T00:55:59.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+qI5C9AtYbQwXvijDxRLanVXDP8=', 'Content-Length': '0'} 2023-10-12T00:55:59.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f18f2d7d17ade70b-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:55:59.688 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.688 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.689 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.689 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.690 DEBUG:boto:Method: GET 2023-10-12T00:55:59.690 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.690 DEBUG:boto:Data: 2023-10-12T00:55:59.691 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.692 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.692 DEBUG:boto:Params: {} 2023-10-12T00:55:59.693 DEBUG:boto:Token: None 2023-10-12T00:55:59.693 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc= 2023-10-12T00:55:59.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc=', 'Content-Length': '0'} 2023-10-12T00:55:59.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c923c9f174d90c0-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.738 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.739 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.739 DEBUG:boto:path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.740 DEBUG:boto:auth_path=/wafmoa-57/e?versionId=null 2023-10-12T00:55:59.740 DEBUG:boto:Method: GET 2023-10-12T00:55:59.741 DEBUG:boto:Path: /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.741 DEBUG:boto:Data: 2023-10-12T00:55:59.742 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.743 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.743 DEBUG:boto:Params: {} 2023-10-12T00:55:59.743 DEBUG:boto:Token: None 2023-10-12T00:55:59.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e?versionId=null 2023-10-12T00:55:59.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc= 2023-10-12T00:55:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/LQZNovDOddVYlYOB2CNPZUMivc=', 'Content-Length': '0'} 2023-10-12T00:55:59.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087eff97f180f44d-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.748 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.748 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.749 DEBUG:boto:path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.749 DEBUG:boto:auth_path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.750 DEBUG:boto:Method: GET 2023-10-12T00:55:59.750 DEBUG:boto:Path: /wafmoa-57/?prefix=e 2023-10-12T00:55:59.750 DEBUG:boto:Data: 2023-10-12T00:55:59.751 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.752 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.752 DEBUG:boto:Params: {} 2023-10-12T00:55:59.753 DEBUG:boto:Token: None 2023-10-12T00:55:59.753 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000112050fe7cc9f848-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.757 DEBUG:boto:b'wafmoa-57e1000falsee2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.758 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.759 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.759 DEBUG:boto:path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.759 DEBUG:boto:auth_path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.760 DEBUG:boto:Method: GET 2023-10-12T00:55:59.760 DEBUG:boto:Path: /wafmoa-57/?prefix=e 2023-10-12T00:55:59.761 DEBUG:boto:Data: 2023-10-12T00:55:59.761 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.762 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.762 DEBUG:boto:Params: {} 2023-10-12T00:55:59.763 DEBUG:boto:Token: None 2023-10-12T00:55:59.763 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deca2214d5e20e15-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.810 DEBUG:boto:b'wafmoa-57e1000falsee2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.811 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.812 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.812 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.813 DEBUG:boto:Path: /wafmoa-57/e 2023-10-12T00:55:59.813 DEBUG:boto:Data: 2023-10-12T00:55:59.813 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.814 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.815 DEBUG:boto:Params: {} 2023-10-12T00:55:59.816 DEBUG:boto:Token: None 2023-10-12T00:55:59.816 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e 2023-10-12T00:55:59.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dVbwtKE4VWEsVDQjoh6vnrbJAgU= 2023-10-12T00:55:59.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dVbwtKE4VWEsVDQjoh6vnrbJAgU=', 'Content-Length': '0'} 2023-10-12T00:55:59.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089fff99760b90563-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.820 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.820 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.820 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.821 DEBUG:boto:Path: /wafmoa-57/e 2023-10-12T00:55:59.821 DEBUG:boto:Data: 2023-10-12T00:55:59.822 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.823 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.823 DEBUG:boto:Params: {} 2023-10-12T00:55:59.823 DEBUG:boto:Token: None 2023-10-12T00:55:59.824 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e 2023-10-12T00:55:59.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dVbwtKE4VWEsVDQjoh6vnrbJAgU= 2023-10-12T00:55:59.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dVbwtKE4VWEsVDQjoh6vnrbJAgU=', 'Content-Length': '0'} 2023-10-12T00:55:59.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f32d30e06b1cf58-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-12T00:55:59.828 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.828 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.829 DEBUG:boto:Method: GET 2023-10-12T00:55:59.829 DEBUG:boto:Path: /wafmoa-57/e 2023-10-12T00:55:59.829 DEBUG:boto:Data: 2023-10-12T00:55:59.830 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.831 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.834 DEBUG:boto:Params: {} 2023-10-12T00:55:59.835 DEBUG:boto:Token: None 2023-10-12T00:55:59.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e 2023-10-12T00:55:59.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vuSbLrVBproud/n9CwmDCC/tRNE= 2023-10-12T00:55:59.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vuSbLrVBproud/n9CwmDCC/tRNE=', 'Content-Length': '0'} 2023-10-12T00:55:59.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f823f0121e406bdd-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.839 DEBUG:boto:path=/wafmoa-57/e 2023-10-12T00:55:59.839 DEBUG:boto:auth_path=/wafmoa-57/e 2023-10-12T00:55:59.840 DEBUG:boto:Method: GET 2023-10-12T00:55:59.840 DEBUG:boto:Path: /wafmoa-57/e 2023-10-12T00:55:59.841 DEBUG:boto:Data: 2023-10-12T00:55:59.841 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.842 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.843 DEBUG:boto:Params: {} 2023-10-12T00:55:59.843 DEBUG:boto:Token: None 2023-10-12T00:55:59.843 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/e 2023-10-12T00:55:59.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vuSbLrVBproud/n9CwmDCC/tRNE= 2023-10-12T00:55:59.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vuSbLrVBproud/n9CwmDCC/tRNE=', 'Content-Length': '0'} 2023-10-12T00:55:59.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d99533ea402905-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.847 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.847 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.848 DEBUG:boto:path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.848 DEBUG:boto:auth_path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.848 DEBUG:boto:Method: GET 2023-10-12T00:55:59.849 DEBUG:boto:Path: /wafmoa-57/?prefix=e 2023-10-12T00:55:59.849 DEBUG:boto:Data: 2023-10-12T00:55:59.850 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.851 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.851 DEBUG:boto:Params: {} 2023-10-12T00:55:59.851 DEBUG:boto:Token: None 2023-10-12T00:55:59.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edee38602db931f0-006527441f-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.902 DEBUG:boto:b'wafmoa-57e1000falsee2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.903 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:55:59.904 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:55:59.904 DEBUG:boto:path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.905 DEBUG:boto:auth_path=/wafmoa-57/?prefix=e 2023-10-12T00:55:59.905 DEBUG:boto:Method: GET 2023-10-12T00:55:59.905 DEBUG:boto:Path: /wafmoa-57/?prefix=e 2023-10-12T00:55:59.906 DEBUG:boto:Data: 2023-10-12T00:55:59.906 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.907 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.907 DEBUG:boto:Params: {} 2023-10-12T00:55:59.908 DEBUG:boto:Token: None 2023-10-12T00:55:59.908 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/ 2023-10-12T00:55:59.909 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU= 2023-10-12T00:55:59.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:19fjSsgvlkWtA3i1c+Jssri2/CU=', 'Content-Length': '0'} 2023-10-12T00:55:59.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c518fe06c68ec5-006527441f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.912 DEBUG:boto:b'wafmoa-57e1000falsee2023-10-12T00:52:32.313Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:55:59.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:55:59.913 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.914 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.914 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.914 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.915 DEBUG:boto:Method: GET 2023-10-12T00:55:59.915 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.916 DEBUG:boto:Data: 2023-10-12T00:55:59.916 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.917 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.917 DEBUG:boto:Params: {} 2023-10-12T00:55:59.918 DEBUG:boto:Token: None 2023-10-12T00:55:59.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM= 2023-10-12T00:55:59.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM=', 'Content-Length': '0'} 2023-10-12T00:55:59.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83651dfcb9cb904-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.922 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.922 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.923 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.923 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.924 DEBUG:boto:Method: GET 2023-10-12T00:55:59.924 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.925 DEBUG:boto:Data: 2023-10-12T00:55:59.925 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.926 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.926 DEBUG:boto:Params: {} 2023-10-12T00:55:59.927 DEBUG:boto:Token: None 2023-10-12T00:55:59.927 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM= 2023-10-12T00:55:59.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM=', 'Content-Length': '0'} 2023-10-12T00:55:59.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087253d471f2433d-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.931 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.931 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.932 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.932 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.932 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.933 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.933 DEBUG:boto:Data: 2023-10-12T00:55:59.934 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.934 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.935 DEBUG:boto:Params: {} 2023-10-12T00:55:59.935 DEBUG:boto:Token: None 2023-10-12T00:55:59.936 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TxrKfk8IowBZYbaOBdfyq7cWDlY= 2023-10-12T00:55:59.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TxrKfk8IowBZYbaOBdfyq7cWDlY=', 'Content-Length': '0'} 2023-10-12T00:55:59.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619a34461f549a97-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.939 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.939 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.940 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.940 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.940 DEBUG:boto:Method: HEAD 2023-10-12T00:55:59.941 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.941 DEBUG:boto:Data: 2023-10-12T00:55:59.942 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.943 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.943 DEBUG:boto:Params: {} 2023-10-12T00:55:59.943 DEBUG:boto:Token: None 2023-10-12T00:55:59.944 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TxrKfk8IowBZYbaOBdfyq7cWDlY= 2023-10-12T00:55:59.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TxrKfk8IowBZYbaOBdfyq7cWDlY=', 'Content-Length': '0'} 2023-10-12T00:55:59.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8edad3654744ab-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:55:59.948 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.948 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.949 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.949 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.949 DEBUG:boto:Method: GET 2023-10-12T00:55:59.950 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.950 DEBUG:boto:Data: 2023-10-12T00:55:59.951 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:55:59.952 DEBUG:boto:Port: 8001 2023-10-12T00:55:59.952 DEBUG:boto:Params: {} 2023-10-12T00:55:59.953 DEBUG:boto:Token: None 2023-10-12T00:55:59.953 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM= 2023-10-12T00:55:59.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM=', 'Content-Length': '0'} 2023-10-12T00:55:59.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e054a69c2b5f72-006527441f-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:55:59.957 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:55:59.957 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:55:59.958 DEBUG:boto:path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.958 DEBUG:boto:auth_path=/wafmoa-57/f?versionId=null 2023-10-12T00:55:59.958 DEBUG:boto:Method: GET 2023-10-12T00:55:59.959 DEBUG:boto:Path: /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.959 DEBUG:boto:Data: 2023-10-12T00:55:59.960 DEBUG:boto:Headers: {} 2023-10-12T00:55:59.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:55:59.961 DEBUG:boto:Port: 8000 2023-10-12T00:55:59.961 DEBUG:boto:Params: {} 2023-10-12T00:55:59.962 DEBUG:boto:Token: None 2023-10-12T00:55:59.962 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:55:59 GMT /wafmoa-57/f?versionId=null 2023-10-12T00:55:59.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM= 2023-10-12T00:55:59.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:55:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jHjFyJhnqBxpSdnOi33xia+l6nM=', 'Content-Length': '0'} 2023-10-12T00:55:59.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3b11a7a11c2230-006527441f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:55:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.006 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.007 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.007 DEBUG:boto:path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.008 DEBUG:boto:auth_path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.008 DEBUG:boto:Method: GET 2023-10-12T00:56:00.009 DEBUG:boto:Path: /wafmoa-57/?prefix=f 2023-10-12T00:56:00.009 DEBUG:boto:Data: 2023-10-12T00:56:00.009 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.010 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.011 DEBUG:boto:Params: {} 2023-10-12T00:56:00.011 DEBUG:boto:Token: None 2023-10-12T00:56:00.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5f5e3c3ecd568a-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.016 DEBUG:boto:b'wafmoa-57f1000falsef2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.016 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.017 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.017 DEBUG:boto:path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.018 DEBUG:boto:auth_path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.018 DEBUG:boto:Method: GET 2023-10-12T00:56:00.019 DEBUG:boto:Path: /wafmoa-57/?prefix=f 2023-10-12T00:56:00.019 DEBUG:boto:Data: 2023-10-12T00:56:00.019 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.020 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.021 DEBUG:boto:Params: {} 2023-10-12T00:56:00.021 DEBUG:boto:Token: None 2023-10-12T00:56:00.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004375360f7a8cdac5-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.025 DEBUG:boto:b'wafmoa-57f1000falsef2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.026 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:56:00.026 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:56:00.027 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.027 DEBUG:boto:Path: /wafmoa-57/f 2023-10-12T00:56:00.028 DEBUG:boto:Data: 2023-10-12T00:56:00.028 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.029 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.029 DEBUG:boto:Params: {} 2023-10-12T00:56:00.030 DEBUG:boto:Token: None 2023-10-12T00:56:00.030 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/f 2023-10-12T00:56:00.031 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Za5bMabAcxvTWFU+ZTOuHzhgo10= 2023-10-12T00:56:00.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Za5bMabAcxvTWFU+ZTOuHzhgo10=', 'Content-Length': '0'} 2023-10-12T00:56:00.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7cc15a0f0136370-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.034 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:56:00.034 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:56:00.035 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.035 DEBUG:boto:Path: /wafmoa-57/f 2023-10-12T00:56:00.035 DEBUG:boto:Data: 2023-10-12T00:56:00.038 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.038 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.039 DEBUG:boto:Params: {} 2023-10-12T00:56:00.039 DEBUG:boto:Token: None 2023-10-12T00:56:00.040 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/f 2023-10-12T00:56:00.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Za5bMabAcxvTWFU+ZTOuHzhgo10= 2023-10-12T00:56:00.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Za5bMabAcxvTWFU+ZTOuHzhgo10=', 'Content-Length': '0'} 2023-10-12T00:56:00.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a563aafc4f5117ad-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-12T00:56:00.044 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:56:00.044 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:56:00.045 DEBUG:boto:Method: GET 2023-10-12T00:56:00.045 DEBUG:boto:Path: /wafmoa-57/f 2023-10-12T00:56:00.046 DEBUG:boto:Data: 2023-10-12T00:56:00.046 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.047 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.047 DEBUG:boto:Params: {} 2023-10-12T00:56:00.048 DEBUG:boto:Token: None 2023-10-12T00:56:00.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/f 2023-10-12T00:56:00.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8YMkpsdzkkIeL6P6CnJfKuIvw8E= 2023-10-12T00:56:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8YMkpsdzkkIeL6P6CnJfKuIvw8E=', 'Content-Length': '0'} 2023-10-12T00:56:00.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9135fba49af671-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.052 DEBUG:boto:path=/wafmoa-57/f 2023-10-12T00:56:00.052 DEBUG:boto:auth_path=/wafmoa-57/f 2023-10-12T00:56:00.052 DEBUG:boto:Method: GET 2023-10-12T00:56:00.053 DEBUG:boto:Path: /wafmoa-57/f 2023-10-12T00:56:00.053 DEBUG:boto:Data: 2023-10-12T00:56:00.054 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.055 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.055 DEBUG:boto:Params: {} 2023-10-12T00:56:00.055 DEBUG:boto:Token: None 2023-10-12T00:56:00.056 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/f 2023-10-12T00:56:00.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8YMkpsdzkkIeL6P6CnJfKuIvw8E= 2023-10-12T00:56:00.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8YMkpsdzkkIeL6P6CnJfKuIvw8E=', 'Content-Length': '0'} 2023-10-12T00:56:00.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbbc988dc6b23c3-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.060 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.060 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.061 DEBUG:boto:path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.061 DEBUG:boto:auth_path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.062 DEBUG:boto:Method: GET 2023-10-12T00:56:00.062 DEBUG:boto:Path: /wafmoa-57/?prefix=f 2023-10-12T00:56:00.062 DEBUG:boto:Data: 2023-10-12T00:56:00.063 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.064 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.064 DEBUG:boto:Params: {} 2023-10-12T00:56:00.065 DEBUG:boto:Token: None 2023-10-12T00:56:00.065 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10833783eb901a5-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.110 DEBUG:boto:b'wafmoa-57f1000falsef2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.111 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.112 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.112 DEBUG:boto:path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.112 DEBUG:boto:auth_path=/wafmoa-57/?prefix=f 2023-10-12T00:56:00.113 DEBUG:boto:Method: GET 2023-10-12T00:56:00.113 DEBUG:boto:Path: /wafmoa-57/?prefix=f 2023-10-12T00:56:00.114 DEBUG:boto:Data: 2023-10-12T00:56:00.114 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.115 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.116 DEBUG:boto:Params: {} 2023-10-12T00:56:00.116 DEBUG:boto:Token: None 2023-10-12T00:56:00.117 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f90b0caaa4dee3-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.120 DEBUG:boto:b'wafmoa-57f1000falsef2023-10-12T00:52:32.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:56:00.121 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.122 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.122 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.123 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.123 DEBUG:boto:Method: GET 2023-10-12T00:56:00.124 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.124 DEBUG:boto:Data: 2023-10-12T00:56:00.124 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.125 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.126 DEBUG:boto:Params: {} 2023-10-12T00:56:00.126 DEBUG:boto:Token: None 2023-10-12T00:56:00.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8= 2023-10-12T00:56:00.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8=', 'Content-Length': '0'} 2023-10-12T00:56:00.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a662650c7229115-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.130 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.131 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.131 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.131 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.132 DEBUG:boto:Method: GET 2023-10-12T00:56:00.132 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.133 DEBUG:boto:Data: 2023-10-12T00:56:00.133 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.134 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.134 DEBUG:boto:Params: {} 2023-10-12T00:56:00.135 DEBUG:boto:Token: None 2023-10-12T00:56:00.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8= 2023-10-12T00:56:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8=', 'Content-Length': '0'} 2023-10-12T00:56:00.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b3a626c9f507b1-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.139 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.140 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.140 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.140 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.141 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.141 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.142 DEBUG:boto:Data: 2023-10-12T00:56:00.142 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.143 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.143 DEBUG:boto:Params: {} 2023-10-12T00:56:00.144 DEBUG:boto:Token: None 2023-10-12T00:56:00.144 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7mhztsrWCAIaEYiosxiM1HHT0LQ= 2023-10-12T00:56:00.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7mhztsrWCAIaEYiosxiM1HHT0LQ=', 'Content-Length': '0'} 2023-10-12T00:56:00.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009328666fb61b4edd-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.147 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.148 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.148 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.148 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.149 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.149 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.150 DEBUG:boto:Data: 2023-10-12T00:56:00.150 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.151 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.151 DEBUG:boto:Params: {} 2023-10-12T00:56:00.152 DEBUG:boto:Token: None 2023-10-12T00:56:00.152 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7mhztsrWCAIaEYiosxiM1HHT0LQ= 2023-10-12T00:56:00.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7mhztsrWCAIaEYiosxiM1HHT0LQ=', 'Content-Length': '0'} 2023-10-12T00:56:00.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4350f989e77e65-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:56:00.156 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.156 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.157 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.157 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.157 DEBUG:boto:Method: GET 2023-10-12T00:56:00.158 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.158 DEBUG:boto:Data: 2023-10-12T00:56:00.159 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.160 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.160 DEBUG:boto:Params: {} 2023-10-12T00:56:00.160 DEBUG:boto:Token: None 2023-10-12T00:56:00.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8= 2023-10-12T00:56:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8=', 'Content-Length': '0'} 2023-10-12T00:56:00.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea9fe609a4fee9d-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.164 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.165 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.165 DEBUG:boto:path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.166 DEBUG:boto:auth_path=/wafmoa-57/g?versionId=null 2023-10-12T00:56:00.166 DEBUG:boto:Method: GET 2023-10-12T00:56:00.166 DEBUG:boto:Path: /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.167 DEBUG:boto:Data: 2023-10-12T00:56:00.167 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.168 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.169 DEBUG:boto:Params: {} 2023-10-12T00:56:00.169 DEBUG:boto:Token: None 2023-10-12T00:56:00.170 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g?versionId=null 2023-10-12T00:56:00.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8= 2023-10-12T00:56:00.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0QDH8qMG9M3FrHbgZZgZiwjTrJ8=', 'Content-Length': '0'} 2023-10-12T00:56:00.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9c6cd71ce9c7ae-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.214 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.216 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.216 DEBUG:boto:path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.217 DEBUG:boto:auth_path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.218 DEBUG:boto:Method: GET 2023-10-12T00:56:00.219 DEBUG:boto:Path: /wafmoa-57/?prefix=g 2023-10-12T00:56:00.220 DEBUG:boto:Data: 2023-10-12T00:56:00.220 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.221 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.222 DEBUG:boto:Params: {} 2023-10-12T00:56:00.222 DEBUG:boto:Token: None 2023-10-12T00:56:00.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000184a8f3b2cd93444-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.227 DEBUG:boto:b'wafmoa-57g1000falseg2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.228 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.228 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.229 DEBUG:boto:path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.230 DEBUG:boto:auth_path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.231 DEBUG:boto:Method: GET 2023-10-12T00:56:00.231 DEBUG:boto:Path: /wafmoa-57/?prefix=g 2023-10-12T00:56:00.232 DEBUG:boto:Data: 2023-10-12T00:56:00.233 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.234 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.234 DEBUG:boto:Params: {} 2023-10-12T00:56:00.235 DEBUG:boto:Token: None 2023-10-12T00:56:00.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2bf40ab0963fd8f-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.238 DEBUG:boto:b'wafmoa-57g1000falseg2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.239 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.240 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.240 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.241 DEBUG:boto:Path: /wafmoa-57/g 2023-10-12T00:56:00.241 DEBUG:boto:Data: 2023-10-12T00:56:00.241 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.242 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.243 DEBUG:boto:Params: {} 2023-10-12T00:56:00.243 DEBUG:boto:Token: None 2023-10-12T00:56:00.244 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g 2023-10-12T00:56:00.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7cKk5k3eVSqxHBf47hB+3lWsUMA= 2023-10-12T00:56:00.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7cKk5k3eVSqxHBf47hB+3lWsUMA=', 'Content-Length': '0'} 2023-10-12T00:56:00.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f225c3d75225ad4f-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.247 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.247 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.248 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.248 DEBUG:boto:Path: /wafmoa-57/g 2023-10-12T00:56:00.248 DEBUG:boto:Data: 2023-10-12T00:56:00.249 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.250 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.250 DEBUG:boto:Params: {} 2023-10-12T00:56:00.251 DEBUG:boto:Token: None 2023-10-12T00:56:00.251 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g 2023-10-12T00:56:00.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7cKk5k3eVSqxHBf47hB+3lWsUMA= 2023-10-12T00:56:00.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7cKk5k3eVSqxHBf47hB+3lWsUMA=', 'Content-Length': '0'} 2023-10-12T00:56:00.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005336ad780893e8a3-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-12T00:56:00.255 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.255 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.256 DEBUG:boto:Method: GET 2023-10-12T00:56:00.256 DEBUG:boto:Path: /wafmoa-57/g 2023-10-12T00:56:00.257 DEBUG:boto:Data: 2023-10-12T00:56:00.257 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.258 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.259 DEBUG:boto:Params: {} 2023-10-12T00:56:00.259 DEBUG:boto:Token: None 2023-10-12T00:56:00.260 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g 2023-10-12T00:56:00.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7NriVEW1DldqGHD5hQBME8cXNA= 2023-10-12T00:56:00.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7NriVEW1DldqGHD5hQBME8cXNA=', 'Content-Length': '0'} 2023-10-12T00:56:00.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4516c746a48111-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.310 DEBUG:boto:path=/wafmoa-57/g 2023-10-12T00:56:00.311 DEBUG:boto:auth_path=/wafmoa-57/g 2023-10-12T00:56:00.311 DEBUG:boto:Method: GET 2023-10-12T00:56:00.312 DEBUG:boto:Path: /wafmoa-57/g 2023-10-12T00:56:00.312 DEBUG:boto:Data: 2023-10-12T00:56:00.313 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.314 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.314 DEBUG:boto:Params: {} 2023-10-12T00:56:00.315 DEBUG:boto:Token: None 2023-10-12T00:56:00.315 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/g 2023-10-12T00:56:00.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7NriVEW1DldqGHD5hQBME8cXNA= 2023-10-12T00:56:00.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7NriVEW1DldqGHD5hQBME8cXNA=', 'Content-Length': '0'} 2023-10-12T00:56:00.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6f5fd26a3d2a76-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.319 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.319 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.320 DEBUG:boto:path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.320 DEBUG:boto:auth_path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.321 DEBUG:boto:Method: GET 2023-10-12T00:56:00.321 DEBUG:boto:Path: /wafmoa-57/?prefix=g 2023-10-12T00:56:00.322 DEBUG:boto:Data: 2023-10-12T00:56:00.322 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.323 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.324 DEBUG:boto:Params: {} 2023-10-12T00:56:00.324 DEBUG:boto:Token: None 2023-10-12T00:56:00.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c80283d01a30e7c5-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.330 DEBUG:boto:b'wafmoa-57g1000falseg2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.331 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.332 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.332 DEBUG:boto:path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.332 DEBUG:boto:auth_path=/wafmoa-57/?prefix=g 2023-10-12T00:56:00.333 DEBUG:boto:Method: GET 2023-10-12T00:56:00.333 DEBUG:boto:Path: /wafmoa-57/?prefix=g 2023-10-12T00:56:00.334 DEBUG:boto:Data: 2023-10-12T00:56:00.334 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.335 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.336 DEBUG:boto:Params: {} 2023-10-12T00:56:00.336 DEBUG:boto:Token: None 2023-10-12T00:56:00.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7fc5f0553cbaa8-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.341 DEBUG:boto:b'wafmoa-57g1000falseg2023-10-12T00:52:32.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:56:00.342 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.342 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.343 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.343 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.343 DEBUG:boto:Method: GET 2023-10-12T00:56:00.344 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.344 DEBUG:boto:Data: 2023-10-12T00:56:00.345 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.346 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.346 DEBUG:boto:Params: {} 2023-10-12T00:56:00.347 DEBUG:boto:Token: None 2023-10-12T00:56:00.347 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw= 2023-10-12T00:56:00.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw=', 'Content-Length': '0'} 2023-10-12T00:56:00.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920c47670018ba1f-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.351 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.351 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.351 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.352 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.352 DEBUG:boto:Method: GET 2023-10-12T00:56:00.353 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.353 DEBUG:boto:Data: 2023-10-12T00:56:00.353 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.354 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.355 DEBUG:boto:Params: {} 2023-10-12T00:56:00.355 DEBUG:boto:Token: None 2023-10-12T00:56:00.356 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw= 2023-10-12T00:56:00.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw=', 'Content-Length': '0'} 2023-10-12T00:56:00.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eaeb2ec640f710b-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.359 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.360 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.360 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.361 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.361 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.361 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.362 DEBUG:boto:Data: 2023-10-12T00:56:00.362 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.363 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.363 DEBUG:boto:Params: {} 2023-10-12T00:56:00.364 DEBUG:boto:Token: None 2023-10-12T00:56:00.364 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GhF6KyGsb1AKFMQoWw31NlXUsU= 2023-10-12T00:56:00.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GhF6KyGsb1AKFMQoWw31NlXUsU=', 'Content-Length': '0'} 2023-10-12T00:56:00.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c29a2b5224bed3cd-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.368 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.368 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.369 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.369 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.369 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.370 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.370 DEBUG:boto:Data: 2023-10-12T00:56:00.371 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.371 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.372 DEBUG:boto:Params: {} 2023-10-12T00:56:00.372 DEBUG:boto:Token: None 2023-10-12T00:56:00.373 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GhF6KyGsb1AKFMQoWw31NlXUsU= 2023-10-12T00:56:00.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GhF6KyGsb1AKFMQoWw31NlXUsU=', 'Content-Length': '0'} 2023-10-12T00:56:00.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33f4ead35924bb4-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:56:00.377 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.377 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.378 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.378 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.378 DEBUG:boto:Method: GET 2023-10-12T00:56:00.379 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.379 DEBUG:boto:Data: 2023-10-12T00:56:00.380 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.381 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.381 DEBUG:boto:Params: {} 2023-10-12T00:56:00.381 DEBUG:boto:Token: None 2023-10-12T00:56:00.382 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw= 2023-10-12T00:56:00.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw=', 'Content-Length': '0'} 2023-10-12T00:56:00.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b9938f44270a6ab-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.385 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.386 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.386 DEBUG:boto:path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.387 DEBUG:boto:auth_path=/wafmoa-57/h?versionId=null 2023-10-12T00:56:00.387 DEBUG:boto:Method: GET 2023-10-12T00:56:00.388 DEBUG:boto:Path: /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.388 DEBUG:boto:Data: 2023-10-12T00:56:00.389 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.389 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.390 DEBUG:boto:Params: {} 2023-10-12T00:56:00.390 DEBUG:boto:Token: None 2023-10-12T00:56:00.391 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h?versionId=null 2023-10-12T00:56:00.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw= 2023-10-12T00:56:00.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sYDkpboVlISuW7DbpmFP9WnY+yw=', 'Content-Length': '0'} 2023-10-12T00:56:00.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3198451343081d-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.434 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.435 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.435 DEBUG:boto:path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.436 DEBUG:boto:auth_path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.436 DEBUG:boto:Method: GET 2023-10-12T00:56:00.437 DEBUG:boto:Path: /wafmoa-57/?prefix=h 2023-10-12T00:56:00.437 DEBUG:boto:Data: 2023-10-12T00:56:00.438 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.439 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.439 DEBUG:boto:Params: {} 2023-10-12T00:56:00.440 DEBUG:boto:Token: None 2023-10-12T00:56:00.440 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f921617c189c181-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.445 DEBUG:boto:b'wafmoa-57h1000falseh2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.446 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.446 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.446 DEBUG:boto:path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.450 DEBUG:boto:auth_path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.451 DEBUG:boto:Method: GET 2023-10-12T00:56:00.451 DEBUG:boto:Path: /wafmoa-57/?prefix=h 2023-10-12T00:56:00.452 DEBUG:boto:Data: 2023-10-12T00:56:00.452 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.453 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.453 DEBUG:boto:Params: {} 2023-10-12T00:56:00.454 DEBUG:boto:Token: None 2023-10-12T00:56:00.454 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c528a72ecb64389-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.458 DEBUG:boto:b'wafmoa-57h1000falseh2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.458 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.459 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.459 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.460 DEBUG:boto:Path: /wafmoa-57/h 2023-10-12T00:56:00.460 DEBUG:boto:Data: 2023-10-12T00:56:00.461 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.461 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.462 DEBUG:boto:Params: {} 2023-10-12T00:56:00.462 DEBUG:boto:Token: None 2023-10-12T00:56:00.463 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h 2023-10-12T00:56:00.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PznerYWt3kvqp3SL+aFNM3ZTZ7U= 2023-10-12T00:56:00.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PznerYWt3kvqp3SL+aFNM3ZTZ7U=', 'Content-Length': '0'} 2023-10-12T00:56:00.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f200a42c4bdc7b-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.466 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.466 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.467 DEBUG:boto:Method: HEAD 2023-10-12T00:56:00.467 DEBUG:boto:Path: /wafmoa-57/h 2023-10-12T00:56:00.468 DEBUG:boto:Data: 2023-10-12T00:56:00.468 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.469 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.469 DEBUG:boto:Params: {} 2023-10-12T00:56:00.470 DEBUG:boto:Token: None 2023-10-12T00:56:00.470 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h 2023-10-12T00:56:00.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PznerYWt3kvqp3SL+aFNM3ZTZ7U= 2023-10-12T00:56:00.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PznerYWt3kvqp3SL+aFNM3ZTZ7U=', 'Content-Length': '0'} 2023-10-12T00:56:00.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7322b69a0bb58b0-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-12T00:56:00.474 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.475 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.475 DEBUG:boto:Method: GET 2023-10-12T00:56:00.475 DEBUG:boto:Path: /wafmoa-57/h 2023-10-12T00:56:00.476 DEBUG:boto:Data: 2023-10-12T00:56:00.476 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.477 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.477 DEBUG:boto:Params: {} 2023-10-12T00:56:00.478 DEBUG:boto:Token: None 2023-10-12T00:56:00.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h 2023-10-12T00:56:00.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EmWH5MBSf7KPx7tFL50cSd3lY50= 2023-10-12T00:56:00.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EmWH5MBSf7KPx7tFL50cSd3lY50=', 'Content-Length': '0'} 2023-10-12T00:56:00.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-57:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a04013c1202d65-0065274420-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.482 DEBUG:boto:path=/wafmoa-57/h 2023-10-12T00:56:00.482 DEBUG:boto:auth_path=/wafmoa-57/h 2023-10-12T00:56:00.483 DEBUG:boto:Method: GET 2023-10-12T00:56:00.483 DEBUG:boto:Path: /wafmoa-57/h 2023-10-12T00:56:00.484 DEBUG:boto:Data: 2023-10-12T00:56:00.484 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.485 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.485 DEBUG:boto:Params: {} 2023-10-12T00:56:00.486 DEBUG:boto:Token: None 2023-10-12T00:56:00.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/h 2023-10-12T00:56:00.487 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EmWH5MBSf7KPx7tFL50cSd3lY50= 2023-10-12T00:56:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EmWH5MBSf7KPx7tFL50cSd3lY50=', 'Content-Length': '0'} 2023-10-12T00:56:00.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:52:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65c344ae56c46ec-0065274420-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.490 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.490 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.491 DEBUG:boto:path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.491 DEBUG:boto:auth_path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.492 DEBUG:boto:Method: GET 2023-10-12T00:56:00.492 DEBUG:boto:Path: /wafmoa-57/?prefix=h 2023-10-12T00:56:00.492 DEBUG:boto:Data: 2023-10-12T00:56:00.493 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.494 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.494 DEBUG:boto:Params: {} 2023-10-12T00:56:00.495 DEBUG:boto:Token: None 2023-10-12T00:56:00.495 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c594fcd8da773b3c-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.499 DEBUG:boto:b'wafmoa-57h1000falseh2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.500 DEBUG:boto:path=/wafmoa-57/ 2023-10-12T00:56:00.500 DEBUG:boto:auth_path=/wafmoa-57/ 2023-10-12T00:56:00.501 DEBUG:boto:path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.501 DEBUG:boto:auth_path=/wafmoa-57/?prefix=h 2023-10-12T00:56:00.502 DEBUG:boto:Method: GET 2023-10-12T00:56:00.502 DEBUG:boto:Path: /wafmoa-57/?prefix=h 2023-10-12T00:56:00.503 DEBUG:boto:Data: 2023-10-12T00:56:00.503 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.504 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.504 DEBUG:boto:Params: {} 2023-10-12T00:56:00.505 DEBUG:boto:Token: None 2023-10-12T00:56:00.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-57/ 2023-10-12T00:56:00.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE= 2023-10-12T00:56:00.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tjzaDKqmWZShR3azoy09mWo9ZWE=', 'Content-Length': '0'} 2023-10-12T00:56:00.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d16685d9fe30721-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.509 DEBUG:boto:b'wafmoa-57h1000falseh2023-10-12T00:52:32.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:00.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-57 zones={a2, a1} 2023-10-12T00:56:00.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-10-12T00:56:00.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.512 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:00.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:00.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:00.589 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:56:00.601 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:56:00.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:00.602 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:56:00.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:00.603 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:56:00.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071952.264425_569.1", 2023-10-12T00:56:00.603 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:52:32.264425Z" 2023-10-12T00:56:00.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:00.604 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:56:00.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.605 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:56:00.606 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071952.264425_569.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.606 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:56:00.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:00.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:00.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:00.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:00.685+0000 7fc3ec8a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:00.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:00.686+0000 7fc3ec8a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:00.699 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:56:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:56:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:56:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:56:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:56:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:56:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:56:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:56:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:56:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:56:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:56:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:56:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071952.264425_569.1", 2023-10-12T00:56:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:52:32.264425Z", 2023-10-12T00:56:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:56:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:56:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:56:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:56:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:56:00Z" 2023-10-12T00:56:00.713 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:00.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071952.264425_569.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:52:32.264425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:56:00Z" } 2023-10-12T00:56:00.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071952.264425_569.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:52:32.264425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:56:00.715 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071952.264425_569.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071952.264425_569.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.716 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:56:00.717 DEBUG:boto:path=/ 2023-10-12T00:56:00.717 DEBUG:boto:auth_path=/ 2023-10-12T00:56:00.717 DEBUG:boto:Method: GET 2023-10-12T00:56:00.718 DEBUG:boto:Path: / 2023-10-12T00:56:00.718 DEBUG:boto:Data: 2023-10-12T00:56:00.719 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.719 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.720 DEBUG:boto:Params: {} 2023-10-12T00:56:00.720 DEBUG:boto:Token: None 2023-10-12T00:56:00.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT / 2023-10-12T00:56:00.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dxz/roezV7orQSDEFn9eTzvYTGI= 2023-10-12T00:56:00.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dxz/roezV7orQSDEFn9eTzvYTGI=', 'Content-Length': '0'} 2023-10-12T00:56:00.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001538721e04b6ae16-0065274420-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.726 DEBUG:boto:path=/ 2023-10-12T00:56:00.727 DEBUG:boto:auth_path=/ 2023-10-12T00:56:00.727 DEBUG:boto:Method: GET 2023-10-12T00:56:00.728 DEBUG:boto:Path: / 2023-10-12T00:56:00.728 DEBUG:boto:Data: 2023-10-12T00:56:00.728 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:00.729 DEBUG:boto:Port: 8001 2023-10-12T00:56:00.730 DEBUG:boto:Params: {} 2023-10-12T00:56:00.730 DEBUG:boto:Token: None 2023-10-12T00:56:00.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:00 GMT / 2023-10-12T00:56:00.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dxz/roezV7orQSDEFn9eTzvYTGI= 2023-10-12T00:56:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dxz/roezV7orQSDEFn9eTzvYTGI=', 'Content-Length': '0'} 2023-10-12T00:56:00.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e528b12a106f7155-0065274420-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-10-12T00:56:00.736 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:00.737 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:00.737 DEBUG:boto:Method: PUT 2023-10-12T00:56:00.738 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:00.739 DEBUG:boto:Data: 2023-10-12T00:56:00.739 DEBUG:boto:Headers: {} 2023-10-12T00:56:00.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:00.741 DEBUG:boto:Port: 8000 2023-10-12T00:56:00.742 DEBUG:boto:Params: {} 2023-10-12T00:56:00.742 DEBUG:boto:Token: None 2023-10-12T00:56:00.743 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:56:00 GMT /wafmoa-58/ 2023-10-12T00:56:00.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kg4MI4wXWNRyuvJL85EUQJ0tV8g= 2023-10-12T00:56:00.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kg4MI4wXWNRyuvJL85EUQJ0tV8g=', 'Content-Length': '0'} 2023-10-12T00:56:00.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001e0c1dd8e2ca61a-0065274420-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:00.760 DEBUG:rgw_multi.tests:created bucket=wafmoa-58 2023-10-12T00:56:00.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.761 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:00.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:00.810 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:00.838 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:56:00.851 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:56:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:56:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:56:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:56:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:56:00.758530Z" 2023-10-12T00:56:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:56:00.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.855 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:56:00.858 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:56:00.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:00.860 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:00.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:00.910 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:00.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:00.938+0000 7feda9b95780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:00.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:00.939+0000 7feda9b95780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:00.953 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:56:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:56:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:56:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:56:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:56:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:56:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:56:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:56:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:56:00.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:56:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:56:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:56:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697071952.264425_569.1", 2023-10-12T00:56:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:52:32.264425Z", 2023-10-12T00:56:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:56:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:56:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:56:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:56:00.968 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:56:00.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:00.968 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:56:00Z" 2023-10-12T00:56:00.968 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:00.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697071952.264425_569.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:52:32.264425Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:56:00Z" } 2023-10-12T00:56:00.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697071952.264425_569.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:52:32.264425Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:56:00.971 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697071952.264425_569.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:00.972 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072160.758530_576.1 target=1_1697071952.264425_569.1 2023-10-12T00:56:05.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:05.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:05.974 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:06.031 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:06.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:06.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:06.062+0000 7fa56f1ac780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:06.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:06.063+0000 7fa56f1ac780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:06.077 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:06.078 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:56:06.078 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:56:06.078 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:56:06.078 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:56:06.079 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:56:06.079 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:06.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.080 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:56:06.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:06.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:56:06.081 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:06.081 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:06.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:06.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:06.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:56:06.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:06.082 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:56:06.082 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:06.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:06.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:06.083 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:56:06.083 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:06.083 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:06.084 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:06.084 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:06.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:06.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:06.085 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:56:06.085 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:06.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:06.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:06.086 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:56:06.086 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:06.086 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:06.087 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:56:06.087 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:06.087 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:06.087 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:06.088 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:56:06.088 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:06.088 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:06.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:06.089 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:56:06.089 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:06.089 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:06.090 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:06.090 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:06.090 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:06.090 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:06.091 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:56:06.091 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:06.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:06.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:06.092 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:06.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.092 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:56:06.092 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:56:06.092 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:56:06.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:06.093 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:56:06Z" 2023-10-12T00:56:06.093 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:06.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:56:06Z" } 2023-10-12T00:56:06.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:56:06.095 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:06.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:06.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:56:06.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:06.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:06.097 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:06.150 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:06.150 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:06.192 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:06.193 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:06.193 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:06.193 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-10-12T00:56:06.193 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:06.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:06.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:06.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:06.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:06.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:06.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-10-12T00:56:06.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:06.197 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006" 2023-10-12T00:56:06.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:06.197 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:06.199 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.200 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.200 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.201 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.202 DEBUG:boto:Data: 2023-10-12T00:56:06.202 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.204 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.204 DEBUG:boto:Params: {} 2023-10-12T00:56:06.205 DEBUG:boto:Token: None 2023-10-12T00:56:06.205 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35b19faf6a3b8b7-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.209 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:06.210 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:06.210 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.211 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:06.211 DEBUG:boto:Data: 2023-10-12T00:56:06.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.212 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.213 DEBUG:boto:Params: {} 2023-10-12T00:56:06.213 DEBUG:boto:Token: None 2023-10-12T00:56:06.214 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:06.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p7Uqgtbwc5/xdHNXzz53jg5gaMw= 2023-10-12T00:56:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p7Uqgtbwc5/xdHNXzz53jg5gaMw='} 2023-10-12T00:56:06.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffcd07ffdc602885-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.221 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.222 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.223 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.223 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.223 DEBUG:boto:Data: 2023-10-12T00:56:06.224 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.225 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.225 DEBUG:boto:Params: {} 2023-10-12T00:56:06.225 DEBUG:boto:Token: None 2023-10-12T00:56:06.226 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f31015f97de8d213-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.230 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:06.230 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:06.231 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.231 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:06.232 DEBUG:boto:Data: 2023-10-12T00:56:06.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.233 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.233 DEBUG:boto:Params: {} 2023-10-12T00:56:06.234 DEBUG:boto:Token: None 2023-10-12T00:56:06.234 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:06.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KIf8oBb3oAQyaQsdHhj2M3KDKhI= 2023-10-12T00:56:06.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KIf8oBb3oAQyaQsdHhj2M3KDKhI='} 2023-10-12T00:56:06.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0ec58c055bf8fef-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.241 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.241 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.242 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.242 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.243 DEBUG:boto:Data: 2023-10-12T00:56:06.243 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.244 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.245 DEBUG:boto:Params: {} 2023-10-12T00:56:06.246 DEBUG:boto:Token: None 2023-10-12T00:56:06.247 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cda32d68f16b3ec5-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.252 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:06.253 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:06.254 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.254 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:06.254 DEBUG:boto:Data: 2023-10-12T00:56:06.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.256 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.256 DEBUG:boto:Params: {} 2023-10-12T00:56:06.257 DEBUG:boto:Token: None 2023-10-12T00:56:06.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:06.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uYBHELUbq+DCQoutDRMuN2oo8Ns= 2023-10-12T00:56:06.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uYBHELUbq+DCQoutDRMuN2oo8Ns='} 2023-10-12T00:56:06.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007095c9f5b1a6bd30-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.303 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.304 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.304 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.304 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.305 DEBUG:boto:Data: 2023-10-12T00:56:06.305 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.306 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.306 DEBUG:boto:Params: {} 2023-10-12T00:56:06.307 DEBUG:boto:Token: None 2023-10-12T00:56:06.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1e3bfcf56ff1214-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.311 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:06.311 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:06.312 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.312 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:06.313 DEBUG:boto:Data: 2023-10-12T00:56:06.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.314 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.314 DEBUG:boto:Params: {} 2023-10-12T00:56:06.315 DEBUG:boto:Token: None 2023-10-12T00:56:06.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:06.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3MUdos4FnDsMgYjaG2euPrYfLl8= 2023-10-12T00:56:06.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3MUdos4FnDsMgYjaG2euPrYfLl8='} 2023-10-12T00:56:06.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cfea49b4fd8df7e-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.322 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.323 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.323 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.324 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.324 DEBUG:boto:Data: 2023-10-12T00:56:06.325 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.325 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.326 DEBUG:boto:Params: {} 2023-10-12T00:56:06.326 DEBUG:boto:Token: None 2023-10-12T00:56:06.327 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ae553127fced4dc-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.330 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:06.331 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:06.331 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.332 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:06.332 DEBUG:boto:Data: 2023-10-12T00:56:06.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.333 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.334 DEBUG:boto:Params: {} 2023-10-12T00:56:06.334 DEBUG:boto:Token: None 2023-10-12T00:56:06.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:06.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gs3f+/t6PwOscOss4AhPCPQ6c/g= 2023-10-12T00:56:06.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gs3f+/t6PwOscOss4AhPCPQ6c/g='} 2023-10-12T00:56:06.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002840d992521546c9-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.383 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.384 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.384 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.384 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.385 DEBUG:boto:Data: 2023-10-12T00:56:06.385 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.386 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.387 DEBUG:boto:Params: {} 2023-10-12T00:56:06.387 DEBUG:boto:Token: None 2023-10-12T00:56:06.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba046f138a21e221-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.391 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:06.392 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:06.392 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.393 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:06.393 DEBUG:boto:Data: 2023-10-12T00:56:06.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.395 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.397 DEBUG:boto:Params: {} 2023-10-12T00:56:06.397 DEBUG:boto:Token: None 2023-10-12T00:56:06.398 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:06.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RH+ynnIyBhWSpE7svxxlJisQJWg= 2023-10-12T00:56:06.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RH+ynnIyBhWSpE7svxxlJisQJWg='} 2023-10-12T00:56:06.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9d83ce45fad535e-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.405 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.405 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.405 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.406 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.406 DEBUG:boto:Data: 2023-10-12T00:56:06.407 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.408 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.408 DEBUG:boto:Params: {} 2023-10-12T00:56:06.408 DEBUG:boto:Token: None 2023-10-12T00:56:06.409 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.409 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f0cb8ba7c3536e8-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.413 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:06.413 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:06.414 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.414 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:06.414 DEBUG:boto:Data: 2023-10-12T00:56:06.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.416 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.417 DEBUG:boto:Params: {} 2023-10-12T00:56:06.417 DEBUG:boto:Token: None 2023-10-12T00:56:06.418 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:06.418 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2AOQ5fjDoEMbgqJpSdk1tK90pSE= 2023-10-12T00:56:06.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2AOQ5fjDoEMbgqJpSdk1tK90pSE='} 2023-10-12T00:56:06.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aea836a21097712b-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.464 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.464 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.465 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.465 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.466 DEBUG:boto:Data: 2023-10-12T00:56:06.466 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.467 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.467 DEBUG:boto:Params: {} 2023-10-12T00:56:06.468 DEBUG:boto:Token: None 2023-10-12T00:56:06.469 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002414c2f90dc9831-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.473 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:06.474 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:06.475 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.475 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:06.476 DEBUG:boto:Data: 2023-10-12T00:56:06.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.477 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.477 DEBUG:boto:Params: {} 2023-10-12T00:56:06.478 DEBUG:boto:Token: None 2023-10-12T00:56:06.478 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:06.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F4V1XZPcLIeIdkbSWoRPjQH6VIY= 2023-10-12T00:56:06.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F4V1XZPcLIeIdkbSWoRPjQH6VIY='} 2023-10-12T00:56:06.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a879a0d0f0998ce-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.486 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.486 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.487 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.487 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.488 DEBUG:boto:Data: 2023-10-12T00:56:06.488 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.489 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.489 DEBUG:boto:Params: {} 2023-10-12T00:56:06.490 DEBUG:boto:Token: None 2023-10-12T00:56:06.490 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.491 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004eafeeef4620f70a-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.494 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:06.495 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:06.495 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.496 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:06.496 DEBUG:boto:Data: 2023-10-12T00:56:06.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.497 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.498 DEBUG:boto:Params: {} 2023-10-12T00:56:06.498 DEBUG:boto:Token: None 2023-10-12T00:56:06.499 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:06.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vd/ZQ12Wd2+qeMg2wTjEkrH5NMQ= 2023-10-12T00:56:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vd/ZQ12Wd2+qeMg2wTjEkrH5NMQ='} 2023-10-12T00:56:06.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdca183f46d650b8-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.505 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:06.506 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:06.506 DEBUG:boto:Method: HEAD 2023-10-12T00:56:06.507 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:06.507 DEBUG:boto:Data: 2023-10-12T00:56:06.508 DEBUG:boto:Headers: {} 2023-10-12T00:56:06.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.509 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.509 DEBUG:boto:Params: {} 2023-10-12T00:56:06.509 DEBUG:boto:Token: None 2023-10-12T00:56:06.510 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/ 2023-10-12T00:56:06.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ= 2023-10-12T00:56:06.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k6WoyO961RH02avlrJ0DkpdXWrQ=', 'Content-Length': '0'} 2023-10-12T00:56:06.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ad5e1529420a61c-0065274426-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.514 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:06.514 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:06.515 DEBUG:boto:Method: PUT 2023-10-12T00:56:06.515 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:06.516 DEBUG:boto:Data: 2023-10-12T00:56:06.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:06.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:06.517 DEBUG:boto:Port: 8000 2023-10-12T00:56:06.518 DEBUG:boto:Params: {} 2023-10-12T00:56:06.518 DEBUG:boto:Token: None 2023-10-12T00:56:06.519 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:06 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:06.519 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rpxyxqAVdd1J9eOqGkLup17RSL4= 2023-10-12T00:56:06.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rpxyxqAVdd1J9eOqGkLup17RSL4='} 2023-10-12T00:56:06.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e646e689587a7cf0-0065274426-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:06.568 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-58 2023-10-12T00:56:06.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:06.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:06.570 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:56:06.629 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:06.629 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:06.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:06.662+0000 7f4b4fc00780 1 waiting to reach incremental sync.. 2023-10-12T00:56:11.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:11.663+0000 7f4b4fc00780 1 waiting to reach incremental sync.. 2023-10-12T00:56:16.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:16.665+0000 7f4b4fc00780 1 bucket sync caught up with source: 2023-10-12T00:56:16.666 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.42.6, 00000000001.78.6, 00000000001.57.6, 00000000001.31.6, 00000000001.29.6, 00000000001.16.6, , 00000000001.45.6, 00000000001.40.6, 00000000001.38.6, 00000000001.36.6] 2023-10-12T00:56:16.666 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.42.6, 00000000001.78.6, 00000000001.57.6, 00000000001.31.6, 00000000001.29.6, 00000000001.16.6, , 00000000001.45.6, 00000000001.40.6, 00000000001.38.6, 00000000001.36.6] 2023-10-12T00:56:16.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:16.665+0000 7f4b4fc00780 0 bucket checkpoint complete 2023-10-12T00:56:16.683 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-58 2023-10-12T00:56:16.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:16.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:16.685 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:56:16.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:16.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:16.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:16.779+0000 7f3223cba780 1 waiting to reach incremental sync.. 2023-10-12T00:56:21.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:21.781+0000 7f3223cba780 1 bucket sync caught up with source: 2023-10-12T00:56:21.782 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.338.6, 00000000001.363.6, 00000000001.305.6, 00000000001.85.6, 00000000001.196.6, 00000000001.164.6, , 00000000001.183.6, 00000000001.247.6, 00000000001.246.6, 00000000001.250.6] 2023-10-12T00:56:21.782 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.338.6, 00000000001.363.6, 00000000001.305.6, 00000000001.85.6, 00000000001.196.6, 00000000001.164.6, , 00000000001.183.6, 00000000001.247.6, 00000000001.246.6, 00000000001.250.6] 2023-10-12T00:56:21.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:21.781+0000 7f3223cba780 0 bucket checkpoint complete 2023-10-12T00:56:21.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-58 zones={a2, a1} 2023-10-12T00:56:21.799 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.800 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.800 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.801 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:21.801 DEBUG:boto:Data: 2023-10-12T00:56:21.801 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.802 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.803 DEBUG:boto:Params: {} 2023-10-12T00:56:21.803 DEBUG:boto:Token: None 2023-10-12T00:56:21.804 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/ 2023-10-12T00:56:21.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z91OQfGXeTi2eyreuSI0eMk2SMQ= 2023-10-12T00:56:21.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z91OQfGXeTi2eyreuSI0eMk2SMQ=', 'Content-Length': '0'} 2023-10-12T00:56:21.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004656bf62b905d44-0065274435-1aea-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.808 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.809 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.809 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.809 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:21.810 DEBUG:boto:Data: 2023-10-12T00:56:21.810 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.811 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.812 DEBUG:boto:Params: {} 2023-10-12T00:56:21.812 DEBUG:boto:Token: None 2023-10-12T00:56:21.813 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/ 2023-10-12T00:56:21.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z91OQfGXeTi2eyreuSI0eMk2SMQ= 2023-10-12T00:56:21.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z91OQfGXeTi2eyreuSI0eMk2SMQ=', 'Content-Length': '0'} 2023-10-12T00:56:21.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000897106e89137513a-0065274435-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.817 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:56:21.817 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.818 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.818 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.818 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.819 DEBUG:boto:Method: GET 2023-10-12T00:56:21.819 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.820 DEBUG:boto:Data: 2023-10-12T00:56:21.820 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.821 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.821 DEBUG:boto:Params: {} 2023-10-12T00:56:21.822 DEBUG:boto:Token: None 2023-10-12T00:56:21.822 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/?versions 2023-10-12T00:56:21.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k= 2023-10-12T00:56:21.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k=', 'Content-Length': '0'} 2023-10-12T00:56:21.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e11509ccd6440f-0065274435-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.827 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-12T00:56:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-12T00:56:21.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-12T00:56:21.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-12T00:56:21.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-12T00:56:21.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-12T00:56:21.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-12T00:56:21.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-12T00:56:21.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-12T00:56:21.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-12T00:56:21.832 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:56:21.832 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.833 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.833 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.833 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.834 DEBUG:boto:Method: GET 2023-10-12T00:56:21.834 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.835 DEBUG:boto:Data: 2023-10-12T00:56:21.835 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.836 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.836 DEBUG:boto:Params: {} 2023-10-12T00:56:21.837 DEBUG:boto:Token: None 2023-10-12T00:56:21.837 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/?versions 2023-10-12T00:56:21.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k= 2023-10-12T00:56:21.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k=', 'Content-Length': '0'} 2023-10-12T00:56:21.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3ec8bee8e29591-0065274435-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.842 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-12T00:56:21.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-12T00:56:21.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-12T00:56:21.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-12T00:56:21.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-12T00:56:21.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-12T00:56:21.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-12T00:56:21.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-12T00:56:21.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-12T00:56:21.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-12T00:56:21.847 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.847 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.848 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.848 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.848 DEBUG:boto:Method: GET 2023-10-12T00:56:21.849 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.849 DEBUG:boto:Data: 2023-10-12T00:56:21.850 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.851 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.851 DEBUG:boto:Params: {} 2023-10-12T00:56:21.852 DEBUG:boto:Token: None 2023-10-12T00:56:21.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/?versions 2023-10-12T00:56:21.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k= 2023-10-12T00:56:21.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k=', 'Content-Length': '0'} 2023-10-12T00:56:21.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000228b4ca7447e440-0065274435-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.856 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.858 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.858 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.858 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.859 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.859 DEBUG:boto:Method: GET 2023-10-12T00:56:21.860 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:21.860 DEBUG:boto:Data: 2023-10-12T00:56:21.861 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.861 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.862 DEBUG:boto:Params: {} 2023-10-12T00:56:21.862 DEBUG:boto:Token: None 2023-10-12T00:56:21.863 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/?versions 2023-10-12T00:56:21.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k= 2023-10-12T00:56:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8XW7zGJaZqVEMFC8MHTr/N+iq8k=', 'Content-Length': '0'} 2023-10-12T00:56:21.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2dfbc9c92334efe-0065274435-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.867 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:21.869 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.870 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.870 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.871 DEBUG:boto:Method: GET 2023-10-12T00:56:21.871 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.871 DEBUG:boto:Data: 2023-10-12T00:56:21.872 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.873 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.873 DEBUG:boto:Params: {} 2023-10-12T00:56:21.874 DEBUG:boto:Token: None 2023-10-12T00:56:21.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE= 2023-10-12T00:56:21.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE=', 'Content-Length': '0'} 2023-10-12T00:56:21.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1377b869e424b58-0065274435-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.918 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.919 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.919 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.920 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.920 DEBUG:boto:Method: GET 2023-10-12T00:56:21.921 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.921 DEBUG:boto:Data: 2023-10-12T00:56:21.922 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.922 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.923 DEBUG:boto:Params: {} 2023-10-12T00:56:21.923 DEBUG:boto:Token: None 2023-10-12T00:56:21.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE= 2023-10-12T00:56:21.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE=', 'Content-Length': '0'} 2023-10-12T00:56:21.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ad82e9ee25dd1d-0065274435-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.928 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.928 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.928 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.929 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.929 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.930 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.930 DEBUG:boto:Data: 2023-10-12T00:56:21.930 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.931 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.932 DEBUG:boto:Params: {} 2023-10-12T00:56:21.932 DEBUG:boto:Token: None 2023-10-12T00:56:21.933 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aHYrFh17iJgy0/5IuzedwUwLe80= 2023-10-12T00:56:21.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aHYrFh17iJgy0/5IuzedwUwLe80=', 'Content-Length': '0'} 2023-10-12T00:56:21.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b7f5d92bfdcfd2-0065274435-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.936 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.937 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.937 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.937 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.938 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.938 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.939 DEBUG:boto:Data: 2023-10-12T00:56:21.939 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.940 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.940 DEBUG:boto:Params: {} 2023-10-12T00:56:21.941 DEBUG:boto:Token: None 2023-10-12T00:56:21.941 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aHYrFh17iJgy0/5IuzedwUwLe80= 2023-10-12T00:56:21.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aHYrFh17iJgy0/5IuzedwUwLe80=', 'Content-Length': '0'} 2023-10-12T00:56:21.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94908634e6ad70a-0065274435-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:21.945 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.946 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.946 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.947 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.947 DEBUG:boto:Method: GET 2023-10-12T00:56:21.948 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.948 DEBUG:boto:Data: 2023-10-12T00:56:21.948 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.949 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.950 DEBUG:boto:Params: {} 2023-10-12T00:56:21.950 DEBUG:boto:Token: None 2023-10-12T00:56:21.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE= 2023-10-12T00:56:21.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE=', 'Content-Length': '0'} 2023-10-12T00:56:21.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a913c833ff2d35a6-0065274435-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.955 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.955 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.956 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.956 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.956 DEBUG:boto:Method: GET 2023-10-12T00:56:21.957 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.957 DEBUG:boto:Data: 2023-10-12T00:56:21.958 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.959 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.959 DEBUG:boto:Params: {} 2023-10-12T00:56:21.960 DEBUG:boto:Token: None 2023-10-12T00:56:21.960 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:21.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE= 2023-10-12T00:56:21.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hNVWpnIB+v6aD3aRINO+TvFIOwE=', 'Content-Length': '0'} 2023-10-12T00:56:21.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103b649395a2df6f-0065274435-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.964 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.964 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.965 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.965 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.966 DEBUG:boto:Method: GET 2023-10-12T00:56:21.966 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.967 DEBUG:boto:Data: 2023-10-12T00:56:21.967 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.968 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.968 DEBUG:boto:Params: {} 2023-10-12T00:56:21.969 DEBUG:boto:Token: None 2023-10-12T00:56:21.969 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/ 2023-10-12T00:56:21.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nTsXQ5XRlwKNnPpFdgvtciBhCOo= 2023-10-12T00:56:21.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nTsXQ5XRlwKNnPpFdgvtciBhCOo=', 'Content-Length': '0'} 2023-10-12T00:56:21.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049e2d9fc71214fd-0065274435-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.974 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.975 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:21.975 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:21.976 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.976 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.976 DEBUG:boto:Method: GET 2023-10-12T00:56:21.977 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:21.977 DEBUG:boto:Data: 2023-10-12T00:56:21.978 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.979 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.979 DEBUG:boto:Params: {} 2023-10-12T00:56:21.980 DEBUG:boto:Token: None 2023-10-12T00:56:21.980 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/ 2023-10-12T00:56:21.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nTsXQ5XRlwKNnPpFdgvtciBhCOo= 2023-10-12T00:56:21.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nTsXQ5XRlwKNnPpFdgvtciBhCOo=', 'Content-Length': '0'} 2023-10-12T00:56:21.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040fec145268b7635-0065274435-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.986 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:21.987 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.987 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.987 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.988 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.988 DEBUG:boto:Data: 2023-10-12T00:56:21.989 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:21.990 DEBUG:boto:Port: 8001 2023-10-12T00:56:21.990 DEBUG:boto:Params: {} 2023-10-12T00:56:21.990 DEBUG:boto:Token: None 2023-10-12T00:56:21.991 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f0chiiGHhWPNE5sUiDlqHKktifg= 2023-10-12T00:56:21.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f0chiiGHhWPNE5sUiDlqHKktifg=', 'Content-Length': '0'} 2023-10-12T00:56:21.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922c6b2efbb26a10-0065274435-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:21.994 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.995 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.995 DEBUG:boto:Method: HEAD 2023-10-12T00:56:21.996 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.996 DEBUG:boto:Data: 2023-10-12T00:56:21.997 DEBUG:boto:Headers: {} 2023-10-12T00:56:21.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:21.997 DEBUG:boto:Port: 8000 2023-10-12T00:56:21.998 DEBUG:boto:Params: {} 2023-10-12T00:56:21.998 DEBUG:boto:Token: None 2023-10-12T00:56:21.999 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:21 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:21.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f0chiiGHhWPNE5sUiDlqHKktifg= 2023-10-12T00:56:22.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f0chiiGHhWPNE5sUiDlqHKktifg=', 'Content-Length': '0'} 2023-10-12T00:56:22.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6629a09cd56bf39-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:22.003 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.003 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.004 DEBUG:boto:Method: GET 2023-10-12T00:56:22.004 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.005 DEBUG:boto:Data: 2023-10-12T00:56:22.005 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.006 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.006 DEBUG:boto:Params: {} 2023-10-12T00:56:22.007 DEBUG:boto:Token: None 2023-10-12T00:56:22.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p9FJXddFx3cKDqyWHSlAD1128Z8= 2023-10-12T00:56:22.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p9FJXddFx3cKDqyWHSlAD1128Z8=', 'Content-Length': '0'} 2023-10-12T00:56:22.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e65dcb8c964c8e5-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.011 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.011 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.012 DEBUG:boto:Method: GET 2023-10-12T00:56:22.012 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.013 DEBUG:boto:Data: 2023-10-12T00:56:22.013 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.014 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.015 DEBUG:boto:Params: {} 2023-10-12T00:56:22.015 DEBUG:boto:Token: None 2023-10-12T00:56:22.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:22.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p9FJXddFx3cKDqyWHSlAD1128Z8= 2023-10-12T00:56:22.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p9FJXddFx3cKDqyWHSlAD1128Z8=', 'Content-Length': '0'} 2023-10-12T00:56:22.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c7bb6f18a6b171-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.062 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.063 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.063 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.064 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.064 DEBUG:boto:Method: GET 2023-10-12T00:56:22.065 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.065 DEBUG:boto:Data: 2023-10-12T00:56:22.066 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.067 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.067 DEBUG:boto:Params: {} 2023-10-12T00:56:22.068 DEBUG:boto:Token: None 2023-10-12T00:56:22.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e386fa1fd728840-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.073 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.073 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.074 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.074 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.075 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.075 DEBUG:boto:Method: GET 2023-10-12T00:56:22.076 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:22.076 DEBUG:boto:Data: 2023-10-12T00:56:22.076 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.077 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.078 DEBUG:boto:Params: {} 2023-10-12T00:56:22.078 DEBUG:boto:Token: None 2023-10-12T00:56:22.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd349090972ecc2e-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.083 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:06.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:22.084 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.085 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.085 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.085 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.086 DEBUG:boto:Method: GET 2023-10-12T00:56:22.086 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.087 DEBUG:boto:Data: 2023-10-12T00:56:22.087 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.088 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.089 DEBUG:boto:Params: {} 2023-10-12T00:56:22.089 DEBUG:boto:Token: None 2023-10-12T00:56:22.090 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM= 2023-10-12T00:56:22.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM=', 'Content-Length': '0'} 2023-10-12T00:56:22.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c813a3946265a424-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.134 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.135 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.136 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.136 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.136 DEBUG:boto:Method: GET 2023-10-12T00:56:22.137 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.137 DEBUG:boto:Data: 2023-10-12T00:56:22.138 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.139 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.139 DEBUG:boto:Params: {} 2023-10-12T00:56:22.140 DEBUG:boto:Token: None 2023-10-12T00:56:22.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM= 2023-10-12T00:56:22.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM=', 'Content-Length': '0'} 2023-10-12T00:56:22.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009615b433fd7f1885-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.144 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.144 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.145 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.145 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.146 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.146 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.147 DEBUG:boto:Data: 2023-10-12T00:56:22.147 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.148 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.148 DEBUG:boto:Params: {} 2023-10-12T00:56:22.149 DEBUG:boto:Token: None 2023-10-12T00:56:22.149 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j+Al3jlfURBMwrN9pTDoi5ZwhAk= 2023-10-12T00:56:22.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j+Al3jlfURBMwrN9pTDoi5ZwhAk=', 'Content-Length': '0'} 2023-10-12T00:56:22.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a95cb97367b6b27-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.153 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.153 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.153 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.154 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.155 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.155 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.156 DEBUG:boto:Data: 2023-10-12T00:56:22.156 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.157 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.157 DEBUG:boto:Params: {} 2023-10-12T00:56:22.158 DEBUG:boto:Token: None 2023-10-12T00:56:22.158 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j+Al3jlfURBMwrN9pTDoi5ZwhAk= 2023-10-12T00:56:22.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j+Al3jlfURBMwrN9pTDoi5ZwhAk=', 'Content-Length': '0'} 2023-10-12T00:56:22.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57b69b59cc83ce6-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:22.162 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.163 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.163 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.164 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.164 DEBUG:boto:Method: GET 2023-10-12T00:56:22.164 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.165 DEBUG:boto:Data: 2023-10-12T00:56:22.165 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.166 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.167 DEBUG:boto:Params: {} 2023-10-12T00:56:22.167 DEBUG:boto:Token: None 2023-10-12T00:56:22.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM= 2023-10-12T00:56:22.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM=', 'Content-Length': '0'} 2023-10-12T00:56:22.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ed0002bb84478e-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.171 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.172 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.172 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.173 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.173 DEBUG:boto:Method: GET 2023-10-12T00:56:22.174 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.174 DEBUG:boto:Data: 2023-10-12T00:56:22.174 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.175 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.176 DEBUG:boto:Params: {} 2023-10-12T00:56:22.176 DEBUG:boto:Token: None 2023-10-12T00:56:22.177 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:22.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM= 2023-10-12T00:56:22.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LqVLgaw+1sNf7wNTJS1poAprbXM=', 'Content-Length': '0'} 2023-10-12T00:56:22.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c765265665f7956-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.181 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.181 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.182 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.182 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.185 DEBUG:boto:Method: GET 2023-10-12T00:56:22.186 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.186 DEBUG:boto:Data: 2023-10-12T00:56:22.187 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.188 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.188 DEBUG:boto:Params: {} 2023-10-12T00:56:22.188 DEBUG:boto:Token: None 2023-10-12T00:56:22.189 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a297a9c49bb95444-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.193 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.194 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.194 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.195 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.195 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.196 DEBUG:boto:Method: GET 2023-10-12T00:56:22.196 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.196 DEBUG:boto:Data: 2023-10-12T00:56:22.197 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.198 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.198 DEBUG:boto:Params: {} 2023-10-12T00:56:22.199 DEBUG:boto:Token: None 2023-10-12T00:56:22.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034fc7c7504ac9277-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.203 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.204 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.204 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.205 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.205 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.206 DEBUG:boto:Data: 2023-10-12T00:56:22.206 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.207 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.207 DEBUG:boto:Params: {} 2023-10-12T00:56:22.208 DEBUG:boto:Token: None 2023-10-12T00:56:22.208 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2syKgCPr6tzl7KwTUEPGt6meRG0= 2023-10-12T00:56:22.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2syKgCPr6tzl7KwTUEPGt6meRG0=', 'Content-Length': '0'} 2023-10-12T00:56:22.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfe2042b5f90360-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.212 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.212 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.213 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.213 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.214 DEBUG:boto:Data: 2023-10-12T00:56:22.214 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.215 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.215 DEBUG:boto:Params: {} 2023-10-12T00:56:22.216 DEBUG:boto:Token: None 2023-10-12T00:56:22.216 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2syKgCPr6tzl7KwTUEPGt6meRG0= 2023-10-12T00:56:22.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2syKgCPr6tzl7KwTUEPGt6meRG0=', 'Content-Length': '0'} 2023-10-12T00:56:22.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1007c3c49c9fad-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:22.220 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.220 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.221 DEBUG:boto:Method: GET 2023-10-12T00:56:22.221 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.222 DEBUG:boto:Data: 2023-10-12T00:56:22.222 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.223 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.223 DEBUG:boto:Params: {} 2023-10-12T00:56:22.224 DEBUG:boto:Token: None 2023-10-12T00:56:22.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BmfnMdxCspbo69eYq59YJvXtJzU= 2023-10-12T00:56:22.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BmfnMdxCspbo69eYq59YJvXtJzU=', 'Content-Length': '0'} 2023-10-12T00:56:22.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de12118d7037efad-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.270 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.271 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.272 DEBUG:boto:Method: GET 2023-10-12T00:56:22.273 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.274 DEBUG:boto:Data: 2023-10-12T00:56:22.274 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.276 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.277 DEBUG:boto:Params: {} 2023-10-12T00:56:22.278 DEBUG:boto:Token: None 2023-10-12T00:56:22.278 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:22.279 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BmfnMdxCspbo69eYq59YJvXtJzU= 2023-10-12T00:56:22.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BmfnMdxCspbo69eYq59YJvXtJzU=', 'Content-Length': '0'} 2023-10-12T00:56:22.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb17352014e81c0e-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.283 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.284 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.284 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.285 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.285 DEBUG:boto:Method: GET 2023-10-12T00:56:22.286 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.286 DEBUG:boto:Data: 2023-10-12T00:56:22.286 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.287 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.288 DEBUG:boto:Params: {} 2023-10-12T00:56:22.288 DEBUG:boto:Token: None 2023-10-12T00:56:22.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004905e827a182a36a-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.293 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.294 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.294 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.295 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.295 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.296 DEBUG:boto:Method: GET 2023-10-12T00:56:22.296 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:22.297 DEBUG:boto:Data: 2023-10-12T00:56:22.297 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.298 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.298 DEBUG:boto:Params: {} 2023-10-12T00:56:22.299 DEBUG:boto:Token: None 2023-10-12T00:56:22.299 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c5e5a71bbc930b-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.304 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:06.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:22.306 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.306 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.307 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.307 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.308 DEBUG:boto:Method: GET 2023-10-12T00:56:22.308 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.308 DEBUG:boto:Data: 2023-10-12T00:56:22.309 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.310 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.311 DEBUG:boto:Params: {} 2023-10-12T00:56:22.311 DEBUG:boto:Token: None 2023-10-12T00:56:22.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8= 2023-10-12T00:56:22.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8=', 'Content-Length': '0'} 2023-10-12T00:56:22.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f0ea4d9f8cced6-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.315 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.316 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.316 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.317 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.317 DEBUG:boto:Method: GET 2023-10-12T00:56:22.318 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.318 DEBUG:boto:Data: 2023-10-12T00:56:22.319 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.320 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.320 DEBUG:boto:Params: {} 2023-10-12T00:56:22.321 DEBUG:boto:Token: None 2023-10-12T00:56:22.321 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8= 2023-10-12T00:56:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8=', 'Content-Length': '0'} 2023-10-12T00:56:22.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae6b332146428c2-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.325 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.325 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.326 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.326 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.327 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.328 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.328 DEBUG:boto:Data: 2023-10-12T00:56:22.329 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.330 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.331 DEBUG:boto:Params: {} 2023-10-12T00:56:22.332 DEBUG:boto:Token: None 2023-10-12T00:56:22.332 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.333 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e3GFMg4k04TO6MKZhbC6TJZUmOQ= 2023-10-12T00:56:22.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e3GFMg4k04TO6MKZhbC6TJZUmOQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb75d49f5be0d843-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.336 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.337 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.337 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.337 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.338 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.338 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.339 DEBUG:boto:Data: 2023-10-12T00:56:22.339 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.340 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.341 DEBUG:boto:Params: {} 2023-10-12T00:56:22.341 DEBUG:boto:Token: None 2023-10-12T00:56:22.342 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.342 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e3GFMg4k04TO6MKZhbC6TJZUmOQ= 2023-10-12T00:56:22.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e3GFMg4k04TO6MKZhbC6TJZUmOQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5301259596ef2d-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:22.346 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.346 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.347 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.347 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.348 DEBUG:boto:Method: GET 2023-10-12T00:56:22.349 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.349 DEBUG:boto:Data: 2023-10-12T00:56:22.350 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.351 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.351 DEBUG:boto:Params: {} 2023-10-12T00:56:22.352 DEBUG:boto:Token: None 2023-10-12T00:56:22.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8= 2023-10-12T00:56:22.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8=', 'Content-Length': '0'} 2023-10-12T00:56:22.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e463b7fb0bfc5bf5-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.357 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.357 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.357 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.358 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.358 DEBUG:boto:Method: GET 2023-10-12T00:56:22.359 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.359 DEBUG:boto:Data: 2023-10-12T00:56:22.360 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.361 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.361 DEBUG:boto:Params: {} 2023-10-12T00:56:22.362 DEBUG:boto:Token: None 2023-10-12T00:56:22.362 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:22.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8= 2023-10-12T00:56:22.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUqwNz+hqVmbjXR+eise6XBeYS8=', 'Content-Length': '0'} 2023-10-12T00:56:22.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314833a9b14f6157-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.366 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.366 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.367 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.367 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.368 DEBUG:boto:Method: GET 2023-10-12T00:56:22.368 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.369 DEBUG:boto:Data: 2023-10-12T00:56:22.369 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.370 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.371 DEBUG:boto:Params: {} 2023-10-12T00:56:22.371 DEBUG:boto:Token: None 2023-10-12T00:56:22.372 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023cab9631579f4d-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.376 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.377 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.377 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.378 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.378 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.379 DEBUG:boto:Method: GET 2023-10-12T00:56:22.379 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.379 DEBUG:boto:Data: 2023-10-12T00:56:22.380 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.381 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.381 DEBUG:boto:Params: {} 2023-10-12T00:56:22.382 DEBUG:boto:Token: None 2023-10-12T00:56:22.382 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aeebcbe3399d0e8-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.386 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.387 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.387 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.387 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.389 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.390 DEBUG:boto:Data: 2023-10-12T00:56:22.391 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.392 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.392 DEBUG:boto:Params: {} 2023-10-12T00:56:22.392 DEBUG:boto:Token: None 2023-10-12T00:56:22.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zj7uy3Du87oj2UBydLSMFg9ctaw= 2023-10-12T00:56:22.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zj7uy3Du87oj2UBydLSMFg9ctaw=', 'Content-Length': '0'} 2023-10-12T00:56:22.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe53cee2671b8bfb-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.396 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.397 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.397 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.398 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.398 DEBUG:boto:Data: 2023-10-12T00:56:22.399 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.399 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.400 DEBUG:boto:Params: {} 2023-10-12T00:56:22.400 DEBUG:boto:Token: None 2023-10-12T00:56:22.401 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zj7uy3Du87oj2UBydLSMFg9ctaw= 2023-10-12T00:56:22.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zj7uy3Du87oj2UBydLSMFg9ctaw=', 'Content-Length': '0'} 2023-10-12T00:56:22.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001effc43dd75ad55e-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:22.405 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.405 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.405 DEBUG:boto:Method: GET 2023-10-12T00:56:22.406 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.406 DEBUG:boto:Data: 2023-10-12T00:56:22.407 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.408 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.408 DEBUG:boto:Params: {} 2023-10-12T00:56:22.409 DEBUG:boto:Token: None 2023-10-12T00:56:22.409 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iv7c/vtI3TzglZzviGO0xeISq3M= 2023-10-12T00:56:22.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iv7c/vtI3TzglZzviGO0xeISq3M=', 'Content-Length': '0'} 2023-10-12T00:56:22.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7185f20e5ce609-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.458 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.459 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.460 DEBUG:boto:Method: GET 2023-10-12T00:56:22.460 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.461 DEBUG:boto:Data: 2023-10-12T00:56:22.462 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.464 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.464 DEBUG:boto:Params: {} 2023-10-12T00:56:22.465 DEBUG:boto:Token: None 2023-10-12T00:56:22.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:22.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iv7c/vtI3TzglZzviGO0xeISq3M= 2023-10-12T00:56:22.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iv7c/vtI3TzglZzviGO0xeISq3M=', 'Content-Length': '0'} 2023-10-12T00:56:22.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116271ca66afd784-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.472 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.473 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.473 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.474 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.475 DEBUG:boto:Method: GET 2023-10-12T00:56:22.476 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.477 DEBUG:boto:Data: 2023-10-12T00:56:22.477 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.479 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.480 DEBUG:boto:Params: {} 2023-10-12T00:56:22.481 DEBUG:boto:Token: None 2023-10-12T00:56:22.482 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359abf25ae8c6d44-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.488 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.489 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.490 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.491 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.492 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.492 DEBUG:boto:Method: GET 2023-10-12T00:56:22.493 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:22.494 DEBUG:boto:Data: 2023-10-12T00:56:22.495 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.496 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.497 DEBUG:boto:Params: {} 2023-10-12T00:56:22.498 DEBUG:boto:Token: None 2023-10-12T00:56:22.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f2d0b420fcbbe2-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.547 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:06.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:22.548 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.549 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.550 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.551 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.552 DEBUG:boto:Method: GET 2023-10-12T00:56:22.552 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.553 DEBUG:boto:Data: 2023-10-12T00:56:22.554 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.555 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.556 DEBUG:boto:Params: {} 2023-10-12T00:56:22.557 DEBUG:boto:Token: None 2023-10-12T00:56:22.558 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.558 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA= 2023-10-12T00:56:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA=', 'Content-Length': '0'} 2023-10-12T00:56:22.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fc571ae384bd94-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.562 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.563 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.563 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.564 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.564 DEBUG:boto:Method: GET 2023-10-12T00:56:22.565 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.565 DEBUG:boto:Data: 2023-10-12T00:56:22.565 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.567 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.568 DEBUG:boto:Params: {} 2023-10-12T00:56:22.568 DEBUG:boto:Token: None 2023-10-12T00:56:22.569 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA= 2023-10-12T00:56:22.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA=', 'Content-Length': '0'} 2023-10-12T00:56:22.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006310029053f21b7a-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.575 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.575 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.576 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.576 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.576 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.577 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.577 DEBUG:boto:Data: 2023-10-12T00:56:22.578 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.578 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.579 DEBUG:boto:Params: {} 2023-10-12T00:56:22.579 DEBUG:boto:Token: None 2023-10-12T00:56:22.580 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.580 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MCoR8Vcpeagpii2XBkj3swv2bFI= 2023-10-12T00:56:22.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MCoR8Vcpeagpii2XBkj3swv2bFI=', 'Content-Length': '0'} 2023-10-12T00:56:22.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed22634c24c45d57-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.583 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.584 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.584 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.584 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.585 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.585 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.586 DEBUG:boto:Data: 2023-10-12T00:56:22.586 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.587 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.587 DEBUG:boto:Params: {} 2023-10-12T00:56:22.588 DEBUG:boto:Token: None 2023-10-12T00:56:22.588 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MCoR8Vcpeagpii2XBkj3swv2bFI= 2023-10-12T00:56:22.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MCoR8Vcpeagpii2XBkj3swv2bFI=', 'Content-Length': '0'} 2023-10-12T00:56:22.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7dc750ad5a0ede7-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:22.593 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.593 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.593 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.594 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.594 DEBUG:boto:Method: GET 2023-10-12T00:56:22.595 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.595 DEBUG:boto:Data: 2023-10-12T00:56:22.596 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.596 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.597 DEBUG:boto:Params: {} 2023-10-12T00:56:22.597 DEBUG:boto:Token: None 2023-10-12T00:56:22.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA= 2023-10-12T00:56:22.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA=', 'Content-Length': '0'} 2023-10-12T00:56:22.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e35405beb53ddfe1-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.602 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.602 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.603 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.603 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.604 DEBUG:boto:Method: GET 2023-10-12T00:56:22.604 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.604 DEBUG:boto:Data: 2023-10-12T00:56:22.605 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.606 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.606 DEBUG:boto:Params: {} 2023-10-12T00:56:22.607 DEBUG:boto:Token: None 2023-10-12T00:56:22.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:22.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA= 2023-10-12T00:56:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r49ps8Ud38YQv5DQAujTHYoKbyA=', 'Content-Length': '0'} 2023-10-12T00:56:22.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f427eea702c072e-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.611 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.611 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.611 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.612 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.613 DEBUG:boto:Method: GET 2023-10-12T00:56:22.613 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.613 DEBUG:boto:Data: 2023-10-12T00:56:22.614 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.615 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.615 DEBUG:boto:Params: {} 2023-10-12T00:56:22.615 DEBUG:boto:Token: None 2023-10-12T00:56:22.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e232e05aeb86e3d0-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.622 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.623 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.624 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.624 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.625 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.626 DEBUG:boto:Method: GET 2023-10-12T00:56:22.627 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.628 DEBUG:boto:Data: 2023-10-12T00:56:22.628 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.630 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.631 DEBUG:boto:Params: {} 2023-10-12T00:56:22.632 DEBUG:boto:Token: None 2023-10-12T00:56:22.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003396b3bd078a63d8-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.678 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.679 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.680 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.680 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.680 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.681 DEBUG:boto:Data: 2023-10-12T00:56:22.681 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.682 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.682 DEBUG:boto:Params: {} 2023-10-12T00:56:22.683 DEBUG:boto:Token: None 2023-10-12T00:56:22.683 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AC9+Sy19BC30r0XzKkfYRbzR07o= 2023-10-12T00:56:22.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AC9+Sy19BC30r0XzKkfYRbzR07o=', 'Content-Length': '0'} 2023-10-12T00:56:22.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8cc288e03c7c180-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.688 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.688 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.689 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.689 DEBUG:boto:Data: 2023-10-12T00:56:22.690 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.690 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.691 DEBUG:boto:Params: {} 2023-10-12T00:56:22.691 DEBUG:boto:Token: None 2023-10-12T00:56:22.692 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AC9+Sy19BC30r0XzKkfYRbzR07o= 2023-10-12T00:56:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AC9+Sy19BC30r0XzKkfYRbzR07o=', 'Content-Length': '0'} 2023-10-12T00:56:22.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e18bdc51c6aa92-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:22.696 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.697 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.697 DEBUG:boto:Method: GET 2023-10-12T00:56:22.698 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.698 DEBUG:boto:Data: 2023-10-12T00:56:22.699 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.700 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.700 DEBUG:boto:Params: {} 2023-10-12T00:56:22.701 DEBUG:boto:Token: None 2023-10-12T00:56:22.701 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FNDk+lxUaCTxrYhr2HDWZ9ZeIzo= 2023-10-12T00:56:22.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FNDk+lxUaCTxrYhr2HDWZ9ZeIzo=', 'Content-Length': '0'} 2023-10-12T00:56:22.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a424417800011e-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.746 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.747 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.747 DEBUG:boto:Method: GET 2023-10-12T00:56:22.748 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.748 DEBUG:boto:Data: 2023-10-12T00:56:22.749 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.750 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.750 DEBUG:boto:Params: {} 2023-10-12T00:56:22.751 DEBUG:boto:Token: None 2023-10-12T00:56:22.751 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:22.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FNDk+lxUaCTxrYhr2HDWZ9ZeIzo= 2023-10-12T00:56:22.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FNDk+lxUaCTxrYhr2HDWZ9ZeIzo=', 'Content-Length': '0'} 2023-10-12T00:56:22.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9b2b5482098123-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.755 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.755 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.756 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.756 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.757 DEBUG:boto:Method: GET 2023-10-12T00:56:22.757 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.758 DEBUG:boto:Data: 2023-10-12T00:56:22.758 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.759 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.759 DEBUG:boto:Params: {} 2023-10-12T00:56:22.760 DEBUG:boto:Token: None 2023-10-12T00:56:22.760 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620bbadc58044467-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.764 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.765 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.765 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.766 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.766 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.767 DEBUG:boto:Method: GET 2023-10-12T00:56:22.767 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:22.768 DEBUG:boto:Data: 2023-10-12T00:56:22.768 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.769 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.770 DEBUG:boto:Params: {} 2023-10-12T00:56:22.771 DEBUG:boto:Token: None 2023-10-12T00:56:22.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004af29054c9fa4965-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.775 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:06.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:22.776 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.776 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.777 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.777 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.778 DEBUG:boto:Method: GET 2023-10-12T00:56:22.779 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.779 DEBUG:boto:Data: 2023-10-12T00:56:22.780 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.780 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.781 DEBUG:boto:Params: {} 2023-10-12T00:56:22.781 DEBUG:boto:Token: None 2023-10-12T00:56:22.782 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ= 2023-10-12T00:56:22.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb400e65e820a72-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.786 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.786 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.786 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.787 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.787 DEBUG:boto:Method: GET 2023-10-12T00:56:22.788 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.788 DEBUG:boto:Data: 2023-10-12T00:56:22.789 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.790 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.790 DEBUG:boto:Params: {} 2023-10-12T00:56:22.791 DEBUG:boto:Token: None 2023-10-12T00:56:22.791 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ= 2023-10-12T00:56:22.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049fd36f876545fd2-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.834 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.835 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.835 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.836 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.836 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.837 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.837 DEBUG:boto:Data: 2023-10-12T00:56:22.838 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.839 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.839 DEBUG:boto:Params: {} 2023-10-12T00:56:22.839 DEBUG:boto:Token: None 2023-10-12T00:56:22.840 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7z/aWJWh5rKvHjwr1O4tj7/cfCo= 2023-10-12T00:56:22.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7z/aWJWh5rKvHjwr1O4tj7/cfCo=', 'Content-Length': '0'} 2023-10-12T00:56:22.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387a9a469063691b-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.844 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.845 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.847 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.848 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.848 DEBUG:boto:Data: 2023-10-12T00:56:22.849 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.850 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.851 DEBUG:boto:Params: {} 2023-10-12T00:56:22.852 DEBUG:boto:Token: None 2023-10-12T00:56:22.852 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7z/aWJWh5rKvHjwr1O4tj7/cfCo= 2023-10-12T00:56:22.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7z/aWJWh5rKvHjwr1O4tj7/cfCo=', 'Content-Length': '0'} 2023-10-12T00:56:22.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f397d4b51bbfe751-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:22.856 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.857 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.857 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.858 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.858 DEBUG:boto:Method: GET 2023-10-12T00:56:22.859 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.859 DEBUG:boto:Data: 2023-10-12T00:56:22.860 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.861 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.861 DEBUG:boto:Params: {} 2023-10-12T00:56:22.862 DEBUG:boto:Token: None 2023-10-12T00:56:22.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ= 2023-10-12T00:56:22.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de1ae3d1f6843c4-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.866 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.867 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.867 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.868 DEBUG:boto:Method: GET 2023-10-12T00:56:22.869 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.869 DEBUG:boto:Data: 2023-10-12T00:56:22.870 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.871 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.871 DEBUG:boto:Params: {} 2023-10-12T00:56:22.872 DEBUG:boto:Token: None 2023-10-12T00:56:22.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:22.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ= 2023-10-12T00:56:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9fN1UO9EI9uH1/YTzx+S3ncYmQ=', 'Content-Length': '0'} 2023-10-12T00:56:22.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8651b2e0a252fa0-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.877 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.877 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.878 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.878 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.878 DEBUG:boto:Method: GET 2023-10-12T00:56:22.879 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.879 DEBUG:boto:Data: 2023-10-12T00:56:22.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.881 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.881 DEBUG:boto:Params: {} 2023-10-12T00:56:22.881 DEBUG:boto:Token: None 2023-10-12T00:56:22.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb85f77ba2617feb-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.886 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.887 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.887 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.888 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.888 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.889 DEBUG:boto:Method: GET 2023-10-12T00:56:22.889 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.889 DEBUG:boto:Data: 2023-10-12T00:56:22.890 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.891 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.892 DEBUG:boto:Params: {} 2023-10-12T00:56:22.892 DEBUG:boto:Token: None 2023-10-12T00:56:22.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031dcd8896ca5aab0-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.897 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.898 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.898 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.899 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.899 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.900 DEBUG:boto:Data: 2023-10-12T00:56:22.900 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.901 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.902 DEBUG:boto:Params: {} 2023-10-12T00:56:22.902 DEBUG:boto:Token: None 2023-10-12T00:56:22.903 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OQuPOjofeL+MTaOeKgJ2pc8bZSA= 2023-10-12T00:56:22.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OQuPOjofeL+MTaOeKgJ2pc8bZSA=', 'Content-Length': '0'} 2023-10-12T00:56:22.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b2d1d58711a297-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.906 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.906 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.907 DEBUG:boto:Method: HEAD 2023-10-12T00:56:22.907 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.908 DEBUG:boto:Data: 2023-10-12T00:56:22.908 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.909 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.909 DEBUG:boto:Params: {} 2023-10-12T00:56:22.910 DEBUG:boto:Token: None 2023-10-12T00:56:22.910 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OQuPOjofeL+MTaOeKgJ2pc8bZSA= 2023-10-12T00:56:22.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OQuPOjofeL+MTaOeKgJ2pc8bZSA=', 'Content-Length': '0'} 2023-10-12T00:56:22.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d5f903c6476b83-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:22.914 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.915 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.915 DEBUG:boto:Method: GET 2023-10-12T00:56:22.916 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.916 DEBUG:boto:Data: 2023-10-12T00:56:22.917 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.917 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.918 DEBUG:boto:Params: {} 2023-10-12T00:56:22.918 DEBUG:boto:Token: None 2023-10-12T00:56:22.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pqewpdsxE0cZwA38tfgQop0Y6Jo= 2023-10-12T00:56:22.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pqewpdsxE0cZwA38tfgQop0Y6Jo=', 'Content-Length': '0'} 2023-10-12T00:56:22.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9ef32fba65a706-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.923 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.923 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.924 DEBUG:boto:Method: GET 2023-10-12T00:56:22.924 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.924 DEBUG:boto:Data: 2023-10-12T00:56:22.925 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.926 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.926 DEBUG:boto:Params: {} 2023-10-12T00:56:22.927 DEBUG:boto:Token: None 2023-10-12T00:56:22.927 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:22.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pqewpdsxE0cZwA38tfgQop0Y6Jo= 2023-10-12T00:56:22.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pqewpdsxE0cZwA38tfgQop0Y6Jo=', 'Content-Length': '0'} 2023-10-12T00:56:22.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073eaecdab91e4773-0065274436-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.931 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.931 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.932 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.932 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.933 DEBUG:boto:Method: GET 2023-10-12T00:56:22.933 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.933 DEBUG:boto:Data: 2023-10-12T00:56:22.934 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.935 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.935 DEBUG:boto:Params: {} 2023-10-12T00:56:22.936 DEBUG:boto:Token: None 2023-10-12T00:56:22.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7e4bcae74e9fd5-0065274436-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.940 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.941 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:22.941 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:22.942 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.942 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.943 DEBUG:boto:Method: GET 2023-10-12T00:56:22.943 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:22.943 DEBUG:boto:Data: 2023-10-12T00:56:22.944 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:22.945 DEBUG:boto:Port: 8000 2023-10-12T00:56:22.945 DEBUG:boto:Params: {} 2023-10-12T00:56:22.946 DEBUG:boto:Token: None 2023-10-12T00:56:22.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/ 2023-10-12T00:56:22.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4= 2023-10-12T00:56:22.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4uRTFCgQ1rgc9Aqzzb6ECEzgYw4=', 'Content-Length': '0'} 2023-10-12T00:56:22.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000532fb6def6605562-0065274436-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:22.950 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:06.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:22.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:22.951 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:22.951 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:22.952 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:22.952 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:22.953 DEBUG:boto:Method: GET 2023-10-12T00:56:22.953 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:22.954 DEBUG:boto:Data: 2023-10-12T00:56:22.954 DEBUG:boto:Headers: {} 2023-10-12T00:56:22.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:22.955 DEBUG:boto:Port: 8001 2023-10-12T00:56:22.955 DEBUG:boto:Params: {} 2023-10-12T00:56:22.956 DEBUG:boto:Token: None 2023-10-12T00:56:22.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:22 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:22.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YsuBYTthphchuxQBtJpvaQ6Vy9s= 2023-10-12T00:56:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YsuBYTthphchuxQBtJpvaQ6Vy9s=', 'Content-Length': '0'} 2023-10-12T00:56:22.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611b0229e3eb8857-0065274436-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.002 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.003 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.003 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.004 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.004 DEBUG:boto:Method: GET 2023-10-12T00:56:23.005 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.005 DEBUG:boto:Data: 2023-10-12T00:56:23.005 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.006 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.007 DEBUG:boto:Params: {} 2023-10-12T00:56:23.007 DEBUG:boto:Token: None 2023-10-12T00:56:23.008 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs= 2023-10-12T00:56:23.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs=', 'Content-Length': '0'} 2023-10-12T00:56:23.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2dc07184b78916-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.011 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.012 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.014 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.014 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.014 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.015 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.015 DEBUG:boto:Data: 2023-10-12T00:56:23.016 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.016 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.017 DEBUG:boto:Params: {} 2023-10-12T00:56:23.017 DEBUG:boto:Token: None 2023-10-12T00:56:23.018 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8pnQwRQkh9GrJ/Ojs+Za5AH0Z0E= 2023-10-12T00:56:23.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8pnQwRQkh9GrJ/Ojs+Za5AH0Z0E=', 'Content-Length': '0'} 2023-10-12T00:56:23.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0135cecce49350-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.021 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.021 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.022 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.022 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.022 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.023 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.023 DEBUG:boto:Data: 2023-10-12T00:56:23.024 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.024 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.025 DEBUG:boto:Params: {} 2023-10-12T00:56:23.025 DEBUG:boto:Token: None 2023-10-12T00:56:23.025 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8pnQwRQkh9GrJ/Ojs+Za5AH0Z0E= 2023-10-12T00:56:23.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8pnQwRQkh9GrJ/Ojs+Za5AH0Z0E=', 'Content-Length': '0'} 2023-10-12T00:56:23.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000688af73e54b1e565-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:23.029 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.030 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.030 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.031 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.031 DEBUG:boto:Method: GET 2023-10-12T00:56:23.031 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.032 DEBUG:boto:Data: 2023-10-12T00:56:23.032 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.033 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.033 DEBUG:boto:Params: {} 2023-10-12T00:56:23.034 DEBUG:boto:Token: None 2023-10-12T00:56:23.034 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.035 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs= 2023-10-12T00:56:23.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs=', 'Content-Length': '0'} 2023-10-12T00:56:23.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47962bb289ff78b-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.038 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.038 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.039 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.039 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.040 DEBUG:boto:Method: GET 2023-10-12T00:56:23.040 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.040 DEBUG:boto:Data: 2023-10-12T00:56:23.041 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.042 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.042 DEBUG:boto:Params: {} 2023-10-12T00:56:23.043 DEBUG:boto:Token: None 2023-10-12T00:56:23.043 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:23.043 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs= 2023-10-12T00:56:23.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+WJeYWLvxFHqTTg7e0Gauac2TPs=', 'Content-Length': '0'} 2023-10-12T00:56:23.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007430267239997405-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.086 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.087 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.087 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.088 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.088 DEBUG:boto:Method: GET 2023-10-12T00:56:23.089 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.089 DEBUG:boto:Data: 2023-10-12T00:56:23.089 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.090 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.091 DEBUG:boto:Params: {} 2023-10-12T00:56:23.091 DEBUG:boto:Token: None 2023-10-12T00:56:23.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010651f747c5407a4-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.096 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.096 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.097 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.097 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.098 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.098 DEBUG:boto:Method: GET 2023-10-12T00:56:23.099 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.099 DEBUG:boto:Data: 2023-10-12T00:56:23.100 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.101 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.101 DEBUG:boto:Params: {} 2023-10-12T00:56:23.101 DEBUG:boto:Token: None 2023-10-12T00:56:23.102 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001414883f4fa8371a-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.106 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.106 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.107 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.107 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.108 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.108 DEBUG:boto:Data: 2023-10-12T00:56:23.108 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.109 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.110 DEBUG:boto:Params: {} 2023-10-12T00:56:23.110 DEBUG:boto:Token: None 2023-10-12T00:56:23.111 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.111 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5T1Ob+u6T4HFjVRyY/7XnIRu2YU= 2023-10-12T00:56:23.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5T1Ob+u6T4HFjVRyY/7XnIRu2YU=', 'Content-Length': '0'} 2023-10-12T00:56:23.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06a3ce95be3da24-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.114 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.115 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.115 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.115 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.116 DEBUG:boto:Data: 2023-10-12T00:56:23.116 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.117 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.117 DEBUG:boto:Params: {} 2023-10-12T00:56:23.118 DEBUG:boto:Token: None 2023-10-12T00:56:23.118 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5T1Ob+u6T4HFjVRyY/7XnIRu2YU= 2023-10-12T00:56:23.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5T1Ob+u6T4HFjVRyY/7XnIRu2YU=', 'Content-Length': '0'} 2023-10-12T00:56:23.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687c18d1b4941479-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:23.122 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.123 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.123 DEBUG:boto:Method: GET 2023-10-12T00:56:23.124 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.124 DEBUG:boto:Data: 2023-10-12T00:56:23.124 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.125 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.126 DEBUG:boto:Params: {} 2023-10-12T00:56:23.126 DEBUG:boto:Token: None 2023-10-12T00:56:23.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:40OlZRysWimyfe95flSd4o9LPb4= 2023-10-12T00:56:23.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:40OlZRysWimyfe95flSd4o9LPb4=', 'Content-Length': '0'} 2023-10-12T00:56:23.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d914aadc56411f-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.130 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.131 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.131 DEBUG:boto:Method: GET 2023-10-12T00:56:23.132 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.132 DEBUG:boto:Data: 2023-10-12T00:56:23.132 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.133 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.134 DEBUG:boto:Params: {} 2023-10-12T00:56:23.134 DEBUG:boto:Token: None 2023-10-12T00:56:23.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:23.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:40OlZRysWimyfe95flSd4o9LPb4= 2023-10-12T00:56:23.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:40OlZRysWimyfe95flSd4o9LPb4=', 'Content-Length': '0'} 2023-10-12T00:56:23.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d154c9e265d4aa2e-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.138 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.139 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.139 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.139 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.140 DEBUG:boto:Method: GET 2023-10-12T00:56:23.140 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.141 DEBUG:boto:Data: 2023-10-12T00:56:23.141 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.142 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.142 DEBUG:boto:Params: {} 2023-10-12T00:56:23.143 DEBUG:boto:Token: None 2023-10-12T00:56:23.143 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d874fb186e5a037-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.147 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.148 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.148 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.149 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.149 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.150 DEBUG:boto:Method: GET 2023-10-12T00:56:23.150 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:23.150 DEBUG:boto:Data: 2023-10-12T00:56:23.151 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.152 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.152 DEBUG:boto:Params: {} 2023-10-12T00:56:23.153 DEBUG:boto:Token: None 2023-10-12T00:56:23.153 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.154 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2772c57f2264ba0-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.158 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:06.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:23.159 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.159 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.160 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.160 DEBUG:boto:Method: GET 2023-10-12T00:56:23.161 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.161 DEBUG:boto:Data: 2023-10-12T00:56:23.162 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.162 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.163 DEBUG:boto:Params: {} 2023-10-12T00:56:23.163 DEBUG:boto:Token: None 2023-10-12T00:56:23.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g= 2023-10-12T00:56:23.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g=', 'Content-Length': '0'} 2023-10-12T00:56:23.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5aba5d7797e920a-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.214 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.215 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.215 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.216 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.216 DEBUG:boto:Method: GET 2023-10-12T00:56:23.217 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.217 DEBUG:boto:Data: 2023-10-12T00:56:23.218 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.218 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.219 DEBUG:boto:Params: {} 2023-10-12T00:56:23.219 DEBUG:boto:Token: None 2023-10-12T00:56:23.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g= 2023-10-12T00:56:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g=', 'Content-Length': '0'} 2023-10-12T00:56:23.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456a15a9d9c14d4a-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.224 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.224 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.225 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.225 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.226 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.226 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.226 DEBUG:boto:Data: 2023-10-12T00:56:23.227 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.228 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.228 DEBUG:boto:Params: {} 2023-10-12T00:56:23.229 DEBUG:boto:Token: None 2023-10-12T00:56:23.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.230 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hM63Y81frih8TlkazcLovdLEfSA= 2023-10-12T00:56:23.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hM63Y81frih8TlkazcLovdLEfSA=', 'Content-Length': '0'} 2023-10-12T00:56:23.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000063aaaef526dfe1f-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.233 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.233 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.233 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.234 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.234 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.235 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.235 DEBUG:boto:Data: 2023-10-12T00:56:23.236 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.236 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.237 DEBUG:boto:Params: {} 2023-10-12T00:56:23.237 DEBUG:boto:Token: None 2023-10-12T00:56:23.238 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hM63Y81frih8TlkazcLovdLEfSA= 2023-10-12T00:56:23.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hM63Y81frih8TlkazcLovdLEfSA=', 'Content-Length': '0'} 2023-10-12T00:56:23.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c18d33173401156-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:23.242 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.242 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.243 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.243 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.244 DEBUG:boto:Method: GET 2023-10-12T00:56:23.244 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.244 DEBUG:boto:Data: 2023-10-12T00:56:23.245 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.246 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.246 DEBUG:boto:Params: {} 2023-10-12T00:56:23.247 DEBUG:boto:Token: None 2023-10-12T00:56:23.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g= 2023-10-12T00:56:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g=', 'Content-Length': '0'} 2023-10-12T00:56:23.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093e0a1acb1aca63c-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.251 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.252 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.252 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.253 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.253 DEBUG:boto:Method: GET 2023-10-12T00:56:23.253 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.254 DEBUG:boto:Data: 2023-10-12T00:56:23.254 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.255 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.256 DEBUG:boto:Params: {} 2023-10-12T00:56:23.256 DEBUG:boto:Token: None 2023-10-12T00:56:23.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:23.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g= 2023-10-12T00:56:23.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/Xq8FxCLFtmjOnsfVC2dUSXZm5g=', 'Content-Length': '0'} 2023-10-12T00:56:23.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df1e9e7b2a11e7e-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.260 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.261 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.261 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.262 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.262 DEBUG:boto:Method: GET 2023-10-12T00:56:23.263 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.263 DEBUG:boto:Data: 2023-10-12T00:56:23.263 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.264 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.265 DEBUG:boto:Params: {} 2023-10-12T00:56:23.265 DEBUG:boto:Token: None 2023-10-12T00:56:23.266 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a55522765b4a4a3-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.270 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.271 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.271 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.271 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.272 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.272 DEBUG:boto:Method: GET 2023-10-12T00:56:23.273 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.273 DEBUG:boto:Data: 2023-10-12T00:56:23.274 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.275 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.275 DEBUG:boto:Params: {} 2023-10-12T00:56:23.276 DEBUG:boto:Token: None 2023-10-12T00:56:23.276 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020304e0a85e692d9-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.280 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.281 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.281 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.282 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.282 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.283 DEBUG:boto:Data: 2023-10-12T00:56:23.283 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.284 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.284 DEBUG:boto:Params: {} 2023-10-12T00:56:23.285 DEBUG:boto:Token: None 2023-10-12T00:56:23.285 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfTEw6s8eFE5sg4V3tKayOj6ErE= 2023-10-12T00:56:23.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfTEw6s8eFE5sg4V3tKayOj6ErE=', 'Content-Length': '0'} 2023-10-12T00:56:23.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b994e9f8d8c8d886-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.289 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.289 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.290 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.290 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.290 DEBUG:boto:Data: 2023-10-12T00:56:23.291 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.292 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.292 DEBUG:boto:Params: {} 2023-10-12T00:56:23.293 DEBUG:boto:Token: None 2023-10-12T00:56:23.293 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfTEw6s8eFE5sg4V3tKayOj6ErE= 2023-10-12T00:56:23.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfTEw6s8eFE5sg4V3tKayOj6ErE=', 'Content-Length': '0'} 2023-10-12T00:56:23.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c0c31cc1940da6-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:23.297 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.297 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.298 DEBUG:boto:Method: GET 2023-10-12T00:56:23.298 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.299 DEBUG:boto:Data: 2023-10-12T00:56:23.299 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.300 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.300 DEBUG:boto:Params: {} 2023-10-12T00:56:23.301 DEBUG:boto:Token: None 2023-10-12T00:56:23.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4vrA1GqejVGxAraHoP2V/7MBRLw= 2023-10-12T00:56:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4vrA1GqejVGxAraHoP2V/7MBRLw=', 'Content-Length': '0'} 2023-10-12T00:56:23.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ab5abf1cf9474b-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.305 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.305 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.306 DEBUG:boto:Method: GET 2023-10-12T00:56:23.306 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.307 DEBUG:boto:Data: 2023-10-12T00:56:23.307 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.308 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.308 DEBUG:boto:Params: {} 2023-10-12T00:56:23.309 DEBUG:boto:Token: None 2023-10-12T00:56:23.309 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:23.310 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4vrA1GqejVGxAraHoP2V/7MBRLw= 2023-10-12T00:56:23.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4vrA1GqejVGxAraHoP2V/7MBRLw=', 'Content-Length': '0'} 2023-10-12T00:56:23.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a77815e2dc0d8be-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.354 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.355 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.355 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.356 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.356 DEBUG:boto:Method: GET 2023-10-12T00:56:23.357 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.357 DEBUG:boto:Data: 2023-10-12T00:56:23.357 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.358 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.359 DEBUG:boto:Params: {} 2023-10-12T00:56:23.359 DEBUG:boto:Token: None 2023-10-12T00:56:23.360 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019e4a3577bad5a0-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.364 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.365 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.365 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.365 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.366 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.366 DEBUG:boto:Method: GET 2023-10-12T00:56:23.367 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:23.367 DEBUG:boto:Data: 2023-10-12T00:56:23.368 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.368 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.369 DEBUG:boto:Params: {} 2023-10-12T00:56:23.369 DEBUG:boto:Token: None 2023-10-12T00:56:23.370 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.370 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000950b2ee7ce40fa5-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.374 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:06.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:23.375 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.375 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.375 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.376 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.376 DEBUG:boto:Method: GET 2023-10-12T00:56:23.377 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.377 DEBUG:boto:Data: 2023-10-12T00:56:23.378 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.378 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.379 DEBUG:boto:Params: {} 2023-10-12T00:56:23.379 DEBUG:boto:Token: None 2023-10-12T00:56:23.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk= 2023-10-12T00:56:23.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk=', 'Content-Length': '0'} 2023-10-12T00:56:23.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002700ce3f2fc68683-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.426 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.427 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.427 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.428 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.428 DEBUG:boto:Method: GET 2023-10-12T00:56:23.428 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.429 DEBUG:boto:Data: 2023-10-12T00:56:23.429 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.430 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.430 DEBUG:boto:Params: {} 2023-10-12T00:56:23.431 DEBUG:boto:Token: None 2023-10-12T00:56:23.431 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk= 2023-10-12T00:56:23.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk=', 'Content-Length': '0'} 2023-10-12T00:56:23.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fca02ddeff28c2-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.435 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.435 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.436 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.436 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.436 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.437 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.437 DEBUG:boto:Data: 2023-10-12T00:56:23.438 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.439 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.439 DEBUG:boto:Params: {} 2023-10-12T00:56:23.439 DEBUG:boto:Token: None 2023-10-12T00:56:23.440 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8QEDZEjuDE8l2wXv8bGAuM4oiJk= 2023-10-12T00:56:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8QEDZEjuDE8l2wXv8bGAuM4oiJk=', 'Content-Length': '0'} 2023-10-12T00:56:23.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aef155c410a8d74a-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.443 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.444 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.444 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.445 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.445 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.445 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.446 DEBUG:boto:Data: 2023-10-12T00:56:23.446 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.447 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.447 DEBUG:boto:Params: {} 2023-10-12T00:56:23.448 DEBUG:boto:Token: None 2023-10-12T00:56:23.448 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8QEDZEjuDE8l2wXv8bGAuM4oiJk= 2023-10-12T00:56:23.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8QEDZEjuDE8l2wXv8bGAuM4oiJk=', 'Content-Length': '0'} 2023-10-12T00:56:23.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a140742798147c82-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:23.452 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.452 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.453 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.453 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.454 DEBUG:boto:Method: GET 2023-10-12T00:56:23.454 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.455 DEBUG:boto:Data: 2023-10-12T00:56:23.455 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.456 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.456 DEBUG:boto:Params: {} 2023-10-12T00:56:23.457 DEBUG:boto:Token: None 2023-10-12T00:56:23.457 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.458 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk= 2023-10-12T00:56:23.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk=', 'Content-Length': '0'} 2023-10-12T00:56:23.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076dfd8cfe8a8de4d-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.461 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.461 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.462 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.462 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.463 DEBUG:boto:Method: GET 2023-10-12T00:56:23.463 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.463 DEBUG:boto:Data: 2023-10-12T00:56:23.464 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.465 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.465 DEBUG:boto:Params: {} 2023-10-12T00:56:23.466 DEBUG:boto:Token: None 2023-10-12T00:56:23.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:23.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk= 2023-10-12T00:56:23.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dJYEAxv9EWbDNGep4AqLTtnhzfk=', 'Content-Length': '0'} 2023-10-12T00:56:23.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000676c250e3c3a925f-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.510 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.511 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.511 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.512 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.512 DEBUG:boto:Method: GET 2023-10-12T00:56:23.513 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.513 DEBUG:boto:Data: 2023-10-12T00:56:23.514 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.515 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.515 DEBUG:boto:Params: {} 2023-10-12T00:56:23.515 DEBUG:boto:Token: None 2023-10-12T00:56:23.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024792412087a4be9-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.520 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.521 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.521 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.522 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.522 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.522 DEBUG:boto:Method: GET 2023-10-12T00:56:23.523 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.523 DEBUG:boto:Data: 2023-10-12T00:56:23.524 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.524 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.525 DEBUG:boto:Params: {} 2023-10-12T00:56:23.525 DEBUG:boto:Token: None 2023-10-12T00:56:23.526 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cc2d64da3fc4c1c-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.530 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.531 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.531 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.531 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.532 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.532 DEBUG:boto:Data: 2023-10-12T00:56:23.533 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.533 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.534 DEBUG:boto:Params: {} 2023-10-12T00:56:23.534 DEBUG:boto:Token: None 2023-10-12T00:56:23.535 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EU8NG6hZgnp5cxjdojrojvOylX0= 2023-10-12T00:56:23.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EU8NG6hZgnp5cxjdojrojvOylX0=', 'Content-Length': '0'} 2023-10-12T00:56:23.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8efff72c53c131-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.538 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.538 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.539 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.539 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.540 DEBUG:boto:Data: 2023-10-12T00:56:23.540 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.541 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.541 DEBUG:boto:Params: {} 2023-10-12T00:56:23.542 DEBUG:boto:Token: None 2023-10-12T00:56:23.542 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EU8NG6hZgnp5cxjdojrojvOylX0= 2023-10-12T00:56:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EU8NG6hZgnp5cxjdojrojvOylX0=', 'Content-Length': '0'} 2023-10-12T00:56:23.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5054d18e9ff8fa1-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:23.546 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.546 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.547 DEBUG:boto:Method: GET 2023-10-12T00:56:23.547 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.548 DEBUG:boto:Data: 2023-10-12T00:56:23.548 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.549 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.550 DEBUG:boto:Params: {} 2023-10-12T00:56:23.550 DEBUG:boto:Token: None 2023-10-12T00:56:23.550 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t9OYyXyUyAGSLHcaRugyp4sS4oc= 2023-10-12T00:56:23.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t9OYyXyUyAGSLHcaRugyp4sS4oc=', 'Content-Length': '0'} 2023-10-12T00:56:23.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40bc747f2652edd-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.594 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.595 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.596 DEBUG:boto:Method: GET 2023-10-12T00:56:23.596 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.596 DEBUG:boto:Data: 2023-10-12T00:56:23.597 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.598 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.598 DEBUG:boto:Params: {} 2023-10-12T00:56:23.599 DEBUG:boto:Token: None 2023-10-12T00:56:23.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:23.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t9OYyXyUyAGSLHcaRugyp4sS4oc= 2023-10-12T00:56:23.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t9OYyXyUyAGSLHcaRugyp4sS4oc=', 'Content-Length': '0'} 2023-10-12T00:56:23.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000798a6e8cf898225-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.603 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.604 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.604 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.604 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.605 DEBUG:boto:Method: GET 2023-10-12T00:56:23.606 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.606 DEBUG:boto:Data: 2023-10-12T00:56:23.606 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.607 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.608 DEBUG:boto:Params: {} 2023-10-12T00:56:23.608 DEBUG:boto:Token: None 2023-10-12T00:56:23.609 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f3e144fac82355-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.613 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.614 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.614 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.614 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.615 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.615 DEBUG:boto:Method: GET 2023-10-12T00:56:23.616 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:23.616 DEBUG:boto:Data: 2023-10-12T00:56:23.617 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.618 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.618 DEBUG:boto:Params: {} 2023-10-12T00:56:23.619 DEBUG:boto:Token: None 2023-10-12T00:56:23.619 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05089904dafc960-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.623 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:06.338Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:23.624 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.624 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.625 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.625 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.625 DEBUG:boto:Method: GET 2023-10-12T00:56:23.626 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.626 DEBUG:boto:Data: 2023-10-12T00:56:23.627 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.631 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.632 DEBUG:boto:Params: {} 2023-10-12T00:56:23.632 DEBUG:boto:Token: None 2023-10-12T00:56:23.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE= 2023-10-12T00:56:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE=', 'Content-Length': '0'} 2023-10-12T00:56:23.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c2b6b697ca8a06-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.636 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.637 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.637 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.638 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.638 DEBUG:boto:Method: GET 2023-10-12T00:56:23.639 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.639 DEBUG:boto:Data: 2023-10-12T00:56:23.639 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.640 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.641 DEBUG:boto:Params: {} 2023-10-12T00:56:23.641 DEBUG:boto:Token: None 2023-10-12T00:56:23.642 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.642 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE= 2023-10-12T00:56:23.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE=', 'Content-Length': '0'} 2023-10-12T00:56:23.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4aa8826dd33fae-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.686 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.687 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.688 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.688 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.689 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.689 DEBUG:boto:Data: 2023-10-12T00:56:23.689 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.690 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.691 DEBUG:boto:Params: {} 2023-10-12T00:56:23.691 DEBUG:boto:Token: None 2023-10-12T00:56:23.692 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rL+5yhnKlJtOjuyeYV5yF+Q5qTU= 2023-10-12T00:56:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rL+5yhnKlJtOjuyeYV5yF+Q5qTU=', 'Content-Length': '0'} 2023-10-12T00:56:23.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c172392560d95a-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.695 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.696 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.696 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.696 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.697 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.697 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.698 DEBUG:boto:Data: 2023-10-12T00:56:23.698 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.699 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.699 DEBUG:boto:Params: {} 2023-10-12T00:56:23.700 DEBUG:boto:Token: None 2023-10-12T00:56:23.700 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rL+5yhnKlJtOjuyeYV5yF+Q5qTU= 2023-10-12T00:56:23.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rL+5yhnKlJtOjuyeYV5yF+Q5qTU=', 'Content-Length': '0'} 2023-10-12T00:56:23.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcaade19f90f56d2-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:23.704 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.705 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.705 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.706 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.706 DEBUG:boto:Method: GET 2023-10-12T00:56:23.707 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.707 DEBUG:boto:Data: 2023-10-12T00:56:23.708 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.708 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.709 DEBUG:boto:Params: {} 2023-10-12T00:56:23.709 DEBUG:boto:Token: None 2023-10-12T00:56:23.710 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE= 2023-10-12T00:56:23.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE=', 'Content-Length': '0'} 2023-10-12T00:56:23.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc90bc1592186ef8-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.714 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.714 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.714 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.715 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.715 DEBUG:boto:Method: GET 2023-10-12T00:56:23.716 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.716 DEBUG:boto:Data: 2023-10-12T00:56:23.717 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.717 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.718 DEBUG:boto:Params: {} 2023-10-12T00:56:23.718 DEBUG:boto:Token: None 2023-10-12T00:56:23.719 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:23.719 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE= 2023-10-12T00:56:23.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+bRIisdtQbMrXg9XzGe2ogcpmmE=', 'Content-Length': '0'} 2023-10-12T00:56:23.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b245a6012c2f5e4-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.722 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.723 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.723 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.724 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.724 DEBUG:boto:Method: GET 2023-10-12T00:56:23.724 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.725 DEBUG:boto:Data: 2023-10-12T00:56:23.725 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.726 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.727 DEBUG:boto:Params: {} 2023-10-12T00:56:23.727 DEBUG:boto:Token: None 2023-10-12T00:56:23.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a329da1194ba21e-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.732 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.732 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.733 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.733 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.734 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.734 DEBUG:boto:Method: GET 2023-10-12T00:56:23.735 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.735 DEBUG:boto:Data: 2023-10-12T00:56:23.735 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.736 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.737 DEBUG:boto:Params: {} 2023-10-12T00:56:23.737 DEBUG:boto:Token: None 2023-10-12T00:56:23.738 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eea7d296dd4cbdb-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.741 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.742 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.742 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.743 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.743 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.744 DEBUG:boto:Data: 2023-10-12T00:56:23.744 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.745 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.745 DEBUG:boto:Params: {} 2023-10-12T00:56:23.746 DEBUG:boto:Token: None 2023-10-12T00:56:23.746 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TDifxDbapx2Gbq94DgqvqCuh1lU= 2023-10-12T00:56:23.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TDifxDbapx2Gbq94DgqvqCuh1lU=', 'Content-Length': '0'} 2023-10-12T00:56:23.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940e195f031155de-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.750 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.750 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.750 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.751 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.751 DEBUG:boto:Data: 2023-10-12T00:56:23.752 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.752 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.753 DEBUG:boto:Params: {} 2023-10-12T00:56:23.753 DEBUG:boto:Token: None 2023-10-12T00:56:23.754 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TDifxDbapx2Gbq94DgqvqCuh1lU= 2023-10-12T00:56:23.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TDifxDbapx2Gbq94DgqvqCuh1lU=', 'Content-Length': '0'} 2023-10-12T00:56:23.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761400835e0d31df-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:23.757 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.758 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.758 DEBUG:boto:Method: GET 2023-10-12T00:56:23.759 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.759 DEBUG:boto:Data: 2023-10-12T00:56:23.760 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.761 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.761 DEBUG:boto:Params: {} 2023-10-12T00:56:23.761 DEBUG:boto:Token: None 2023-10-12T00:56:23.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sydT0O9eHLtlZTJsDJnQO/z+NqA= 2023-10-12T00:56:23.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sydT0O9eHLtlZTJsDJnQO/z+NqA=', 'Content-Length': '0'} 2023-10-12T00:56:23.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4341e46229147da-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.806 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.807 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.808 DEBUG:boto:Method: GET 2023-10-12T00:56:23.809 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.810 DEBUG:boto:Data: 2023-10-12T00:56:23.811 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.811 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.812 DEBUG:boto:Params: {} 2023-10-12T00:56:23.812 DEBUG:boto:Token: None 2023-10-12T00:56:23.813 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:23.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sydT0O9eHLtlZTJsDJnQO/z+NqA= 2023-10-12T00:56:23.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sydT0O9eHLtlZTJsDJnQO/z+NqA=', 'Content-Length': '0'} 2023-10-12T00:56:23.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7dbd4814352f55-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.817 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.818 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.818 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.819 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.819 DEBUG:boto:Method: GET 2023-10-12T00:56:23.820 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.821 DEBUG:boto:Data: 2023-10-12T00:56:23.822 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.823 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.824 DEBUG:boto:Params: {} 2023-10-12T00:56:23.825 DEBUG:boto:Token: None 2023-10-12T00:56:23.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde99b66f00f70c3-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.830 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.831 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.832 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.832 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.833 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.833 DEBUG:boto:Method: GET 2023-10-12T00:56:23.834 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:23.834 DEBUG:boto:Data: 2023-10-12T00:56:23.835 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.836 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.837 DEBUG:boto:Params: {} 2023-10-12T00:56:23.838 DEBUG:boto:Token: None 2023-10-12T00:56:23.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0a24f8abda2754-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.843 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:06.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:23.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.845 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.846 DEBUG:boto:Method: GET 2023-10-12T00:56:23.847 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.847 DEBUG:boto:Data: 2023-10-12T00:56:23.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.849 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.849 DEBUG:boto:Params: {} 2023-10-12T00:56:23.850 DEBUG:boto:Token: None 2023-10-12T00:56:23.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho= 2023-10-12T00:56:23.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho=', 'Content-Length': '0'} 2023-10-12T00:56:23.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007628b2a6fb30dab7-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.856 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.857 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.857 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.858 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.859 DEBUG:boto:Method: GET 2023-10-12T00:56:23.859 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.859 DEBUG:boto:Data: 2023-10-12T00:56:23.860 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.861 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.861 DEBUG:boto:Params: {} 2023-10-12T00:56:23.862 DEBUG:boto:Token: None 2023-10-12T00:56:23.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho= 2023-10-12T00:56:23.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho=', 'Content-Length': '0'} 2023-10-12T00:56:23.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0564927e06403c0-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.906 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.907 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.908 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.908 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.909 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.910 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.910 DEBUG:boto:Data: 2023-10-12T00:56:23.911 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.911 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.912 DEBUG:boto:Params: {} 2023-10-12T00:56:23.912 DEBUG:boto:Token: None 2023-10-12T00:56:23.913 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c6qrpdCR6ead2aDIFuqYDp+1pNA= 2023-10-12T00:56:23.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c6qrpdCR6ead2aDIFuqYDp+1pNA=', 'Content-Length': '0'} 2023-10-12T00:56:23.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e97a85bd331174-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.916 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.917 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.917 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.918 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.919 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.920 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.920 DEBUG:boto:Data: 2023-10-12T00:56:23.921 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.922 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.923 DEBUG:boto:Params: {} 2023-10-12T00:56:23.924 DEBUG:boto:Token: None 2023-10-12T00:56:23.924 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c6qrpdCR6ead2aDIFuqYDp+1pNA= 2023-10-12T00:56:23.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c6qrpdCR6ead2aDIFuqYDp+1pNA=', 'Content-Length': '0'} 2023-10-12T00:56:23.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009958aeccb517957-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:23.928 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.929 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.929 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.930 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.930 DEBUG:boto:Method: GET 2023-10-12T00:56:23.931 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.931 DEBUG:boto:Data: 2023-10-12T00:56:23.932 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.933 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.933 DEBUG:boto:Params: {} 2023-10-12T00:56:23.934 DEBUG:boto:Token: None 2023-10-12T00:56:23.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.935 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho= 2023-10-12T00:56:23.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho=', 'Content-Length': '0'} 2023-10-12T00:56:23.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866a156f23a45480-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.938 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.939 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.939 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.939 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.940 DEBUG:boto:Method: GET 2023-10-12T00:56:23.940 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.941 DEBUG:boto:Data: 2023-10-12T00:56:23.941 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.942 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.943 DEBUG:boto:Params: {} 2023-10-12T00:56:23.943 DEBUG:boto:Token: None 2023-10-12T00:56:23.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:23.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho= 2023-10-12T00:56:23.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3e+YcL1n61byphAuJbatAgsS0ho=', 'Content-Length': '0'} 2023-10-12T00:56:23.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007736a3f0ebb17930-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.948 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.948 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.949 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.949 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.950 DEBUG:boto:Method: GET 2023-10-12T00:56:23.950 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.951 DEBUG:boto:Data: 2023-10-12T00:56:23.952 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.954 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.954 DEBUG:boto:Params: {} 2023-10-12T00:56:23.955 DEBUG:boto:Token: None 2023-10-12T00:56:23.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cbdcbe0ee31712b-0065274437-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.960 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.961 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:23.961 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:23.962 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.962 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.963 DEBUG:boto:Method: GET 2023-10-12T00:56:23.963 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:23.964 DEBUG:boto:Data: 2023-10-12T00:56:23.964 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.965 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.965 DEBUG:boto:Params: {} 2023-10-12T00:56:23.966 DEBUG:boto:Token: None 2023-10-12T00:56:23.966 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/ 2023-10-12T00:56:23.967 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do= 2023-10-12T00:56:23.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UHt84F93GRh9I3Ro+Envwuwy/do=', 'Content-Length': '0'} 2023-10-12T00:56:23.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbccf1bda8f4e8bf-0065274437-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.970 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:23.971 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.971 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.972 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.972 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.972 DEBUG:boto:Data: 2023-10-12T00:56:23.973 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.974 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.974 DEBUG:boto:Params: {} 2023-10-12T00:56:23.975 DEBUG:boto:Token: None 2023-10-12T00:56:23.975 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EMPYgpiJ//9rmDnCD5IxVt8DmI0= 2023-10-12T00:56:23.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EMPYgpiJ//9rmDnCD5IxVt8DmI0=', 'Content-Length': '0'} 2023-10-12T00:56:23.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb76161f9da09a7-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.979 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.979 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.980 DEBUG:boto:Method: HEAD 2023-10-12T00:56:23.980 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.980 DEBUG:boto:Data: 2023-10-12T00:56:23.981 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:23.982 DEBUG:boto:Port: 8000 2023-10-12T00:56:23.982 DEBUG:boto:Params: {} 2023-10-12T00:56:23.983 DEBUG:boto:Token: None 2023-10-12T00:56:23.983 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.983 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EMPYgpiJ//9rmDnCD5IxVt8DmI0= 2023-10-12T00:56:23.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EMPYgpiJ//9rmDnCD5IxVt8DmI0=', 'Content-Length': '0'} 2023-10-12T00:56:23.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f16834541d1203-0065274437-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:23.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:23.987 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.987 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.988 DEBUG:boto:Method: GET 2023-10-12T00:56:23.988 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.989 DEBUG:boto:Data: 2023-10-12T00:56:23.989 DEBUG:boto:Headers: {} 2023-10-12T00:56:23.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:23.990 DEBUG:boto:Port: 8001 2023-10-12T00:56:23.991 DEBUG:boto:Params: {} 2023-10-12T00:56:23.992 DEBUG:boto:Token: None 2023-10-12T00:56:23.992 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:23 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:23.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GfedaqiTM5VZsuSUx9JQ7z2mMso= 2023-10-12T00:56:23.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GfedaqiTM5VZsuSUx9JQ7z2mMso=', 'Content-Length': '0'} 2023-10-12T00:56:23.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15e96b76c7adeca-0065274437-1aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:24.040 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:24.041 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:24.042 DEBUG:boto:Method: GET 2023-10-12T00:56:24.042 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:24.043 DEBUG:boto:Data: 2023-10-12T00:56:24.044 DEBUG:boto:Headers: {} 2023-10-12T00:56:24.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:24.045 DEBUG:boto:Port: 8000 2023-10-12T00:56:24.045 DEBUG:boto:Params: {} 2023-10-12T00:56:24.046 DEBUG:boto:Token: None 2023-10-12T00:56:24.047 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:24 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:24.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXdv3Elq12D3q7Mlxca4u0n1wsg= 2023-10-12T00:56:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXdv3Elq12D3q7Mlxca4u0n1wsg=', 'Content-Length': '0'} 2023-10-12T00:56:24.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241cb52b210207c4-0065274438-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:24.050 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:24.051 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:24.051 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.051 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.052 DEBUG:boto:Method: GET 2023-10-12T00:56:24.052 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.053 DEBUG:boto:Data: 2023-10-12T00:56:24.053 DEBUG:boto:Headers: {} 2023-10-12T00:56:24.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:24.054 DEBUG:boto:Port: 8001 2023-10-12T00:56:24.054 DEBUG:boto:Params: {} 2023-10-12T00:56:24.055 DEBUG:boto:Token: None 2023-10-12T00:56:24.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:24 GMT /wafmoa-58/ 2023-10-12T00:56:24.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QOV2l9KQsh+HIttDldjTWSeJcLk= 2023-10-12T00:56:24.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QOV2l9KQsh+HIttDldjTWSeJcLk=', 'Content-Length': '0'} 2023-10-12T00:56:24.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9306629ac4885d2-0065274438-1aea-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:24.059 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:24.060 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:24.060 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:24.061 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.061 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.062 DEBUG:boto:Method: GET 2023-10-12T00:56:24.063 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:24.064 DEBUG:boto:Data: 2023-10-12T00:56:24.064 DEBUG:boto:Headers: {} 2023-10-12T00:56:24.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:24.066 DEBUG:boto:Port: 8000 2023-10-12T00:56:24.067 DEBUG:boto:Params: {} 2023-10-12T00:56:24.067 DEBUG:boto:Token: None 2023-10-12T00:56:24.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:24 GMT /wafmoa-58/ 2023-10-12T00:56:24.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QOV2l9KQsh+HIttDldjTWSeJcLk= 2023-10-12T00:56:24.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QOV2l9KQsh+HIttDldjTWSeJcLk=', 'Content-Length': '0'} 2023-10-12T00:56:24.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008176d546bb7f0a37-0065274438-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:24.072 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:06.421Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:24.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-58 zones={a2, a1} 2023-10-12T00:56:24.074 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T00:56:24.074 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T00:56:24.075 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:56:24.075+0000 7ff011973700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 158492) UID: 0 2023-10-12T00:56:24.075 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T00:56:24.075+0000 7ff015bdc7c0 -1 shutting down 2023-10-12T00:56:28.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T00:56:30.173 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T00:56:30.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:30.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:30.175 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:30.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:30.238 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:30.283 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:30.284 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:30.287 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-10-12T00:56:30.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:30.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:30.324 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:30.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:30.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:30.417 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:30.417 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:30.418 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-10-12T00:56:30.418 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-10-12T00:56:30.418 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000024", 2023-10-12T00:56:30.418 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-10-12T00:56:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000028", 2023-10-12T00:56:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000032", 2023-10-12T00:56:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:30.420 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-10-12T00:56:30.420 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-10-12T00:56:30.420 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-10-12T00:56:30.420 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-10-12T00:56:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-10-12T00:56:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-10-12T00:56:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000076" 2023-10-12T00:56:30.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:30.422 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:30.423 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.423 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.424 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.425 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.425 DEBUG:boto:Data: 2023-10-12T00:56:30.426 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.427 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.427 DEBUG:boto:Params: {} 2023-10-12T00:56:30.428 DEBUG:boto:Token: None 2023-10-12T00:56:30.428 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d856a2e4de7a1d-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.433 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:30.433 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:30.434 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.434 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:30.434 DEBUG:boto:Data: 2023-10-12T00:56:30.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.436 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.436 DEBUG:boto:Params: {} 2023-10-12T00:56:30.437 DEBUG:boto:Token: None 2023-10-12T00:56:30.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:30.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OlZgKDSm3Wha1OYBl3uQ7NsRQ2I= 2023-10-12T00:56:30.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OlZgKDSm3Wha1OYBl3uQ7NsRQ2I='} 2023-10-12T00:56:30.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ca0b9bd25831f4c-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.444 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.444 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.445 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.445 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.445 DEBUG:boto:Data: 2023-10-12T00:56:30.446 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.447 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.447 DEBUG:boto:Params: {} 2023-10-12T00:56:30.448 DEBUG:boto:Token: None 2023-10-12T00:56:30.448 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054df0ea02cf7b98c-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.452 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:30.452 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:30.453 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.453 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:30.454 DEBUG:boto:Data: 2023-10-12T00:56:30.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.455 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.456 DEBUG:boto:Params: {} 2023-10-12T00:56:30.456 DEBUG:boto:Token: None 2023-10-12T00:56:30.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:30.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3cPpNQiDQkQE/e/DEETm52qAnsU= 2023-10-12T00:56:30.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3cPpNQiDQkQE/e/DEETm52qAnsU='} 2023-10-12T00:56:30.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0838508931b3453-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.507 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.508 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.508 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.508 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.509 DEBUG:boto:Data: 2023-10-12T00:56:30.509 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.510 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.510 DEBUG:boto:Params: {} 2023-10-12T00:56:30.511 DEBUG:boto:Token: None 2023-10-12T00:56:30.511 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc7c6cabe796e05e-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.515 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:30.516 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:30.516 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.517 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:30.517 DEBUG:boto:Data: 2023-10-12T00:56:30.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.518 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.519 DEBUG:boto:Params: {} 2023-10-12T00:56:30.519 DEBUG:boto:Token: None 2023-10-12T00:56:30.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:30.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eHGNURh+hcHoqgHC45PNbeP1oZw= 2023-10-12T00:56:30.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eHGNURh+hcHoqgHC45PNbeP1oZw='} 2023-10-12T00:56:30.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac9c07373ccdf923-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.527 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.527 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.528 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.528 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.528 DEBUG:boto:Data: 2023-10-12T00:56:30.529 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.529 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.530 DEBUG:boto:Params: {} 2023-10-12T00:56:30.530 DEBUG:boto:Token: None 2023-10-12T00:56:30.531 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8df0a9412e59599-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.535 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:30.536 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:30.536 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.536 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:30.537 DEBUG:boto:Data: 2023-10-12T00:56:30.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.538 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.539 DEBUG:boto:Params: {} 2023-10-12T00:56:30.539 DEBUG:boto:Token: None 2023-10-12T00:56:30.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:30.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kmb2OxG1jzRnLlIAg2SG4ZCUqhE= 2023-10-12T00:56:30.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kmb2OxG1jzRnLlIAg2SG4ZCUqhE='} 2023-10-12T00:56:30.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044c994ed897be24e-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.587 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.587 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.588 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.588 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.589 DEBUG:boto:Data: 2023-10-12T00:56:30.589 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.590 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.590 DEBUG:boto:Params: {} 2023-10-12T00:56:30.591 DEBUG:boto:Token: None 2023-10-12T00:56:30.591 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eb69f0ef42f5a8c-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.595 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:30.595 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:30.596 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.596 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:30.597 DEBUG:boto:Data: 2023-10-12T00:56:30.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.598 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.598 DEBUG:boto:Params: {} 2023-10-12T00:56:30.599 DEBUG:boto:Token: None 2023-10-12T00:56:30.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:30.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pteaidCx0Z3e/oUBT35Gl7uDSVA= 2023-10-12T00:56:30.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pteaidCx0Z3e/oUBT35Gl7uDSVA='} 2023-10-12T00:56:30.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4141a0037214015-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.606 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.606 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.607 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.607 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.607 DEBUG:boto:Data: 2023-10-12T00:56:30.608 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.608 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.609 DEBUG:boto:Params: {} 2023-10-12T00:56:30.609 DEBUG:boto:Token: None 2023-10-12T00:56:30.610 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088119cb5d84a8413-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.613 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:30.614 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:30.614 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.615 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:30.615 DEBUG:boto:Data: 2023-10-12T00:56:30.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.617 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.617 DEBUG:boto:Params: {} 2023-10-12T00:56:30.617 DEBUG:boto:Token: None 2023-10-12T00:56:30.618 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:30.618 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WrSCB1l9zxiBg0vP9t0upOaZX5k= 2023-10-12T00:56:30.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WrSCB1l9zxiBg0vP9t0upOaZX5k='} 2023-10-12T00:56:30.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000077211ad86052df0-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.663 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.663 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.664 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.664 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.665 DEBUG:boto:Data: 2023-10-12T00:56:30.665 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.666 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.666 DEBUG:boto:Params: {} 2023-10-12T00:56:30.667 DEBUG:boto:Token: None 2023-10-12T00:56:30.667 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000995c593a165ce19d-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.671 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:30.672 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:30.672 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.672 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:30.673 DEBUG:boto:Data: 2023-10-12T00:56:30.673 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.674 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.674 DEBUG:boto:Params: {} 2023-10-12T00:56:30.675 DEBUG:boto:Token: None 2023-10-12T00:56:30.675 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:30.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NBFoo6kMZ/4eomRqnTkNNTVl8Wk= 2023-10-12T00:56:30.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NBFoo6kMZ/4eomRqnTkNNTVl8Wk='} 2023-10-12T00:56:30.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6d8786ee03709f3-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.682 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.682 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.683 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.683 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.684 DEBUG:boto:Data: 2023-10-12T00:56:30.684 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.685 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.685 DEBUG:boto:Params: {} 2023-10-12T00:56:30.686 DEBUG:boto:Token: None 2023-10-12T00:56:30.686 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f82c5b986fdc4ed-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.690 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:30.691 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:30.691 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.692 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:30.692 DEBUG:boto:Data: 2023-10-12T00:56:30.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.693 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.694 DEBUG:boto:Params: {} 2023-10-12T00:56:30.694 DEBUG:boto:Token: None 2023-10-12T00:56:30.695 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:30.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1v5pjXHIV24otmXzC2/KH6i0h3Q= 2023-10-12T00:56:30.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1v5pjXHIV24otmXzC2/KH6i0h3Q='} 2023-10-12T00:56:30.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000248aced497f72fb4-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.739 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.739 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.740 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.740 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.741 DEBUG:boto:Data: 2023-10-12T00:56:30.741 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.742 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.742 DEBUG:boto:Params: {} 2023-10-12T00:56:30.743 DEBUG:boto:Token: None 2023-10-12T00:56:30.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c036c6a3979efcc-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.747 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:30.748 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:30.748 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.748 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:30.749 DEBUG:boto:Data: 2023-10-12T00:56:30.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.750 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.750 DEBUG:boto:Params: {} 2023-10-12T00:56:30.751 DEBUG:boto:Token: None 2023-10-12T00:56:30.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:30.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uknqdNHK91ONoPyFWl29UiI6/qQ= 2023-10-12T00:56:30.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uknqdNHK91ONoPyFWl29UiI6/qQ='} 2023-10-12T00:56:30.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039fd8f6f83b24382-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.759 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.759 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.760 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.760 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.761 DEBUG:boto:Data: 2023-10-12T00:56:30.761 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.762 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.762 DEBUG:boto:Params: {} 2023-10-12T00:56:30.763 DEBUG:boto:Token: None 2023-10-12T00:56:30.763 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e784cf486279c412-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.767 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:30.768 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:30.768 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.768 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:30.769 DEBUG:boto:Data: 2023-10-12T00:56:30.769 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.770 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.770 DEBUG:boto:Params: {} 2023-10-12T00:56:30.771 DEBUG:boto:Token: None 2023-10-12T00:56:30.771 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:30.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r2/lSDQ9xr4plZmuX45uMDEQg9A= 2023-10-12T00:56:30.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r2/lSDQ9xr4plZmuX45uMDEQg9A='} 2023-10-12T00:56:30.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004016271adc6b257d-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.815 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.815 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.816 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.816 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.817 DEBUG:boto:Data: 2023-10-12T00:56:30.817 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.818 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.818 DEBUG:boto:Params: {} 2023-10-12T00:56:30.819 DEBUG:boto:Token: None 2023-10-12T00:56:30.819 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.820 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f70b24611cd9f693-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.823 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:30.824 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:30.824 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.824 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:30.825 DEBUG:boto:Data: 2023-10-12T00:56:30.825 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.826 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.827 DEBUG:boto:Params: {} 2023-10-12T00:56:30.827 DEBUG:boto:Token: None 2023-10-12T00:56:30.828 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:30.828 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uh9hOS9w7WohA7BQ/uodCTaq+II= 2023-10-12T00:56:30.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uh9hOS9w7WohA7BQ/uodCTaq+II='} 2023-10-12T00:56:30.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fe67923a89edff7-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.834 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.834 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.835 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.835 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.836 DEBUG:boto:Data: 2023-10-12T00:56:30.836 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.837 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.837 DEBUG:boto:Params: {} 2023-10-12T00:56:30.838 DEBUG:boto:Token: None 2023-10-12T00:56:30.838 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034e0301281681661-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.843 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:30.843 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:30.844 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.844 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:30.844 DEBUG:boto:Data: 2023-10-12T00:56:30.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.846 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.846 DEBUG:boto:Params: {} 2023-10-12T00:56:30.847 DEBUG:boto:Token: None 2023-10-12T00:56:30.847 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:30.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ug7/r//V71mWId4iz98BHjBkdOw= 2023-10-12T00:56:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ug7/r//V71mWId4iz98BHjBkdOw='} 2023-10-12T00:56:30.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000905ae135bb85d993-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.891 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.891 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.892 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.892 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.893 DEBUG:boto:Data: 2023-10-12T00:56:30.893 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.894 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.894 DEBUG:boto:Params: {} 2023-10-12T00:56:30.895 DEBUG:boto:Token: None 2023-10-12T00:56:30.895 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009da1ca775b621a46-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.900 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:30.900 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:30.901 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.901 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:30.901 DEBUG:boto:Data: 2023-10-12T00:56:30.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.902 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.903 DEBUG:boto:Params: {} 2023-10-12T00:56:30.903 DEBUG:boto:Token: None 2023-10-12T00:56:30.904 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:30.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cLZ4LKo8ZAyMyJlFfGsk78Nr/YY= 2023-10-12T00:56:30.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cLZ4LKo8ZAyMyJlFfGsk78Nr/YY='} 2023-10-12T00:56:30.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c2c9762365beafe-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.910 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.910 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.911 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.911 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.912 DEBUG:boto:Data: 2023-10-12T00:56:30.912 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.913 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.913 DEBUG:boto:Params: {} 2023-10-12T00:56:30.914 DEBUG:boto:Token: None 2023-10-12T00:56:30.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5f7a271d1b91963-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.918 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:30.919 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:30.919 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.920 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:30.920 DEBUG:boto:Data: 2023-10-12T00:56:30.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.921 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.922 DEBUG:boto:Params: {} 2023-10-12T00:56:30.922 DEBUG:boto:Token: None 2023-10-12T00:56:30.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:30.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jXQ7zX442usEgEanMCUInM1LGqg= 2023-10-12T00:56:30.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jXQ7zX442usEgEanMCUInM1LGqg='} 2023-10-12T00:56:30.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daca052f9f1080c9-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.967 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.968 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.968 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.969 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.969 DEBUG:boto:Data: 2023-10-12T00:56:30.969 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.970 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.971 DEBUG:boto:Params: {} 2023-10-12T00:56:30.971 DEBUG:boto:Token: None 2023-10-12T00:56:30.972 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000843be000dcf154d2-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.976 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:30.976 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:30.976 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.977 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:30.977 DEBUG:boto:Data: 2023-10-12T00:56:30.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:30.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.978 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.979 DEBUG:boto:Params: {} 2023-10-12T00:56:30.979 DEBUG:boto:Token: None 2023-10-12T00:56:30.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:30.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oujxjrBr8QBVO+neJUt94CEGm/s= 2023-10-12T00:56:30.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oujxjrBr8QBVO+neJUt94CEGm/s='} 2023-10-12T00:56:30.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025b88ce157708606-006527443e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.987 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:30.987 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:30.987 DEBUG:boto:Method: HEAD 2023-10-12T00:56:30.988 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:30.988 DEBUG:boto:Data: 2023-10-12T00:56:30.989 DEBUG:boto:Headers: {} 2023-10-12T00:56:30.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:30.989 DEBUG:boto:Port: 8000 2023-10-12T00:56:30.990 DEBUG:boto:Params: {} 2023-10-12T00:56:30.990 DEBUG:boto:Token: None 2023-10-12T00:56:30.993 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:30 GMT /wafmoa-58/ 2023-10-12T00:56:30.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE= 2023-10-12T00:56:30.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ahsUTlLFycFzYU5lMzW9cdX9iE=', 'Content-Length': '0'} 2023-10-12T00:56:30.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b05dbd0ef2e2cd7-006527443e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:30.997 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:30.998 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:30.998 DEBUG:boto:Method: PUT 2023-10-12T00:56:30.999 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:30.999 DEBUG:boto:Data: 2023-10-12T00:56:30.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.000 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.001 DEBUG:boto:Params: {} 2023-10-12T00:56:31.001 DEBUG:boto:Token: None 2023-10-12T00:56:31.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:31.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UJUogu2MzB2WacozBaFW02CzbTU= 2023-10-12T00:56:31.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UJUogu2MzB2WacozBaFW02CzbTU='} 2023-10-12T00:56:31.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfd4cf5579a014e2-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:31.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:31.048 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:31.143 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:31.143 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:31.149 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 22 2023-10-12T00:56:31.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:31.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:31.194 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:31.285 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:31.286 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-10-12T00:56:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-10-12T00:56:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-10-12T00:56:31.287 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000025", 2023-10-12T00:56:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-10-12T00:56:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-10-12T00:56:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-10-12T00:56:31.288 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-10-12T00:56:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-10-12T00:56:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:31.289 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-10-12T00:56:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:31.290 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007" 2023-10-12T00:56:31.291 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:31.291 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:31.292 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.292 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.293 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.293 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.294 DEBUG:boto:Data: 2023-10-12T00:56:31.294 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.295 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.295 DEBUG:boto:Params: {} 2023-10-12T00:56:31.296 DEBUG:boto:Token: None 2023-10-12T00:56:31.297 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb70eccf77d67687-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.301 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:31.302 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:31.302 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.302 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:31.303 DEBUG:boto:Data: 2023-10-12T00:56:31.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.304 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.304 DEBUG:boto:Params: {} 2023-10-12T00:56:31.305 DEBUG:boto:Token: None 2023-10-12T00:56:31.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:31.306 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:72xkf6/dfMlY0pceTtsrYntI+C4= 2023-10-12T00:56:31.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:72xkf6/dfMlY0pceTtsrYntI+C4='} 2023-10-12T00:56:31.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0dc2adf0997e82f-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.312 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.312 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.313 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.313 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.314 DEBUG:boto:Data: 2023-10-12T00:56:31.314 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.315 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.315 DEBUG:boto:Params: {} 2023-10-12T00:56:31.316 DEBUG:boto:Token: None 2023-10-12T00:56:31.316 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077b6cb9125040f99-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.320 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:31.321 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:31.322 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.322 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:31.323 DEBUG:boto:Data: 2023-10-12T00:56:31.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.325 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.325 DEBUG:boto:Params: {} 2023-10-12T00:56:31.326 DEBUG:boto:Token: None 2023-10-12T00:56:31.326 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:31.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YR3/Fk4tmbg1NRdkf/pV//DUG6I= 2023-10-12T00:56:31.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YR3/Fk4tmbg1NRdkf/pV//DUG6I='} 2023-10-12T00:56:31.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1c9ca1546c08055-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.336 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.336 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.336 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.337 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.337 DEBUG:boto:Data: 2023-10-12T00:56:31.338 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.338 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.339 DEBUG:boto:Params: {} 2023-10-12T00:56:31.339 DEBUG:boto:Token: None 2023-10-12T00:56:31.340 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005947455dc71a8958-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.344 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:31.344 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:31.345 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.345 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:31.346 DEBUG:boto:Data: 2023-10-12T00:56:31.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.347 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.347 DEBUG:boto:Params: {} 2023-10-12T00:56:31.348 DEBUG:boto:Token: None 2023-10-12T00:56:31.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:31.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MT7UIJX8PfwSDHpXpHWEAF18+7Q= 2023-10-12T00:56:31.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MT7UIJX8PfwSDHpXpHWEAF18+7Q='} 2023-10-12T00:56:31.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb9812180b3a508c-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.399 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.401 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.401 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.402 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.403 DEBUG:boto:Data: 2023-10-12T00:56:31.403 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.404 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.405 DEBUG:boto:Params: {} 2023-10-12T00:56:31.405 DEBUG:boto:Token: None 2023-10-12T00:56:31.406 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019561f74e4fe5275-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.410 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:31.411 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:31.411 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.412 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:31.412 DEBUG:boto:Data: 2023-10-12T00:56:31.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.414 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.414 DEBUG:boto:Params: {} 2023-10-12T00:56:31.415 DEBUG:boto:Token: None 2023-10-12T00:56:31.415 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:31.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uB3iYP5eBm3EJiH9hBj+/cx1V0Q= 2023-10-12T00:56:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uB3iYP5eBm3EJiH9hBj+/cx1V0Q='} 2023-10-12T00:56:31.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b1d33188798151b-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.422 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.423 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.423 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.424 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.424 DEBUG:boto:Data: 2023-10-12T00:56:31.425 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.425 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.426 DEBUG:boto:Params: {} 2023-10-12T00:56:31.426 DEBUG:boto:Token: None 2023-10-12T00:56:31.427 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.427 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d43a256899f6b3e2-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.431 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:31.431 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:31.432 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.432 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:31.433 DEBUG:boto:Data: 2023-10-12T00:56:31.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.434 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.434 DEBUG:boto:Params: {} 2023-10-12T00:56:31.435 DEBUG:boto:Token: None 2023-10-12T00:56:31.435 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:31.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uzJLgtlWqqs+G9SwRCkzDzzhQWg= 2023-10-12T00:56:31.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uzJLgtlWqqs+G9SwRCkzDzzhQWg='} 2023-10-12T00:56:31.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5393070865448af-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.483 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.484 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.485 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.485 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.486 DEBUG:boto:Data: 2023-10-12T00:56:31.487 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.488 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.489 DEBUG:boto:Params: {} 2023-10-12T00:56:31.489 DEBUG:boto:Token: None 2023-10-12T00:56:31.490 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.491 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b1ee00e03943b1a-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.496 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:31.496 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:31.497 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.497 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:31.498 DEBUG:boto:Data: 2023-10-12T00:56:31.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.500 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.500 DEBUG:boto:Params: {} 2023-10-12T00:56:31.500 DEBUG:boto:Token: None 2023-10-12T00:56:31.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:31.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fbBNdqVXohBxcQXYWryHD4DQt9o= 2023-10-12T00:56:31.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fbBNdqVXohBxcQXYWryHD4DQt9o='} 2023-10-12T00:56:31.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bd8b64ff1f00156-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.508 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.509 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.509 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.510 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.510 DEBUG:boto:Data: 2023-10-12T00:56:31.511 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.511 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.512 DEBUG:boto:Params: {} 2023-10-12T00:56:31.512 DEBUG:boto:Token: None 2023-10-12T00:56:31.513 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c22f5ade5d35c65-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.517 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:31.517 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:31.518 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.518 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:31.519 DEBUG:boto:Data: 2023-10-12T00:56:31.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.520 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.520 DEBUG:boto:Params: {} 2023-10-12T00:56:31.521 DEBUG:boto:Token: None 2023-10-12T00:56:31.521 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:31.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FWVc6nvTJvjTaFVJp7JAg/eScmA= 2023-10-12T00:56:31.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FWVc6nvTJvjTaFVJp7JAg/eScmA='} 2023-10-12T00:56:31.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e52031d3b4e665f-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.571 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.572 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.572 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.573 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.573 DEBUG:boto:Data: 2023-10-12T00:56:31.573 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.574 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.575 DEBUG:boto:Params: {} 2023-10-12T00:56:31.575 DEBUG:boto:Token: None 2023-10-12T00:56:31.575 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b22fb0ffe97892c-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.580 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:31.580 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:31.581 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.581 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:31.582 DEBUG:boto:Data: 2023-10-12T00:56:31.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.583 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.583 DEBUG:boto:Params: {} 2023-10-12T00:56:31.584 DEBUG:boto:Token: None 2023-10-12T00:56:31.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:31.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a5cMKaLNeYSsnKDmEX5gB2ZyGVI= 2023-10-12T00:56:31.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a5cMKaLNeYSsnKDmEX5gB2ZyGVI='} 2023-10-12T00:56:31.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009191825f89cb7f7-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.591 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.591 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.591 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.592 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.592 DEBUG:boto:Data: 2023-10-12T00:56:31.593 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.594 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.594 DEBUG:boto:Params: {} 2023-10-12T00:56:31.595 DEBUG:boto:Token: None 2023-10-12T00:56:31.595 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8e4e82034f6d7cd-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.599 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:31.600 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:31.600 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.601 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:31.601 DEBUG:boto:Data: 2023-10-12T00:56:31.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.602 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.603 DEBUG:boto:Params: {} 2023-10-12T00:56:31.604 DEBUG:boto:Token: None 2023-10-12T00:56:31.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:31.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ez7ylHngmL7opwYkkzgRIsUDlh4= 2023-10-12T00:56:31.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ez7ylHngmL7opwYkkzgRIsUDlh4='} 2023-10-12T00:56:31.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005190eb68bc5c219d-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.651 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.652 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.652 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.653 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.653 DEBUG:boto:Data: 2023-10-12T00:56:31.654 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.655 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.656 DEBUG:boto:Params: {} 2023-10-12T00:56:31.657 DEBUG:boto:Token: None 2023-10-12T00:56:31.658 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8f19e1f79c594ae-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.663 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:31.664 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:31.664 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.665 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:31.665 DEBUG:boto:Data: 2023-10-12T00:56:31.666 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.667 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.667 DEBUG:boto:Params: {} 2023-10-12T00:56:31.668 DEBUG:boto:Token: None 2023-10-12T00:56:31.668 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:31.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:k4iMGrO98JxnKDNKxVjdjz/Qea8= 2023-10-12T00:56:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:k4iMGrO98JxnKDNKxVjdjz/Qea8='} 2023-10-12T00:56:31.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa04a8ef27cf3652-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.676 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.677 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.677 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.678 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.678 DEBUG:boto:Data: 2023-10-12T00:56:31.679 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.679 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.680 DEBUG:boto:Params: {} 2023-10-12T00:56:31.680 DEBUG:boto:Token: None 2023-10-12T00:56:31.681 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076c8f00fa8ef56b6-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:31.687 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:31.688 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.689 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:31.690 DEBUG:boto:Data: 2023-10-12T00:56:31.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.692 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.693 DEBUG:boto:Params: {} 2023-10-12T00:56:31.693 DEBUG:boto:Token: None 2023-10-12T00:56:31.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:31.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NTAKKht+yIMO8/9n74dZZeaDRpU= 2023-10-12T00:56:31.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NTAKKht+yIMO8/9n74dZZeaDRpU='} 2023-10-12T00:56:31.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c015e37eb9c0d395-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.744 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.745 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.745 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.746 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.747 DEBUG:boto:Data: 2023-10-12T00:56:31.748 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.749 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.750 DEBUG:boto:Params: {} 2023-10-12T00:56:31.750 DEBUG:boto:Token: None 2023-10-12T00:56:31.751 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad96806fb13cedf9-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.757 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:31.757 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:31.758 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.758 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:31.759 DEBUG:boto:Data: 2023-10-12T00:56:31.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.760 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.760 DEBUG:boto:Params: {} 2023-10-12T00:56:31.761 DEBUG:boto:Token: None 2023-10-12T00:56:31.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:31.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mc5+5VOov7P9bjZ8Keoxhbb61cs= 2023-10-12T00:56:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mc5+5VOov7P9bjZ8Keoxhbb61cs='} 2023-10-12T00:56:31.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000625b8b8946df1dbb-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.770 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.770 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.771 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.771 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.771 DEBUG:boto:Data: 2023-10-12T00:56:31.772 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.773 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.773 DEBUG:boto:Params: {} 2023-10-12T00:56:31.773 DEBUG:boto:Token: None 2023-10-12T00:56:31.774 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b72fa002555e7f-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.778 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:31.779 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:31.779 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.780 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:31.780 DEBUG:boto:Data: 2023-10-12T00:56:31.780 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.781 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.782 DEBUG:boto:Params: {} 2023-10-12T00:56:31.782 DEBUG:boto:Token: None 2023-10-12T00:56:31.783 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:31.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lrx0d5KMwrbNiafNSH9D+AsS7Wk= 2023-10-12T00:56:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lrx0d5KMwrbNiafNSH9D+AsS7Wk='} 2023-10-12T00:56:31.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7e1c9a7117778aa-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.827 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.828 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.828 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.828 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.829 DEBUG:boto:Data: 2023-10-12T00:56:31.829 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.830 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.831 DEBUG:boto:Params: {} 2023-10-12T00:56:31.831 DEBUG:boto:Token: None 2023-10-12T00:56:31.832 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0e181c0e697b55a-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.836 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:31.836 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:31.836 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.837 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:31.837 DEBUG:boto:Data: 2023-10-12T00:56:31.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.839 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.839 DEBUG:boto:Params: {} 2023-10-12T00:56:31.839 DEBUG:boto:Token: None 2023-10-12T00:56:31.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:31.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qn0ImAvvj2WUFZyljdSd7BPLw/4= 2023-10-12T00:56:31.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qn0ImAvvj2WUFZyljdSd7BPLw/4='} 2023-10-12T00:56:31.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004645c5d3046776ab-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.848 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.848 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.849 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.849 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.850 DEBUG:boto:Data: 2023-10-12T00:56:31.850 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.851 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.851 DEBUG:boto:Params: {} 2023-10-12T00:56:31.852 DEBUG:boto:Token: None 2023-10-12T00:56:31.852 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024cca234da2869b7-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.856 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:31.857 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:31.857 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.858 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:31.858 DEBUG:boto:Data: 2023-10-12T00:56:31.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.859 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.860 DEBUG:boto:Params: {} 2023-10-12T00:56:31.860 DEBUG:boto:Token: None 2023-10-12T00:56:31.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:31.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iUfNqVyWAluDm9yZQ9YVNkE2zU0= 2023-10-12T00:56:31.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iUfNqVyWAluDm9yZQ9YVNkE2zU0='} 2023-10-12T00:56:31.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7f46641fcfcc314-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.911 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.912 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.912 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.912 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.913 DEBUG:boto:Data: 2023-10-12T00:56:31.913 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.914 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.915 DEBUG:boto:Params: {} 2023-10-12T00:56:31.915 DEBUG:boto:Token: None 2023-10-12T00:56:31.915 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a15bb1149027460-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.920 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:31.920 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:31.921 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.921 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:31.922 DEBUG:boto:Data: 2023-10-12T00:56:31.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.923 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.923 DEBUG:boto:Params: {} 2023-10-12T00:56:31.924 DEBUG:boto:Token: None 2023-10-12T00:56:31.924 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:31.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WjRRpVl+3J+XNdwwMm1qbafFC2c= 2023-10-12T00:56:31.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WjRRpVl+3J+XNdwwMm1qbafFC2c='} 2023-10-12T00:56:31.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fcecfcec620ca2d-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.932 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.932 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.933 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.933 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.934 DEBUG:boto:Data: 2023-10-12T00:56:31.934 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.935 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.935 DEBUG:boto:Params: {} 2023-10-12T00:56:31.936 DEBUG:boto:Token: None 2023-10-12T00:56:31.936 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fc6328b5b28c389-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.940 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:31.940 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:31.941 DEBUG:boto:Method: PUT 2023-10-12T00:56:31.941 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:31.942 DEBUG:boto:Data: 2023-10-12T00:56:31.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:31.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.943 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.943 DEBUG:boto:Params: {} 2023-10-12T00:56:31.944 DEBUG:boto:Token: None 2023-10-12T00:56:31.944 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:31.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UJUogu2MzB2WacozBaFW02CzbTU= 2023-10-12T00:56:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UJUogu2MzB2WacozBaFW02CzbTU='} 2023-10-12T00:56:31.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d7b09dfdba0d76a-006527443f-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.991 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:31.991 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:31.992 DEBUG:boto:Method: HEAD 2023-10-12T00:56:31.992 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:31.993 DEBUG:boto:Data: 2023-10-12T00:56:31.993 DEBUG:boto:Headers: {} 2023-10-12T00:56:31.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:31.994 DEBUG:boto:Port: 8000 2023-10-12T00:56:31.994 DEBUG:boto:Params: {} 2023-10-12T00:56:31.995 DEBUG:boto:Token: None 2023-10-12T00:56:31.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:31 GMT /wafmoa-58/ 2023-10-12T00:56:31.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI= 2023-10-12T00:56:31.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:31 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACstoESD06vfEtfJJ5TscW0U7fI=', 'Content-Length': '0'} 2023-10-12T00:56:31.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083d2432033231a01-006527443f-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:31.999 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.000 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.000 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.001 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.001 DEBUG:boto:Data: 2023-10-12T00:56:32.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.002 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.003 DEBUG:boto:Params: {} 2023-10-12T00:56:32.003 DEBUG:boto:Token: None 2023-10-12T00:56:32.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ir8Lwb3X+zQRQUnqPtti8dx3dm0= 2023-10-12T00:56:32.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ir8Lwb3X+zQRQUnqPtti8dx3dm0='} 2023-10-12T00:56:32.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c26f654f3530edf-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:32.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:32.013 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:32.109 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:32.110 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:32.116 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 26 2023-10-12T00:56:32.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:32.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:32.163 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:32.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:32.215 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:32.254 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000058", 2023-10-12T00:56:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-10-12T00:56:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000046", 2023-10-12T00:56:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-10-12T00:56:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000050", 2023-10-12T00:56:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000032", 2023-10-12T00:56:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000039", 2023-10-12T00:56:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-10-12T00:56:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-10-12T00:56:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000040", 2023-10-12T00:56:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-10-12T00:56:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000045", 2023-10-12T00:56:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-10-12T00:56:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-10-12T00:56:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-10-12T00:56:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000055", 2023-10-12T00:56:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008" 2023-10-12T00:56:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:32.260 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:32.261 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.261 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.262 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.262 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.263 DEBUG:boto:Data: 2023-10-12T00:56:32.263 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.264 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.265 DEBUG:boto:Params: {} 2023-10-12T00:56:32.265 DEBUG:boto:Token: None 2023-10-12T00:56:32.266 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ddf891fe5c95321-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.270 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.271 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.271 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.272 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.272 DEBUG:boto:Data: 2023-10-12T00:56:32.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.273 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.274 DEBUG:boto:Params: {} 2023-10-12T00:56:32.274 DEBUG:boto:Token: None 2023-10-12T00:56:32.275 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:32.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ir8Lwb3X+zQRQUnqPtti8dx3dm0= 2023-10-12T00:56:32.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ir8Lwb3X+zQRQUnqPtti8dx3dm0='} 2023-10-12T00:56:32.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000df52142f8e13655-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.283 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.283 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.284 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.284 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.284 DEBUG:boto:Data: 2023-10-12T00:56:32.285 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.286 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.287 DEBUG:boto:Params: {} 2023-10-12T00:56:32.287 DEBUG:boto:Token: None 2023-10-12T00:56:32.288 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a8df51a5176cca4-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.292 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:32.293 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:32.293 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.294 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:32.294 DEBUG:boto:Data: 2023-10-12T00:56:32.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.295 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.296 DEBUG:boto:Params: {} 2023-10-12T00:56:32.296 DEBUG:boto:Token: None 2023-10-12T00:56:32.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:32.298 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JjM2FnY89IMhA46zUgsun16opRY= 2023-10-12T00:56:32.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JjM2FnY89IMhA46zUgsun16opRY='} 2023-10-12T00:56:32.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0dfc2b881556509-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.347 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.348 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.348 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.348 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.349 DEBUG:boto:Data: 2023-10-12T00:56:32.349 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.350 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.351 DEBUG:boto:Params: {} 2023-10-12T00:56:32.351 DEBUG:boto:Token: None 2023-10-12T00:56:32.352 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b43a6373f9d32a1-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.356 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:32.357 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:32.357 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.357 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:32.358 DEBUG:boto:Data: 2023-10-12T00:56:32.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.359 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.360 DEBUG:boto:Params: {} 2023-10-12T00:56:32.360 DEBUG:boto:Token: None 2023-10-12T00:56:32.361 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:32.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DC7LEo/cisicp3SfXi/i8MvpXsI= 2023-10-12T00:56:32.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DC7LEo/cisicp3SfXi/i8MvpXsI='} 2023-10-12T00:56:32.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000017560177c1a107-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.369 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.369 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.370 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.370 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.370 DEBUG:boto:Data: 2023-10-12T00:56:32.371 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.372 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.372 DEBUG:boto:Params: {} 2023-10-12T00:56:32.373 DEBUG:boto:Token: None 2023-10-12T00:56:32.373 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051dbb58417665eb7-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.378 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:32.378 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:32.379 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.379 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:32.379 DEBUG:boto:Data: 2023-10-12T00:56:32.380 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.381 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.381 DEBUG:boto:Params: {} 2023-10-12T00:56:32.382 DEBUG:boto:Token: None 2023-10-12T00:56:32.382 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:32.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gAxq/y40iKU8ebvcJfWxIh3npXk= 2023-10-12T00:56:32.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gAxq/y40iKU8ebvcJfWxIh3npXk='} 2023-10-12T00:56:32.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dabb37fa07c68d1d-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.431 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.432 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.432 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.433 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.433 DEBUG:boto:Data: 2023-10-12T00:56:32.434 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.434 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.435 DEBUG:boto:Params: {} 2023-10-12T00:56:32.436 DEBUG:boto:Token: None 2023-10-12T00:56:32.436 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33a682bb2342ab2-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.440 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:32.441 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:32.442 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.442 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:32.443 DEBUG:boto:Data: 2023-10-12T00:56:32.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.444 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.444 DEBUG:boto:Params: {} 2023-10-12T00:56:32.445 DEBUG:boto:Token: None 2023-10-12T00:56:32.445 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:32.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8uQmbjbUhu5jf92GoWUH9x8BlPA= 2023-10-12T00:56:32.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8uQmbjbUhu5jf92GoWUH9x8BlPA='} 2023-10-12T00:56:32.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf74f6a05206d12e-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.452 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.453 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.453 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.454 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.454 DEBUG:boto:Data: 2023-10-12T00:56:32.455 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.456 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.456 DEBUG:boto:Params: {} 2023-10-12T00:56:32.457 DEBUG:boto:Token: None 2023-10-12T00:56:32.458 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006677245f160ed1ce-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.463 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:32.464 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:32.464 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.465 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:32.465 DEBUG:boto:Data: 2023-10-12T00:56:32.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.467 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.467 DEBUG:boto:Params: {} 2023-10-12T00:56:32.467 DEBUG:boto:Token: None 2023-10-12T00:56:32.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:32.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sLPHdzq7CVAgeGocmNei/ucPTXs= 2023-10-12T00:56:32.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sLPHdzq7CVAgeGocmNei/ucPTXs='} 2023-10-12T00:56:32.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ee4eeb894560dd1-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.515 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.516 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.516 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.516 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.517 DEBUG:boto:Data: 2023-10-12T00:56:32.517 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.518 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.519 DEBUG:boto:Params: {} 2023-10-12T00:56:32.520 DEBUG:boto:Token: None 2023-10-12T00:56:32.520 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ab377180dd31891-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.526 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:32.527 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:32.527 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.527 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:32.528 DEBUG:boto:Data: 2023-10-12T00:56:32.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.529 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.529 DEBUG:boto:Params: {} 2023-10-12T00:56:32.530 DEBUG:boto:Token: None 2023-10-12T00:56:32.530 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:32.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PFMI9cj+je2SbyXARPyBeHliTVw= 2023-10-12T00:56:32.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PFMI9cj+je2SbyXARPyBeHliTVw='} 2023-10-12T00:56:32.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000179eedf627b646a2-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.538 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.538 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.539 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.539 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.540 DEBUG:boto:Data: 2023-10-12T00:56:32.540 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.541 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.542 DEBUG:boto:Params: {} 2023-10-12T00:56:32.542 DEBUG:boto:Token: None 2023-10-12T00:56:32.543 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074d5a37a986ab0fe-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.547 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:32.548 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:32.548 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.548 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:32.549 DEBUG:boto:Data: 2023-10-12T00:56:32.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.550 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.551 DEBUG:boto:Params: {} 2023-10-12T00:56:32.552 DEBUG:boto:Token: None 2023-10-12T00:56:32.552 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:32.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mcik53E9jo3+JUZ0dcp3ejKFNtI= 2023-10-12T00:56:32.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mcik53E9jo3+JUZ0dcp3ejKFNtI='} 2023-10-12T00:56:32.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffb964daca3710bf-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.603 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.604 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.604 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.604 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.605 DEBUG:boto:Data: 2023-10-12T00:56:32.605 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.606 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.607 DEBUG:boto:Params: {} 2023-10-12T00:56:32.607 DEBUG:boto:Token: None 2023-10-12T00:56:32.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcc98362f07f1693-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.612 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:32.612 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:32.613 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.613 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:32.613 DEBUG:boto:Data: 2023-10-12T00:56:32.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.615 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.615 DEBUG:boto:Params: {} 2023-10-12T00:56:32.615 DEBUG:boto:Token: None 2023-10-12T00:56:32.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:32.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VcS05JJ+xnm90I+A4bJeAczZt94= 2023-10-12T00:56:32.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VcS05JJ+xnm90I+A4bJeAczZt94='} 2023-10-12T00:56:32.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da2763a3b76329aa-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.622 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.623 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.623 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.624 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.624 DEBUG:boto:Data: 2023-10-12T00:56:32.624 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.625 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.626 DEBUG:boto:Params: {} 2023-10-12T00:56:32.626 DEBUG:boto:Token: None 2023-10-12T00:56:32.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094a680e4ac189391-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.632 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:32.632 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:32.633 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.633 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:32.633 DEBUG:boto:Data: 2023-10-12T00:56:32.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.635 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.635 DEBUG:boto:Params: {} 2023-10-12T00:56:32.636 DEBUG:boto:Token: None 2023-10-12T00:56:32.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:32.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rSIPpzqcYReJ0SKLnoFp4A+9dqQ= 2023-10-12T00:56:32.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rSIPpzqcYReJ0SKLnoFp4A+9dqQ='} 2023-10-12T00:56:32.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deb9a95ee35a0768-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.683 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.684 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.684 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.685 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.685 DEBUG:boto:Data: 2023-10-12T00:56:32.685 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.686 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.687 DEBUG:boto:Params: {} 2023-10-12T00:56:32.687 DEBUG:boto:Token: None 2023-10-12T00:56:32.688 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.688 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003659fd3d429a6753-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.692 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:32.693 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:32.693 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.694 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:32.694 DEBUG:boto:Data: 2023-10-12T00:56:32.694 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.695 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.696 DEBUG:boto:Params: {} 2023-10-12T00:56:32.696 DEBUG:boto:Token: None 2023-10-12T00:56:32.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:32.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WjtHbDplUQq2yn7pU7KYe0jj2EE= 2023-10-12T00:56:32.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WjtHbDplUQq2yn7pU7KYe0jj2EE='} 2023-10-12T00:56:32.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5c105c14fc17ca6-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.703 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.703 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.704 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.704 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.705 DEBUG:boto:Data: 2023-10-12T00:56:32.705 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.706 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.706 DEBUG:boto:Params: {} 2023-10-12T00:56:32.707 DEBUG:boto:Token: None 2023-10-12T00:56:32.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b1b97db54af84fe-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.711 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:32.712 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:32.712 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.713 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:32.713 DEBUG:boto:Data: 2023-10-12T00:56:32.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.714 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.715 DEBUG:boto:Params: {} 2023-10-12T00:56:32.715 DEBUG:boto:Token: None 2023-10-12T00:56:32.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:32.716 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:09htMG1vA/KEog/eHOYvCYxpKNA= 2023-10-12T00:56:32.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:09htMG1vA/KEog/eHOYvCYxpKNA='} 2023-10-12T00:56:32.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c472e51494730f6b-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.763 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.764 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.764 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.764 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.765 DEBUG:boto:Data: 2023-10-12T00:56:32.765 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.766 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.766 DEBUG:boto:Params: {} 2023-10-12T00:56:32.767 DEBUG:boto:Token: None 2023-10-12T00:56:32.767 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096b5bec091280b46-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.772 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:32.772 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:32.772 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.773 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:32.773 DEBUG:boto:Data: 2023-10-12T00:56:32.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.775 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.775 DEBUG:boto:Params: {} 2023-10-12T00:56:32.775 DEBUG:boto:Token: None 2023-10-12T00:56:32.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:32.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ywQu3YnIsPXdIxGNoMtEmnv23Rc= 2023-10-12T00:56:32.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ywQu3YnIsPXdIxGNoMtEmnv23Rc='} 2023-10-12T00:56:32.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000636794053cb106a6-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.783 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.783 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.784 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.784 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.785 DEBUG:boto:Data: 2023-10-12T00:56:32.785 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.786 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.786 DEBUG:boto:Params: {} 2023-10-12T00:56:32.787 DEBUG:boto:Token: None 2023-10-12T00:56:32.787 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d991416e3a3640-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.791 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:32.792 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:32.792 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.793 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:32.793 DEBUG:boto:Data: 2023-10-12T00:56:32.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.794 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.795 DEBUG:boto:Params: {} 2023-10-12T00:56:32.795 DEBUG:boto:Token: None 2023-10-12T00:56:32.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:32.796 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RAxJjztQu++gZenvfKuKLBwnhcw= 2023-10-12T00:56:32.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RAxJjztQu++gZenvfKuKLBwnhcw='} 2023-10-12T00:56:32.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3fb8c2a9f72f92d-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.839 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.839 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.840 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.840 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.840 DEBUG:boto:Data: 2023-10-12T00:56:32.841 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.842 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.842 DEBUG:boto:Params: {} 2023-10-12T00:56:32.842 DEBUG:boto:Token: None 2023-10-12T00:56:32.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.843 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f581c2dd5e3d7a7d-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.847 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:32.847 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:32.848 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.848 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:32.848 DEBUG:boto:Data: 2023-10-12T00:56:32.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.850 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.850 DEBUG:boto:Params: {} 2023-10-12T00:56:32.850 DEBUG:boto:Token: None 2023-10-12T00:56:32.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:32.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFuC5HcogvOeX0lx2VKDB9DoLBk= 2023-10-12T00:56:32.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFuC5HcogvOeX0lx2VKDB9DoLBk='} 2023-10-12T00:56:32.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f67c87e7798e8e62-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.857 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.857 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.858 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.858 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.859 DEBUG:boto:Data: 2023-10-12T00:56:32.859 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.860 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.860 DEBUG:boto:Params: {} 2023-10-12T00:56:32.861 DEBUG:boto:Token: None 2023-10-12T00:56:32.861 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015e94738b4e1199a-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.865 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:32.865 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:32.866 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.866 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:32.866 DEBUG:boto:Data: 2023-10-12T00:56:32.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.868 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.868 DEBUG:boto:Params: {} 2023-10-12T00:56:32.868 DEBUG:boto:Token: None 2023-10-12T00:56:32.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:32.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xfuEqHy0EwGmEGWS/HnO8oQi7BM= 2023-10-12T00:56:32.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xfuEqHy0EwGmEGWS/HnO8oQi7BM='} 2023-10-12T00:56:32.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ff0a3a7df4ea8f9-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.915 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.915 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.916 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.916 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.916 DEBUG:boto:Data: 2023-10-12T00:56:32.917 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.917 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.918 DEBUG:boto:Params: {} 2023-10-12T00:56:32.918 DEBUG:boto:Token: None 2023-10-12T00:56:32.919 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e78663f2b201bb37-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.923 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:32.923 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:32.924 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.924 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:32.924 DEBUG:boto:Data: 2023-10-12T00:56:32.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.926 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.926 DEBUG:boto:Params: {} 2023-10-12T00:56:32.927 DEBUG:boto:Token: None 2023-10-12T00:56:32.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:32.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1Xx+SKq+bR1gDC33QS01y3D1r1o= 2023-10-12T00:56:32.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1Xx+SKq+bR1gDC33QS01y3D1r1o='} 2023-10-12T00:56:32.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cdcc1ab8eecc80a-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.935 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.935 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.936 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.936 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.936 DEBUG:boto:Data: 2023-10-12T00:56:32.937 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.938 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.938 DEBUG:boto:Params: {} 2023-10-12T00:56:32.938 DEBUG:boto:Token: None 2023-10-12T00:56:32.939 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7fd49ababecc793-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.943 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:32.943 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:32.944 DEBUG:boto:Method: PUT 2023-10-12T00:56:32.944 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:32.945 DEBUG:boto:Data: 2023-10-12T00:56:32.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:32.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.946 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.946 DEBUG:boto:Params: {} 2023-10-12T00:56:32.947 DEBUG:boto:Token: None 2023-10-12T00:56:32.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:32.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SK0dmnNzPHEItC6KYclCoPq+dM= 2023-10-12T00:56:32.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SK0dmnNzPHEItC6KYclCoPq+dM='} 2023-10-12T00:56:32.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047be8eb5401d2f81-0065274440-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.991 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:32.991 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:32.992 DEBUG:boto:Method: HEAD 2023-10-12T00:56:32.992 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:32.993 DEBUG:boto:Data: 2023-10-12T00:56:32.993 DEBUG:boto:Headers: {} 2023-10-12T00:56:32.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:32.994 DEBUG:boto:Port: 8000 2023-10-12T00:56:32.994 DEBUG:boto:Params: {} 2023-10-12T00:56:32.995 DEBUG:boto:Token: None 2023-10-12T00:56:32.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:32 GMT /wafmoa-58/ 2023-10-12T00:56:32.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk= 2023-10-12T00:56:32.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:32 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ztaJnWORAH5YnMvA343/wBlvEtk=', 'Content-Length': '0'} 2023-10-12T00:56:32.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b12bec9130c357d5-0065274440-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:32.999 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:33.000 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:33.000 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.001 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:33.001 DEBUG:boto:Data: 2023-10-12T00:56:33.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.002 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.003 DEBUG:boto:Params: {} 2023-10-12T00:56:33.003 DEBUG:boto:Token: None 2023-10-12T00:56:33.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:33.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWEmplLXz+4/T+I8kLxt2HSLozY= 2023-10-12T00:56:33.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWEmplLXz+4/T+I8kLxt2HSLozY='} 2023-10-12T00:56:33.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e242b9728c32ecd1-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.010 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.010 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.010 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.011 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.011 DEBUG:boto:Data: 2023-10-12T00:56:33.012 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.012 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.013 DEBUG:boto:Params: {} 2023-10-12T00:56:33.013 DEBUG:boto:Token: None 2023-10-12T00:56:33.014 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.014 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004293efb1726c270c-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.018 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:33.018 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:33.019 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.019 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:33.019 DEBUG:boto:Data: 2023-10-12T00:56:33.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.021 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.021 DEBUG:boto:Params: {} 2023-10-12T00:56:33.021 DEBUG:boto:Token: None 2023-10-12T00:56:33.022 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:33.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KJt5bYazN7+9fv9ZoxAmBG87Jo0= 2023-10-12T00:56:33.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KJt5bYazN7+9fv9ZoxAmBG87Jo0='} 2023-10-12T00:56:33.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090eb748960b46c70-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.067 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.067 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.068 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.068 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.069 DEBUG:boto:Data: 2023-10-12T00:56:33.069 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.070 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.070 DEBUG:boto:Params: {} 2023-10-12T00:56:33.071 DEBUG:boto:Token: None 2023-10-12T00:56:33.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f4d6caa8aff0751-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.076 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:33.076 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:33.077 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.077 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:33.077 DEBUG:boto:Data: 2023-10-12T00:56:33.078 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.079 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.079 DEBUG:boto:Params: {} 2023-10-12T00:56:33.080 DEBUG:boto:Token: None 2023-10-12T00:56:33.080 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:33.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z8n4oC/edrMQFPbWZZHsGXNYqEs= 2023-10-12T00:56:33.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z8n4oC/edrMQFPbWZZHsGXNYqEs='} 2023-10-12T00:56:33.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cf2b9dc93483d6a-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:33.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:33.088 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:33.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:33.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:33.185 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:33.185 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:33.195 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 35 2023-10-12T00:56:33.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:33.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:33.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:33.338 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:33.338 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:33.338 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-10-12T00:56:33.338 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000059", 2023-10-12T00:56:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000027", 2023-10-12T00:56:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-10-12T00:56:33.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000049", 2023-10-12T00:56:33.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-10-12T00:56:33.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008", 2023-10-12T00:56:33.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-10-12T00:56:33.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000031", 2023-10-12T00:56:33.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000022", 2023-10-12T00:56:33.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-10-12T00:56:33.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-10-12T00:56:33.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000037", 2023-10-12T00:56:33.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-10-12T00:56:33.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:33.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:33.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000042", 2023-10-12T00:56:33.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:33.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:33.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-10-12T00:56:33.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000025", 2023-10-12T00:56:33.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-10-12T00:56:33.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:33.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-10-12T00:56:33.344 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-10-12T00:56:33.344 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021" 2023-10-12T00:56:33.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:33.344 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:33.345 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.346 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.347 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.347 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.347 DEBUG:boto:Data: 2023-10-12T00:56:33.348 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.349 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.349 DEBUG:boto:Params: {} 2023-10-12T00:56:33.349 DEBUG:boto:Token: None 2023-10-12T00:56:33.350 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007831d34abd883836-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.355 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:33.355 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:33.356 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.356 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:33.356 DEBUG:boto:Data: 2023-10-12T00:56:33.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.357 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.358 DEBUG:boto:Params: {} 2023-10-12T00:56:33.358 DEBUG:boto:Token: None 2023-10-12T00:56:33.359 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:33.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sd5oBOiJd/e4Bgpyi83lHETSbLg= 2023-10-12T00:56:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sd5oBOiJd/e4Bgpyi83lHETSbLg='} 2023-10-12T00:56:33.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000641f018f4ec44f46-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.366 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.367 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.367 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.368 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.368 DEBUG:boto:Data: 2023-10-12T00:56:33.369 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.369 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.370 DEBUG:boto:Params: {} 2023-10-12T00:56:33.370 DEBUG:boto:Token: None 2023-10-12T00:56:33.371 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005271fb51288e7393-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.375 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:33.376 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:33.376 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.377 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:33.377 DEBUG:boto:Data: 2023-10-12T00:56:33.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.378 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.379 DEBUG:boto:Params: {} 2023-10-12T00:56:33.379 DEBUG:boto:Token: None 2023-10-12T00:56:33.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:33.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hJbKuxidj4Tn9EKFUb51YaFsyTk= 2023-10-12T00:56:33.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hJbKuxidj4Tn9EKFUb51YaFsyTk='} 2023-10-12T00:56:33.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ce6a72b6237f45e-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.423 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.424 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.424 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.424 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.425 DEBUG:boto:Data: 2023-10-12T00:56:33.425 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.426 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.426 DEBUG:boto:Params: {} 2023-10-12T00:56:33.427 DEBUG:boto:Token: None 2023-10-12T00:56:33.427 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4cb11740ca41dc1-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.432 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:33.433 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:33.433 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.434 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:33.434 DEBUG:boto:Data: 2023-10-12T00:56:33.434 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.435 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.436 DEBUG:boto:Params: {} 2023-10-12T00:56:33.436 DEBUG:boto:Token: None 2023-10-12T00:56:33.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:33.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MggCPNQ4GdVWM9M5wHrnbXvPpZE= 2023-10-12T00:56:33.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MggCPNQ4GdVWM9M5wHrnbXvPpZE='} 2023-10-12T00:56:33.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f3f3d8924c5ea6b-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.443 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.443 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.444 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.444 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.445 DEBUG:boto:Data: 2023-10-12T00:56:33.445 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.446 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.446 DEBUG:boto:Params: {} 2023-10-12T00:56:33.447 DEBUG:boto:Token: None 2023-10-12T00:56:33.447 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a04364924ff479c7-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.452 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:33.453 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:33.454 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.454 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:33.455 DEBUG:boto:Data: 2023-10-12T00:56:33.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.457 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.457 DEBUG:boto:Params: {} 2023-10-12T00:56:33.458 DEBUG:boto:Token: None 2023-10-12T00:56:33.458 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:33.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kddx7U49ZFSc0kFg9OCFLyE23+Y= 2023-10-12T00:56:33.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kddx7U49ZFSc0kFg9OCFLyE23+Y='} 2023-10-12T00:56:33.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000023b759e92605f41-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.507 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.507 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.508 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.508 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.509 DEBUG:boto:Data: 2023-10-12T00:56:33.509 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.510 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.510 DEBUG:boto:Params: {} 2023-10-12T00:56:33.511 DEBUG:boto:Token: None 2023-10-12T00:56:33.511 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6b6913e7d1c644b-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.516 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:33.517 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:33.517 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.517 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:33.518 DEBUG:boto:Data: 2023-10-12T00:56:33.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.519 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.520 DEBUG:boto:Params: {} 2023-10-12T00:56:33.520 DEBUG:boto:Token: None 2023-10-12T00:56:33.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:33.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xVx0aXsf7MWdtF43nvMmPCbEJnw= 2023-10-12T00:56:33.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xVx0aXsf7MWdtF43nvMmPCbEJnw='} 2023-10-12T00:56:33.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cd8c18541afa054-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.528 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.528 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.529 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.529 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.529 DEBUG:boto:Data: 2023-10-12T00:56:33.530 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.531 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.531 DEBUG:boto:Params: {} 2023-10-12T00:56:33.531 DEBUG:boto:Token: None 2023-10-12T00:56:33.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d5cc9f3182a6b11-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.537 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:33.537 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:33.537 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.538 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:33.539 DEBUG:boto:Data: 2023-10-12T00:56:33.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.540 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.540 DEBUG:boto:Params: {} 2023-10-12T00:56:33.541 DEBUG:boto:Token: None 2023-10-12T00:56:33.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:33.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0VPoNrWQMUQYGW2aWh88blWjX+8= 2023-10-12T00:56:33.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0VPoNrWQMUQYGW2aWh88blWjX+8='} 2023-10-12T00:56:33.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038a937fff892ed6a-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.588 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.588 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.589 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.589 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.590 DEBUG:boto:Data: 2023-10-12T00:56:33.591 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.592 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.593 DEBUG:boto:Params: {} 2023-10-12T00:56:33.593 DEBUG:boto:Token: None 2023-10-12T00:56:33.594 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d10fdb0b8baf9ac1-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.599 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:33.599 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:33.600 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.600 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:33.601 DEBUG:boto:Data: 2023-10-12T00:56:33.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.602 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.602 DEBUG:boto:Params: {} 2023-10-12T00:56:33.603 DEBUG:boto:Token: None 2023-10-12T00:56:33.603 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:33.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ad1uS/yNBxvyRM/tmZvJnv0bYH0= 2023-10-12T00:56:33.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ad1uS/yNBxvyRM/tmZvJnv0bYH0='} 2023-10-12T00:56:33.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025cf64468e43b9f5-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.609 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.610 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.610 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.611 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.611 DEBUG:boto:Data: 2023-10-12T00:56:33.611 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.612 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.613 DEBUG:boto:Params: {} 2023-10-12T00:56:33.613 DEBUG:boto:Token: None 2023-10-12T00:56:33.613 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.614 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003536036b055d77ce-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.618 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:33.618 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:33.619 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.619 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:33.619 DEBUG:boto:Data: 2023-10-12T00:56:33.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.621 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.621 DEBUG:boto:Params: {} 2023-10-12T00:56:33.621 DEBUG:boto:Token: None 2023-10-12T00:56:33.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:33.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fNEZHecsh9wLBasdgdhX9hlBC+s= 2023-10-12T00:56:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fNEZHecsh9wLBasdgdhX9hlBC+s='} 2023-10-12T00:56:33.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e00daf28c49a1329-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.667 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.667 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.668 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.668 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.668 DEBUG:boto:Data: 2023-10-12T00:56:33.669 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.670 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.670 DEBUG:boto:Params: {} 2023-10-12T00:56:33.670 DEBUG:boto:Token: None 2023-10-12T00:56:33.671 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eac3b61982c572d0-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.676 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:33.677 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:33.677 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.678 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:33.679 DEBUG:boto:Data: 2023-10-12T00:56:33.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.681 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.681 DEBUG:boto:Params: {} 2023-10-12T00:56:33.682 DEBUG:boto:Token: None 2023-10-12T00:56:33.682 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:33.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2PP8uEhTMA5JMiE4ZWLAnksFOJc= 2023-10-12T00:56:33.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2PP8uEhTMA5JMiE4ZWLAnksFOJc='} 2023-10-12T00:56:33.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cc31cde682dfc08-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.690 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.690 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.691 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.691 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.691 DEBUG:boto:Data: 2023-10-12T00:56:33.692 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.693 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.693 DEBUG:boto:Params: {} 2023-10-12T00:56:33.693 DEBUG:boto:Token: None 2023-10-12T00:56:33.694 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ebb6ec2cbdc2e8a-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.698 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:33.699 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:33.699 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.699 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:33.700 DEBUG:boto:Data: 2023-10-12T00:56:33.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.701 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.701 DEBUG:boto:Params: {} 2023-10-12T00:56:33.702 DEBUG:boto:Token: None 2023-10-12T00:56:33.702 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:33.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XPdE47iyP4KhS+hx8aZ2xeEga8A= 2023-10-12T00:56:33.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XPdE47iyP4KhS+hx8aZ2xeEga8A='} 2023-10-12T00:56:33.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f073f31e56f0ab18-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.751 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.752 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.752 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.753 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.754 DEBUG:boto:Data: 2023-10-12T00:56:33.755 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.756 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.757 DEBUG:boto:Params: {} 2023-10-12T00:56:33.758 DEBUG:boto:Token: None 2023-10-12T00:56:33.758 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a46c15e0d672ea3c-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.764 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:33.764 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:33.765 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.765 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:33.765 DEBUG:boto:Data: 2023-10-12T00:56:33.766 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.767 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.767 DEBUG:boto:Params: {} 2023-10-12T00:56:33.768 DEBUG:boto:Token: None 2023-10-12T00:56:33.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:33.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lNFiVaicmWyRsrVHz1JohyxpWxA= 2023-10-12T00:56:33.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lNFiVaicmWyRsrVHz1JohyxpWxA='} 2023-10-12T00:56:33.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000154614b382da2ddb-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.776 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.777 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.777 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.778 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.778 DEBUG:boto:Data: 2023-10-12T00:56:33.778 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.779 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.780 DEBUG:boto:Params: {} 2023-10-12T00:56:33.780 DEBUG:boto:Token: None 2023-10-12T00:56:33.781 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000583daf1b9271118e-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.786 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:33.786 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:33.787 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.787 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:33.788 DEBUG:boto:Data: 2023-10-12T00:56:33.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.789 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.789 DEBUG:boto:Params: {} 2023-10-12T00:56:33.790 DEBUG:boto:Token: None 2023-10-12T00:56:33.790 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:33.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zsb6MRcSe2sHbQNAe5dhdOr7KzE= 2023-10-12T00:56:33.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zsb6MRcSe2sHbQNAe5dhdOr7KzE='} 2023-10-12T00:56:33.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb4277492444eb9c-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.835 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.836 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.836 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.836 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.837 DEBUG:boto:Data: 2023-10-12T00:56:33.837 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.838 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.839 DEBUG:boto:Params: {} 2023-10-12T00:56:33.839 DEBUG:boto:Token: None 2023-10-12T00:56:33.840 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7ac84d9ffdd5251-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:33.845 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:33.845 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.846 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:33.846 DEBUG:boto:Data: 2023-10-12T00:56:33.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.847 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.848 DEBUG:boto:Params: {} 2023-10-12T00:56:33.848 DEBUG:boto:Token: None 2023-10-12T00:56:33.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:33.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:REvZ2dVVwoxPU3UkUL8NMGPG/aQ= 2023-10-12T00:56:33.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:REvZ2dVVwoxPU3UkUL8NMGPG/aQ='} 2023-10-12T00:56:33.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d07a4a07d73ee4a5-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.855 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.856 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.856 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.856 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.857 DEBUG:boto:Data: 2023-10-12T00:56:33.857 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.859 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.859 DEBUG:boto:Params: {} 2023-10-12T00:56:33.860 DEBUG:boto:Token: None 2023-10-12T00:56:33.861 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000234ef07d19e222a5-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.867 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:33.867 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:33.868 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.868 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:33.868 DEBUG:boto:Data: 2023-10-12T00:56:33.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.869 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.870 DEBUG:boto:Params: {} 2023-10-12T00:56:33.870 DEBUG:boto:Token: None 2023-10-12T00:56:33.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:33.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mgB6HUINkQB2/AH6hhp7cy32NuA= 2023-10-12T00:56:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mgB6HUINkQB2/AH6hhp7cy32NuA='} 2023-10-12T00:56:33.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088cfc5ad5e394487-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.915 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.916 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.916 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.916 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.917 DEBUG:boto:Data: 2023-10-12T00:56:33.917 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.918 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.919 DEBUG:boto:Params: {} 2023-10-12T00:56:33.919 DEBUG:boto:Token: None 2023-10-12T00:56:33.920 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006412dcbb94696f97-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.924 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:33.925 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:33.925 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.925 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:33.926 DEBUG:boto:Data: 2023-10-12T00:56:33.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.927 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.927 DEBUG:boto:Params: {} 2023-10-12T00:56:33.928 DEBUG:boto:Token: None 2023-10-12T00:56:33.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:33.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:81iqVbgbzQQn6ySa0Kv3Ig7UohE= 2023-10-12T00:56:33.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:81iqVbgbzQQn6ySa0Kv3Ig7UohE='} 2023-10-12T00:56:33.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022b306aea42be0d4-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.936 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:33.937 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:33.938 DEBUG:boto:Method: HEAD 2023-10-12T00:56:33.938 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:33.939 DEBUG:boto:Data: 2023-10-12T00:56:33.940 DEBUG:boto:Headers: {} 2023-10-12T00:56:33.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.941 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.941 DEBUG:boto:Params: {} 2023-10-12T00:56:33.941 DEBUG:boto:Token: None 2023-10-12T00:56:33.942 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/ 2023-10-12T00:56:33.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k= 2023-10-12T00:56:33.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mlwk/MwNoHK2LYtg9oAhWslFq9k=', 'Content-Length': '0'} 2023-10-12T00:56:33.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T00:56:33.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000505505b2baa60f-0065274441-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.946 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:33.947 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:33.947 DEBUG:boto:Method: PUT 2023-10-12T00:56:33.948 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:33.948 DEBUG:boto:Data: 2023-10-12T00:56:33.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:33.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:33.949 DEBUG:boto:Port: 8000 2023-10-12T00:56:33.950 DEBUG:boto:Params: {} 2023-10-12T00:56:33.950 DEBUG:boto:Token: None 2023-10-12T00:56:33.951 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:33 GMT /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:33.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SlDAU9EIa2Y/RFMEOb1riSC+j3c= 2023-10-12T00:56:33.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SlDAU9EIa2Y/RFMEOb1riSC+j3c='} 2023-10-12T00:56:33.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3a88b66ee116a5e-0065274441-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:33.999 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.000 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.000 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.000 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.001 DEBUG:boto:Data: 2023-10-12T00:56:34.001 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.002 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.002 DEBUG:boto:Params: {} 2023-10-12T00:56:34.003 DEBUG:boto:Token: None 2023-10-12T00:56:34.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000021a42fe0bdf9b3a-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.008 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.008 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.009 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.009 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.009 DEBUG:boto:Data: 2023-10-12T00:56:34.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.011 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.011 DEBUG:boto:Params: {} 2023-10-12T00:56:34.012 DEBUG:boto:Token: None 2023-10-12T00:56:34.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:596yF1smgRcmKBdaMMfDNdlWdkE= 2023-10-12T00:56:34.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:596yF1smgRcmKBdaMMfDNdlWdkE='} 2023-10-12T00:56:34.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f9f7b9e005d60d2-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.019 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.020 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.020 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.021 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.021 DEBUG:boto:Data: 2023-10-12T00:56:34.021 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.022 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.023 DEBUG:boto:Params: {} 2023-10-12T00:56:34.023 DEBUG:boto:Token: None 2023-10-12T00:56:34.023 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac8f89b4d4f82b83-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.028 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:34.028 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:34.029 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.029 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:34.030 DEBUG:boto:Data: 2023-10-12T00:56:34.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.031 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.031 DEBUG:boto:Params: {} 2023-10-12T00:56:34.032 DEBUG:boto:Token: None 2023-10-12T00:56:34.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:34.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PFydnS1DwZUh7oiFSxtWEDu1Epk= 2023-10-12T00:56:34.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PFydnS1DwZUh7oiFSxtWEDu1Epk='} 2023-10-12T00:56:34.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a9131325a2769c6-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.084 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.084 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.085 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.086 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.086 DEBUG:boto:Data: 2023-10-12T00:56:34.087 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.088 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.089 DEBUG:boto:Params: {} 2023-10-12T00:56:34.089 DEBUG:boto:Token: None 2023-10-12T00:56:34.089 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3eeef8aa7a8d978-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.094 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:34.095 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:34.095 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.095 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:34.096 DEBUG:boto:Data: 2023-10-12T00:56:34.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.097 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.098 DEBUG:boto:Params: {} 2023-10-12T00:56:34.098 DEBUG:boto:Token: None 2023-10-12T00:56:34.099 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:34.099 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vLOdtB8fFz/QAhzutdg7EXIGklQ= 2023-10-12T00:56:34.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vLOdtB8fFz/QAhzutdg7EXIGklQ='} 2023-10-12T00:56:34.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d49cb7dca76665cf-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.106 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.107 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.107 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.108 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.108 DEBUG:boto:Data: 2023-10-12T00:56:34.108 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.109 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.110 DEBUG:boto:Params: {} 2023-10-12T00:56:34.110 DEBUG:boto:Token: None 2023-10-12T00:56:34.111 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.111 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daa5cc210b65554d-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.116 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:34.116 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:34.116 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.117 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:34.117 DEBUG:boto:Data: 2023-10-12T00:56:34.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.119 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.119 DEBUG:boto:Params: {} 2023-10-12T00:56:34.119 DEBUG:boto:Token: None 2023-10-12T00:56:34.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:34.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cjkJh3sHG/eNqC1TwlukGFKINfU= 2023-10-12T00:56:34.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cjkJh3sHG/eNqC1TwlukGFKINfU='} 2023-10-12T00:56:34.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e00d50fb944dd327-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.168 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.168 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.168 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.169 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.169 DEBUG:boto:Data: 2023-10-12T00:56:34.170 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.170 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.171 DEBUG:boto:Params: {} 2023-10-12T00:56:34.171 DEBUG:boto:Token: None 2023-10-12T00:56:34.172 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab2f288a9d635450-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.176 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:34.177 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:34.177 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.178 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:34.178 DEBUG:boto:Data: 2023-10-12T00:56:34.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.179 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.179 DEBUG:boto:Params: {} 2023-10-12T00:56:34.180 DEBUG:boto:Token: None 2023-10-12T00:56:34.180 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:34.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cOva4sqSaxx120d25VRnp5C0ix8= 2023-10-12T00:56:34.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cOva4sqSaxx120d25VRnp5C0ix8='} 2023-10-12T00:56:34.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e29554c1698cfd56-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.187 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.188 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.188 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.189 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.190 DEBUG:boto:Data: 2023-10-12T00:56:34.191 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.192 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.192 DEBUG:boto:Params: {} 2023-10-12T00:56:34.193 DEBUG:boto:Token: None 2023-10-12T00:56:34.194 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6d6cea825db737a-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.200 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:34.201 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:34.201 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.202 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:34.203 DEBUG:boto:Data: 2023-10-12T00:56:34.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.204 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.205 DEBUG:boto:Params: {} 2023-10-12T00:56:34.205 DEBUG:boto:Token: None 2023-10-12T00:56:34.206 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:34.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vJ6D7lm+G8Gj4jMvYW/boBE6H3g= 2023-10-12T00:56:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vJ6D7lm+G8Gj4jMvYW/boBE6H3g='} 2023-10-12T00:56:34.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007555b1ebc4d2f2d1-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.251 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.252 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.252 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.253 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.253 DEBUG:boto:Data: 2023-10-12T00:56:34.254 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.254 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.255 DEBUG:boto:Params: {} 2023-10-12T00:56:34.255 DEBUG:boto:Token: None 2023-10-12T00:56:34.256 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c59a04886eda8422-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.260 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:34.261 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:34.261 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.262 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:34.262 DEBUG:boto:Data: 2023-10-12T00:56:34.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.263 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.264 DEBUG:boto:Params: {} 2023-10-12T00:56:34.264 DEBUG:boto:Token: None 2023-10-12T00:56:34.265 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:34.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Vt6QjNz5hgGyl5ISYOffEdUSgvw= 2023-10-12T00:56:34.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Vt6QjNz5hgGyl5ISYOffEdUSgvw='} 2023-10-12T00:56:34.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de4d7f1f6a77521e-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.271 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.272 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.272 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.272 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.273 DEBUG:boto:Data: 2023-10-12T00:56:34.273 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.274 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.274 DEBUG:boto:Params: {} 2023-10-12T00:56:34.275 DEBUG:boto:Token: None 2023-10-12T00:56:34.275 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa42df49ef9a2539-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.280 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.280 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.281 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.281 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.282 DEBUG:boto:Data: 2023-10-12T00:56:34.282 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.283 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.283 DEBUG:boto:Params: {} 2023-10-12T00:56:34.284 DEBUG:boto:Token: None 2023-10-12T00:56:34.284 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C1+ihv3SKX/hRpLfTbz1yyli4+M= 2023-10-12T00:56:34.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C1+ihv3SKX/hRpLfTbz1yyli4+M='} 2023-10-12T00:56:34.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7dba9600d2ee61f-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.335 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.336 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.337 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.337 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.338 DEBUG:boto:Data: 2023-10-12T00:56:34.339 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.339 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.340 DEBUG:boto:Params: {} 2023-10-12T00:56:34.340 DEBUG:boto:Token: None 2023-10-12T00:56:34.341 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.341 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24e581981be3e23-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.345 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:34.345 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:34.346 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.346 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:34.347 DEBUG:boto:Data: 2023-10-12T00:56:34.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.348 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.348 DEBUG:boto:Params: {} 2023-10-12T00:56:34.349 DEBUG:boto:Token: None 2023-10-12T00:56:34.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:34.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V8wOoGenu/7Ar5M1U57+C8amxIg= 2023-10-12T00:56:34.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V8wOoGenu/7Ar5M1U57+C8amxIg='} 2023-10-12T00:56:34.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048fd16979bb95fc1-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.357 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.358 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.358 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.359 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.359 DEBUG:boto:Data: 2023-10-12T00:56:34.359 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.360 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.361 DEBUG:boto:Params: {} 2023-10-12T00:56:34.361 DEBUG:boto:Token: None 2023-10-12T00:56:34.362 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.362 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8d6216d6104731e-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.367 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:34.367 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:34.368 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.369 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:34.369 DEBUG:boto:Data: 2023-10-12T00:56:34.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.370 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.371 DEBUG:boto:Params: {} 2023-10-12T00:56:34.371 DEBUG:boto:Token: None 2023-10-12T00:56:34.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:34.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WPW17Ma/ZSZpJgjTI6WRxMiMD0E= 2023-10-12T00:56:34.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WPW17Ma/ZSZpJgjTI6WRxMiMD0E='} 2023-10-12T00:56:34.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a14cb2d2dfc66a6-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.379 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.380 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.380 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.381 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.381 DEBUG:boto:Data: 2023-10-12T00:56:34.381 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.383 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.383 DEBUG:boto:Params: {} 2023-10-12T00:56:34.384 DEBUG:boto:Token: None 2023-10-12T00:56:34.385 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.386 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000afda7ebfa409b24-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.391 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:34.391 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:34.391 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.392 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:34.392 DEBUG:boto:Data: 2023-10-12T00:56:34.392 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.393 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.393 DEBUG:boto:Params: {} 2023-10-12T00:56:34.394 DEBUG:boto:Token: None 2023-10-12T00:56:34.394 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:34.395 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0mnNM4RpMDjXaj5bbsGL1POfJzo= 2023-10-12T00:56:34.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0mnNM4RpMDjXaj5bbsGL1POfJzo='} 2023-10-12T00:56:34.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d12860b39f2b356a-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:34.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:34.441 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:34.492 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:34.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:34.541 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:34.542 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:34.551 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 42 2023-10-12T00:56:34.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:34.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:34.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:34.661 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:34.661 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:34.701 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:34.702 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:34.702 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-10-12T00:56:34.702 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-10-12T00:56:34.702 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000467", 2023-10-12T00:56:34.703 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-10-12T00:56:34.703 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000056", 2023-10-12T00:56:34.703 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:34.703 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000023", 2023-10-12T00:56:34.704 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000317", 2023-10-12T00:56:34.704 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000022", 2023-10-12T00:56:34.704 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:34.704 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-10-12T00:56:34.705 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-10-12T00:56:34.705 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000271", 2023-10-12T00:56:34.705 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008", 2023-10-12T00:56:34.705 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000018", 2023-10-12T00:56:34.706 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-10-12T00:56:34.706 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-10-12T00:56:34.706 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000162", 2023-10-12T00:56:34.706 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-10-12T00:56:34.706 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:34.707 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000423", 2023-10-12T00:56:34.707 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:34.707 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:34.707 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:34.707 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:34.708 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000283", 2023-10-12T00:56:34.708 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:34.708 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000061", 2023-10-12T00:56:34.708 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000050", 2023-10-12T00:56:34.708 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000029", 2023-10-12T00:56:34.709 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000133" 2023-10-12T00:56:34.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:34.709 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:34.710 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.711 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.711 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.712 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.712 DEBUG:boto:Data: 2023-10-12T00:56:34.713 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.713 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.714 DEBUG:boto:Params: {} 2023-10-12T00:56:34.714 DEBUG:boto:Token: None 2023-10-12T00:56:34.715 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c65f21b005c7da4-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.721 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:34.721 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:34.722 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.723 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:34.724 DEBUG:boto:Data: 2023-10-12T00:56:34.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.726 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.726 DEBUG:boto:Params: {} 2023-10-12T00:56:34.727 DEBUG:boto:Token: None 2023-10-12T00:56:34.727 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:34.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:awGxljOKMpFIDCUajgj2B1V8zTU= 2023-10-12T00:56:34.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:awGxljOKMpFIDCUajgj2B1V8zTU='} 2023-10-12T00:56:34.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054849cabca82ab02-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.734 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.734 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.735 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.735 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.736 DEBUG:boto:Data: 2023-10-12T00:56:34.736 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.737 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.738 DEBUG:boto:Params: {} 2023-10-12T00:56:34.738 DEBUG:boto:Token: None 2023-10-12T00:56:34.739 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000267612518981aace-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.744 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:34.744 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:34.745 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.745 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:34.745 DEBUG:boto:Data: 2023-10-12T00:56:34.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.747 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.748 DEBUG:boto:Params: {} 2023-10-12T00:56:34.748 DEBUG:boto:Token: None 2023-10-12T00:56:34.749 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:34.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EWjK4Ghpa/qlpNLBYnOafmJGyKg= 2023-10-12T00:56:34.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EWjK4Ghpa/qlpNLBYnOafmJGyKg='} 2023-10-12T00:56:34.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001864c0999bae938a-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.795 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.796 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.796 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.797 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.797 DEBUG:boto:Data: 2023-10-12T00:56:34.797 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.798 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.799 DEBUG:boto:Params: {} 2023-10-12T00:56:34.800 DEBUG:boto:Token: None 2023-10-12T00:56:34.800 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d47f101a720c4cc-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.805 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.806 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.806 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.807 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.807 DEBUG:boto:Data: 2023-10-12T00:56:34.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.808 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.809 DEBUG:boto:Params: {} 2023-10-12T00:56:34.809 DEBUG:boto:Token: None 2023-10-12T00:56:34.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:34.810 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:596yF1smgRcmKBdaMMfDNdlWdkE= 2023-10-12T00:56:34.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:596yF1smgRcmKBdaMMfDNdlWdkE='} 2023-10-12T00:56:34.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4241d3acd690e25-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.818 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.818 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.818 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.819 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.819 DEBUG:boto:Data: 2023-10-12T00:56:34.820 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.821 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.821 DEBUG:boto:Params: {} 2023-10-12T00:56:34.821 DEBUG:boto:Token: None 2023-10-12T00:56:34.822 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000958db97b40e43525-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.827 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:34.827 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:34.828 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.828 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:34.829 DEBUG:boto:Data: 2023-10-12T00:56:34.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.830 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.830 DEBUG:boto:Params: {} 2023-10-12T00:56:34.831 DEBUG:boto:Token: None 2023-10-12T00:56:34.831 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:34.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g4dhVYmR1vGV235zQdqKloDYAas= 2023-10-12T00:56:34.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g4dhVYmR1vGV235zQdqKloDYAas='} 2023-10-12T00:56:34.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab9ea049c7e2aef8-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.880 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.881 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.881 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.881 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.882 DEBUG:boto:Data: 2023-10-12T00:56:34.882 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.883 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.884 DEBUG:boto:Params: {} 2023-10-12T00:56:34.884 DEBUG:boto:Token: None 2023-10-12T00:56:34.885 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009182aac36d29c7cf-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.890 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:34.890 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:34.890 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.891 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:34.891 DEBUG:boto:Data: 2023-10-12T00:56:34.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.892 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.893 DEBUG:boto:Params: {} 2023-10-12T00:56:34.893 DEBUG:boto:Token: None 2023-10-12T00:56:34.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:34.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+7LGZV3XH+SIDeoKrUkdncRNgRY= 2023-10-12T00:56:34.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+7LGZV3XH+SIDeoKrUkdncRNgRY='} 2023-10-12T00:56:34.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021b34d43fe5ec785-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.901 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.902 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.902 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.903 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.903 DEBUG:boto:Data: 2023-10-12T00:56:34.903 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.904 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.905 DEBUG:boto:Params: {} 2023-10-12T00:56:34.906 DEBUG:boto:Token: None 2023-10-12T00:56:34.907 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e6326dbe58e2d30-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.912 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.913 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.914 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.915 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.915 DEBUG:boto:Data: 2023-10-12T00:56:34.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.916 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.917 DEBUG:boto:Params: {} 2023-10-12T00:56:34.917 DEBUG:boto:Token: None 2023-10-12T00:56:34.918 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:34.918 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C1+ihv3SKX/hRpLfTbz1yyli4+M= 2023-10-12T00:56:34.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C1+ihv3SKX/hRpLfTbz1yyli4+M='} 2023-10-12T00:56:34.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e92ac85a4af3437-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.967 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.967 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.968 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.968 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.969 DEBUG:boto:Data: 2023-10-12T00:56:34.969 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.970 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.971 DEBUG:boto:Params: {} 2023-10-12T00:56:34.971 DEBUG:boto:Token: None 2023-10-12T00:56:34.971 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078496e1ed8957596-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.976 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:34.977 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:34.977 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.977 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:34.978 DEBUG:boto:Data: 2023-10-12T00:56:34.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:34.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.979 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.979 DEBUG:boto:Params: {} 2023-10-12T00:56:34.980 DEBUG:boto:Token: None 2023-10-12T00:56:34.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:34.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fQdIHPb5oESAidUeAr1Tfs0G7hQ= 2023-10-12T00:56:34.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fQdIHPb5oESAidUeAr1Tfs0G7hQ='} 2023-10-12T00:56:34.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093880cfb43fad612-0065274442-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.987 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:34.987 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:34.987 DEBUG:boto:Method: HEAD 2023-10-12T00:56:34.988 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:34.988 DEBUG:boto:Data: 2023-10-12T00:56:34.988 DEBUG:boto:Headers: {} 2023-10-12T00:56:34.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:34.989 DEBUG:boto:Port: 8000 2023-10-12T00:56:34.990 DEBUG:boto:Params: {} 2023-10-12T00:56:34.990 DEBUG:boto:Token: None 2023-10-12T00:56:34.991 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:34 GMT /wafmoa-58/ 2023-10-12T00:56:34.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k= 2023-10-12T00:56:34.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DqCaoHTsk1Stw2uhrnhaNMNxY+k=', 'Content-Length': '0'} 2023-10-12T00:56:34.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6aa371ec76c490d-0065274442-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:34.996 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:34.997 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:34.997 DEBUG:boto:Method: PUT 2023-10-12T00:56:34.998 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:34.999 DEBUG:boto:Data: 2023-10-12T00:56:35.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.001 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.002 DEBUG:boto:Params: {} 2023-10-12T00:56:35.002 DEBUG:boto:Token: None 2023-10-12T00:56:35.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:35.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z+UH8pErrMj8aXCeVvKWFXDtqoM= 2023-10-12T00:56:35.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z+UH8pErrMj8aXCeVvKWFXDtqoM='} 2023-10-12T00:56:35.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d75ada883e1a7258-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.051 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.052 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.052 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.052 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.053 DEBUG:boto:Data: 2023-10-12T00:56:35.053 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.054 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.054 DEBUG:boto:Params: {} 2023-10-12T00:56:35.055 DEBUG:boto:Token: None 2023-10-12T00:56:35.055 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a72dcc50c2ccf970-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.060 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:35.061 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:35.061 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.062 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:35.062 DEBUG:boto:Data: 2023-10-12T00:56:35.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.063 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.064 DEBUG:boto:Params: {} 2023-10-12T00:56:35.064 DEBUG:boto:Token: None 2023-10-12T00:56:35.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:35.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NWRYFeaQQSgPa2WrmwW3eSZ9luM= 2023-10-12T00:56:35.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NWRYFeaQQSgPa2WrmwW3eSZ9luM='} 2023-10-12T00:56:35.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e0115eefa8b6d93-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.071 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.072 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.072 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.073 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.073 DEBUG:boto:Data: 2023-10-12T00:56:35.073 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.074 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.075 DEBUG:boto:Params: {} 2023-10-12T00:56:35.075 DEBUG:boto:Token: None 2023-10-12T00:56:35.077 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.077 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000879b1ab59182ef9a-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.082 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:35.082 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:35.083 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.083 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:35.084 DEBUG:boto:Data: 2023-10-12T00:56:35.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.085 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.085 DEBUG:boto:Params: {} 2023-10-12T00:56:35.086 DEBUG:boto:Token: None 2023-10-12T00:56:35.086 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:35.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7nDiSQfXiwslyprr5u6Mp9gLSg= 2023-10-12T00:56:35.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7nDiSQfXiwslyprr5u6Mp9gLSg='} 2023-10-12T00:56:35.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8a69ee0837a8f2b-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.132 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.133 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.133 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.133 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.134 DEBUG:boto:Data: 2023-10-12T00:56:35.134 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.135 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.135 DEBUG:boto:Params: {} 2023-10-12T00:56:35.136 DEBUG:boto:Token: None 2023-10-12T00:56:35.136 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb5cdc44cd921a62-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.141 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:35.142 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:35.142 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.142 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:35.143 DEBUG:boto:Data: 2023-10-12T00:56:35.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.144 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.145 DEBUG:boto:Params: {} 2023-10-12T00:56:35.145 DEBUG:boto:Token: None 2023-10-12T00:56:35.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:35.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xQM0EgaCC8d6fMaAJaHcp4T2E9k= 2023-10-12T00:56:35.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xQM0EgaCC8d6fMaAJaHcp4T2E9k='} 2023-10-12T00:56:35.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f61415bd6906106-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.152 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.153 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.153 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.154 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.154 DEBUG:boto:Data: 2023-10-12T00:56:35.154 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.155 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.156 DEBUG:boto:Params: {} 2023-10-12T00:56:35.156 DEBUG:boto:Token: None 2023-10-12T00:56:35.156 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000228edfbc99cac8bb-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.161 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:35.162 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:35.162 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.163 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:35.163 DEBUG:boto:Data: 2023-10-12T00:56:35.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.164 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.165 DEBUG:boto:Params: {} 2023-10-12T00:56:35.165 DEBUG:boto:Token: None 2023-10-12T00:56:35.166 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:35.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2+Yb8tfNULeTPNu954ld94iko54= 2023-10-12T00:56:35.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2+Yb8tfNULeTPNu954ld94iko54='} 2023-10-12T00:56:35.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0ef50015b214bbc-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.173 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.174 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.174 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.175 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.175 DEBUG:boto:Data: 2023-10-12T00:56:35.176 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.176 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.177 DEBUG:boto:Params: {} 2023-10-12T00:56:35.177 DEBUG:boto:Token: None 2023-10-12T00:56:35.178 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.178 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045070c0b0f346994-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.182 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:35.183 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:35.183 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.184 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:35.184 DEBUG:boto:Data: 2023-10-12T00:56:35.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.185 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.186 DEBUG:boto:Params: {} 2023-10-12T00:56:35.186 DEBUG:boto:Token: None 2023-10-12T00:56:35.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:35.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0lskqhwvl2on+DfsUv2Fc2h/VaY= 2023-10-12T00:56:35.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0lskqhwvl2on+DfsUv2Fc2h/VaY='} 2023-10-12T00:56:35.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044b8aef05ee8ae75-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.231 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.232 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.232 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.233 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.233 DEBUG:boto:Data: 2023-10-12T00:56:35.234 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.234 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.235 DEBUG:boto:Params: {} 2023-10-12T00:56:35.235 DEBUG:boto:Token: None 2023-10-12T00:56:35.236 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a99a66b5094811a-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.241 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:35.242 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:35.242 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.243 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:35.243 DEBUG:boto:Data: 2023-10-12T00:56:35.244 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.244 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.245 DEBUG:boto:Params: {} 2023-10-12T00:56:35.245 DEBUG:boto:Token: None 2023-10-12T00:56:35.245 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:35.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:84rDOx5UpEzbZhHWdycxEQ9A7Os= 2023-10-12T00:56:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:84rDOx5UpEzbZhHWdycxEQ9A7Os='} 2023-10-12T00:56:35.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000077f5ecbd853d2a9-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.253 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.253 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.254 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.254 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.254 DEBUG:boto:Data: 2023-10-12T00:56:35.255 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.256 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.256 DEBUG:boto:Params: {} 2023-10-12T00:56:35.256 DEBUG:boto:Token: None 2023-10-12T00:56:35.257 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062202a1c52da2cfc-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.262 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:35.262 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:35.263 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.263 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:35.263 DEBUG:boto:Data: 2023-10-12T00:56:35.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.265 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.265 DEBUG:boto:Params: {} 2023-10-12T00:56:35.266 DEBUG:boto:Token: None 2023-10-12T00:56:35.266 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:35.267 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xEKGqjj+sbREv6tdRRvBhf+Ud5o= 2023-10-12T00:56:35.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xEKGqjj+sbREv6tdRRvBhf+Ud5o='} 2023-10-12T00:56:35.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efa85d276e59ebe1-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.311 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.312 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.312 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.313 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.313 DEBUG:boto:Data: 2023-10-12T00:56:35.314 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.315 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.315 DEBUG:boto:Params: {} 2023-10-12T00:56:35.316 DEBUG:boto:Token: None 2023-10-12T00:56:35.316 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000562ddc2ce0ee0b00-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.321 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:35.322 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:35.322 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.323 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:35.324 DEBUG:boto:Data: 2023-10-12T00:56:35.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.325 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.325 DEBUG:boto:Params: {} 2023-10-12T00:56:35.326 DEBUG:boto:Token: None 2023-10-12T00:56:35.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:35.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FiV+/FrPDuh8qoAtKDiNmTbhWJA= 2023-10-12T00:56:35.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FiV+/FrPDuh8qoAtKDiNmTbhWJA='} 2023-10-12T00:56:35.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a69c612b2eefcf84-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.334 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.335 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.335 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.336 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.336 DEBUG:boto:Data: 2023-10-12T00:56:35.337 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.337 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.338 DEBUG:boto:Params: {} 2023-10-12T00:56:35.338 DEBUG:boto:Token: None 2023-10-12T00:56:35.339 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac36f68ce8102998-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.344 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:35.345 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:35.345 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.346 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:35.346 DEBUG:boto:Data: 2023-10-12T00:56:35.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.347 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.348 DEBUG:boto:Params: {} 2023-10-12T00:56:35.348 DEBUG:boto:Token: None 2023-10-12T00:56:35.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:35.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NrtdyR74nBqlWAOHg7Qb9Mvimg8= 2023-10-12T00:56:35.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NrtdyR74nBqlWAOHg7Qb9Mvimg8='} 2023-10-12T00:56:35.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ffb67b8e4c008ee-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.396 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.397 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.397 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.398 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.398 DEBUG:boto:Data: 2023-10-12T00:56:35.399 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.400 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.400 DEBUG:boto:Params: {} 2023-10-12T00:56:35.400 DEBUG:boto:Token: None 2023-10-12T00:56:35.401 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce9ce38e81cf171c-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.407 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:35.407 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:35.408 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.408 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:35.409 DEBUG:boto:Data: 2023-10-12T00:56:35.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.410 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.410 DEBUG:boto:Params: {} 2023-10-12T00:56:35.411 DEBUG:boto:Token: None 2023-10-12T00:56:35.411 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:35.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MJkFoBlzojASosnBYW0HyR2CADU= 2023-10-12T00:56:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MJkFoBlzojASosnBYW0HyR2CADU='} 2023-10-12T00:56:35.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a748774a700118c-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.419 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.419 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.420 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.420 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.421 DEBUG:boto:Data: 2023-10-12T00:56:35.421 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.422 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.422 DEBUG:boto:Params: {} 2023-10-12T00:56:35.423 DEBUG:boto:Token: None 2023-10-12T00:56:35.423 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000facc3477fde46613-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.428 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:35.429 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:35.429 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.429 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:35.430 DEBUG:boto:Data: 2023-10-12T00:56:35.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.432 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.433 DEBUG:boto:Params: {} 2023-10-12T00:56:35.433 DEBUG:boto:Token: None 2023-10-12T00:56:35.434 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:35.435 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XpaDwpaL/J/5ZlJ6aByZQkxZqCo= 2023-10-12T00:56:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XpaDwpaL/J/5ZlJ6aByZQkxZqCo='} 2023-10-12T00:56:35.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055b1631a0afe5202-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.479 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.480 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.480 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.481 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.481 DEBUG:boto:Data: 2023-10-12T00:56:35.482 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.482 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.483 DEBUG:boto:Params: {} 2023-10-12T00:56:35.485 DEBUG:boto:Token: None 2023-10-12T00:56:35.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb34180ea5622315-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.491 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:35.492 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:35.492 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.493 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:35.493 DEBUG:boto:Data: 2023-10-12T00:56:35.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.495 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.496 DEBUG:boto:Params: {} 2023-10-12T00:56:35.496 DEBUG:boto:Token: None 2023-10-12T00:56:35.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:35.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGOdHbRY2W1W1oBRoF2yIGQXsB0= 2023-10-12T00:56:35.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGOdHbRY2W1W1oBRoF2yIGQXsB0='} 2023-10-12T00:56:35.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f7da652066fc44e-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.504 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.505 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.505 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.505 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.506 DEBUG:boto:Data: 2023-10-12T00:56:35.506 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.507 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.507 DEBUG:boto:Params: {} 2023-10-12T00:56:35.508 DEBUG:boto:Token: None 2023-10-12T00:56:35.508 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d582f53fbb70036-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.513 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:35.514 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:35.514 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.515 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:35.515 DEBUG:boto:Data: 2023-10-12T00:56:35.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.516 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.517 DEBUG:boto:Params: {} 2023-10-12T00:56:35.517 DEBUG:boto:Token: None 2023-10-12T00:56:35.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:35.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EgQTOwKCLwAzsshtdPPXC0+J+CE= 2023-10-12T00:56:35.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EgQTOwKCLwAzsshtdPPXC0+J+CE='} 2023-10-12T00:56:35.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e598864b6ad8d63-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.524 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.525 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.526 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.526 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.527 DEBUG:boto:Data: 2023-10-12T00:56:35.528 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.529 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.530 DEBUG:boto:Params: {} 2023-10-12T00:56:35.530 DEBUG:boto:Token: None 2023-10-12T00:56:35.531 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3a0187ed8d2d726-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.537 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:35.537 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:35.538 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.539 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:35.539 DEBUG:boto:Data: 2023-10-12T00:56:35.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.541 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.541 DEBUG:boto:Params: {} 2023-10-12T00:56:35.541 DEBUG:boto:Token: None 2023-10-12T00:56:35.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:35.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yHH8ztGJUWvUKni4wOulRQ+93fM= 2023-10-12T00:56:35.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yHH8ztGJUWvUKni4wOulRQ+93fM='} 2023-10-12T00:56:35.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd574f18e8c16ab-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.591 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.592 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.592 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.593 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.593 DEBUG:boto:Data: 2023-10-12T00:56:35.593 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.594 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.595 DEBUG:boto:Params: {} 2023-10-12T00:56:35.595 DEBUG:boto:Token: None 2023-10-12T00:56:35.596 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4e586239dffa717-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.601 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:35.601 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:35.602 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.602 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:35.603 DEBUG:boto:Data: 2023-10-12T00:56:35.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.604 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.604 DEBUG:boto:Params: {} 2023-10-12T00:56:35.605 DEBUG:boto:Token: None 2023-10-12T00:56:35.605 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:35.606 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uy3udQzU5pWC6NICthX1JJl2lAg= 2023-10-12T00:56:35.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uy3udQzU5pWC6NICthX1JJl2lAg='} 2023-10-12T00:56:35.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000993c1b450d5699aa-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.613 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.613 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.614 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.614 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.614 DEBUG:boto:Data: 2023-10-12T00:56:35.615 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.616 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.616 DEBUG:boto:Params: {} 2023-10-12T00:56:35.616 DEBUG:boto:Token: None 2023-10-12T00:56:35.617 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.618 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009537da16b6c13440-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.622 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:35.622 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:35.623 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.623 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:35.623 DEBUG:boto:Data: 2023-10-12T00:56:35.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.625 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.625 DEBUG:boto:Params: {} 2023-10-12T00:56:35.626 DEBUG:boto:Token: None 2023-10-12T00:56:35.626 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:35.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9ctCHihea/12/2tcWY3E36DljU8= 2023-10-12T00:56:35.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9ctCHihea/12/2tcWY3E36DljU8='} 2023-10-12T00:56:35.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047367d6bf7efd4dc-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.671 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.672 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.672 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.673 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.673 DEBUG:boto:Data: 2023-10-12T00:56:35.673 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.674 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.675 DEBUG:boto:Params: {} 2023-10-12T00:56:35.675 DEBUG:boto:Token: None 2023-10-12T00:56:35.676 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a13fd14fc490e211-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.681 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:35.681 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:35.681 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.682 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:35.682 DEBUG:boto:Data: 2023-10-12T00:56:35.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.683 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.684 DEBUG:boto:Params: {} 2023-10-12T00:56:35.684 DEBUG:boto:Token: None 2023-10-12T00:56:35.685 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:35.685 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e8rMV5GXyahK4uW/TkSuQA0KwYE= 2023-10-12T00:56:35.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e8rMV5GXyahK4uW/TkSuQA0KwYE='} 2023-10-12T00:56:35.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032fa7b45e97d2bd2-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.692 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.693 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.693 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.693 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.694 DEBUG:boto:Data: 2023-10-12T00:56:35.694 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.695 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.695 DEBUG:boto:Params: {} 2023-10-12T00:56:35.696 DEBUG:boto:Token: None 2023-10-12T00:56:35.696 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5d5e5f2048215a8-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.701 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:35.702 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:35.702 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.703 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:35.703 DEBUG:boto:Data: 2023-10-12T00:56:35.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.704 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.705 DEBUG:boto:Params: {} 2023-10-12T00:56:35.705 DEBUG:boto:Token: None 2023-10-12T00:56:35.706 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:35.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tc20DYxSbVCHnk+n5qG0hTV65mw= 2023-10-12T00:56:35.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tc20DYxSbVCHnk+n5qG0hTV65mw='} 2023-10-12T00:56:35.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec4a2341bd7c79f0-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.755 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.756 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.756 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.757 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.757 DEBUG:boto:Data: 2023-10-12T00:56:35.757 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.758 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.759 DEBUG:boto:Params: {} 2023-10-12T00:56:35.759 DEBUG:boto:Token: None 2023-10-12T00:56:35.760 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d16812e7c9c59659-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.765 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:35.765 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:35.766 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.766 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:35.767 DEBUG:boto:Data: 2023-10-12T00:56:35.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.768 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.768 DEBUG:boto:Params: {} 2023-10-12T00:56:35.769 DEBUG:boto:Token: None 2023-10-12T00:56:35.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:35.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jljY3h7F3pv+FbnimJBb6TInaFE= 2023-10-12T00:56:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jljY3h7F3pv+FbnimJBb6TInaFE='} 2023-10-12T00:56:35.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a186985090dbeba4-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.777 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:35.778 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:35.778 DEBUG:boto:Method: HEAD 2023-10-12T00:56:35.778 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:35.779 DEBUG:boto:Data: 2023-10-12T00:56:35.779 DEBUG:boto:Headers: {} 2023-10-12T00:56:35.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.780 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.781 DEBUG:boto:Params: {} 2023-10-12T00:56:35.781 DEBUG:boto:Token: None 2023-10-12T00:56:35.782 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/ 2023-10-12T00:56:35.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo= 2023-10-12T00:56:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nx5duNJdoUdGm88m4rkfJBXiKPo=', 'Content-Length': '0'} 2023-10-12T00:56:35.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000863b559429f3ce02-0065274443-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.786 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:35.787 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:35.787 DEBUG:boto:Method: PUT 2023-10-12T00:56:35.788 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:35.788 DEBUG:boto:Data: 2023-10-12T00:56:35.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:35.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:35.789 DEBUG:boto:Port: 8000 2023-10-12T00:56:35.790 DEBUG:boto:Params: {} 2023-10-12T00:56:35.790 DEBUG:boto:Token: None 2023-10-12T00:56:35.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:35 GMT /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:35.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NpxiTzklr1xW9OiKk0XVxZ3M160= 2023-10-12T00:56:35.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NpxiTzklr1xW9OiKk0XVxZ3M160='} 2023-10-12T00:56:35.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f94223d6e58c6e9-0065274443-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:35.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:35.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:35.836 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:35.938 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:56:35.939 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-58 2023-10-12T00:56:35.950 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 54 2023-10-12T00:56:36.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:36.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:36.010 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:36.061 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:36.062 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:36.101 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-10-12T00:56:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000060", 2023-10-12T00:56:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-10-12T00:56:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-10-12T00:56:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-10-12T00:56:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-10-12T00:56:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000065", 2023-10-12T00:56:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-10-12T00:56:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-10-12T00:56:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000048", 2023-10-12T00:56:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-10-12T00:56:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-10-12T00:56:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-10-12T00:56:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000062", 2023-10-12T00:56:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-10-12T00:56:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000040", 2023-10-12T00:56:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000024", 2023-10-12T00:56:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-10-12T00:56:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-10-12T00:56:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-10-12T00:56:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000045", 2023-10-12T00:56:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-10-12T00:56:36.107 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-10-12T00:56:36.107 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-10-12T00:56:36.107 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000039", 2023-10-12T00:56:36.107 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000027", 2023-10-12T00:56:36.107 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-10-12T00:56:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-10-12T00:56:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000053", 2023-10-12T00:56:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-10-12T00:56:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-10-12T00:56:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000036", 2023-10-12T00:56:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-10-12T00:56:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-10-12T00:56:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-10-12T00:56:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000057", 2023-10-12T00:56:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000018", 2023-10-12T00:56:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008" 2023-10-12T00:56:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:36.110 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:36.111 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.112 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.112 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.113 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.113 DEBUG:boto:Data: 2023-10-12T00:56:36.113 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.114 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.115 DEBUG:boto:Params: {} 2023-10-12T00:56:36.115 DEBUG:boto:Token: None 2023-10-12T00:56:36.116 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b63c496206fd2471-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.121 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:36.121 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:36.122 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.122 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:36.123 DEBUG:boto:Data: 2023-10-12T00:56:36.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.124 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.125 DEBUG:boto:Params: {} 2023-10-12T00:56:36.125 DEBUG:boto:Token: None 2023-10-12T00:56:36.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:36.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5t4eYFJfa/yr4XH6RhfC5dQy/pk= 2023-10-12T00:56:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5t4eYFJfa/yr4XH6RhfC5dQy/pk='} 2023-10-12T00:56:36.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fc65a72afa06714-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.132 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.133 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.133 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.134 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.134 DEBUG:boto:Data: 2023-10-12T00:56:36.134 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.135 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.135 DEBUG:boto:Params: {} 2023-10-12T00:56:36.136 DEBUG:boto:Token: None 2023-10-12T00:56:36.136 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007120b6ea71109b91-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.141 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:36.141 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:36.142 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.142 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:36.143 DEBUG:boto:Data: 2023-10-12T00:56:36.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.144 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.145 DEBUG:boto:Params: {} 2023-10-12T00:56:36.145 DEBUG:boto:Token: None 2023-10-12T00:56:36.145 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:36.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ioO64zH5b/an5At5L6K74IpOb8w= 2023-10-12T00:56:36.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ioO64zH5b/an5At5L6K74IpOb8w='} 2023-10-12T00:56:36.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d62c4b569381988-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.191 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.192 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.192 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.193 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.193 DEBUG:boto:Data: 2023-10-12T00:56:36.193 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.194 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.195 DEBUG:boto:Params: {} 2023-10-12T00:56:36.195 DEBUG:boto:Token: None 2023-10-12T00:56:36.196 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000127e32a96c67b61d-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.201 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:36.201 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:36.202 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.202 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:36.203 DEBUG:boto:Data: 2023-10-12T00:56:36.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.204 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.204 DEBUG:boto:Params: {} 2023-10-12T00:56:36.205 DEBUG:boto:Token: None 2023-10-12T00:56:36.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:36.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eJsxlGR7gKk+k9KG+aNXr3UxYmw= 2023-10-12T00:56:36.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eJsxlGR7gKk+k9KG+aNXr3UxYmw='} 2023-10-12T00:56:36.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c38ecc6e99ffdc6-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.213 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.214 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.214 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.215 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.215 DEBUG:boto:Data: 2023-10-12T00:56:36.215 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.216 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.217 DEBUG:boto:Params: {} 2023-10-12T00:56:36.217 DEBUG:boto:Token: None 2023-10-12T00:56:36.218 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002618a1181d8e834f-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.223 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:36.223 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:36.224 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.224 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:36.224 DEBUG:boto:Data: 2023-10-12T00:56:36.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.226 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.226 DEBUG:boto:Params: {} 2023-10-12T00:56:36.227 DEBUG:boto:Token: None 2023-10-12T00:56:36.227 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:36.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mOSCu+J5+ZT9Vbtoqr4/ed1QUpI= 2023-10-12T00:56:36.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mOSCu+J5+ZT9Vbtoqr4/ed1QUpI='} 2023-10-12T00:56:36.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040a20e341c1dffec-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.271 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.272 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.272 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.272 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.273 DEBUG:boto:Data: 2023-10-12T00:56:36.273 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.274 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.275 DEBUG:boto:Params: {} 2023-10-12T00:56:36.275 DEBUG:boto:Token: None 2023-10-12T00:56:36.276 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005643c4309cfce699-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.281 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:36.281 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:36.282 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.282 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:36.283 DEBUG:boto:Data: 2023-10-12T00:56:36.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.284 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.284 DEBUG:boto:Params: {} 2023-10-12T00:56:36.285 DEBUG:boto:Token: None 2023-10-12T00:56:36.286 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:36.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zR82kQhgVP000Ih7D8wxHHY7Vtw= 2023-10-12T00:56:36.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zR82kQhgVP000Ih7D8wxHHY7Vtw='} 2023-10-12T00:56:36.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e577e0e0acd430f-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.293 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.294 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.294 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.294 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.295 DEBUG:boto:Data: 2023-10-12T00:56:36.295 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.297 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.298 DEBUG:boto:Params: {} 2023-10-12T00:56:36.299 DEBUG:boto:Token: None 2023-10-12T00:56:36.299 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004af0d36565cb5f4e-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.306 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:36.307 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:36.308 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.309 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:36.310 DEBUG:boto:Data: 2023-10-12T00:56:36.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.312 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.313 DEBUG:boto:Params: {} 2023-10-12T00:56:36.314 DEBUG:boto:Token: None 2023-10-12T00:56:36.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:36.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PNYioPbDzcd7tYy3DAwOPjitRvg= 2023-10-12T00:56:36.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PNYioPbDzcd7tYy3DAwOPjitRvg='} 2023-10-12T00:56:36.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027950a96b580bee6-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.363 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.364 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.364 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.365 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.365 DEBUG:boto:Data: 2023-10-12T00:56:36.366 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.367 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.367 DEBUG:boto:Params: {} 2023-10-12T00:56:36.368 DEBUG:boto:Token: None 2023-10-12T00:56:36.368 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009220aa4c7498c10b-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.373 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:36.373 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:36.374 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.374 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:36.375 DEBUG:boto:Data: 2023-10-12T00:56:36.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.376 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.376 DEBUG:boto:Params: {} 2023-10-12T00:56:36.377 DEBUG:boto:Token: None 2023-10-12T00:56:36.377 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:36.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EOy3vnKtek/FDq/YRijQD5680UY= 2023-10-12T00:56:36.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EOy3vnKtek/FDq/YRijQD5680UY='} 2023-10-12T00:56:36.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a678af10e07ac436-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.385 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.385 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.386 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.386 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.387 DEBUG:boto:Data: 2023-10-12T00:56:36.387 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.388 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.388 DEBUG:boto:Params: {} 2023-10-12T00:56:36.389 DEBUG:boto:Token: None 2023-10-12T00:56:36.389 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2853d07a8c23527-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.394 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:36.395 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:36.395 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.396 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:36.396 DEBUG:boto:Data: 2023-10-12T00:56:36.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.397 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.398 DEBUG:boto:Params: {} 2023-10-12T00:56:36.398 DEBUG:boto:Token: None 2023-10-12T00:56:36.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:36.399 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iN6PjGhhlDSXy+ABh2NKzGBzXJU= 2023-10-12T00:56:36.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iN6PjGhhlDSXy+ABh2NKzGBzXJU='} 2023-10-12T00:56:36.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e72dfe89dc8faf82-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.443 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.443 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.444 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.444 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.445 DEBUG:boto:Data: 2023-10-12T00:56:36.445 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.446 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.446 DEBUG:boto:Params: {} 2023-10-12T00:56:36.447 DEBUG:boto:Token: None 2023-10-12T00:56:36.447 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1ef43cf7ee01fe5-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.452 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:36.452 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:36.453 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.453 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:36.453 DEBUG:boto:Data: 2023-10-12T00:56:36.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.455 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.455 DEBUG:boto:Params: {} 2023-10-12T00:56:36.455 DEBUG:boto:Token: None 2023-10-12T00:56:36.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:36.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8FEaqCqIdg5NOu8YA3Zr17yf8dA= 2023-10-12T00:56:36.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8FEaqCqIdg5NOu8YA3Zr17yf8dA='} 2023-10-12T00:56:36.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab182b24d312e2bc-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.463 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.463 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.464 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.464 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.465 DEBUG:boto:Data: 2023-10-12T00:56:36.465 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.466 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.467 DEBUG:boto:Params: {} 2023-10-12T00:56:36.467 DEBUG:boto:Token: None 2023-10-12T00:56:36.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc4c8ff29bceca2c-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.472 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:36.473 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:36.473 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.474 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:36.474 DEBUG:boto:Data: 2023-10-12T00:56:36.474 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.475 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.476 DEBUG:boto:Params: {} 2023-10-12T00:56:36.476 DEBUG:boto:Token: None 2023-10-12T00:56:36.476 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:36.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/UmdcjmFmDIk7Xgg1kFApZNazIs= 2023-10-12T00:56:36.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/UmdcjmFmDIk7Xgg1kFApZNazIs='} 2023-10-12T00:56:36.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deb05b341da9af96-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.523 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.524 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.524 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.524 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.525 DEBUG:boto:Data: 2023-10-12T00:56:36.525 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.526 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.527 DEBUG:boto:Params: {} 2023-10-12T00:56:36.527 DEBUG:boto:Token: None 2023-10-12T00:56:36.527 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd9a83601fe7cf31-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.532 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:36.533 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:36.533 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.534 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:36.534 DEBUG:boto:Data: 2023-10-12T00:56:36.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.536 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.536 DEBUG:boto:Params: {} 2023-10-12T00:56:36.537 DEBUG:boto:Token: None 2023-10-12T00:56:36.537 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:36.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YXHXiR6MYCeNg8DJoWnV9Nw2tvQ= 2023-10-12T00:56:36.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YXHXiR6MYCeNg8DJoWnV9Nw2tvQ='} 2023-10-12T00:56:36.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006490aa5098fa1c2a-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.547 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.548 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.549 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.549 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.550 DEBUG:boto:Data: 2023-10-12T00:56:36.551 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.552 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.553 DEBUG:boto:Params: {} 2023-10-12T00:56:36.554 DEBUG:boto:Token: None 2023-10-12T00:56:36.555 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000523536dcfe764dcd-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.562 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:36.563 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:36.563 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.564 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:36.565 DEBUG:boto:Data: 2023-10-12T00:56:36.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.566 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.567 DEBUG:boto:Params: {} 2023-10-12T00:56:36.567 DEBUG:boto:Token: None 2023-10-12T00:56:36.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:36.568 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GSsZ9dWRd3afBEODnO/BPhaEiCE= 2023-10-12T00:56:36.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GSsZ9dWRd3afBEODnO/BPhaEiCE='} 2023-10-12T00:56:36.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098b359e3eada1014-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.619 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.620 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.620 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.621 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.621 DEBUG:boto:Data: 2023-10-12T00:56:36.621 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.623 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.623 DEBUG:boto:Params: {} 2023-10-12T00:56:36.624 DEBUG:boto:Token: None 2023-10-12T00:56:36.625 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040f4a425f6b7ec8f-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.630 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:36.631 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:36.631 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.631 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:36.632 DEBUG:boto:Data: 2023-10-12T00:56:36.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.633 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.634 DEBUG:boto:Params: {} 2023-10-12T00:56:36.634 DEBUG:boto:Token: None 2023-10-12T00:56:36.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:36.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GCDnkelezAg1nrrjMpiybsaceec= 2023-10-12T00:56:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GCDnkelezAg1nrrjMpiybsaceec='} 2023-10-12T00:56:36.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c227573441172fd-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.643 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.643 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.644 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.644 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.644 DEBUG:boto:Data: 2023-10-12T00:56:36.645 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.646 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.646 DEBUG:boto:Params: {} 2023-10-12T00:56:36.647 DEBUG:boto:Token: None 2023-10-12T00:56:36.647 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aaee36e74f10f86-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.653 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:36.654 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:36.655 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.656 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:36.656 DEBUG:boto:Data: 2023-10-12T00:56:36.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.658 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.659 DEBUG:boto:Params: {} 2023-10-12T00:56:36.659 DEBUG:boto:Token: None 2023-10-12T00:56:36.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:36.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LhpE6cpdC+vcA3ky2rinTn3q75k= 2023-10-12T00:56:36.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LhpE6cpdC+vcA3ky2rinTn3q75k='} 2023-10-12T00:56:36.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2615c2cc2521e1c-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.712 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.712 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.713 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.713 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.714 DEBUG:boto:Data: 2023-10-12T00:56:36.714 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.718 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.718 DEBUG:boto:Params: {} 2023-10-12T00:56:36.719 DEBUG:boto:Token: None 2023-10-12T00:56:36.719 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe73891aac9249f1-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.725 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:36.725 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:36.725 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.726 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:36.726 DEBUG:boto:Data: 2023-10-12T00:56:36.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.728 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.728 DEBUG:boto:Params: {} 2023-10-12T00:56:36.729 DEBUG:boto:Token: None 2023-10-12T00:56:36.729 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:36.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:auHV17JOIRihBRHHlRbRZB+CYbU= 2023-10-12T00:56:36.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:auHV17JOIRihBRHHlRbRZB+CYbU='} 2023-10-12T00:56:36.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f374b55b988e0ace-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.737 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.738 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.738 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.739 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.739 DEBUG:boto:Data: 2023-10-12T00:56:36.739 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.740 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.741 DEBUG:boto:Params: {} 2023-10-12T00:56:36.741 DEBUG:boto:Token: None 2023-10-12T00:56:36.742 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000277372ff7b38a81e-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.747 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:36.747 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:36.748 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.748 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:36.749 DEBUG:boto:Data: 2023-10-12T00:56:36.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.751 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.752 DEBUG:boto:Params: {} 2023-10-12T00:56:36.753 DEBUG:boto:Token: None 2023-10-12T00:56:36.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:36.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iDZe1Yvv2dh52siTTDWBKBVqWjQ= 2023-10-12T00:56:36.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iDZe1Yvv2dh52siTTDWBKBVqWjQ='} 2023-10-12T00:56:36.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba59f85c681bec93-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.799 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.800 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.801 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.801 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.802 DEBUG:boto:Data: 2023-10-12T00:56:36.803 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.804 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.804 DEBUG:boto:Params: {} 2023-10-12T00:56:36.805 DEBUG:boto:Token: None 2023-10-12T00:56:36.805 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000068e3ca2ae24f6d8-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.810 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:36.811 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:36.811 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.812 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:36.812 DEBUG:boto:Data: 2023-10-12T00:56:36.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.813 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.814 DEBUG:boto:Params: {} 2023-10-12T00:56:36.814 DEBUG:boto:Token: None 2023-10-12T00:56:36.815 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:36.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1/Ihrq8oSO97tZh8ykZmhJuViYg= 2023-10-12T00:56:36.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1/Ihrq8oSO97tZh8ykZmhJuViYg='} 2023-10-12T00:56:36.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004207a5f78024ea5e-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.822 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.823 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.823 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.824 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.825 DEBUG:boto:Data: 2023-10-12T00:56:36.825 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.827 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.827 DEBUG:boto:Params: {} 2023-10-12T00:56:36.828 DEBUG:boto:Token: None 2023-10-12T00:56:36.828 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000465d83ef318cbedb-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.833 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:36.834 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:36.834 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.834 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:36.835 DEBUG:boto:Data: 2023-10-12T00:56:36.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.836 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.837 DEBUG:boto:Params: {} 2023-10-12T00:56:36.837 DEBUG:boto:Token: None 2023-10-12T00:56:36.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:36.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oNzoWRo05v9p1BId/LYEg2NIBx8= 2023-10-12T00:56:36.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oNzoWRo05v9p1BId/LYEg2NIBx8='} 2023-10-12T00:56:36.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad0cf3e1a68da260-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.887 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.888 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.888 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.889 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.889 DEBUG:boto:Data: 2023-10-12T00:56:36.889 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.890 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.891 DEBUG:boto:Params: {} 2023-10-12T00:56:36.891 DEBUG:boto:Token: None 2023-10-12T00:56:36.892 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7bb7e6dade7f183-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.897 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:36.897 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:36.898 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.898 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:36.899 DEBUG:boto:Data: 2023-10-12T00:56:36.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.900 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.900 DEBUG:boto:Params: {} 2023-10-12T00:56:36.901 DEBUG:boto:Token: None 2023-10-12T00:56:36.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:36.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gky59lCDleZyxgV7za/wIeVo/Ag= 2023-10-12T00:56:36.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gky59lCDleZyxgV7za/wIeVo/Ag='} 2023-10-12T00:56:36.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8cf7c6438a47f5f-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.908 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.908 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.909 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.909 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.909 DEBUG:boto:Data: 2023-10-12T00:56:36.910 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.911 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.911 DEBUG:boto:Params: {} 2023-10-12T00:56:36.911 DEBUG:boto:Token: None 2023-10-12T00:56:36.912 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccca42c9b2a7cd4b-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.917 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:36.917 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:36.918 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.918 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:36.918 DEBUG:boto:Data: 2023-10-12T00:56:36.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.921 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.921 DEBUG:boto:Params: {} 2023-10-12T00:56:36.922 DEBUG:boto:Token: None 2023-10-12T00:56:36.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:36.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TAS7MyVOcK4PjSiYsTmEpWLapxw= 2023-10-12T00:56:36.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TAS7MyVOcK4PjSiYsTmEpWLapxw='} 2023-10-12T00:56:36.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000808b9edc9aa910a2-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.967 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.968 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.968 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.968 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.969 DEBUG:boto:Data: 2023-10-12T00:56:36.969 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.970 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.971 DEBUG:boto:Params: {} 2023-10-12T00:56:36.971 DEBUG:boto:Token: None 2023-10-12T00:56:36.971 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046fae010652e85ac-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.976 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:36.976 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:36.977 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.977 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:36.978 DEBUG:boto:Data: 2023-10-12T00:56:36.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:36.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.979 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.979 DEBUG:boto:Params: {} 2023-10-12T00:56:36.980 DEBUG:boto:Token: None 2023-10-12T00:56:36.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:36.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MABugbqlHGaUxNnEBGuWR2wPzvk= 2023-10-12T00:56:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MABugbqlHGaUxNnEBGuWR2wPzvk='} 2023-10-12T00:56:36.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fe202a05ead463f-0065274444-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.988 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:36.988 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:36.989 DEBUG:boto:Method: HEAD 2023-10-12T00:56:36.989 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:36.990 DEBUG:boto:Data: 2023-10-12T00:56:36.990 DEBUG:boto:Headers: {} 2023-10-12T00:56:36.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:36.991 DEBUG:boto:Port: 8000 2023-10-12T00:56:36.991 DEBUG:boto:Params: {} 2023-10-12T00:56:36.992 DEBUG:boto:Token: None 2023-10-12T00:56:36.992 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:36 GMT /wafmoa-58/ 2023-10-12T00:56:36.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M= 2023-10-12T00:56:36.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hrLbfZMFj4yemuJvYvBGMcGfp8M=', 'Content-Length': '0'} 2023-10-12T00:56:36.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047370a83404b6282-0065274444-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:36.997 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:36.997 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:36.998 DEBUG:boto:Method: PUT 2023-10-12T00:56:36.998 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:36.999 DEBUG:boto:Data: 2023-10-12T00:56:36.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.000 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.000 DEBUG:boto:Params: {} 2023-10-12T00:56:37.001 DEBUG:boto:Token: None 2023-10-12T00:56:37.001 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:37.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RRQ6oR57AVjBYGGrtc8SUYlL/fk= 2023-10-12T00:56:37.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RRQ6oR57AVjBYGGrtc8SUYlL/fk='} 2023-10-12T00:56:37.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000943eabeeec1e35d-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.047 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.048 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.048 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.049 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.049 DEBUG:boto:Data: 2023-10-12T00:56:37.050 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.050 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.051 DEBUG:boto:Params: {} 2023-10-12T00:56:37.051 DEBUG:boto:Token: None 2023-10-12T00:56:37.052 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4db63060b19786f-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.056 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:37.057 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:37.057 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.058 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:37.058 DEBUG:boto:Data: 2023-10-12T00:56:37.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.059 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.060 DEBUG:boto:Params: {} 2023-10-12T00:56:37.060 DEBUG:boto:Token: None 2023-10-12T00:56:37.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:37.062 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N8DpGCUiEJ85aPYrwTcPxyFU/Ys= 2023-10-12T00:56:37.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N8DpGCUiEJ85aPYrwTcPxyFU/Ys='} 2023-10-12T00:56:37.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc5d85d9ce8a0d6b-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.070 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.070 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.071 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.071 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.071 DEBUG:boto:Data: 2023-10-12T00:56:37.072 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.073 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.073 DEBUG:boto:Params: {} 2023-10-12T00:56:37.074 DEBUG:boto:Token: None 2023-10-12T00:56:37.074 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048517e3c1b28821e-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.079 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:37.080 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:37.080 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.081 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:37.081 DEBUG:boto:Data: 2023-10-12T00:56:37.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.083 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.083 DEBUG:boto:Params: {} 2023-10-12T00:56:37.084 DEBUG:boto:Token: None 2023-10-12T00:56:37.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:37.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C+rbS3hpHuIPLbIhlQX7O1ed8Ms= 2023-10-12T00:56:37.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C+rbS3hpHuIPLbIhlQX7O1ed8Ms='} 2023-10-12T00:56:37.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002efb9c2f55a827b8-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.131 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.132 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.132 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.133 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.133 DEBUG:boto:Data: 2023-10-12T00:56:37.133 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.134 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.135 DEBUG:boto:Params: {} 2023-10-12T00:56:37.135 DEBUG:boto:Token: None 2023-10-12T00:56:37.136 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c56460496c6943a-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.141 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:37.142 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:37.142 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.143 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:37.143 DEBUG:boto:Data: 2023-10-12T00:56:37.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.144 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.145 DEBUG:boto:Params: {} 2023-10-12T00:56:37.145 DEBUG:boto:Token: None 2023-10-12T00:56:37.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:37.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JwcvGU/xUuYB53h0j5pKZ30u+Jw= 2023-10-12T00:56:37.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JwcvGU/xUuYB53h0j5pKZ30u+Jw='} 2023-10-12T00:56:37.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000faddded20d2bb229-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.153 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.153 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.154 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.154 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.154 DEBUG:boto:Data: 2023-10-12T00:56:37.155 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.156 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.156 DEBUG:boto:Params: {} 2023-10-12T00:56:37.157 DEBUG:boto:Token: None 2023-10-12T00:56:37.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c6378e69b9f868b-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.162 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:37.162 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:37.163 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.164 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:37.165 DEBUG:boto:Data: 2023-10-12T00:56:37.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.167 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.168 DEBUG:boto:Params: {} 2023-10-12T00:56:37.169 DEBUG:boto:Token: None 2023-10-12T00:56:37.170 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:37.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WgNDPtnXhr0MzwGePkVtCwdlMXM= 2023-10-12T00:56:37.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WgNDPtnXhr0MzwGePkVtCwdlMXM='} 2023-10-12T00:56:37.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000320e4f91cadb6cd8-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.219 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.220 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.220 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.221 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.221 DEBUG:boto:Data: 2023-10-12T00:56:37.221 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.222 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.222 DEBUG:boto:Params: {} 2023-10-12T00:56:37.223 DEBUG:boto:Token: None 2023-10-12T00:56:37.223 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099e6ba718e1c62b6-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.228 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:37.228 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:37.229 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.229 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:37.230 DEBUG:boto:Data: 2023-10-12T00:56:37.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.231 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.231 DEBUG:boto:Params: {} 2023-10-12T00:56:37.232 DEBUG:boto:Token: None 2023-10-12T00:56:37.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:37.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kf6ItJ8I2R9q4MSVPFdDA2sokQo= 2023-10-12T00:56:37.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kf6ItJ8I2R9q4MSVPFdDA2sokQo='} 2023-10-12T00:56:37.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d95fc61ad822edc-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.239 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.240 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.240 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.241 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.241 DEBUG:boto:Data: 2023-10-12T00:56:37.241 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.242 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.243 DEBUG:boto:Params: {} 2023-10-12T00:56:37.243 DEBUG:boto:Token: None 2023-10-12T00:56:37.243 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3a6761b5cb2316-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.248 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:37.249 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:37.249 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.249 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:37.250 DEBUG:boto:Data: 2023-10-12T00:56:37.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.251 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.252 DEBUG:boto:Params: {} 2023-10-12T00:56:37.252 DEBUG:boto:Token: None 2023-10-12T00:56:37.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:37.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JhisHKc07deWcfO8XYWrBZ/IP4c= 2023-10-12T00:56:37.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JhisHKc07deWcfO8XYWrBZ/IP4c='} 2023-10-12T00:56:37.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017c6c79f979caca4-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.299 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.299 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.300 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.300 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.301 DEBUG:boto:Data: 2023-10-12T00:56:37.301 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.302 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.302 DEBUG:boto:Params: {} 2023-10-12T00:56:37.303 DEBUG:boto:Token: None 2023-10-12T00:56:37.303 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.304 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f5e82874e1f7045-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.308 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:37.308 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:37.308 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.309 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:37.309 DEBUG:boto:Data: 2023-10-12T00:56:37.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.311 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.311 DEBUG:boto:Params: {} 2023-10-12T00:56:37.312 DEBUG:boto:Token: None 2023-10-12T00:56:37.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:37.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:835LKDkaIl9zBEyu9ka3mWaDgQE= 2023-10-12T00:56:37.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:835LKDkaIl9zBEyu9ka3mWaDgQE='} 2023-10-12T00:56:37.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b692252992fad7-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.320 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.321 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.321 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.322 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.322 DEBUG:boto:Data: 2023-10-12T00:56:37.322 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.323 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.324 DEBUG:boto:Params: {} 2023-10-12T00:56:37.324 DEBUG:boto:Token: None 2023-10-12T00:56:37.324 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cceb348897e26ba3-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.329 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:37.330 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:37.330 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.331 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:37.331 DEBUG:boto:Data: 2023-10-12T00:56:37.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.332 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.333 DEBUG:boto:Params: {} 2023-10-12T00:56:37.333 DEBUG:boto:Token: None 2023-10-12T00:56:37.334 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:37.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JfWQCsy3MSh055Enc9XqS5TvL6M= 2023-10-12T00:56:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JfWQCsy3MSh055Enc9XqS5TvL6M='} 2023-10-12T00:56:37.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e22055d26946fec8-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.383 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.383 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.384 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.384 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.385 DEBUG:boto:Data: 2023-10-12T00:56:37.385 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.386 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.386 DEBUG:boto:Params: {} 2023-10-12T00:56:37.387 DEBUG:boto:Token: None 2023-10-12T00:56:37.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edc28a5bc70c96f6-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.392 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:37.392 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:37.393 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.393 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:37.394 DEBUG:boto:Data: 2023-10-12T00:56:37.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.395 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.395 DEBUG:boto:Params: {} 2023-10-12T00:56:37.396 DEBUG:boto:Token: None 2023-10-12T00:56:37.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:37.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YvQp3HxJOC7MAjCa/W4hunSx3B4= 2023-10-12T00:56:37.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YvQp3HxJOC7MAjCa/W4hunSx3B4='} 2023-10-12T00:56:37.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b05631a31453842e-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.404 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.404 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.404 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.405 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.405 DEBUG:boto:Data: 2023-10-12T00:56:37.406 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.407 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.407 DEBUG:boto:Params: {} 2023-10-12T00:56:37.407 DEBUG:boto:Token: None 2023-10-12T00:56:37.408 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d035ec0fc765d2af-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.412 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:37.413 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:37.413 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.414 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:37.414 DEBUG:boto:Data: 2023-10-12T00:56:37.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.415 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.416 DEBUG:boto:Params: {} 2023-10-12T00:56:37.416 DEBUG:boto:Token: None 2023-10-12T00:56:37.416 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:37.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eGq9zkZaygqpwUZi1UOaftlfJz0= 2023-10-12T00:56:37.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eGq9zkZaygqpwUZi1UOaftlfJz0='} 2023-10-12T00:56:37.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050d25733ff6e2a87-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.463 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.464 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.464 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.465 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.465 DEBUG:boto:Data: 2023-10-12T00:56:37.465 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.466 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.467 DEBUG:boto:Params: {} 2023-10-12T00:56:37.467 DEBUG:boto:Token: None 2023-10-12T00:56:37.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000db71eef2bc38dcd-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.473 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:37.473 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:37.473 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.474 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:37.474 DEBUG:boto:Data: 2023-10-12T00:56:37.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.475 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.476 DEBUG:boto:Params: {} 2023-10-12T00:56:37.476 DEBUG:boto:Token: None 2023-10-12T00:56:37.477 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:37.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F6mRTvznp8R/YYFJEqdmS++Ykjo= 2023-10-12T00:56:37.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F6mRTvznp8R/YYFJEqdmS++Ykjo='} 2023-10-12T00:56:37.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e48ce550aa163756-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.484 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:37.485 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:37.485 DEBUG:boto:Method: HEAD 2023-10-12T00:56:37.486 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:37.486 DEBUG:boto:Data: 2023-10-12T00:56:37.486 DEBUG:boto:Headers: {} 2023-10-12T00:56:37.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.487 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.488 DEBUG:boto:Params: {} 2023-10-12T00:56:37.488 DEBUG:boto:Token: None 2023-10-12T00:56:37.489 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/ 2023-10-12T00:56:37.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw= 2023-10-12T00:56:37.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pP1MkMV5QOn6IHuQ90cd5Ln8Mmw=', 'Content-Length': '0'} 2023-10-12T00:56:37.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000548968e1406512fe-0065274445-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.493 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:37.494 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:37.494 DEBUG:boto:Method: PUT 2023-10-12T00:56:37.495 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:37.495 DEBUG:boto:Data: 2023-10-12T00:56:37.495 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:56:37.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:37.496 DEBUG:boto:Port: 8000 2023-10-12T00:56:37.497 DEBUG:boto:Params: {} 2023-10-12T00:56:37.497 DEBUG:boto:Token: None 2023-10-12T00:56:37.498 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:56:37 GMT /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:37.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FfPeHTJiTj7Mbe6P3ny2N5sd12U= 2023-10-12T00:56:37.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:56:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FfPeHTJiTj7Mbe6P3ny2N5sd12U='} 2023-10-12T00:56:37.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000844f695b61e0588f-0065274445-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:37.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:37.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:56:37.548 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:56:37.600 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:37.601 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:37.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 2 2023-10-12T00:56:37.668 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-12T00:56:37.668 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 2 2023-10-12T00:56:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000001.78.6 2023-10-12T00:56:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.665+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.666+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.666+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 2 2023-10-12T00:56:37.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.666+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000001.31.6 2023-10-12T00:56:37.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.666+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 2 2023-10-12T00:56:37.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000001.16.6 2023-10-12T00:56:37.673 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 2 2023-10-12T00:56:37.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 2 2023-10-12T00:56:37.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.57.6 2023-10-12T00:56:37.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.667+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-12T00:56:37.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 2 2023-10-12T00:56:37.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 2 2023-10-12T00:56:37.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 2 2023-10-12T00:56:37.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 2 2023-10-12T00:56:37.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 2 2023-10-12T00:56:37.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000001.29.6 2023-10-12T00:56:37.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-12T00:56:37.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000001.36.6 2023-10-12T00:56:37.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.668+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position=00000000001.38.6 2023-10-12T00:56:37.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.718+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.718+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.718+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:37.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:37.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:37.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:37.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:37.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:37.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:37.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:37.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:37.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:37.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:37.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:37.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:37.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:37.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:37.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:37.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:37.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-12T00:56:37.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000020 2023-10-12T00:56:37.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:30.603169+0000 2023-10-12T00:56:37.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-10-12T00:56:37.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000020[0] 2023-10-12T00:56:37.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000001.55.6 2023-10-12T00:56:37.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-12T00:56:37.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000028 2023-10-12T00:56:37.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:30.926692+0000 2023-10-12T00:56:37.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-12T00:56:37.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-12T00:56:37.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000000 2023-10-12T00:56:37.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:30.460596+0000 2023-10-12T00:56:37.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-12T00:56:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2023-10-12T00:56:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-12T00:56:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000000[0] 2023-10-12T00:56:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-12T00:56:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:37.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.723+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000028 2023-10-12T00:56:37.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-12T00:56:37.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000026 2023-10-12T00:56:37.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:30.850900+0000 2023-10-12T00:56:37.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-10-12T00:56:37.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000026[0] 2023-10-12T00:56:37.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-12T00:56:37.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-12T00:56:37.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000010 2023-10-12T00:56:37.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:30.775211+0000 2023-10-12T00:56:37.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-12T00:56:37.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000010[0] 2023-10-12T00:56:37.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-12T00:56:37.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-12T00:56:37.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000013 2023-10-12T00:56:37.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:30.755349+0000 2023-10-12T00:56:37.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-12T00:56:37.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:37.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000004 2023-10-12T00:56:37.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:30.699068+0000 2023-10-12T00:56:37.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-12T00:56:37.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000013[0] 2023-10-12T00:56:37.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-12T00:56:37.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000004[0] 2023-10-12T00:56:37.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-12T00:56:37.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:37.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000001 2023-10-12T00:56:37.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:30.543068+0000 2023-10-12T00:56:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.103.6 op=0 op_state=1 2023-10-12T00:56:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-10-12T00:56:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000002.103.6 2023-10-12T00:56:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-12T00:56:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000033 2023-10-12T00:56:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:30.831636+0000 2023-10-12T00:56:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.48.6 op=0 op_state=1 2023-10-12T00:56:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000001 2023-10-12T00:56:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000033[0] 2023-10-12T00:56:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000001.48.6 2023-10-12T00:56:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.738+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000013]: success 2023-10-12T00:56:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.738+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:1 marker=00000000002.40.6 timestamp=2023-10-12T00:56:30.755349+0000 2023-10-12T00:56:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.738+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000000]: success 2023-10-12T00:56:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.738+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:1 marker=00000000002.53.6 timestamp=2023-10-12T00:56:30.460596+0000 2023-10-12T00:56:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000026]: success 2023-10-12T00:56:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:1 marker=00000000001.45.6 timestamp=2023-10-12T00:56:30.850900+0000 2023-10-12T00:56:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000004]: success 2023-10-12T00:56:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:1 marker=00000000002.35.6 timestamp=2023-10-12T00:56:30.699068+0000 2023-10-12T00:56:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000020]: success 2023-10-12T00:56:37.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.739+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:1 marker=00000000001.55.6 timestamp=2023-10-12T00:56:30.603169+0000 2023-10-12T00:56:37.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000001.105.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:37.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000016 2023-10-12T00:56:37.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:31.005843+0000 2023-10-12T00:56:37.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.105.6 op=0 op_state=1 2023-10-12T00:56:37.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000016[0] 2023-10-12T00:56:37.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000001.105.6 2023-10-12T00:56:37.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-12T00:56:37.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000007 2023-10-12T00:56:37.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:30.679493+0000 2023-10-12T00:56:37.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-12T00:56:37.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000007[0] 2023-10-12T00:56:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-12T00:56:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-12T00:56:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000030 2023-10-12T00:56:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:30.621759+0000 2023-10-12T00:56:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-10-12T00:56:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2023-10-12T00:56:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000001.66.6 2023-10-12T00:56:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000021 2023-10-12T00:56:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:30.907505+0000 2023-10-12T00:56:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-10-12T00:56:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000030 2023-10-12T00:56:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000001.50.6 2023-10-12T00:56:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-12T00:56:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000024 2023-10-12T00:56:37.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:30.441053+0000 2023-10-12T00:56:37.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-10-12T00:56:37.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: : injecting data sync error on key=obj00000000000000000024 2023-10-12T00:56:37.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-12T00:56:37.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-12T00:56:37.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9/obj00000000000000000019 2023-10-12T00:56:37.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: entry->timestamp=2023-10-12T00:56:30.983853+0000 2023-10-12T00:56:37.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.33.6 op=0 op_state=1 2023-10-12T00:56:37.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:37.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000024 2023-10-12T00:56:37.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000019[0] 2023-10-12T00:56:37.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.765+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position=00000000001.33.6 2023-10-12T00:56:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: parsed entry: id=00000000001.20.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-10-12T00:56:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16/obj00000000000000000076 2023-10-12T00:56:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: entry->timestamp=2023-10-12T00:56:30.523646+0000 2023-10-12T00:56:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.20.6 op=0 op_state=1 2023-10-12T00:56:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000076[0] 2023-10-12T00:56:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position=00000000001.20.6 2023-10-12T00:56:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.768+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: a sync operation returned error 2023-10-12T00:56:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.768+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: backing out with sync_status=-5 2023-10-12T00:56:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.768+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.768+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.769+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: a sync operation returned error 2023-10-12T00:56:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.769+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: backing out with sync_status=-5 2023-10-12T00:56:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.769+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.769+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.769+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.770+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.770+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.770+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.771+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.771+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.772+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000076]: success 2023-10-12T00:56:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.772+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16:1 marker=00000000001.20.6 timestamp=2023-10-12T00:56:30.523646+0000 2023-10-12T00:56:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.772+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000019]: success 2023-10-12T00:56:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.772+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9:1 marker=00000000001.33.6 timestamp=2023-10-12T00:56:30.983853+0000 2023-10-12T00:56:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.778+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000033]: success 2023-10-12T00:56:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.778+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:1 marker=00000000001.48.6 timestamp=2023-10-12T00:56:30.831636+0000 2023-10-12T00:56:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.779+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000010]: success 2023-10-12T00:56:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.779+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8:1 marker=00000000002.44.6 timestamp=2023-10-12T00:56:30.775211+0000 2023-10-12T00:56:37.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: a sync operation returned error 2023-10-12T00:56:37.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: backing out with sync_status=-5 2023-10-12T00:56:37.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:37.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:37.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:37.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.806+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.807+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: a sync operation returned error 2023-10-12T00:56:37.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.807+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: backing out with sync_status=-5 2023-10-12T00:56:37.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.807+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:37.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.807+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:37.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.808+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.808+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:37.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.808+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.810+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000007]: success 2023-10-12T00:56:37.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.810+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:1 marker=00000000001.37.6 timestamp=2023-10-12T00:56:30.679493+0000 2023-10-12T00:56:37.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.810+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000016]: success 2023-10-12T00:56:37.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.810+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:1 marker=00000000001.105.6 timestamp=2023-10-12T00:56:31.005843+0000 2023-10-12T00:56:37.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.811+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000021]: success 2023-10-12T00:56:37.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.811+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:1 marker=00000000001.50.6 timestamp=2023-10-12T00:56:30.907505+0000 2023-10-12T00:56:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-12T00:56:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000028 2023-10-12T00:56:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:30.926692+0000 2023-10-12T00:56:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-12T00:56:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2023-10-12T00:56:37.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-12T00:56:37.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:37.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.812+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000028 2023-10-12T00:56:37.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:37.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000001 2023-10-12T00:56:37.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:30.543068+0000 2023-10-12T00:56:37.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.103.6 op=0 op_state=1 2023-10-12T00:56:37.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:37.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.813+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000002.103.6 2023-10-12T00:56:37.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.818+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000001]: success 2023-10-12T00:56:37.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.818+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:1 marker=00000000002.103.6 timestamp=2023-10-12T00:56:30.543068+0000 2023-10-12T00:56:37.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-12T00:56:37.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000030 2023-10-12T00:56:37.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:30.621759+0000 2023-10-12T00:56:37.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-10-12T00:56:37.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000030[0] 2023-10-12T00:56:37.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.849+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000001.66.6 2023-10-12T00:56:37.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-12T00:56:37.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000024 2023-10-12T00:56:37.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:30.441053+0000 2023-10-12T00:56:37.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-10-12T00:56:37.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000024[0] 2023-10-12T00:56:37.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.850+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-12T00:56:37.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.854+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000024]: success 2023-10-12T00:56:37.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.854+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:1 marker=00000000001.42.6 timestamp=2023-10-12T00:56:30.441053+0000 2023-10-12T00:56:37.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.855+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: a sync operation returned error 2023-10-12T00:56:37.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.855+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: backing out with sync_status=-5 2023-10-12T00:56:37.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.855+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:37.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.855+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:37.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.855+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.856+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:37.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.856+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.895+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000030]: success 2023-10-12T00:56:37.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.895+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:1 marker=00000000001.66.6 timestamp=2023-10-12T00:56:30.621759+0000 2023-10-12T00:56:37.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-12T00:56:37.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000028 2023-10-12T00:56:37.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:30.926692+0000 2023-10-12T00:56:37.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-12T00:56:37.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000028[0] 2023-10-12T00:56:37.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.899+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-12T00:56:37.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.943+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000028]: success 2023-10-12T00:56:37.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.943+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4:1 marker=00000000001.61.6 timestamp=2023-10-12T00:56:30.926692+0000 2023-10-12T00:56:37.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.948+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.948+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:37.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:37.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:37.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.949+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:37.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:37.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:37.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.950+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000021 2023-10-12T00:56:37.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:31.608420+0000 2023-10-12T00:56:37.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-12T00:56:37.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:37.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.951+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-12T00:56:37.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:37.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:37.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:37.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000016 2023-10-12T00:56:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:31.948307+0000 2023-10-12T00:56:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-12T00:56:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000004 2023-10-12T00:56:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:31.844017+0000 2023-10-12T00:56:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-12T00:56:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000016[0] 2023-10-12T00:56:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-12T00:56:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000004[0] 2023-10-12T00:56:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-12T00:56:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.952+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-12T00:56:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000025 2023-10-12T00:56:37.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:31.697844+0000 2023-10-12T00:56:37.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-10-12T00:56:37.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000025[0] 2023-10-12T00:56:37.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.953+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000001.69.6 2023-10-12T00:56:37.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket shard: 0 2023-10-12T00:56:37.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.954+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-12T00:56:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9/obj00000000000000000003 2023-10-12T00:56:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: entry->timestamp=2023-10-12T00:56:31.352356+0000 2023-10-12T00:56:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-12T00:56:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000009 2023-10-12T00:56:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:32.007963+0000 2023-10-12T00:56:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-12T00:56:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000003[0] 2023-10-12T00:56:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-12T00:56:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000009[0] 2023-10-12T00:56:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-12T00:56:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000004]: success 2023-10-12T00:56:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.955+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4:2 marker=00000000002.52.6 timestamp=2023-10-12T00:56:31.844017+0000 2023-10-12T00:56:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.956+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000021]: success 2023-10-12T00:56:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.956+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:2 marker=00000000002.38.6 timestamp=2023-10-12T00:56:31.608420+0000 2023-10-12T00:56:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-12T00:56:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000015 2023-10-12T00:56:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:31.588242+0000 2023-10-12T00:56:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-12T00:56:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000015[0] 2023-10-12T00:56:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.957+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-12T00:56:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000002.115.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-12T00:56:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000012 2023-10-12T00:56:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:31.672693+0000 2023-10-12T00:56:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.115.6 op=0 op_state=1 2023-10-12T00:56:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000012[0] 2023-10-12T00:56:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.958+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000002.115.6 2023-10-12T00:56:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000016]: success 2023-10-12T00:56:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:2 marker=00000000002.50.6 timestamp=2023-10-12T00:56:31.948307+0000 2023-10-12T00:56:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-12T00:56:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000006 2023-10-12T00:56:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:31.418996+0000 2023-10-12T00:56:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-12T00:56:37.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-12T00:56:37.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16/obj00000000000000000011 2023-10-12T00:56:37.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: entry->timestamp=2023-10-12T00:56:31.928329+0000 2023-10-12T00:56:37.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.25.6 op=0 op_state=1 2023-10-12T00:56:37.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-12T00:56:37.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000007 2023-10-12T00:56:37.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:31.786563+0000 2023-10-12T00:56:37.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-12T00:56:37.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.959+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000006[0] 2023-10-12T00:56:37.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.960+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-12T00:56:37.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.960+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000011[0] 2023-10-12T00:56:37.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.960+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position=00000000002.25.6 2023-10-12T00:56:37.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.960+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000007[0] 2023-10-12T00:56:37.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.960+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-12T00:56:37.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.964+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000007]: success 2023-10-12T00:56:37.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.964+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18:2 marker=00000000002.49.6 timestamp=2023-10-12T00:56:31.786563+0000 2023-10-12T00:56:37.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.967+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:entry[obj00000000000000000003]: success 2023-10-12T00:56:37.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.967+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9:2 marker=00000000002.48.6 timestamp=2023-10-12T00:56:31.352356+0000 2023-10-12T00:56:37.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.967+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000012]: success 2023-10-12T00:56:37.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.968+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:2 marker=00000000002.115.6 timestamp=2023-10-12T00:56:31.672693+0000 2023-10-12T00:56:37.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.971+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000011]: success 2023-10-12T00:56:37.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.971+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16:2 marker=00000000002.25.6 timestamp=2023-10-12T00:56:31.928329+0000 2023-10-12T00:56:37.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-12T00:56:37.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000010 2023-10-12T00:56:37.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:31.439352+0000 2023-10-12T00:56:37.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-12T00:56:37.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000010[0] 2023-10-12T00:56:37.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.991+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-12T00:56:37.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.993+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000010]: success 2023-10-12T00:56:37.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.993+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:2 marker=00000000002.50.6 timestamp=2023-10-12T00:56:31.439352+0000 2023-10-12T00:56:37.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000002.113.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-12T00:56:37.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000019 2023-10-12T00:56:37.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:31.504785+0000 2023-10-12T00:56:37.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.113.6 op=0 op_state=1 2023-10-12T00:56:37.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000019[0] 2023-10-12T00:56:37.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.994+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000002.113.6 2023-10-12T00:56:37.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.995+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000019]: success 2023-10-12T00:56:37.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.995+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8:2 marker=00000000002.113.6 timestamp=2023-10-12T00:56:31.504785+0000 2023-10-12T00:56:37.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:37.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000005 2023-10-12T00:56:37.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:31.330095+0000 2023-10-12T00:56:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-12T00:56:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000005[0] 2023-10-12T00:56:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-12T00:56:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-12T00:56:38.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000000 2023-10-12T00:56:38.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:31.524858+0000 2023-10-12T00:56:38.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-10-12T00:56:38.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000000[0] 2023-10-12T00:56:38.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.997+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000002.83.6 2023-10-12T00:56:38.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:38.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17/obj00000000000000000001 2023-10-12T00:56:38.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: entry->timestamp=2023-10-12T00:56:31.765710+0000 2023-10-12T00:56:38.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-12T00:56:38.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.998+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-12T00:56:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000002 2023-10-12T00:56:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:31.864606+0000 2023-10-12T00:56:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-10-12T00:56:38.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000002[0] 2023-10-12T00:56:38.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000002.26.6 2023-10-12T00:56:38.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000001.23.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-12T00:56:38.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000014 2023-10-12T00:56:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:31.309171+0000 2023-10-12T00:56:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.23.6 op=0 op_state=1 2023-10-12T00:56:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000014[0] 2023-10-12T00:56:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000001.23.6 2023-10-12T00:56:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000000]: success 2023-10-12T00:56:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:37.999+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:2 marker=00000000002.83.6 timestamp=2023-10-12T00:56:31.524858+0000 2023-10-12T00:56:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.002+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: success 2023-10-12T00:56:38.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.002+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17:2 marker=00000000002.30.6 timestamp=2023-10-12T00:56:31.765710+0000 2023-10-12T00:56:38.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.005+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000015]: success 2023-10-12T00:56:38.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.005+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:2 marker=00000000002.67.6 timestamp=2023-10-12T00:56:31.588242+0000 2023-10-12T00:56:38.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.005+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000025]: success 2023-10-12T00:56:38.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.005+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:2 marker=00000000001.69.6 timestamp=2023-10-12T00:56:31.697844+0000 2023-10-12T00:56:38.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.007+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000002]: success 2023-10-12T00:56:38.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.007+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:2 marker=00000000002.26.6 timestamp=2023-10-12T00:56:31.864606+0000 2023-10-12T00:56:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.008+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000009]: success 2023-10-12T00:56:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.008+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:2 marker=00000000001.61.6 timestamp=2023-10-12T00:56:32.007963+0000 2023-10-12T00:56:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.011+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000006]: success 2023-10-12T00:56:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.011+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:2 marker=00000000002.65.6 timestamp=2023-10-12T00:56:31.418996+0000 2023-10-12T00:56:38.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.044+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000005]: success 2023-10-12T00:56:38.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.044+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:2 marker=00000000002.47.6 timestamp=2023-10-12T00:56:31.330095+0000 2023-10-12T00:56:38.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.044+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000014]: success 2023-10-12T00:56:38.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.044+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:2 marker=00000000001.23.6 timestamp=2023-10-12T00:56:31.309171+0000 2023-10-12T00:56:38.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:38.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:38.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:38.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.052+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:38.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.053+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:38.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:38.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:38.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:38.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.054+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:38.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.055+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:38.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:38.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:38.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.056+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:38.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:38.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:38.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:38.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket shard: 0 2023-10-12T00:56:38.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket shard: 0 2023-10-12T00:56:38.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.057+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000002 2023-10-12T00:56:38.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:32.386045+0000 2023-10-12T00:56:38.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-12T00:56:38.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-12T00:56:38.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000040 2023-10-12T00:56:38.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:33.007365+0000 2023-10-12T00:56:38.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-12T00:56:38.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000002[0] 2023-10-12T00:56:38.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-12T00:56:38.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000040[0] 2023-10-12T00:56:38.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-12T00:56:38.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-12T00:56:38.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000006 2023-10-12T00:56:38.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:32.556007+0000 2023-10-12T00:56:38.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-12T00:56:38.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000006[0] 2023-10-12T00:56:38.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-12T00:56:38.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000002.121.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-12T00:56:38.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000013 2023-10-12T00:56:38.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:32.639972+0000 2023-10-12T00:56:38.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.121.6 op=0 op_state=1 2023-10-12T00:56:38.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.058+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket shard: 0 2023-10-12T00:56:38.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000013[0] 2023-10-12T00:56:38.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000002.121.6 2023-10-12T00:56:38.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket shard: 0 2023-10-12T00:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.059+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19/obj00000000000000000021 2023-10-12T00:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: entry->timestamp=2023-10-12T00:56:33.083808+0000 2023-10-12T00:56:38.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-12T00:56:38.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:38.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-12T00:56:38.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:38.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17/obj00000000000000000001 2023-10-12T00:56:38.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: entry->timestamp=2023-10-12T00:56:32.534201+0000 2023-10-12T00:56:38.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-12T00:56:38.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:38.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.061+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-12T00:56:38.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-12T00:56:38.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000000 2023-10-12T00:56:38.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:32.930834+0000 2023-10-12T00:56:38.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-12T00:56:38.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000000[0] 2023-10-12T00:56:38.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.062+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-12T00:56:38.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000002]: success 2023-10-12T00:56:38.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:3 marker=00000000002.39.6 timestamp=2023-10-12T00:56:32.386045+0000 2023-10-12T00:56:38.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000006]: success 2023-10-12T00:56:38.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:3 marker=00000000002.69.6 timestamp=2023-10-12T00:56:32.556007+0000 2023-10-12T00:56:38.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000013]: success 2023-10-12T00:56:38.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.064+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:3 marker=00000000002.121.6 timestamp=2023-10-12T00:56:32.639972+0000 2023-10-12T00:56:38.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-12T00:56:38.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22/obj00000000000000000008 2023-10-12T00:56:38.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: entry->timestamp=2023-10-12T00:56:32.950832+0000 2023-10-12T00:56:38.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-12T00:56:38.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000008[0] 2023-10-12T00:56:38.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.065+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-12T00:56:38.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-10-12T00:56:38.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000055 2023-10-12T00:56:38.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: entry->timestamp=2023-10-12T00:56:32.449698+0000 2023-10-12T00:56:38.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-10-12T00:56:38.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000055[0] 2023-10-12T00:56:38.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-12T00:56:38.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-12T00:56:38.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000003 2023-10-12T00:56:38.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:32.799510+0000 2023-10-12T00:56:38.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-12T00:56:38.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000003[0] 2023-10-12T00:56:38.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-12T00:56:38.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000001]: success 2023-10-12T00:56:38.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17:3 marker=00000000002.36.6 timestamp=2023-10-12T00:56:32.534201+0000 2023-10-12T00:56:38.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000021]: success 2023-10-12T00:56:38.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.066+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19:3 marker=00000000002.58.6 timestamp=2023-10-12T00:56:33.083808+0000 2023-10-12T00:56:38.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.068+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000000]: success 2023-10-12T00:56:38.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.068+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18:3 marker=00000000002.55.6 timestamp=2023-10-12T00:56:32.930834+0000 2023-10-12T00:56:38.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.068+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000040]: success 2023-10-12T00:56:38.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.068+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:3 marker=00000000001.49.6 timestamp=2023-10-12T00:56:33.007365+0000 2023-10-12T00:56:38.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.069+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000003]: success 2023-10-12T00:56:38.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.069+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20:3 marker=00000000002.79.6 timestamp=2023-10-12T00:56:32.799510+0000 2023-10-12T00:56:38.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-12T00:56:38.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000026 2023-10-12T00:56:38.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:32.471921+0000 2023-10-12T00:56:38.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-12T00:56:38.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000026[0] 2023-10-12T00:56:38.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-12T00:56:38.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-10-12T00:56:38.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000058 2023-10-12T00:56:38.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:32.301020+0000 2023-10-12T00:56:38.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-12T00:56:38.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-12T00:56:38.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000014 2023-10-12T00:56:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:32.719683+0000 2023-10-12T00:56:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-10-12T00:56:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-10-12T00:56:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000046 2023-10-12T00:56:38.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:32.700534+0000 2023-10-12T00:56:38.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-10-12T00:56:38.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-12T00:56:38.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000011 2023-10-12T00:56:38.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:32.365125+0000 2023-10-12T00:56:38.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-10-12T00:56:38.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000058[0] 2023-10-12T00:56:38.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-12T00:56:38.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000014[0] 2023-10-12T00:56:38.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000002.77.6 2023-10-12T00:56:38.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000046[0] 2023-10-12T00:56:38.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000001.46.6 2023-10-12T00:56:38.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000011[0] 2023-10-12T00:56:38.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.095+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-12T00:56:38.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000001.30.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-12T00:56:38.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000050 2023-10-12T00:56:38.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:33.025597+0000 2023-10-12T00:56:38.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.30.6 op=0 op_state=1 2023-10-12T00:56:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000050[0] 2023-10-12T00:56:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.096+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000001.30.6 2023-10-12T00:56:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-12T00:56:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000039 2023-10-12T00:56:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:32.854623+0000 2023-10-12T00:56:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-10-12T00:56:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000039[0] 2023-10-12T00:56:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000001.32.6 2023-10-12T00:56:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000026]: success 2023-10-12T00:56:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:3 marker=00000000002.47.6 timestamp=2023-10-12T00:56:32.471921+0000 2023-10-12T00:56:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000014]: success 2023-10-12T00:56:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.098+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4:3 marker=00000000002.77.6 timestamp=2023-10-12T00:56:32.719683+0000 2023-10-12T00:56:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.099+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: success 2023-10-12T00:56:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.099+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:3 marker=00000000002.75.6 timestamp=2023-10-12T00:56:32.365125+0000 2023-10-12T00:56:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-12T00:56:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000045 2023-10-12T00:56:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:32.619773+0000 2023-10-12T00:56:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-12T00:56:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000045[0] 2023-10-12T00:56:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-12T00:56:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000009 2023-10-12T00:56:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:32.278960+0000 2023-10-12T00:56:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-12T00:56:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-12T00:56:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000012 2023-10-12T00:56:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:32.779556+0000 2023-10-12T00:56:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-12T00:56:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-12T00:56:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000032 2023-10-12T00:56:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: entry->timestamp=2023-10-12T00:56:32.872802+0000 2023-10-12T00:56:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-10-12T00:56:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000009[0] 2023-10-12T00:56:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-12T00:56:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000012[0] 2023-10-12T00:56:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-12T00:56:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000032[0] 2023-10-12T00:56:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.100+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position=00000000001.52.6 2023-10-12T00:56:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000009]: success 2023-10-12T00:56:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:3 marker=00000000002.37.6 timestamp=2023-10-12T00:56:32.278960+0000 2023-10-12T00:56:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000050]: success 2023-10-12T00:56:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:3 marker=00000000001.30.6 timestamp=2023-10-12T00:56:33.025597+0000 2023-10-12T00:56:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000012]: success 2023-10-12T00:56:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.104+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:3 marker=00000000002.42.6 timestamp=2023-10-12T00:56:32.779556+0000 2023-10-12T00:56:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.106+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000046]: success 2023-10-12T00:56:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.106+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:3 marker=00000000001.46.6 timestamp=2023-10-12T00:56:32.700534+0000 2023-10-12T00:56:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.112+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000008]: success 2023-10-12T00:56:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.112+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22:3 marker=00000000002.57.6 timestamp=2023-10-12T00:56:32.950832+0000 2023-10-12T00:56:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.114+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000055]: success 2023-10-12T00:56:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.114+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21:3 marker=00000000001.67.6 timestamp=2023-10-12T00:56:32.449698+0000 2023-10-12T00:56:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.143+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000058]: success 2023-10-12T00:56:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.143+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:3 marker=00000000001.37.6 timestamp=2023-10-12T00:56:32.301020+0000 2023-10-12T00:56:38.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000032]: success 2023-10-12T00:56:38.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6:3 marker=00000000001.52.6 timestamp=2023-10-12T00:56:32.872802+0000 2023-10-12T00:56:38.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000045]: success 2023-10-12T00:56:38.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:3 marker=00000000001.58.6 timestamp=2023-10-12T00:56:32.619773+0000 2023-10-12T00:56:38.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000039]: success 2023-10-12T00:56:38.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.149+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8:3 marker=00000000001.32.6 timestamp=2023-10-12T00:56:32.854623+0000 2023-10-12T00:56:38.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:38.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:38.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.174+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:38.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.175+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:38.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:38.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:38.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:38.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.176+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.177+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.177+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.177+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.177+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.177+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.178+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket shard: 0 2023-10-12T00:56:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000031 2023-10-12T00:56:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:34.398094+0000 2023-10-12T00:56:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-12T00:56:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000031[0] 2023-10-12T00:56:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.179+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-12T00:56:38.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket shard: 0 2023-10-12T00:56:38.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:38.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket shard: 0 2023-10-12T00:56:38.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket shard: 0 2023-10-12T00:56:38.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:38.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket shard: 0 2023-10-12T00:56:38.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket shard: 0 2023-10-12T00:56:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.180+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.181+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.181+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.181+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket shard: 0 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.181+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.181+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket shard: 0 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000037 2023-10-12T00:56:38.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:33.607083+0000 2023-10-12T00:56:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-12T00:56:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: : injecting data sync error on key=obj00000000000000000037 2023-10-12T00:56:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-12T00:56:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-12T00:56:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17/obj00000000000000000012 2023-10-12T00:56:38.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: entry->timestamp=2023-10-12T00:56:34.209713+0000 2023-10-12T00:56:38.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-12T00:56:38.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-12T00:56:38.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000022 2023-10-12T00:56:38.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:33.852643+0000 2023-10-12T00:56:38.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-12T00:56:38.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-12T00:56:38.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000010 2023-10-12T00:56:38.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:33.524240+0000 2023-10-12T00:56:38.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-12T00:56:38.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000037 2023-10-12T00:56:38.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000012[0] 2023-10-12T00:56:38.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-12T00:56:38.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:38.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16/obj00000000000000000005 2023-10-12T00:56:38.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: entry->timestamp=2023-10-12T00:56:33.462109+0000 2023-10-12T00:56:38.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-12T00:56:38.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000022[0] 2023-10-12T00:56:38.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-12T00:56:38.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-12T00:56:38.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-12T00:56:38.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000005[0] 2023-10-12T00:56:38.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-12T00:56:38.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.182+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000010 2023-10-12T00:56:38.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.183+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket shard: 0 2023-10-12T00:56:38.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.183+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-12T00:56:38.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000030 2023-10-12T00:56:38.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:33.932299+0000 2023-10-12T00:56:38.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-12T00:56:38.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2023-10-12T00:56:38.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-12T00:56:38.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:38.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000001 2023-10-12T00:56:38.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: entry->timestamp=2023-10-12T00:56:33.383453+0000 2023-10-12T00:56:38.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-12T00:56:38.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000030 2023-10-12T00:56:38.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:38.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.186+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-12T00:56:38.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000012]: success 2023-10-12T00:56:38.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17:4 marker=00000000002.42.6 timestamp=2023-10-12T00:56:34.209713+0000 2023-10-12T00:56:38.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-10-12T00:56:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19/obj00000000000000000042 2023-10-12T00:56:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: entry->timestamp=2023-10-12T00:56:34.184135+0000 2023-10-12T00:56:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-10-12T00:56:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28/obj00000000000000000021 2023-10-12T00:56:38.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: entry->timestamp=2023-10-12T00:56:34.036451+0000 2023-10-12T00:56:38.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-12T00:56:38.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000042[0] 2023-10-12T00:56:38.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position=00000000001.55.6 2023-10-12T00:56:38.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: a sync operation returned error 2023-10-12T00:56:38.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:38.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-12T00:56:38.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.187+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: backing out with sync_status=-5 2023-10-12T00:56:38.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: a sync operation returned error 2023-10-12T00:56:38.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: backing out with sync_status=-5 2023-10-12T00:56:38.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.188+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:38.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24/obj00000000000000000016 2023-10-12T00:56:38.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: entry->timestamp=2023-10-12T00:56:34.288054+0000 2023-10-12T00:56:38.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-12T00:56:38.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000016[0] 2023-10-12T00:56:38.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.189+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-12T00:56:38.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.191+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000005]: success 2023-10-12T00:56:38.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.191+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16:4 marker=00000000002.41.6 timestamp=2023-10-12T00:56:33.462109+0000 2023-10-12T00:56:38.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.191+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000001]: success 2023-10-12T00:56:38.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.191+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21:4 marker=00000000002.61.6 timestamp=2023-10-12T00:56:33.383453+0000 2023-10-12T00:56:38.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000016]: success 2023-10-12T00:56:38.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24:4 marker=00000000002.69.6 timestamp=2023-10-12T00:56:34.288054+0000 2023-10-12T00:56:38.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:38.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.193+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000031]: success 2023-10-12T00:56:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:4 marker=00000000001.39.6 timestamp=2023-10-12T00:56:34.398094+0000 2023-10-12T00:56:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000021]: success 2023-10-12T00:56:38.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.194+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28:4 marker=00000000002.39.6 timestamp=2023-10-12T00:56:34.036451+0000 2023-10-12T00:56:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.196+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000022]: success 2023-10-12T00:56:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.196+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:4 marker=00000000002.62.6 timestamp=2023-10-12T00:56:33.852643+0000 2023-10-12T00:56:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-10-12T00:56:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000059 2023-10-12T00:56:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:33.954755+0000 2023-10-12T00:56:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-10-12T00:56:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000059[0] 2023-10-12T00:56:38.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000001.54.6 2023-10-12T00:56:38.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-12T00:56:38.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000006 2023-10-12T00:56:38.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:33.625424+0000 2023-10-12T00:56:38.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-12T00:56:38.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:38.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000004 2023-10-12T00:56:38.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:33.706100+0000 2023-10-12T00:56:38.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-12T00:56:38.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000006[0] 2023-10-12T00:56:38.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-12T00:56:38.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2023-10-12T00:56:38.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-12T00:56:38.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.216+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000004 2023-10-12T00:56:38.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-12T00:56:38.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000011 2023-10-12T00:56:38.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:33.362556+0000 2023-10-12T00:56:38.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-12T00:56:38.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000011[0] 2023-10-12T00:56:38.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.217+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-12T00:56:38.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-10-12T00:56:38.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000049 2023-10-12T00:56:38.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:34.268790+0000 2023-10-12T00:56:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-10-12T00:56:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000049[0] 2023-10-12T00:56:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.218+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-12T00:56:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-12T00:56:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000019 2023-10-12T00:56:38.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: entry->timestamp=2023-10-12T00:56:33.686077+0000 2023-10-12T00:56:38.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.90.6 op=0 op_state=1 2023-10-12T00:56:38.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000019[0] 2023-10-12T00:56:38.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position=00000000002.90.6 2023-10-12T00:56:38.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000006]: success 2023-10-12T00:56:38.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:4 marker=00000000002.60.6 timestamp=2023-10-12T00:56:33.625424+0000 2023-10-12T00:56:38.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000011]: success 2023-10-12T00:56:38.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.219+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4:4 marker=00000000002.47.6 timestamp=2023-10-12T00:56:33.362556+0000 2023-10-12T00:56:38.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.220+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000002 2023-10-12T00:56:38.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:34.375732+0000 2023-10-12T00:56:38.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.25.6 op=0 op_state=1 2023-10-12T00:56:38.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-10-12T00:56:38.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000002.25.6 2023-10-12T00:56:38.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-12T00:56:38.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000020 2023-10-12T00:56:38.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:33.545126+0000 2023-10-12T00:56:38.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-10-12T00:56:38.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-12T00:56:38.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000015 2023-10-12T00:56:38.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:34.124025+0000 2023-10-12T00:56:38.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-10-12T00:56:38.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000002 2023-10-12T00:56:38.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000020[0] 2023-10-12T00:56:38.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000002.88.6 2023-10-12T00:56:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000015[0] 2023-10-12T00:56:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.221+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000002.76.6 2023-10-12T00:56:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-12T00:56:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000003 2023-10-12T00:56:38.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:34.015979+0000 2023-10-12T00:56:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.25.6 op=0 op_state=1 2023-10-12T00:56:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000003[0] 2023-10-12T00:56:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.25.6 2023-10-12T00:56:38.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000019]: success 2023-10-12T00:56:38.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.222+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6:4 marker=00000000002.90.6 timestamp=2023-10-12T00:56:33.686077+0000 2023-10-12T00:56:38.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.223+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000020]: success 2023-10-12T00:56:38.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:4 marker=00000000002.88.6 timestamp=2023-10-12T00:56:33.545126+0000 2023-10-12T00:56:38.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-12T00:56:38.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25/obj00000000000000000000 2023-10-12T00:56:38.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: entry->timestamp=2023-10-12T00:56:33.772208+0000 2023-10-12T00:56:38.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-12T00:56:38.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000000[0] 2023-10-12T00:56:38.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-12T00:56:38.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000003]: success 2023-10-12T00:56:38.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18:4 marker=00000000002.25.6 timestamp=2023-10-12T00:56:34.015979+0000 2023-10-12T00:56:38.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000015]: success 2023-10-12T00:56:38.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.224+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:4 marker=00000000002.76.6 timestamp=2023-10-12T00:56:34.124025+0000 2023-10-12T00:56:38.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-12T00:56:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000013 2023-10-12T00:56:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:33.794426+0000 2023-10-12T00:56:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-10-12T00:56:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000013[0] 2023-10-12T00:56:38.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.225+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-12T00:56:38.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-12T00:56:38.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26/obj00000000000000000014 2023-10-12T00:56:38.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: entry->timestamp=2023-10-12T00:56:34.102925+0000 2023-10-12T00:56:38.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-12T00:56:38.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000014[0] 2023-10-12T00:56:38.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-12T00:56:38.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000000]: success 2023-10-12T00:56:38.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.226+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25:4 marker=00000000002.37.6 timestamp=2023-10-12T00:56:33.772208+0000 2023-10-12T00:56:38.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-12T00:56:38.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23/obj00000000000000000025 2023-10-12T00:56:38.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: entry->timestamp=2023-10-12T00:56:33.874787+0000 2023-10-12T00:56:38.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.24.6 op=0 op_state=1 2023-10-12T00:56:38.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000025[0] 2023-10-12T00:56:38.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position=00000000002.24.6 2023-10-12T00:56:38.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-12T00:56:38.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000027 2023-10-12T00:56:38.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:33.440596+0000 2023-10-12T00:56:38.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-12T00:56:38.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22/obj00000000000000000009 2023-10-12T00:56:38.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: entry->timestamp=2023-10-12T00:56:34.353113+0000 2023-10-12T00:56:38.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-12T00:56:38.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000027[0] 2023-10-12T00:56:38.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-12T00:56:38.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000009[0] 2023-10-12T00:56:38.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.227+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-12T00:56:38.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.228+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000013]: success 2023-10-12T00:56:38.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.228+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:4 marker=00000000002.75.6 timestamp=2023-10-12T00:56:33.794426+0000 2023-10-12T00:56:38.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.229+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000014]: success 2023-10-12T00:56:38.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.229+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26:4 marker=00000000002.59.6 timestamp=2023-10-12T00:56:34.102925+0000 2023-10-12T00:56:38.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.230+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000025]: success 2023-10-12T00:56:38.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.230+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23:4 marker=00000000002.24.6 timestamp=2023-10-12T00:56:33.874787+0000 2023-10-12T00:56:38.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: a sync operation returned error 2023-10-12T00:56:38.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: backing out with sync_status=-5 2023-10-12T00:56:38.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000009]: success 2023-10-12T00:56:38.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.232+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22:4 marker=00000000002.36.6 timestamp=2023-10-12T00:56:34.353113+0000 2023-10-12T00:56:38.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.234+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.234+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.234+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-10-12T00:56:38.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000037 2023-10-12T00:56:38.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:33.607083+0000 2023-10-12T00:56:38.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-12T00:56:38.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000037[0] 2023-10-12T00:56:38.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-12T00:56:38.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-12T00:56:38.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000010 2023-10-12T00:56:38.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:33.524240+0000 2023-10-12T00:56:38.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-12T00:56:38.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000010[0] 2023-10-12T00:56:38.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.236+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-12T00:56:38.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.237+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000042]: success 2023-10-12T00:56:38.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.237+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19:4 marker=00000000001.55.6 timestamp=2023-10-12T00:56:34.184135+0000 2023-10-12T00:56:38.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.239+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000010]: success 2023-10-12T00:56:38.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.239+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20:4 marker=00000000002.63.6 timestamp=2023-10-12T00:56:33.524240+0000 2023-10-12T00:56:38.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.259+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: a sync operation returned error 2023-10-12T00:56:38.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.259+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: backing out with sync_status=-5 2023-10-12T00:56:38.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.259+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.259+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.259+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.260+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.260+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: a sync operation returned error 2023-10-12T00:56:38.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: backing out with sync_status=-5 2023-10-12T00:56:38.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000059]: success 2023-10-12T00:56:38.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.263+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:4 marker=00000000001.54.6 timestamp=2023-10-12T00:56:33.954755+0000 2023-10-12T00:56:38.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.264+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.265+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:38.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.265+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.266+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000049]: success 2023-10-12T00:56:38.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.266+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:4 marker=00000000001.67.6 timestamp=2023-10-12T00:56:34.268790+0000 2023-10-12T00:56:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.274+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000027]: success 2023-10-12T00:56:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.274+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:4 marker=00000000001.58.6 timestamp=2023-10-12T00:56:33.440596+0000 2023-10-12T00:56:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-12T00:56:38.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000030 2023-10-12T00:56:38.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:33.932299+0000 2023-10-12T00:56:38.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-12T00:56:38.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000030[0] 2023-10-12T00:56:38.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.276+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-12T00:56:38.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.278+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000030]: success 2023-10-12T00:56:38.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.278+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27:4 marker=00000000002.45.6 timestamp=2023-10-12T00:56:33.932299+0000 2023-10-12T00:56:38.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.281+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000037]: success 2023-10-12T00:56:38.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.281+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:4 marker=00000000001.44.6 timestamp=2023-10-12T00:56:33.607083+0000 2023-10-12T00:56:38.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:38.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000004 2023-10-12T00:56:38.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:33.706100+0000 2023-10-12T00:56:38.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-12T00:56:38.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000004[0] 2023-10-12T00:56:38.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.302+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-12T00:56:38.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.304+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000004]: success 2023-10-12T00:56:38.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.304+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:4 marker=00000000002.57.6 timestamp=2023-10-12T00:56:33.706100+0000 2023-10-12T00:56:38.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000002 2023-10-12T00:56:38.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:34.375732+0000 2023-10-12T00:56:38.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.25.6 op=0 op_state=1 2023-10-12T00:56:38.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000002[0] 2023-10-12T00:56:38.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.307+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000002.25.6 2023-10-12T00:56:38.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.309+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000002]: success 2023-10-12T00:56:38.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.309+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:4 marker=00000000002.25.6 timestamp=2023-10-12T00:56:34.375732+0000 2023-10-12T00:56:38.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.353+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:38.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:38.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:38.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:38.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:38.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.354+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:38.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.355+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.356+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:38.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:38.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:38.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:38.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-10-12T00:56:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000023 2023-10-12T00:56:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: entry->timestamp=2023-10-12T00:56:35.006581+0000 2023-10-12T00:56:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-12T00:56:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000001.129.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-10-12T00:56:38.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000317 2023-10-12T00:56:38.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:34.752965+0000 2023-10-12T00:56:38.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.129.6 op=0 op_state=1 2023-10-12T00:56:38.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000023[0] 2023-10-12T00:56:38.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-12T00:56:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000004 2023-10-12T00:56:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:35.270000+0000 2023-10-12T00:56:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-12T00:56:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-10-12T00:56:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000467 2023-10-12T00:56:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:35.545947+0000 2023-10-12T00:56:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-12T00:56:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000317[0] 2023-10-12T00:56:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000001.129.6 2023-10-12T00:56:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000004[0] 2023-10-12T00:56:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-12T00:56:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: : injecting data sync error on key=obj00000000000000000467 2023-10-12T00:56:38.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-12T00:56:38.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.357+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000467 2023-10-12T00:56:38.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:38.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket shard: 0 2023-10-12T00:56:38.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:38.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.359+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:38.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.360+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket shard: 0 2023-10-12T00:56:38.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket shard: 0 2023-10-12T00:56:38.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: sync status for source bucket shard: 0 2023-10-12T00:56:38.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket shard: 0 2023-10-12T00:56:38.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket shard: 0 2023-10-12T00:56:38.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: sync status for source bucket shard: 0 2023-10-12T00:56:38.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:38.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket shard: 0 2023-10-12T00:56:38.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket shard: 0 2023-10-12T00:56:38.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket shard: 0 2023-10-12T00:56:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: a sync operation returned error 2023-10-12T00:56:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-12T00:56:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16/obj00000000000000000017 2023-10-12T00:56:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: entry->timestamp=2023-10-12T00:56:34.731045+0000 2023-10-12T00:56:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.96.6 op=0 op_state=1 2023-10-12T00:56:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-10-12T00:56:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17/obj00000000000000000162 2023-10-12T00:56:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: entry->timestamp=2023-10-12T00:56:35.353205+0000 2023-10-12T00:56:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.362+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-10-12T00:56:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000017[0] 2023-10-12T00:56:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position=00000000002.96.6 2023-10-12T00:56:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000162[0] 2023-10-12T00:56:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position=00000000001.47.6 2023-10-12T00:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: backing out with sync_status=-5 2023-10-12T00:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-12T00:56:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000018 2023-10-12T00:56:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:35.149720+0000 2023-10-12T00:56:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-12T00:56:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000018[0] 2023-10-12T00:56:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.363+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-12T00:56:38.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket shard: 0 2023-10-12T00:56:38.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000004]: success 2023-10-12T00:56:38.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:5 marker=00000000002.48.6 timestamp=2023-10-12T00:56:35.270000+0000 2023-10-12T00:56:38.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.364+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-10-12T00:56:38.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30/obj00000000000000000133 2023-10-12T00:56:38.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: entry->timestamp=2023-10-12T00:56:34.984078+0000 2023-10-12T00:56:38.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-10-12T00:56:38.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000133[0] 2023-10-12T00:56:38.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.366+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: listing bilog for incremental sync; position=00000000001.83.6 2023-10-12T00:56:38.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:38.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000016 2023-10-12T00:56:38.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:34.921551+0000 2023-10-12T00:56:38.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-10-12T00:56:38.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2023-10-12T00:56:38.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.81.6 2023-10-12T00:56:38.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.367+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000016 2023-10-12T00:56:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-12T00:56:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000423 2023-10-12T00:56:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:35.068894+0000 2023-10-12T00:56:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.88.6 op=0 op_state=1 2023-10-12T00:56:38.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2023-10-12T00:56:38.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000001.88.6 2023-10-12T00:56:38.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.368+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000423 2023-10-12T00:56:38.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-12T00:56:38.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24/obj00000000000000000011 2023-10-12T00:56:38.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: entry->timestamp=2023-10-12T00:56:34.897897+0000 2023-10-12T00:56:38.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-12T00:56:38.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-12T00:56:38.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000061 2023-10-12T00:56:38.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:34.835363+0000 2023-10-12T00:56:38.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-12T00:56:38.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-10-12T00:56:38.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25/obj00000000000000000283 2023-10-12T00:56:38.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: entry->timestamp=2023-10-12T00:56:35.794968+0000 2023-10-12T00:56:38.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-12T00:56:38.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000011[0] 2023-10-12T00:56:38.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-12T00:56:38.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: : injecting data sync error on key=obj00000000000000000061 2023-10-12T00:56:38.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-12T00:56:38.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000283[0] 2023-10-12T00:56:38.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-12T00:56:38.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:38.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23/obj00000000000000000001 2023-10-12T00:56:38.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: entry->timestamp=2023-10-12T00:56:35.190905+0000 2023-10-12T00:56:38.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-12T00:56:38.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000061 2023-10-12T00:56:38.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:38.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.369+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-12T00:56:38.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000002 2023-10-12T00:56:38.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: entry->timestamp=2023-10-12T00:56:35.688636+0000 2023-10-12T00:56:38.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-12T00:56:38.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-10-12T00:56:38.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-12T00:56:38.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.370+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000002 2023-10-12T00:56:38.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.373+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000317]: success 2023-10-12T00:56:38.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.373+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:5 marker=00000000001.129.6 timestamp=2023-10-12T00:56:34.752965+0000 2023-10-12T00:56:38.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.373+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.373+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000023]: success 2023-10-12T00:56:38.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.373+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6:5 marker=00000000002.38.6 timestamp=2023-10-12T00:56:35.006581+0000 2023-10-12T00:56:38.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.374+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:38.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.374+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.375+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000001]: success 2023-10-12T00:56:38.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.375+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23:5 marker=00000000002.69.6 timestamp=2023-10-12T00:56:35.190905+0000 2023-10-12T00:56:38.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.376+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000011]: success 2023-10-12T00:56:38.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.376+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24:5 marker=00000000002.49.6 timestamp=2023-10-12T00:56:34.897897+0000 2023-10-12T00:56:38.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.377+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000133]: success 2023-10-12T00:56:38.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.377+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30:5 marker=00000000001.83.6 timestamp=2023-10-12T00:56:34.984078+0000 2023-10-12T00:56:38.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-12T00:56:38.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000006 2023-10-12T00:56:38.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:35.249298+0000 2023-10-12T00:56:38.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-12T00:56:38.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000006[0] 2023-10-12T00:56:38.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.396+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-12T00:56:38.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-12T00:56:38.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000019 2023-10-12T00:56:38.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:35.090370+0000 2023-10-12T00:56:38.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-12T00:56:38.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000019[0] 2023-10-12T00:56:38.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.397+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-12T00:56:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000005 2023-10-12T00:56:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:35.773496+0000 2023-10-12T00:56:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-12T00:56:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-10-12T00:56:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-12T00:56:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000005 2023-10-12T00:56:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000006]: success 2023-10-12T00:56:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:5 marker=00000000002.64.6 timestamp=2023-10-12T00:56:35.249298+0000 2023-10-12T00:56:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000019]: success 2023-10-12T00:56:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.398+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:5 marker=00000000002.48.6 timestamp=2023-10-12T00:56:35.090370+0000 2023-10-12T00:56:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-12T00:56:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000008 2023-10-12T00:56:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:35.609039+0000 2023-10-12T00:56:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-12T00:56:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000008[0] 2023-10-12T00:56:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.399+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-12T00:56:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-12T00:56:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000022 2023-10-12T00:56:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:35.630045+0000 2023-10-12T00:56:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-12T00:56:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2023-10-12T00:56:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-12T00:56:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-10-12T00:56:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000271 2023-10-12T00:56:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:35.438872+0000 2023-10-12T00:56:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-10-12T00:56:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000022 2023-10-12T00:56:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000271[0] 2023-10-12T00:56:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.400+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000001.50.6 2023-10-12T00:56:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.401+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000008]: success 2023-10-12T00:56:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.401+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:5 marker=00000000002.42.6 timestamp=2023-10-12T00:56:35.609039+0000 2023-10-12T00:56:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:35.415275+0000 2023-10-12T00:56:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.131.6 op=0 op_state=1 2023-10-12T00:56:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2023-10-12T00:56:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000002.131.6 2023-10-12T00:56:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.402+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000021 2023-10-12T00:56:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:35.330462+0000 2023-10-12T00:56:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-12T00:56:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.403+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-12T00:56:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-12T00:56:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000007 2023-10-12T00:56:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:35.709483+0000 2023-10-12T00:56:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-12T00:56:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000007[0] 2023-10-12T00:56:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.404+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-12T00:56:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-12T00:56:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22/obj00000000000000000012 2023-10-12T00:56:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: entry->timestamp=2023-10-12T00:56:35.500728+0000 2023-10-12T00:56:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-12T00:56:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000012[0] 2023-10-12T00:56:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-12T00:56:38.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000021]: success 2023-10-12T00:56:38.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.405+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:5 marker=00000000002.50.6 timestamp=2023-10-12T00:56:35.330462+0000 2023-10-12T00:56:38.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.406+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000007]: success 2023-10-12T00:56:38.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.406+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:5 marker=00000000002.67.6 timestamp=2023-10-12T00:56:35.709483+0000 2023-10-12T00:56:38.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.406+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:entry[obj00000000000000000012]: success 2023-10-12T00:56:38.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22:5 marker=00000000002.52.6 timestamp=2023-10-12T00:56:35.500728+0000 2023-10-12T00:56:38.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-10-12T00:56:38.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29/obj00000000000000000029 2023-10-12T00:56:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: entry->timestamp=2023-10-12T00:56:35.521540+0000 2023-10-12T00:56:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-12T00:56:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000029[0] 2023-10-12T00:56:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.407+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-12T00:56:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-12T00:56:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19/obj00000000000000000003 2023-10-12T00:56:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: entry->timestamp=2023-10-12T00:56:34.814048+0000 2023-10-12T00:56:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-12T00:56:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000003[0] 2023-10-12T00:56:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.408+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-12T00:56:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-12T00:56:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28/obj00000000000000000050 2023-10-12T00:56:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: entry->timestamp=2023-10-12T00:56:35.169727+0000 2023-10-12T00:56:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-12T00:56:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000050[0] 2023-10-12T00:56:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.409+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-12T00:56:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.410+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000003]: success 2023-10-12T00:56:38.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.410+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19:5 marker=00000000002.65.6 timestamp=2023-10-12T00:56:34.814048+0000 2023-10-12T00:56:38.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.411+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000050]: success 2023-10-12T00:56:38.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.411+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28:5 marker=00000000002.45.6 timestamp=2023-10-12T00:56:35.169727+0000 2023-10-12T00:56:38.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: a sync operation returned error 2023-10-12T00:56:38.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: a sync operation returned error 2023-10-12T00:56:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: backing out with sync_status=-5 2023-10-12T00:56:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: backing out with sync_status=-5 2023-10-12T00:56:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.412+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: a sync operation returned error 2023-10-12T00:56:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: a sync operation returned error 2023-10-12T00:56:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: backing out with sync_status=-5 2023-10-12T00:56:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: backing out with sync_status=-5 2023-10-12T00:56:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000018]: success 2023-10-12T00:56:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.413+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:5 marker=00000000002.51.6 timestamp=2023-10-12T00:56:35.149720+0000 2023-10-12T00:56:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.414+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000017]: success 2023-10-12T00:56:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.414+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16:5 marker=00000000002.96.6 timestamp=2023-10-12T00:56:34.731045+0000 2023-10-12T00:56:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.416+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.416+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.416+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:38.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-10-12T00:56:38.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000467 2023-10-12T00:56:38.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:35.545947+0000 2023-10-12T00:56:38.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-12T00:56:38.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000467[0] 2023-10-12T00:56:38.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-12T00:56:38.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket shard: 0 2023-10-12T00:56:38.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000162]: success 2023-10-12T00:56:38.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.417+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17:5 marker=00000000001.47.6 timestamp=2023-10-12T00:56:35.353205+0000 2023-10-12T00:56:38.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.418+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.418+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.418+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.419+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.419+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000283]: success 2023-10-12T00:56:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.419+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25:5 marker=00000000001.44.6 timestamp=2023-10-12T00:56:35.794968+0000 2023-10-12T00:56:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.440+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: a sync operation returned error 2023-10-12T00:56:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.440+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: backing out with sync_status=-5 2023-10-12T00:56:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.440+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.440+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.441+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.441+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.441+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.442+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: a sync operation returned error 2023-10-12T00:56:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.442+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: backing out with sync_status=-5 2023-10-12T00:56:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.442+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.442+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.443+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.443+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.443+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.444+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: a sync operation returned error 2023-10-12T00:56:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.444+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: backing out with sync_status=-5 2023-10-12T00:56:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.444+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.444+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.444+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.445+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.445+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.446+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000271]: success 2023-10-12T00:56:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.446+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:5 marker=00000000001.50.6 timestamp=2023-10-12T00:56:35.438872+0000 2023-10-12T00:56:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.452+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000029]: success 2023-10-12T00:56:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.452+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29:5 marker=00000000001.49.6 timestamp=2023-10-12T00:56:35.521540+0000 2023-10-12T00:56:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000016 2023-10-12T00:56:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:34.921551+0000 2023-10-12T00:56:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-10-12T00:56:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000016[0] 2023-10-12T00:56:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.460+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.81.6 2023-10-12T00:56:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000002 2023-10-12T00:56:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: entry->timestamp=2023-10-12T00:56:35.688636+0000 2023-10-12T00:56:38.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-12T00:56:38.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000002[0] 2023-10-12T00:56:38.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-12T00:56:38.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-12T00:56:38.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000423 2023-10-12T00:56:38.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:35.068894+0000 2023-10-12T00:56:38.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.88.6 op=0 op_state=1 2023-10-12T00:56:38.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-12T00:56:38.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000061 2023-10-12T00:56:38.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:34.835363+0000 2023-10-12T00:56:38.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-12T00:56:38.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000423[0] 2023-10-12T00:56:38.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000001.88.6 2023-10-12T00:56:38.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000061[0] 2023-10-12T00:56:38.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.461+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-12T00:56:38.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.463+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000016]: success 2023-10-12T00:56:38.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.463+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18:5 marker=00000000002.81.6 timestamp=2023-10-12T00:56:34.921551+0000 2023-10-12T00:56:38.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.464+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000002]: success 2023-10-12T00:56:38.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.464+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21:5 marker=00000000002.68.6 timestamp=2023-10-12T00:56:35.688636+0000 2023-10-12T00:56:38.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.466+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000467]: success 2023-10-12T00:56:38.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.466+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:5 marker=00000000001.44.6 timestamp=2023-10-12T00:56:35.545947+0000 2023-10-12T00:56:38.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.467+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000061]: success 2023-10-12T00:56:38.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.467+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27:5 marker=00000000001.53.6 timestamp=2023-10-12T00:56:34.835363+0000 2023-10-12T00:56:38.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.468+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000423]: success 2023-10-12T00:56:38.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.468+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20:5 marker=00000000001.88.6 timestamp=2023-10-12T00:56:35.068894+0000 2023-10-12T00:56:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000005 2023-10-12T00:56:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:35.773496+0000 2023-10-12T00:56:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-12T00:56:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000005[0] 2023-10-12T00:56:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.484+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-12T00:56:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-12T00:56:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000022 2023-10-12T00:56:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:35.630045+0000 2023-10-12T00:56:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-12T00:56:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000022[0] 2023-10-12T00:56:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.485+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-12T00:56:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.486+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000005]: success 2023-10-12T00:56:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.486+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:5 marker=00000000002.55.6 timestamp=2023-10-12T00:56:35.773496+0000 2023-10-12T00:56:38.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:35.415275+0000 2023-10-12T00:56:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.131.6 op=0 op_state=1 2023-10-12T00:56:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2023-10-12T00:56:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000002.131.6 2023-10-12T00:56:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000022]: success 2023-10-12T00:56:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8:5 marker=00000000002.66.6 timestamp=2023-10-12T00:56:35.630045+0000 2023-10-12T00:56:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.487+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: a sync operation returned error 2023-10-12T00:56:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: backing out with sync_status=-5 2023-10-12T00:56:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.529+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:35.415275+0000 2023-10-12T00:56:38.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.131.6 op=0 op_state=1 2023-10-12T00:56:38.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2023-10-12T00:56:38.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000002.131.6 2023-10-12T00:56:38.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.572+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.614+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: a sync operation returned error 2023-10-12T00:56:38.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.614+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: backing out with sync_status=-5 2023-10-12T00:56:38.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.614+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.614+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.614+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.615+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.615+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000002.131.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000009 2023-10-12T00:56:38.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:35.415275+0000 2023-10-12T00:56:38.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.131.6 op=0 op_state=1 2023-10-12T00:56:38.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000009[0] 2023-10-12T00:56:38.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.657+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000002.131.6 2023-10-12T00:56:38.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.659+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000009]: success 2023-10-12T00:56:38.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.659+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:5 marker=00000000002.131.6 timestamp=2023-10-12T00:56:35.415275+0000 2023-10-12T00:56:38.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.703+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: sync status for source bucket shard: 0 2023-10-12T00:56:38.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: sync status for source bucket shard: 0 2023-10-12T00:56:38.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: sync status for source bucket shard: 0 2023-10-12T00:56:38.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: sync status for source bucket shard: 0 2023-10-12T00:56:38.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: sync status for source bucket shard: 0 2023-10-12T00:56:38.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: sync status for source bucket shard: 0 2023-10-12T00:56:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.704+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.705+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.705+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.705+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: sync status for source bucket shard: 0 2023-10-12T00:56:38.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:38.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-12T00:56:38.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3/obj00000000000000000001 2023-10-12T00:56:38.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: entry->timestamp=2023-10-12T00:56:37.315952+0000 2023-10-12T00:56:38.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-12T00:56:38.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: sync status for source bucket shard: 0 2023-10-12T00:56:38.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: sync status for source bucket shard: 0 2023-10-12T00:56:38.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000001[0] 2023-10-12T00:56:38.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-12T00:56:38.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:9]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.706+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-12T00:56:38.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2/obj00000000000000000011 2023-10-12T00:56:38.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: entry->timestamp=2023-10-12T00:56:37.174733+0000 2023-10-12T00:56:38.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-12T00:56:38.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000011[0] 2023-10-12T00:56:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-12T00:56:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-10-12T00:56:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0/obj00000000000000000060 2023-10-12T00:56:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: entry->timestamp=2023-10-12T00:56:37.005482+0000 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4/obj00000000000000000033 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: entry->timestamp=2023-10-12T00:56:36.459455+0000 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-12T00:56:38.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-10-12T00:56:38.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5/obj00000000000000000065 2023-10-12T00:56:38.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: entry->timestamp=2023-10-12T00:56:36.129863+0000 2023-10-12T00:56:38.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.707+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-10-12T00:56:38.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000060[0] 2023-10-12T00:56:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: listing bilog for incremental sync; position=00000000001.60.6 2023-10-12T00:56:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000033[0] 2023-10-12T00:56:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-12T00:56:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000065[0] 2023-10-12T00:56:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.708+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: listing bilog for incremental sync; position=00000000001.57.6 2023-10-12T00:56:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: sync status for source bucket shard: 0 2023-10-12T00:56:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: sync status for source bucket shard: 0 2023-10-12T00:56:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: sync status for source bucket shard: 0 2023-10-12T00:56:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: sync status for source bucket shard: 0 2023-10-12T00:56:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: sync status for source bucket shard: 0 2023-10-12T00:56:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: sync status for source bucket shard: 0 2023-10-12T00:56:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: sync status for source bucket shard: 0 2023-10-12T00:56:38.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: sync status for source bucket shard: 0 2023-10-12T00:56:38.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: sync status for source bucket shard: 0 2023-10-12T00:56:38.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: sync status for source bucket shard: 0 2023-10-12T00:56:38.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: sync status for source bucket shard: 0 2023-10-12T00:56:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.709+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.710+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: sync status for source bucket shard: 0 2023-10-12T00:56:38.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: sync status for source bucket shard: 0 2023-10-12T00:56:38.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:22]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: sync status for source bucket shard: 0 2023-10-12T00:56:38.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: sync status for source bucket shard: 0 2023-10-12T00:56:38.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.711+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.712+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:entry[obj00000000000000000033]: success 2023-10-12T00:56:38.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.712+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:4:6 marker=00000000002.30.6 timestamp=2023-10-12T00:56:36.459455+0000 2023-10-12T00:56:38.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.712+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:entry[obj00000000000000000001]: success 2023-10-12T00:56:38.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.712+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:3:6 marker=00000000002.50.6 timestamp=2023-10-12T00:56:37.315952+0000 2023-10-12T00:56:38.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: sync status for source bucket shard: 0 2023-10-12T00:56:38.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: sync status for source bucket shard: 0 2023-10-12T00:56:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: sync status for source bucket shard: 0 2023-10-12T00:56:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: sync status for source bucket shard: 0 2023-10-12T00:56:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: sync status for source bucket shard: 0 2023-10-12T00:56:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: sync status for source bucket shard: 0 2023-10-12T00:56:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:31]: sync status for source bucket shard: 0 2023-10-12T00:56:38.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: sync status for source bucket shard: 0 2023-10-12T00:56:38.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: sync status for source bucket shard: 0 2023-10-12T00:56:38.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:31]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:31]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-12T00:56:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11/obj00000000000000000030 2023-10-12T00:56:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: entry->timestamp=2023-10-12T00:56:36.209436+0000 2023-10-12T00:56:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-12T00:56:38.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-10-12T00:56:38.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12/obj00000000000000000062 2023-10-12T00:56:38.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: entry->timestamp=2023-10-12T00:56:36.290658+0000 2023-10-12T00:56:38.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-10-12T00:56:38.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-12T00:56:38.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14/obj00000000000000000040 2023-10-12T00:56:38.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: entry->timestamp=2023-10-12T00:56:36.984268+0000 2023-10-12T00:56:38.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-10-12T00:56:38.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-12T00:56:38.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15/obj00000000000000000024 2023-10-12T00:56:38.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: entry->timestamp=2023-10-12T00:56:36.403096+0000 2023-10-12T00:56:38.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-10-12T00:56:38.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:entry[obj00000000000000000011]: success 2023-10-12T00:56:38.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:2:6 marker=00000000002.64.6 timestamp=2023-10-12T00:56:37.174733+0000 2023-10-12T00:56:38.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000030[0] 2023-10-12T00:56:38.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-12T00:56:38.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000062[0] 2023-10-12T00:56:38.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: listing bilog for incremental sync; position=00000000001.51.6 2023-10-12T00:56:38.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000040[0] 2023-10-12T00:56:38.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: listing bilog for incremental sync; position=00000000002.81.6 2023-10-12T00:56:38.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000024[0] 2023-10-12T00:56:38.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.715+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-12T00:56:38.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-12T00:56:38.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21/obj00000000000000000017 2023-10-12T00:56:38.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: entry->timestamp=2023-10-12T00:56:36.842056+0000 2023-10-12T00:56:38.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-12T00:56:38.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-12T00:56:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20/obj00000000000000000006 2023-10-12T00:56:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: entry->timestamp=2023-10-12T00:56:37.066038+0000 2023-10-12T00:56:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-12T00:56:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-12T00:56:38.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23/obj00000000000000000039 2023-10-12T00:56:38.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: entry->timestamp=2023-10-12T00:56:36.319766+0000 2023-10-12T00:56:38.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.96.6 op=0 op_state=1 2023-10-12T00:56:38.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-12T00:56:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10/obj00000000000000000026 2023-10-12T00:56:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: entry->timestamp=2023-10-12T00:56:36.664293+0000 2023-10-12T00:56:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-12T00:56:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000017[0] 2023-10-12T00:56:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-12T00:56:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000006[0] 2023-10-12T00:56:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-12T00:56:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000039[0] 2023-10-12T00:56:38.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: listing bilog for incremental sync; position=00000000002.96.6 2023-10-12T00:56:38.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-12T00:56:38.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16/obj00000000000000000012 2023-10-12T00:56:38.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: entry->timestamp=2023-10-12T00:56:36.926587+0000 2023-10-12T00:56:38.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-10-12T00:56:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000026[0] 2023-10-12T00:56:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-12T00:56:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-12T00:56:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17/obj00000000000000000002 2023-10-12T00:56:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: entry->timestamp=2023-10-12T00:56:37.236044+0000 2023-10-12T00:56:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-12T00:56:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-10-12T00:56:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000053 2023-10-12T00:56:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:36.905327+0000 2023-10-12T00:56:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.717+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.98.6 op=0 op_state=1 2023-10-12T00:56:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000012[0] 2023-10-12T00:56:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: listing bilog for incremental sync; position=00000000002.74.6 2023-10-12T00:56:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000002[0] 2023-10-12T00:56:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-12T00:56:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: : injecting data sync error on key=obj00000000000000000053 2023-10-12T00:56:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000002.98.6 2023-10-12T00:56:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.719+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000053 2023-10-12T00:56:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: sync status for source bucket shard: 0 2023-10-12T00:56:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.720+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: sync status for source bucket shard: 0 2023-10-12T00:56:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-12T00:56:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24/obj00000000000000000027 2023-10-12T00:56:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: entry->timestamp=2023-10-12T00:56:37.420497+0000 2023-10-12T00:56:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-12T00:56:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-12T00:56:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33/obj00000000000000000019 2023-10-12T00:56:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: entry->timestamp=2023-10-12T00:56:36.733471+0000 2023-10-12T00:56:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-12T00:56:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000027[0] 2023-10-12T00:56:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-12T00:56:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000019[0] 2023-10-12T00:56:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-12T00:56:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-12T00:56:38.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13/obj00000000000000000009 2023-10-12T00:56:38.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: entry->timestamp=2023-10-12T00:56:36.231322+0000 2023-10-12T00:56:38.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-12T00:56:38.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000009[0] 2023-10-12T00:56:38.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.721+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-12T00:56:38.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.722+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:entry[obj00000000000000000040]: success 2023-10-12T00:56:38.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.722+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:14:6 marker=00000000002.81.6 timestamp=2023-10-12T00:56:36.984268+0000 2023-10-12T00:56:38.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.724+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:entry[obj00000000000000000030]: success 2023-10-12T00:56:38.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:11:6 marker=00000000002.61.6 timestamp=2023-10-12T00:56:36.209436+0000 2023-10-12T00:56:38.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.724+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:entry[obj00000000000000000024]: success 2023-10-12T00:56:38.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.724+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:15:6 marker=00000000002.75.6 timestamp=2023-10-12T00:56:36.403096+0000 2023-10-12T00:56:38.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-12T00:56:38.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36/obj00000000000000000008 2023-10-12T00:56:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: entry->timestamp=2023-10-12T00:56:37.088238+0000 2023-10-12T00:56:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-12T00:56:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000008[0] 2023-10-12T00:56:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-12T00:56:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:entry[obj00000000000000000065]: success 2023-10-12T00:56:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.725+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:5:6 marker=00000000001.57.6 timestamp=2023-10-12T00:56:36.129863+0000 2023-10-12T00:56:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:entry[obj00000000000000000060]: success 2023-10-12T00:56:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:0:6 marker=00000000001.60.6 timestamp=2023-10-12T00:56:37.005482+0000 2023-10-12T00:56:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:entry[obj00000000000000000017]: success 2023-10-12T00:56:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:21:6 marker=00000000002.46.6 timestamp=2023-10-12T00:56:36.842056+0000 2023-10-12T00:56:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:entry[obj00000000000000000039]: success 2023-10-12T00:56:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.726+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:23:6 marker=00000000002.96.6 timestamp=2023-10-12T00:56:36.319766+0000 2023-10-12T00:56:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:entry[obj00000000000000000006]: success 2023-10-12T00:56:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:20:6 marker=00000000002.30.6 timestamp=2023-10-12T00:56:37.066038+0000 2023-10-12T00:56:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:entry[obj00000000000000000027]: success 2023-10-12T00:56:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:24:6 marker=00000000002.67.6 timestamp=2023-10-12T00:56:37.420497+0000 2023-10-12T00:56:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: a sync operation returned error 2023-10-12T00:56:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: backing out with sync_status=-5 2023-10-12T00:56:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.728+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-12T00:56:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32/obj00000000000000000014 2023-10-12T00:56:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: entry->timestamp=2023-10-12T00:56:36.149479+0000 2023-10-12T00:56:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-12T00:56:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000014[0] 2023-10-12T00:56:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-12T00:56:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:entry[obj00000000000000000026]: success 2023-10-12T00:56:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:10:6 marker=00000000002.37.6 timestamp=2023-10-12T00:56:36.664293+0000 2023-10-12T00:56:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:entry[obj00000000000000000002]: success 2023-10-12T00:56:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.729+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:17:6 marker=00000000002.62.6 timestamp=2023-10-12T00:56:37.236044+0000 2023-10-12T00:56:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.730+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:entry[obj00000000000000000008]: success 2023-10-12T00:56:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.730+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:36:6 marker=00000000002.70.6 timestamp=2023-10-12T00:56:37.088238+0000 2023-10-12T00:56:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.731+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:entry[obj00000000000000000012]: success 2023-10-12T00:56:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.731+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:16:6 marker=00000000002.74.6 timestamp=2023-10-12T00:56:36.926587+0000 2023-10-12T00:56:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.731+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.731+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:entry[obj00000000000000000019]: success 2023-10-12T00:56:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.731+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:33:6 marker=00000000002.60.6 timestamp=2023-10-12T00:56:36.733471+0000 2023-10-12T00:56:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.732+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:entry[obj00000000000000000009]: success 2023-10-12T00:56:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.732+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:13:6 marker=00000000002.65.6 timestamp=2023-10-12T00:56:36.231322+0000 2023-10-12T00:56:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.732+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:entry[obj00000000000000000014]: success 2023-10-12T00:56:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.732+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:32:6 marker=00000000002.59.6 timestamp=2023-10-12T00:56:36.149479+0000 2023-10-12T00:56:38.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.733+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: sync status for source bucket shard: 0 2023-10-12T00:56:38.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.733+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-12T00:56:38.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1/obj00000000000000000000 2023-10-12T00:56:38.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: entry->timestamp=2023-10-12T00:56:36.381044+0000 2023-10-12T00:56:38.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-12T00:56:38.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000000[0] 2023-10-12T00:56:38.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.747+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-12T00:56:38.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-12T00:56:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7/obj00000000000000000016 2023-10-12T00:56:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: entry->timestamp=2023-10-12T00:56:36.480592+0000 2023-10-12T00:56:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-12T00:56:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000016[0] 2023-10-12T00:56:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-12T00:56:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:entry[obj00000000000000000000]: success 2023-10-12T00:56:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.748+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:1:6 marker=00000000002.43.6 timestamp=2023-10-12T00:56:36.381044+0000 2023-10-12T00:56:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000005 2023-10-12T00:56:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: entry->timestamp=2023-10-12T00:56:36.638932+0000 2023-10-12T00:56:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-12T00:56:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-10-12T00:56:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-12T00:56:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-10-12T00:56:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8/obj00000000000000000048 2023-10-12T00:56:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: entry->timestamp=2023-10-12T00:56:37.150450+0000 2023-10-12T00:56:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-12T00:56:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T00:56:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 0 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: ERROR: failed to sync object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000005 2023-10-12T00:56:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000048[0] 2023-10-12T00:56:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-12T00:56:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:entry[obj00000000000000000016]: success 2023-10-12T00:56:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.749+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:7:6 marker=00000000002.60.6 timestamp=2023-10-12T00:56:36.480592+0000 2023-10-12T00:56:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-12T00:56:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19/obj00000000000000000045 2023-10-12T00:56:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: entry->timestamp=2023-10-12T00:56:36.758568+0000 2023-10-12T00:56:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-12T00:56:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000045[0] 2023-10-12T00:56:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.756+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-12T00:56:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.757+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:entry[obj00000000000000000045]: success 2023-10-12T00:56:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.757+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:19:6 marker=00000000002.62.6 timestamp=2023-10-12T00:56:36.758568+0000 2023-10-12T00:56:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-12T00:56:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26/obj00000000000000000021 2023-10-12T00:56:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: entry->timestamp=2023-10-12T00:56:36.542526+0000 2023-10-12T00:56:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-10-12T00:56:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000021[0] 2023-10-12T00:56:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: listing bilog for incremental sync; position=00000000002.33.6 2023-10-12T00:56:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-12T00:56:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28/obj00000000000000000020 2023-10-12T00:56:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: entry->timestamp=2023-10-12T00:56:37.480929+0000 2023-10-12T00:56:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-12T00:56:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-12T00:56:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29/obj00000000000000000004 2023-10-12T00:56:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: entry->timestamp=2023-10-12T00:56:36.571899+0000 2023-10-12T00:56:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-12T00:56:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: parsed entry: id=00000000001.74.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-10-12T00:56:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34/obj00000000000000000057 2023-10-12T00:56:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: entry->timestamp=2023-10-12T00:56:36.818841+0000 2023-10-12T00:56:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.74.6 op=0 op_state=1 2023-10-12T00:56:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000020[0] 2023-10-12T00:56:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-12T00:56:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000004[0] 2023-10-12T00:56:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-12T00:56:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000057[0] 2023-10-12T00:56:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.759+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: listing bilog for incremental sync; position=00000000001.74.6 2023-10-12T00:56:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-12T00:56:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35/obj00000000000000000018 2023-10-12T00:56:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: entry->timestamp=2023-10-12T00:56:37.337502+0000 2023-10-12T00:56:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-12T00:56:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000018[0] 2023-10-12T00:56:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.760+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-12T00:56:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.761+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:entry[obj00000000000000000020]: success 2023-10-12T00:56:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:28:6 marker=00000000002.63.6 timestamp=2023-10-12T00:56:37.480929+0000 2023-10-12T00:56:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.761+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:entry[obj00000000000000000021]: success 2023-10-12T00:56:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.761+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:26:6 marker=00000000002.33.6 timestamp=2023-10-12T00:56:36.542526+0000 2023-10-12T00:56:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:entry[obj00000000000000000004]: success 2023-10-12T00:56:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:29:6 marker=00000000002.66.6 timestamp=2023-10-12T00:56:36.571899+0000 2023-10-12T00:56:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:entry[obj00000000000000000018]: success 2023-10-12T00:56:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:35:6 marker=00000000002.68.6 timestamp=2023-10-12T00:56:37.337502+0000 2023-10-12T00:56:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-12T00:56:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25/obj00000000000000000010 2023-10-12T00:56:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: entry->timestamp=2023-10-12T00:56:37.256487+0000 2023-10-12T00:56:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-10-12T00:56:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: parsed entry: id=00000000001.64.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-10-12T00:56:38.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30/obj00000000000000000036 2023-10-12T00:56:38.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: entry->timestamp=2023-10-12T00:56:37.501675+0000 2023-10-12T00:56:38.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000001.64.6 op=0 op_state=1 2023-10-12T00:56:38.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000010[0] 2023-10-12T00:56:38.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: listing bilog for incremental sync; position=00000000002.77.6 2023-10-12T00:56:38.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000036[0] 2023-10-12T00:56:38.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.763+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: listing bilog for incremental sync; position=00000000001.64.6 2023-10-12T00:56:38.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-12T00:56:38.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18/obj00000000000000000013 2023-10-12T00:56:38.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: entry->timestamp=2023-10-12T00:56:37.399944+0000 2023-10-12T00:56:38.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-12T00:56:38.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000013[0] 2023-10-12T00:56:38.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.764+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-12T00:56:38.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.766+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:entry[obj00000000000000000010]: success 2023-10-12T00:56:38.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:25:6 marker=00000000002.77.6 timestamp=2023-10-12T00:56:37.256487+0000 2023-10-12T00:56:38.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.766+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:entry[obj00000000000000000013]: success 2023-10-12T00:56:38.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.766+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:18:6 marker=00000000002.62.6 timestamp=2023-10-12T00:56:37.399944+0000 2023-10-12T00:56:38.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.768+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:entry[obj00000000000000000062]: success 2023-10-12T00:56:38.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.768+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:12:6 marker=00000000001.51.6 timestamp=2023-10-12T00:56:36.290658+0000 2023-10-12T00:56:38.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-10-12T00:56:38.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27/obj00000000000000000053 2023-10-12T00:56:38.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: entry->timestamp=2023-10-12T00:56:36.905327+0000 2023-10-12T00:56:38.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.98.6 op=0 op_state=1 2023-10-12T00:56:38.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000053[0] 2023-10-12T00:56:38.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.776+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: listing bilog for incremental sync; position=00000000002.98.6 2023-10-12T00:56:38.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: a sync operation returned error 2023-10-12T00:56:38.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: backing out with sync_status=-5 2023-10-12T00:56:38.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: incremental sync on bucket failed, retcode=-5 2023-10-12T00:56:38.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: ERROR: incremental sync failed. error: -5 2023-10-12T00:56:38.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T00:56:38.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: sync status for source bucket shard: 0 2023-10-12T00:56:38.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.791+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position= 2023-10-12T00:56:38.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.794+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:entry[obj00000000000000000048]: success 2023-10-12T00:56:38.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.794+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:8:6 marker=00000000001.58.6 timestamp=2023-10-12T00:56:37.150450+0000 2023-10-12T00:56:38.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.805+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:entry[obj00000000000000000057]: success 2023-10-12T00:56:38.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.805+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:34:6 marker=00000000001.74.6 timestamp=2023-10-12T00:56:36.818841+0000 2023-10-12T00:56:38.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.809+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:entry[obj00000000000000000036]: success 2023-10-12T00:56:38.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.809+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:30:6 marker=00000000001.64.6 timestamp=2023-10-12T00:56:37.501675+0000 2023-10-12T00:56:38.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.820+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:entry[obj00000000000000000053]: success 2023-10-12T00:56:38.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.820+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:27:6 marker=00000000002.98.6 timestamp=2023-10-12T00:56:36.905327+0000 2023-10-12T00:56:38.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-12T00:56:38.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: syncing object: wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6/obj00000000000000000005 2023-10-12T00:56:38.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: entry->timestamp=2023-10-12T00:56:36.638932+0000 2023-10-12T00:56:38.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-12T00:56:38.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 5 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-58[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23])/obj00000000000000000005[0] 2023-10-12T00:56:38.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.833+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-12T00:56:38.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.834+0000 7f1a61894780 10 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:entry[obj00000000000000000005]: success 2023-10-12T00:56:38.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:38.834+0000 7f1a61894780 20 RGW-SYNC:bucket[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23<-wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]:inc_sync[wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6]: updating marker marker_oid=bucket.sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23:6:6 marker=00000000002.35.6 timestamp=2023-10-12T00:56:36.638932+0000 2023-10-12T00:56:38.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:38.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:38.897 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:38.987 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:38.987 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:39.020 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:56:39.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.020 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000000", 2023-10-12T00:56:39.021 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.021 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.021 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.022 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.022 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.022 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.022 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.023 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.023 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.381044Z", 2023-10-12T00:56:39.023 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.023 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.023 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.024 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.024 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.024 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.024 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.025 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.025 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11995530094521795638", 2023-10-12T00:56:39.025 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.026 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.026 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000001", 2023-10-12T00:56:39.027 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.027 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.027 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23 2023-10-12T00:56:39.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.028 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.028 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.028 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.028 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.029 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.029 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.315952Z", 2023-10-12T00:56:39.029 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.029 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.030 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.030 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.030 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.030 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.030 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.031 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.031 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17633442079324699351", 2023-10-12T00:56:39.031 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.032 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.032 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.032 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000002", 2023-10-12T00:56:39.033 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.033 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.033 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17 2023-10-12T00:56:39.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.034 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.034 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.034 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.034 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.035 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.035 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.236044Z", 2023-10-12T00:56:39.035 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.035 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.036 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.036 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.036 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.036 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.036 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.5590652008903356124", 2023-10-12T00:56:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.037 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.038 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.038 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000003", 2023-10-12T00:56:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T00:56:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.040 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.040 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.040 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.040 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.814048Z", 2023-10-12T00:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.041 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.042 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.042 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.042 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.042 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.042 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.043 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15286375173593828901", 2023-10-12T00:56:39.043 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.043 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.044 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.044 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000004", 2023-10-12T00:56:39.044 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.045 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.045 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-10-12T00:56:39.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.046 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.046 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.046 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.046 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.046 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.047 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.571899Z", 2023-10-12T00:56:39.047 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.047 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.047 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.048 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.048 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.048 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.048 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.048 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.049 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11003237150028861460", 2023-10-12T00:56:39.049 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.049 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.049 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000005", 2023-10-12T00:56:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.051 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.051 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.051 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T00:56:39.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.052 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.052 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.052 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.052 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.053 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.053 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.638932Z", 2023-10-12T00:56:39.053 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.053 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.053 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.054 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.054 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.054 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.054 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.054 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.055 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6638997938621215485", 2023-10-12T00:56:39.055 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.056 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.056 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000006", 2023-10-12T00:56:39.057 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.057 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.057 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-10-12T00:56:39.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.058 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.058 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.058 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.058 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.059 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.059 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.066038Z", 2023-10-12T00:56:39.059 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.059 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.060 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.060 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.060 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.060 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.060 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.061 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.061 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.13573152022484094315", 2023-10-12T00:56:39.061 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.062 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.063 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000007", 2023-10-12T00:56:39.063 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.063 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.063 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.064 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.064 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.064 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.064 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.065 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.065 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.709483Z", 2023-10-12T00:56:39.065 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.065 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.066 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.066 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.066 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.066 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.067 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.067 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.067 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17026460106486479344", 2023-10-12T00:56:39.067 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.068 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.068 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.068 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000008", 2023-10-12T00:56:39.069 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.069 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.069 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T00:56:39.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.070 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.070 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.070 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.070 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.071 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.071 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.088238Z", 2023-10-12T00:56:39.071 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.071 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.071 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.072 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.072 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.072 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.072 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.072 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.073 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3385471272013539256", 2023-10-12T00:56:39.073 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.073 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.074 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000009", 2023-10-12T00:56:39.074 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.075 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.075 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18 2023-10-12T00:56:39.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.075 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.231322Z", 2023-10-12T00:56:39.077 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.077 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.077 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.077 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.077 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.078 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.078 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.078 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.079 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4657300581627330540", 2023-10-12T00:56:39.079 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.079 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.079 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.080 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000010", 2023-10-12T00:56:39.080 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.080 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.081 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T00:56:39.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.081 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.081 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.082 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.082 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.082 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.082 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.256487Z", 2023-10-12T00:56:39.083 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.083 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.083 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.083 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.084 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.084 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.084 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.084 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.085 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1713276196518407332", 2023-10-12T00:56:39.085 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.085 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.085 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.086 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000011", 2023-10-12T00:56:39.086 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.087 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.087 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.087 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.087 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.088 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.088 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.088 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.088 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.088 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.174733Z", 2023-10-12T00:56:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3606907839162576088", 2023-10-12T00:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000012", 2023-10-12T00:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T00:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.926587Z", 2023-10-12T00:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.9262671728942321826", 2023-10-12T00:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000013", 2023-10-12T00:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23 2023-10-12T00:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.399944Z", 2023-10-12T00:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12706397975162881070", 2023-10-12T00:56:39.103 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.103 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.104 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000014", 2023-10-12T00:56:39.104 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.104 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.104 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 19 2023-10-12T00:56:39.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.105 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.105 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.105 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.106 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.106 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.149479Z", 2023-10-12T00:56:39.106 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.106 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.107 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.107 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.107 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.107 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.107 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.108 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.108 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10187921590498892168", 2023-10-12T00:56:39.108 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.109 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.109 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.109 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000015", 2023-10-12T00:56:39.110 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.110 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.110 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.111 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.111 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.111 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.111 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.112 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.124025Z", 2023-10-12T00:56:39.112 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.112 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.112 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.113 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.113 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.113 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.113 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.114 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.114 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16144649280648237863", 2023-10-12T00:56:39.114 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000016", 2023-10-12T00:56:39.115 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.116 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.116 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14 2023-10-12T00:56:39.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.117 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.117 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.117 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.117 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.118 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.480592Z", 2023-10-12T00:56:39.118 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.118 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.118 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.119 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.119 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.119 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.119 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.119 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.120 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16046425751715164054", 2023-10-12T00:56:39.120 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.120 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.120 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.121 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000017", 2023-10-12T00:56:39.121 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.121 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.122 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 15 2023-10-12T00:56:39.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.123 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.123 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.124 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.125 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.126 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.842056Z", 2023-10-12T00:56:39.126 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.127 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.127 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.127 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.128 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.128 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.128 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.128 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.129 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12469609618769551968", 2023-10-12T00:56:39.130 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.130 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.131 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000018", 2023-10-12T00:56:39.131 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.131 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 16 2023-10-12T00:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.337502Z", 2023-10-12T00:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16294117814024208072", 2023-10-12T00:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000019", 2023-10-12T00:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-10-12T00:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.733471Z", 2023-10-12T00:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17542845853521021646", 2023-10-12T00:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000020", 2023-10-12T00:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T00:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.146 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.146 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.146 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.147 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.480929Z", 2023-10-12T00:56:39.147 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.147 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.147 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.148 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.148 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.148 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.148 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.148 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.149 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16468790072000788310", 2023-10-12T00:56:39.149 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.149 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.150 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000021", 2023-10-12T00:56:39.151 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.151 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.151 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T00:56:39.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.152 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.152 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.152 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.152 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.542526Z", 2023-10-12T00:56:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.154 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.154 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.154 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.154 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.155 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.155 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.7246479063579761706", 2023-10-12T00:56:39.155 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.156 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.156 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.156 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000022", 2023-10-12T00:56:39.157 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.157 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.157 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.157 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T00:56:39.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.158 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.158 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.158 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.158 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.159 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.159 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.630045Z", 2023-10-12T00:56:39.159 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.159 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.160 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.160 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.160 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.160 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.160 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.161 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.161 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.5131426728109921500", 2023-10-12T00:56:39.161 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.162 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.163 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000023", 2023-10-12T00:56:39.163 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.163 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.164 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.164 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.164 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.164 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.165 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.165 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.165 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.006581Z", 2023-10-12T00:56:39.165 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.166 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.166 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.166 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.166 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.166 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.167 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.167 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.167 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15517955744750793304", 2023-10-12T00:56:39.167 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.168 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.169 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000024", 2023-10-12T00:56:39.169 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.169 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.169 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.169 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.170 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.170 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.170 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.170 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.171 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.171 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.403096Z", 2023-10-12T00:56:39.171 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.172 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.172 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.172 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.172 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.172 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.173 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.173 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.173 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16658250464777711490", 2023-10-12T00:56:39.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.174 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.175 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000025", 2023-10-12T00:56:39.175 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.175 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.175 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.175 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.176 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.176 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.176 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.176 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.177 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.177 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.874787Z", 2023-10-12T00:56:39.177 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.177 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.177 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.178 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.178 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.178 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.178 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.178 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.179 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.9858315457738130567", 2023-10-12T00:56:39.179 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000026", 2023-10-12T00:56:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.187 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.664293Z", 2023-10-12T00:56:39.187 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.187 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.188 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.188 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.188 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.188 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.188 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.189 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.189 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11700734654193737244", 2023-10-12T00:56:39.189 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.189 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.190 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.190 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000027", 2023-10-12T00:56:39.191 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.191 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.191 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.191 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20 2023-10-12T00:56:39.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.192 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.192 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.192 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.192 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.192 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.193 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.420497Z", 2023-10-12T00:56:39.193 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.193 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.193 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.193 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.194 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.194 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.194 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.194 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.195 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.5823813148983831175", 2023-10-12T00:56:39.195 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.195 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.195 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.196 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000028", 2023-10-12T00:56:39.196 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.196 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.197 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.197 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.197 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.198 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.198 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.198 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.198 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:30.926692Z", 2023-10-12T00:56:39.198 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.199 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.199 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.199 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.199 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.199 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.200 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.200 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.200 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15765419147654103241", 2023-10-12T00:56:39.201 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.201 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.202 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.203 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000029", 2023-10-12T00:56:39.203 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.204 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.204 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.205 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.206 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.206 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.206 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.207 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.207 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.521540Z", 2023-10-12T00:56:39.208 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.208 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.208 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.209 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.209 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.210 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.210 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.211 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.211 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3339850564814867811", 2023-10-12T00:56:39.212 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.213 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.214 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000030", 2023-10-12T00:56:39.214 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.215 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.215 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.215 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.216 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.217 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.217 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.217 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.218 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.218 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.209436Z", 2023-10-12T00:56:39.219 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.219 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.219 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.220 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.220 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.221 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.221 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.221 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.222 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.7798243093859270086", 2023-10-12T00:56:39.222 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.223 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.223 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000031", 2023-10-12T00:56:39.224 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.224 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.224 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:56:39.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.225 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.225 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.225 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.225 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.225 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.226 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.398094Z", 2023-10-12T00:56:39.226 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.226 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.226 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.227 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.227 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.227 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.227 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.227 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15071402477581186410", 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000032", 2023-10-12T00:56:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T00:56:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.872802Z", 2023-10-12T00:56:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2301519350987663609", 2023-10-12T00:56:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000033", 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.459455Z", 2023-10-12T00:56:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12328651417086780092", 2023-10-12T00:56:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000036", 2023-10-12T00:56:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.501675Z", 2023-10-12T00:56:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.9533954777371662479", 2023-10-12T00:56:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000037", 2023-10-12T00:56:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 37 2023-10-12T00:56:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.247 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.247 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.247 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.247 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.247 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.607083Z", 2023-10-12T00:56:39.248 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.248 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.248 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.248 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.248 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2724506553779730933", 2023-10-12T00:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000039", 2023-10-12T00:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.319766Z", 2023-10-12T00:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2852197581415562982", 2023-10-12T00:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000040", 2023-10-12T00:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.984268Z", 2023-10-12T00:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2297401647718811199", 2023-10-12T00:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000042", 2023-10-12T00:56:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.184135Z", 2023-10-12T00:56:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16327049213777870166", 2023-10-12T00:56:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000045", 2023-10-12T00:56:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.758568Z", 2023-10-12T00:56:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.13428036839749184659", 2023-10-12T00:56:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000046", 2023-10-12T00:56:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 19 2023-10-12T00:56:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.700534Z", 2023-10-12T00:56:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17708441537622408358", 2023-10-12T00:56:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000048", 2023-10-12T00:56:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.150450Z", 2023-10-12T00:56:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.18076849473080504873", 2023-10-12T00:56:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000049", 2023-10-12T00:56:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T00:56:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.268790Z", 2023-10-12T00:56:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16018599222533640734", 2023-10-12T00:56:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000050", 2023-10-12T00:56:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.169727Z", 2023-10-12T00:56:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15055340046076955580", 2023-10-12T00:56:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000053", 2023-10-12T00:56:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:56:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.905327Z", 2023-10-12T00:56:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.14469920897653571423", 2023-10-12T00:56:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000055", 2023-10-12T00:56:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.301 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.301 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.301 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.449698Z", 2023-10-12T00:56:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.14948844231580897582", 2023-10-12T00:56:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000057", 2023-10-12T00:56:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.818841Z", 2023-10-12T00:56:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4757954013765626462", 2023-10-12T00:56:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000058", 2023-10-12T00:56:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.314 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.314 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.315 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.315 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.315 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.315 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.316 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.301020Z", 2023-10-12T00:56:39.316 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.316 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.316 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.317 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.317 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.317 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.317 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.318 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.318 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11592198063574574345", 2023-10-12T00:56:39.318 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.318 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.319 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.319 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000059", 2023-10-12T00:56:39.320 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.320 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.320 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.954755Z", 2023-10-12T00:56:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.323 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.323 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.323 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.323 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.324 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.324 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.324 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.15251593412287228510", 2023-10-12T00:56:39.325 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.325 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.325 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.326 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000060", 2023-10-12T00:56:39.326 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.326 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.327 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.327 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.327 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.328 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.328 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.328 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.328 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.005482Z", 2023-10-12T00:56:39.329 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.329 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.329 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.329 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.329 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.330 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.330 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.330 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.331 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.667635275551335261", 2023-10-12T00:56:39.331 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.331 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.331 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.332 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000061", 2023-10-12T00:56:39.332 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.332 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.333 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.333 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.333 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.333 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.334 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.334 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.334 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.334 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.835363Z", 2023-10-12T00:56:39.335 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.335 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.335 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.335 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.336 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.336 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.336 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.337 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12366497865552539384", 2023-10-12T00:56:39.337 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.337 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.338 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000062", 2023-10-12T00:56:39.338 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.339 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.339 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.340 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.290658Z", 2023-10-12T00:56:39.341 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.341 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.341 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.342 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.342 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.342 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.342 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.343 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.343 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6798040761314919183", 2023-10-12T00:56:39.343 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.343 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.344 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.344 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000065", 2023-10-12T00:56:39.345 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.345 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.345 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T00:56:39.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.346 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.346 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.346 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.346 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.347 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.347 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.129863Z", 2023-10-12T00:56:39.347 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.347 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.348 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.348 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.348 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.348 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.349 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.349 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6901302927611160340", 2023-10-12T00:56:39.350 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.350 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.350 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.351 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000076", 2023-10-12T00:56:39.351 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.351 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.351 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.352 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T00:56:39.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.352 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.352 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.353 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.353 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.353 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.354 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:30.523646Z", 2023-10-12T00:56:39.354 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.354 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.354 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.355 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.355 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.355 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.356 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.356 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.356 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12437824204715194659", 2023-10-12T00:56:39.357 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.357 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.357 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.358 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000133", 2023-10-12T00:56:39.358 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.359 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.359 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 38 2023-10-12T00:56:39.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.360 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.360 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.360 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.360 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.360 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.361 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.984078Z", 2023-10-12T00:56:39.361 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.361 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.361 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.362 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.362 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.362 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.362 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.363 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.363 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10630761193724892690", 2023-10-12T00:56:39.363 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.364 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.364 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000162", 2023-10-12T00:56:39.365 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.365 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.365 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.366 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.366 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.366 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.367 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.367 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.353205Z", 2023-10-12T00:56:39.367 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.368 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.368 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.368 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.368 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.369 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.369 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.369 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.369 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2304549678139574510", 2023-10-12T00:56:39.370 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.370 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.371 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000271", 2023-10-12T00:56:39.371 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.371 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.372 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.372 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.372 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.373 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.373 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.373 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.373 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.373 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.438872Z", 2023-10-12T00:56:39.374 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.374 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.374 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.374 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.375 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.375 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.375 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.375 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.376 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6174825527606399490", 2023-10-12T00:56:39.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.376 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.376 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.377 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000283", 2023-10-12T00:56:39.377 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.378 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.378 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.378 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.379 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.379 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.379 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.379 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.380 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.380 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.794968Z", 2023-10-12T00:56:39.380 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.380 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.380 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.381 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.381 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.381 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.381 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.382 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.382 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6887167377395992297", 2023-10-12T00:56:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.384 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000317", 2023-10-12T00:56:39.384 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.384 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.384 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.385 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.385 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.385 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.386 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.386 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.386 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.386 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.752965Z", 2023-10-12T00:56:39.386 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.387 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.387 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.387 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.387 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.388 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.388 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.388 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1757741520555774858", 2023-10-12T00:56:39.389 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.389 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.391 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.391 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000423", 2023-10-12T00:56:39.392 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.392 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.392 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.393 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.393 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.393 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.394 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.394 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.394 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.068894Z", 2023-10-12T00:56:39.394 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.395 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.395 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.395 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.395 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.395 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.396 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.396 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.396 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.9079562439905471891", 2023-10-12T00:56:39.397 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.397 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000467", 2023-10-12T00:56:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T00:56:39.399 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.399 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.399 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.400 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.400 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.400 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.400 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.545947Z", 2023-10-12T00:56:39.401 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.401 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.401 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.401 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.402 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.402 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.402 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.402 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.403 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3446789670649468587", 2023-10-12T00:56:39.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.403 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.404 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:39.404 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:56:39.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:39.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:39.406 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:39.459 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:39.460 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:39.490 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:56:39.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.491 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000000", 2023-10-12T00:56:39.491 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.491 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.491 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.492 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.492 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.492 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.492 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.493 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.493 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.493 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.381044Z", 2023-10-12T00:56:39.493 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.493 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.494 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.494 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.494 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.494 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.495 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.495 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.495 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_czV40XhbiqInIB8-OZFt-kEVVd1c0qT", 2023-10-12T00:56:39.495 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.496 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.496 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000001", 2023-10-12T00:56:39.497 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.497 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.497 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.498 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.498 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.498 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.499 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.499 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.499 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.315952Z", 2023-10-12T00:56:39.499 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.499 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.500 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.500 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.500 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.500 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.501 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.501 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.501 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_eMzTQScZ0fJInxHMJ0EFlV6v2pi-v8a", 2023-10-12T00:56:39.501 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.502 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.502 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000002", 2023-10-12T00:56:39.503 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.503 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.503 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.503 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.504 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.504 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.504 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.504 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.505 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.505 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.236044Z", 2023-10-12T00:56:39.505 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.505 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.505 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.506 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.506 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.506 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.506 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.506 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.507 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_GDViGCvjdNeN-JcdC3FzMj5-UD4qKWo", 2023-10-12T00:56:39.507 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.508 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.508 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000003", 2023-10-12T00:56:39.509 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.509 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.509 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.510 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.510 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.510 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.510 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.511 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.511 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.814048Z", 2023-10-12T00:56:39.511 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.511 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.511 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.512 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.512 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.512 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.512 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.512 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.513 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_GXhWJjKMFVxD6QoQAvkKi4826iYwMVK", 2023-10-12T00:56:39.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.513 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.514 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000004", 2023-10-12T00:56:39.514 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.514 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.515 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.515 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.515 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.516 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.516 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.516 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.516 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.516 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.571899Z", 2023-10-12T00:56:39.517 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.517 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.517 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.517 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.517 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.518 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.518 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.518 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.518 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_O3O4PWp_v0nV9kVA1t3IJXhqhUA1hTS", 2023-10-12T00:56:39.519 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.519 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.519 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.520 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000005", 2023-10-12T00:56:39.520 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.520 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.520 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.521 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.521 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.521 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.522 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.522 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.522 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.522 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.638932Z", 2023-10-12T00:56:39.522 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.523 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.523 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.523 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.523 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.523 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.524 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.524 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.524 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_4bOOsWo5VJWptyGJ9rHwSnFofd8x7k1", 2023-10-12T00:56:39.525 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.525 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000006", 2023-10-12T00:56:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.527 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.527 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.527 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.527 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.528 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.528 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.066038Z", 2023-10-12T00:56:39.528 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.528 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.529 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.529 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.529 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.529 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.529 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.530 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.530 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_7_q3vGQHs_AqNncmwofS0mz5_o6HWcQ", 2023-10-12T00:56:39.530 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.531 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.531 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000007", 2023-10-12T00:56:39.532 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.532 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.532 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.533 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.533 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.533 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.533 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.533 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.534 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.709483Z", 2023-10-12T00:56:39.534 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.534 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.534 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.535 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.535 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.535 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.535 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.535 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.536 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_6H2NhaXbEg5Joeo1PLKFS-4YWMHaUUX", 2023-10-12T00:56:39.536 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.536 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.536 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.537 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000008", 2023-10-12T00:56:39.537 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.538 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.538 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T00:56:39.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.538 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.539 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.088238Z", 2023-10-12T00:56:39.540 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.540 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.540 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.540 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.540 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.541 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.541 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.541 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.542 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_U9nRmXUFCyzGvqrxh6KU1yDtC46dSS7", 2023-10-12T00:56:39.542 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.542 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.543 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000009", 2023-10-12T00:56:39.543 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.543 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.544 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.544 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.544 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.545 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.545 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.545 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.545 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.231322Z", 2023-10-12T00:56:39.546 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.546 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.546 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.546 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.546 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.547 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.547 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.547 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.547 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yZvD41Z8wyFh3y_yH8qhr6mejaQYVVA", 2023-10-12T00:56:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000010", 2023-10-12T00:56:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.551 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.551 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.551 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.256487Z", 2023-10-12T00:56:39.551 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.553 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.553 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.553 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.554 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.554 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.554 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.554 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.555 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_M2LIX5SRrazzE2x-4lG3lBG0c-yp5wC", 2023-10-12T00:56:39.555 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.555 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.555 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.556 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000011", 2023-10-12T00:56:39.556 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.556 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.557 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.557 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.557 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.558 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.558 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.558 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.558 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.558 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.174733Z", 2023-10-12T00:56:39.559 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.559 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.559 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.559 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.559 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.560 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.560 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.560 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.560 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_8OOzRexiJnNb1A4ZdsRIOT9rSKK1ZUy", 2023-10-12T00:56:39.561 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.561 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.561 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.562 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000012", 2023-10-12T00:56:39.562 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.562 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.563 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.563 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.563 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.563 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.564 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.564 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.564 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.926587Z", 2023-10-12T00:56:39.564 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.564 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.565 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.565 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.565 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.565 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.566 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.566 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.566 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_X1fs3r7Qkqm18gvqjNrTMUUCUQLDdz_", 2023-10-12T00:56:39.566 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.567 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.567 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000013", 2023-10-12T00:56:39.568 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.568 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.568 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.569 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.399944Z", 2023-10-12T00:56:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.570 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.571 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_f6g6KHbYgEqB2hIKfzGtV4AZQ_Bl1ff", 2023-10-12T00:56:39.572 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.573 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000014", 2023-10-12T00:56:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.575 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.149479Z", 2023-10-12T00:56:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.576 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.577 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ccBGe-aE1gShZ7H9BSgWkMqWwhxOOvg", 2023-10-12T00:56:39.578 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.579 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000015", 2023-10-12T00:56:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.580 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T00:56:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.581 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.124025Z", 2023-10-12T00:56:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.582 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.583 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_96DWT5z5T8OpzjTDju82bOyeZzuE0Cy", 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000016", 2023-10-12T00:56:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.586 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.480592Z", 2023-10-12T00:56:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_gDr-42MWL6OJOk9qutBf8Km6mbg0wkF", 2023-10-12T00:56:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000017", 2023-10-12T00:56:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.842056Z", 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_7f126HVyQYRV-YDkj3uuvAYM4enBwgF", 2023-10-12T00:56:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000018", 2023-10-12T00:56:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.337502Z", 2023-10-12T00:56:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CMz632UuSizD86ycvt4-ojLdJbRMi7q", 2023-10-12T00:56:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000019", 2023-10-12T00:56:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.733471Z", 2023-10-12T00:56:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_dMh4j7fVy6rBR8xEesdS_oyMTJgZDku", 2023-10-12T00:56:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000020", 2023-10-12T00:56:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.608 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.480929Z", 2023-10-12T00:56:39.609 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.610 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.611 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_nPrQXB7wPRer5AH7ulhCHg-JGhTs4i5", 2023-10-12T00:56:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000021", 2023-10-12T00:56:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.613 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.614 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.542526Z", 2023-10-12T00:56:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.615 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.616 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_biRnTDT5gqmyuv3voZIF05Q5LpV4NTQ", 2023-10-12T00:56:39.617 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.618 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000022", 2023-10-12T00:56:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.620 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.630045Z", 2023-10-12T00:56:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.621 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.622 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2jIoIzvVZ91JIdU_G4emyNpcO4fPgVr", 2023-10-12T00:56:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.623 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000023", 2023-10-12T00:56:39.624 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.626 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.006581Z", 2023-10-12T00:56:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.627 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.628 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kZvJsGUcbDTowUFHXj0hVCkWo5z8soD", 2023-10-12T00:56:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.629 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000024", 2023-10-12T00:56:39.630 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.631 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.632 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.403096Z", 2023-10-12T00:56:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.633 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yj3kvBTqqQuuJY2qJyqgTyhSONuQlKl", 2023-10-12T00:56:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.635 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000025", 2023-10-12T00:56:39.636 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.638 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.874787Z", 2023-10-12T00:56:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.639 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.640 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_EdBKkWkgXT9tEzCZfDJ1jVJJOtR1hvJ", 2023-10-12T00:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.641 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000026", 2023-10-12T00:56:39.642 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.643 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.644 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.664293Z", 2023-10-12T00:56:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.645 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.646 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ljeen0V2kea2HQthXoZlfir6wOqbR42", 2023-10-12T00:56:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.647 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000027", 2023-10-12T00:56:39.648 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T00:56:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.649 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.650 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.420497Z", 2023-10-12T00:56:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.651 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.652 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_tArUgO8RQwRgYjb5UuY0KEE4EyjbbqM", 2023-10-12T00:56:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.653 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000028", 2023-10-12T00:56:39.654 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.655 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.656 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:30.926692Z", 2023-10-12T00:56:39.657 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.658 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.659 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_HanmqKpWIZ-S1vl1EUNHc0Z70sN_qrM", 2023-10-12T00:56:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000029", 2023-10-12T00:56:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.661 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.662 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.521540Z", 2023-10-12T00:56:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.663 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.664 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_twZmYEHkegc6KJXttzmX1c_JjSSA-4a", 2023-10-12T00:56:39.665 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.666 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.666 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000030", 2023-10-12T00:56:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.668 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.668 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.668 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.669 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.669 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.669 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.209436Z", 2023-10-12T00:56:39.669 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.669 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.670 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.670 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.670 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.670 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.671 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.671 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.671 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_gkJVi_nWd9_3IqL0eueYPOGMhoc8ZD2", 2023-10-12T00:56:39.671 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.672 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.672 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.672 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000031", 2023-10-12T00:56:39.673 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.673 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.673 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:56:39.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.674 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.674 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.674 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.674 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.675 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.675 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.398094Z", 2023-10-12T00:56:39.675 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.675 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.675 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.676 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.676 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.676 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.676 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.676 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_AsHgI7tCe-c6eshUbblAngdFXnTPXIr", 2023-10-12T00:56:39.677 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.678 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.678 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000032", 2023-10-12T00:56:39.678 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.679 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.679 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T00:56:39.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.680 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.680 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.680 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.680 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.680 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.872802Z", 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.681 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.682 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.682 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.682 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.683 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_drCklDYWxMKdXsxlUBx6iMzCrr5UgFB", 2023-10-12T00:56:39.683 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.683 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.684 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000033", 2023-10-12T00:56:39.684 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.684 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.684 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.685 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.685 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.685 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.685 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.686 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.686 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.686 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.459455Z", 2023-10-12T00:56:39.686 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.686 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.687 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.687 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.687 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.687 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.687 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.688 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.688 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "__L8XL3CkyjmFKHQzEDFyH5RbCUwEjGh", 2023-10-12T00:56:39.688 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.688 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.689 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.689 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000036", 2023-10-12T00:56:39.689 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.690 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.690 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.691 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.691 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.691 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.691 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.691 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.692 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.501675Z", 2023-10-12T00:56:39.692 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.692 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.693 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.693 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.693 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.693 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.693 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.694 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.694 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yN0A2CoLP3pU4EmnehWqdTxheQX0K0O", 2023-10-12T00:56:39.694 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.695 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.695 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000037", 2023-10-12T00:56:39.696 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.696 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.696 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 37 2023-10-12T00:56:39.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.697 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.697 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.697 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.697 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.698 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.698 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.607083Z", 2023-10-12T00:56:39.698 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.698 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.699 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.699 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.699 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.700 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.700 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Q7O8oO1NzbDIHxWgc67VpKxwfN3Cuk3", 2023-10-12T00:56:39.700 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.700 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.701 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.701 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000039", 2023-10-12T00:56:39.701 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.702 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.702 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.702 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.703 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.703 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.703 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.703 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.703 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.704 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.319766Z", 2023-10-12T00:56:39.704 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.704 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.704 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.704 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.705 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.705 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.705 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.705 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.706 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_g8LqGFJmeV-E4NijzmDnrH5IRF5GPyn", 2023-10-12T00:56:39.706 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.706 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.707 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000040", 2023-10-12T00:56:39.707 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.708 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.708 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.708 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.709 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.709 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.709 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.709 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.984268Z", 2023-10-12T00:56:39.710 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.710 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.710 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.710 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.711 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.711 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.711 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.711 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.712 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_5WHqTYfbae5af3Rifn1rW-UsxvyIOg7", 2023-10-12T00:56:39.712 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.712 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.712 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.713 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000042", 2023-10-12T00:56:39.713 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.713 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.713 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.714 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.714 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.714 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.714 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.715 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.715 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.715 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.184135Z", 2023-10-12T00:56:39.715 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.715 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.717 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.717 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yU2wMDxfV6aWS09YMyADj-zKEQEHZLp", 2023-10-12T00:56:39.717 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.718 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000045", 2023-10-12T00:56:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.720 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.758568Z", 2023-10-12T00:56:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.721 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.722 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ImUxG-s7aPWBvVcnuG8aCZc9amKXttT", 2023-10-12T00:56:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.723 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000046", 2023-10-12T00:56:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T00:56:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.700534Z", 2023-10-12T00:56:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_6ovv29TG3uKgskMqGonfrENcq41WhE6", 2023-10-12T00:56:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000048", 2023-10-12T00:56:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.150450Z", 2023-10-12T00:56:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_uRKECqjVRgIf_V1MWgjo6hc_8lwydI9", 2023-10-12T00:56:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000049", 2023-10-12T00:56:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.268790Z", 2023-10-12T00:56:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.739 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_FwWsj7Z-aEFEGlAFoT_VVWevWwF4A5C", 2023-10-12T00:56:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.740 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000050", 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.742 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.169727Z", 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.743 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.744 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_llIwFXA8AOlaF3BxkbC5rDbiTUXV6GT", 2023-10-12T00:56:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.746 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000053", 2023-10-12T00:56:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:56:39.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.748 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.905327Z", 2023-10-12T00:56:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.749 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Zz2S8RHb-HKLZJHxE_HfSagNCvfMog-", 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000055", 2023-10-12T00:56:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.449698Z", 2023-10-12T00:56:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_3cDSGm2qiElHIX7NrODYKXx7fQ3cq9u", 2023-10-12T00:56:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000057", 2023-10-12T00:56:39.757 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.758 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.758 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.758 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.759 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.759 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.759 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.759 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.759 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.760 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.818841Z", 2023-10-12T00:56:39.760 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.760 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.760 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.761 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.761 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.761 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.761 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.761 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.762 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kaTOgD3PQ12gbMbI1bxfXEwlMIosNBf", 2023-10-12T00:56:39.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.762 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.763 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000058", 2023-10-12T00:56:39.763 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.764 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.764 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.764 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.764 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.765 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.765 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.765 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.765 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.765 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:32.301020Z", 2023-10-12T00:56:39.766 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.766 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.766 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.766 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.767 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.767 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.767 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.767 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JwTO8kmKehfpDAeKNj01KlsbNOENO-O", 2023-10-12T00:56:39.768 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.768 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.768 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.769 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000059", 2023-10-12T00:56:39.769 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.769 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.770 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.770 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.770 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.770 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:33.954755Z", 2023-10-12T00:56:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CIOlgEFfuqVADUErXk59Ut7KLZKJfHP", 2023-10-12T00:56:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000060", 2023-10-12T00:56:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T00:56:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:37.005482Z", 2023-10-12T00:56:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.779 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yMtys55UaM2dzAFqZb5EHDaxM8TQIPV", 2023-10-12T00:56:39.779 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.779 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.780 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000061", 2023-10-12T00:56:39.780 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.780 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.781 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-10-12T00:56:39.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.781 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.781 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.782 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.782 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.782 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.782 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.835363Z", 2023-10-12T00:56:39.782 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.783 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.783 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.783 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.783 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.784 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.784 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.784 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.784 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CXwiy9Me10vxVMpTGJ-D7VszuE49s4z", 2023-10-12T00:56:39.785 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.785 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.785 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.786 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000062", 2023-10-12T00:56:39.786 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.786 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.786 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.787 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T00:56:39.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.787 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.787 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.787 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.788 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.788 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.788 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.290658Z", 2023-10-12T00:56:39.788 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.788 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.789 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.789 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.789 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.789 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.790 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.790 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.790 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_n15QRmDpke6hztMvCQBATYeHgFcokSt", 2023-10-12T00:56:39.790 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.791 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.791 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.792 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000065", 2023-10-12T00:56:39.792 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.792 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.792 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.793 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.793 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.793 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.793 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.794 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.794 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:36.129863Z", 2023-10-12T00:56:39.794 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.794 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.795 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.795 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.795 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.795 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.796 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.796 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.796 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_V0pcm_gCq3PX8l4OPFYVE4Ja7hL2VLC", 2023-10-12T00:56:39.797 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.797 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.797 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.798 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000076", 2023-10-12T00:56:39.798 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.798 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:30.523646Z", 2023-10-12T00:56:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.801 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.801 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.801 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.801 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.802 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.802 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.802 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_PU_Vf_86zoC1S00tIbvofc_rVg9KLFK", 2023-10-12T00:56:39.802 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.803 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.803 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.805 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000133", 2023-10-12T00:56:39.805 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.806 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 38 2023-10-12T00:56:39.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.807 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.807 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.807 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.807 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.807 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T00:56:39.808 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.984078Z", 2023-10-12T00:56:39.808 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.808 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.808 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.808 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.809 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.809 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.809 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.809 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.810 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_WDVhNioorUY3WEsrXr82on-mpeTh9TY", 2023-10-12T00:56:39.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.810 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.811 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000162", 2023-10-12T00:56:39.811 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.811 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.812 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.812 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.812 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.353205Z", 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.813 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.814 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.814 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.814 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.814 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.815 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.815 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.815 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Uek9KNwl5-RfU3rzZs6Ssd849dvccGb", 2023-10-12T00:56:39.815 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.816 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.818 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000271", 2023-10-12T00:56:39.818 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.818 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.819 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.819 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.819 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.819 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.820 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.820 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.820 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.438872Z", 2023-10-12T00:56:39.820 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.820 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.821 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.821 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.821 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.821 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.821 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.822 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.822 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_uoO8mnUS23A608VW2DMT-WVaTBbZtp_", 2023-10-12T00:56:39.822 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.823 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.823 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000283", 2023-10-12T00:56:39.824 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.824 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.824 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.825 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.825 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.825 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.825 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.825 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.826 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.794968Z", 2023-10-12T00:56:39.826 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.826 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.826 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.827 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.827 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.827 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.827 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.828 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_E1jBcn0Hq_GySt7x0X-hV2SFxBbUhfu", 2023-10-12T00:56:39.828 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.828 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.829 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000317", 2023-10-12T00:56:39.829 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.830 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.830 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.830 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T00:56:39.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.830 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.831 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.831 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.831 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.831 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T00:56:39.831 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:34.752965Z", 2023-10-12T00:56:39.832 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.832 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.832 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.832 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.833 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.833 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.833 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.833 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.834 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_TtfnqE5bk_vdmqCtfsdrkvG9o_NnEyM", 2023-10-12T00:56:39.834 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.834 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.834 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.835 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000423", 2023-10-12T00:56:39.835 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.835 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T00:56:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.836 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.837 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.837 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.837 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.837 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.068894Z", 2023-10-12T00:56:39.838 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.838 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.838 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.838 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.838 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.839 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.839 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.839 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.840 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_wCbJZehqx2Z_dIRIu-vq3_u1eptb9pd", 2023-10-12T00:56:39.840 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.840 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:39.841 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000467", 2023-10-12T00:56:39.841 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T00:56:39.841 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T00:56:39.841 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T00:56:39.842 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T00:56:39.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.842 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T00:56:39.842 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T00:56:39.843 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T00:56:39.843 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T00:56:39.843 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T00:56:39.843 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:56:35.545947Z", 2023-10-12T00:56:39.843 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T00:56:39.844 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T00:56:39.844 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T00:56:39.844 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T00:56:39.844 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T00:56:39.844 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T00:56:39.845 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T00:56:39.845 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T00:56:39.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:39.845 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_vTfR4cPsoWqw1Ki3_KHXSRM-isCmcj2", 2023-10-12T00:56:39.845 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T00:56:39.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T00:56:39.846 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T00:56:39.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:39.846 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:56:39.847 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T00:56:39.848 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T00:56:39.850 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T00:56:39.851 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T00:56:39.873 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:56:39.873 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:56:39.873 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T00:56:40.874 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:40.898 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T00:56:40.899 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T00:56:40.899 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T00:56:40.900 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T00:56:40.900 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-58 2023-10-12T00:56:40.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:40.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:40.902 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:56:40.999 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:41.000 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:41.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:41.040+0000 7f39dffb2780 1 bucket sync caught up with source: 2023-10-12T00:56:41.041 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.60.6, 00000000002.43.6, 00000000002.64.6, 00000000002.50.6, 00000000002.30.6, 00000000001.57.6, 00000000002.35.6, 00000000002.60.6, 00000000001.58.6, , 00000000002.37.6, 00000000002.61.6, 00000000001.51.6, 00000000002.65.6, 00000000002.81.6, 00000000002.75.6, 00000000002.74.6, 00000000002.62.6, 00000000002.62.6, 00000000002.62.6, 00000000002.30.6, 00000000002.46.6, , 00000000002.96.6, 00000000002.67.6, 00000000002.77.6, 00000000002.33.6, 00000000002.98.6, 00000000002.63.6, 00000000002.66.6, 00000000001.64.6, , 00000000002.59.6, 00000000002.60.6, 00000000001.74.6, 00000000002.68.6, 00000000002.70.6] 2023-10-12T00:56:41.041 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.60.6, 00000000002.43.6, 00000000002.64.6, 00000000002.50.6, 00000000002.30.6, 00000000001.57.6, 00000000002.35.6, 00000000002.60.6, 00000000001.58.6, , 00000000002.37.6, 00000000002.61.6, 00000000001.51.6, 00000000002.65.6, 00000000002.81.6, 00000000002.75.6, 00000000002.74.6, 00000000002.62.6, 00000000002.62.6, 00000000002.62.6, 00000000002.30.6, 00000000002.46.6, , 00000000002.96.6, 00000000002.67.6, 00000000002.77.6, 00000000002.33.6, 00000000002.98.6, 00000000002.63.6, 00000000002.66.6, 00000000001.64.6, , 00000000002.59.6, 00000000002.60.6, 00000000001.74.6, 00000000002.68.6, 00000000002.70.6] 2023-10-12T00:56:41.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:41.040+0000 7f39dffb2780 0 bucket checkpoint complete 2023-10-12T00:56:41.054 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-58 2023-10-12T00:56:41.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:41.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:56:41.056 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:56:41.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:41.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:41.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:41.146+0000 7f35d5379780 1 bucket sync caught up with source: 2023-10-12T00:56:41.148 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000005.346.6, 00000000006.373.6, 00000000006.315.6, 00000000010.103.6, 00000000005.204.6, 00000000008.178.6, 00000000007.119.6, 00000000006.193.6, 00000000007.271.6, 00000000007.269.6, 00000000004.256.6] 2023-10-12T00:56:41.148 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000005.346.6, 00000000006.373.6, 00000000006.315.6, 00000000010.103.6, 00000000005.204.6, 00000000008.178.6, 00000000007.119.6, 00000000006.193.6, 00000000007.271.6, 00000000007.269.6, 00000000004.256.6] 2023-10-12T00:56:41.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:41.146+0000 7f35d5379780 0 bucket checkpoint complete 2023-10-12T00:56:41.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-58 zones={a2, a1} 2023-10-12T00:56:41.161 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.161 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.161 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.162 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:41.162 DEBUG:boto:Data: 2023-10-12T00:56:41.163 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.164 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.164 DEBUG:boto:Params: {} 2023-10-12T00:56:41.164 DEBUG:boto:Token: None 2023-10-12T00:56:41.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/ 2023-10-12T00:56:41.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw= 2023-10-12T00:56:41.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw=', 'Content-Length': '0'} 2023-10-12T00:56:41.167 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-12T00:56:41.168 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:56:41.734 DEBUG:boto:Token: None 2023-10-12T00:56:41.735 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/ 2023-10-12T00:56:41.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw= 2023-10-12T00:56:41.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw=', 'Content-Length': '0'} 2023-10-12T00:56:41.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007416e5a9c6262076-0065274449-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.743 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.743 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.743 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.744 DEBUG:boto:Path: /wafmoa-58/ 2023-10-12T00:56:41.744 DEBUG:boto:Data: 2023-10-12T00:56:41.745 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.745 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.746 DEBUG:boto:Params: {} 2023-10-12T00:56:41.746 DEBUG:boto:Token: None 2023-10-12T00:56:41.747 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/ 2023-10-12T00:56:41.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw= 2023-10-12T00:56:41.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MAnA+i8UT1pf8FDHs7wEtk1wchw=', 'Content-Length': '0'} 2023-10-12T00:56:41.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd83b5c62e19ee76-0065274449-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.752 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:56:41.752 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.753 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.753 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.754 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.754 DEBUG:boto:Method: GET 2023-10-12T00:56:41.754 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.755 DEBUG:boto:Data: 2023-10-12T00:56:41.755 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.756 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.757 DEBUG:boto:Params: {} 2023-10-12T00:56:41.757 DEBUG:boto:Token: None 2023-10-12T00:56:41.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/?versions 2023-10-12T00:56:41.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg= 2023-10-12T00:56:41.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg=', 'Content-Length': '0'} 2023-10-12T00:56:41.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007699c3647770aa72-0065274449-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.762 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-12T00:56:41.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-12T00:56:41.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-12T00:56:41.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-12T00:56:41.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-12T00:56:41.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-12T00:56:41.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-12T00:56:41.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-12T00:56:41.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-12T00:56:41.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-12T00:56:41.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-12T00:56:41.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-12T00:56:41.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-12T00:56:41.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-12T00:56:41.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-12T00:56:41.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-12T00:56:41.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-12T00:56:41.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-12T00:56:41.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-12T00:56:41.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-12T00:56:41.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-12T00:56:41.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-12T00:56:41.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-12T00:56:41.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-12T00:56:41.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-12T00:56:41.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-12T00:56:41.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-12T00:56:41.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-12T00:56:41.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-12T00:56:41.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-12T00:56:41.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-12T00:56:41.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-12T00:56:41.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-12T00:56:41.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-12T00:56:41.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-12T00:56:41.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-12T00:56:41.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-12T00:56:41.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-12T00:56:41.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-12T00:56:41.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-12T00:56:41.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-12T00:56:41.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-12T00:56:41.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-12T00:56:41.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-12T00:56:41.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-12T00:56:41.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-12T00:56:41.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-12T00:56:41.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-12T00:56:41.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-12T00:56:41.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-12T00:56:41.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-12T00:56:41.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-12T00:56:41.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-12T00:56:41.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-12T00:56:41.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-12T00:56:41.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-12T00:56:41.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-12T00:56:41.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-12T00:56:41.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-12T00:56:41.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-12T00:56:41.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-12T00:56:41.788 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:56:41.789 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.789 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.790 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.790 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.790 DEBUG:boto:Method: GET 2023-10-12T00:56:41.791 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.791 DEBUG:boto:Data: 2023-10-12T00:56:41.792 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.792 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.793 DEBUG:boto:Params: {} 2023-10-12T00:56:41.793 DEBUG:boto:Token: None 2023-10-12T00:56:41.794 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/?versions 2023-10-12T00:56:41.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg= 2023-10-12T00:56:41.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg=', 'Content-Length': '0'} 2023-10-12T00:56:41.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733d8990cd30b55d-0065274449-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.800 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-12T00:56:41.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-12T00:56:41.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-12T00:56:41.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-12T00:56:41.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-12T00:56:41.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-12T00:56:41.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-12T00:56:41.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-12T00:56:41.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-12T00:56:41.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-12T00:56:41.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-12T00:56:41.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-12T00:56:41.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-12T00:56:41.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-12T00:56:41.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-12T00:56:41.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-12T00:56:41.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-12T00:56:41.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-12T00:56:41.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-12T00:56:41.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-12T00:56:41.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-12T00:56:41.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-12T00:56:41.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-12T00:56:41.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-12T00:56:41.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-12T00:56:41.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-12T00:56:41.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-12T00:56:41.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-12T00:56:41.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-12T00:56:41.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-12T00:56:41.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-12T00:56:41.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-12T00:56:41.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-12T00:56:41.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-12T00:56:41.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-12T00:56:41.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-12T00:56:41.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-12T00:56:41.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-12T00:56:41.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-12T00:56:41.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-12T00:56:41.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-12T00:56:41.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-12T00:56:41.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-12T00:56:41.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-12T00:56:41.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-12T00:56:41.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-12T00:56:41.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-12T00:56:41.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-12T00:56:41.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-12T00:56:41.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-12T00:56:41.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-12T00:56:41.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-12T00:56:41.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-12T00:56:41.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-12T00:56:41.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-12T00:56:41.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-12T00:56:41.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-12T00:56:41.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-12T00:56:41.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-12T00:56:41.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-12T00:56:41.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-12T00:56:41.825 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.826 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.826 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.827 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.827 DEBUG:boto:Method: GET 2023-10-12T00:56:41.828 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.828 DEBUG:boto:Data: 2023-10-12T00:56:41.828 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.829 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.830 DEBUG:boto:Params: {} 2023-10-12T00:56:41.830 DEBUG:boto:Token: None 2023-10-12T00:56:41.831 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/?versions 2023-10-12T00:56:41.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg= 2023-10-12T00:56:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg=', 'Content-Length': '0'} 2023-10-12T00:56:41.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007757864e4908c0d9-0065274449-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.835 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.840 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.840 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.841 DEBUG:boto:path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.841 DEBUG:boto:auth_path=/wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.842 DEBUG:boto:Method: GET 2023-10-12T00:56:41.842 DEBUG:boto:Path: /wafmoa-58/?versions&max-keys=999 2023-10-12T00:56:41.843 DEBUG:boto:Data: 2023-10-12T00:56:41.843 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.844 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.844 DEBUG:boto:Params: {} 2023-10-12T00:56:41.845 DEBUG:boto:Token: None 2023-10-12T00:56:41.845 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/?versions 2023-10-12T00:56:41.846 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg= 2023-10-12T00:56:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8+u047LtJtGA+GZj0Ju80+7oEg=', 'Content-Length': '0'} 2023-10-12T00:56:41.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f232a2036b8e6a14-0065274449-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.851 DEBUG:boto:b'wafmoa-58999falseobj00000000000000000000nulltrue2023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:41.855 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.856 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.856 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.857 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.857 DEBUG:boto:Method: GET 2023-10-12T00:56:41.858 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.858 DEBUG:boto:Data: 2023-10-12T00:56:41.859 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.861 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.861 DEBUG:boto:Params: {} 2023-10-12T00:56:41.862 DEBUG:boto:Token: None 2023-10-12T00:56:41.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08= 2023-10-12T00:56:41.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08=', 'Content-Length': '0'} 2023-10-12T00:56:41.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f639b0a2b21a2cb-0065274449-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.867 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.867 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.869 DEBUG:boto:Method: GET 2023-10-12T00:56:41.869 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.870 DEBUG:boto:Data: 2023-10-12T00:56:41.870 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.871 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.872 DEBUG:boto:Params: {} 2023-10-12T00:56:41.872 DEBUG:boto:Token: None 2023-10-12T00:56:41.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08= 2023-10-12T00:56:41.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08=', 'Content-Length': '0'} 2023-10-12T00:56:41.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e838a3321be582c9-0065274449-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.877 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.877 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.878 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.878 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.879 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.879 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.879 DEBUG:boto:Data: 2023-10-12T00:56:41.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.881 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.881 DEBUG:boto:Params: {} 2023-10-12T00:56:41.882 DEBUG:boto:Token: None 2023-10-12T00:56:41.883 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8pLG1MKCBbNT1SfdO8PB0QbIn8= 2023-10-12T00:56:41.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8pLG1MKCBbNT1SfdO8PB0QbIn8=', 'Content-Length': '0'} 2023-10-12T00:56:41.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c25f5ec3004513-0065274449-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.886 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.887 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.887 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.888 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.888 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.888 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.889 DEBUG:boto:Data: 2023-10-12T00:56:41.889 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.890 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.890 DEBUG:boto:Params: {} 2023-10-12T00:56:41.891 DEBUG:boto:Token: None 2023-10-12T00:56:41.891 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.892 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8pLG1MKCBbNT1SfdO8PB0QbIn8= 2023-10-12T00:56:41.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8pLG1MKCBbNT1SfdO8PB0QbIn8=', 'Content-Length': '0'} 2023-10-12T00:56:41.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009671ced239bde635-0065274449-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:41.899 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.899 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.900 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.900 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.901 DEBUG:boto:Method: GET 2023-10-12T00:56:41.901 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.902 DEBUG:boto:Data: 2023-10-12T00:56:41.902 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.903 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.903 DEBUG:boto:Params: {} 2023-10-12T00:56:41.904 DEBUG:boto:Token: None 2023-10-12T00:56:41.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08= 2023-10-12T00:56:41.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08=', 'Content-Length': '0'} 2023-10-12T00:56:41.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003541aca6005593e8-0065274449-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.908 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.908 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.909 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.909 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.910 DEBUG:boto:Method: GET 2023-10-12T00:56:41.910 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.911 DEBUG:boto:Data: 2023-10-12T00:56:41.911 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.912 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.912 DEBUG:boto:Params: {} 2023-10-12T00:56:41.913 DEBUG:boto:Token: None 2023-10-12T00:56:41.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000?versionId=null 2023-10-12T00:56:41.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08= 2023-10-12T00:56:41.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VWzLlXpCLVG1GSHa3R5eEO7wS08=', 'Content-Length': '0'} 2023-10-12T00:56:41.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabea8ff6b9b5b47-0065274449-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.917 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.918 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.918 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.919 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.919 DEBUG:boto:Method: GET 2023-10-12T00:56:41.920 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.920 DEBUG:boto:Data: 2023-10-12T00:56:41.920 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.921 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.922 DEBUG:boto:Params: {} 2023-10-12T00:56:41.922 DEBUG:boto:Token: None 2023-10-12T00:56:41.923 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/ 2023-10-12T00:56:41.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qny1wylQqLpmyaReunExM3vvhXI= 2023-10-12T00:56:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qny1wylQqLpmyaReunExM3vvhXI=', 'Content-Length': '0'} 2023-10-12T00:56:41.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017118deda6b42a1c-0065274449-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.971 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.972 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:41.972 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:41.973 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.973 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.974 DEBUG:boto:Method: GET 2023-10-12T00:56:41.974 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:41.975 DEBUG:boto:Data: 2023-10-12T00:56:41.975 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.976 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.976 DEBUG:boto:Params: {} 2023-10-12T00:56:41.977 DEBUG:boto:Token: None 2023-10-12T00:56:41.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/ 2023-10-12T00:56:41.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qny1wylQqLpmyaReunExM3vvhXI= 2023-10-12T00:56:41.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qny1wylQqLpmyaReunExM3vvhXI=', 'Content-Length': '0'} 2023-10-12T00:56:41.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d32ef6f713f73181-0065274449-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.983 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:41.984 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.984 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.985 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.985 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.986 DEBUG:boto:Data: 2023-10-12T00:56:41.986 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:41.987 DEBUG:boto:Port: 8001 2023-10-12T00:56:41.987 DEBUG:boto:Params: {} 2023-10-12T00:56:41.988 DEBUG:boto:Token: None 2023-10-12T00:56:41.988 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QpFWKsJbv6GMhD8Bd7KCfjvu9S8= 2023-10-12T00:56:41.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QpFWKsJbv6GMhD8Bd7KCfjvu9S8=', 'Content-Length': '0'} 2023-10-12T00:56:41.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3159ba1b2e65c3f-0065274449-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:41.992 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.992 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.993 DEBUG:boto:Method: HEAD 2023-10-12T00:56:41.993 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.994 DEBUG:boto:Data: 2023-10-12T00:56:41.994 DEBUG:boto:Headers: {} 2023-10-12T00:56:41.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:41.995 DEBUG:boto:Port: 8000 2023-10-12T00:56:41.995 DEBUG:boto:Params: {} 2023-10-12T00:56:41.996 DEBUG:boto:Token: None 2023-10-12T00:56:41.996 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:41 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:41.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QpFWKsJbv6GMhD8Bd7KCfjvu9S8= 2023-10-12T00:56:41.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QpFWKsJbv6GMhD8Bd7KCfjvu9S8=', 'Content-Length': '0'} 2023-10-12T00:56:42.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9da48d7ec6368db-0065274449-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-12T00:56:42.001 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.002 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.002 DEBUG:boto:Method: GET 2023-10-12T00:56:42.003 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.003 DEBUG:boto:Data: 2023-10-12T00:56:42.003 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.004 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.005 DEBUG:boto:Params: {} 2023-10-12T00:56:42.005 DEBUG:boto:Token: None 2023-10-12T00:56:42.006 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wSaxFwD6Tiymf2FYIjpOaBbyamE= 2023-10-12T00:56:42.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wSaxFwD6Tiymf2FYIjpOaBbyamE=', 'Content-Length': '0'} 2023-10-12T00:56:42.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573b031ee9a92408-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.012 DEBUG:boto:path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.012 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.013 DEBUG:boto:Method: GET 2023-10-12T00:56:42.013 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.013 DEBUG:boto:Data: 2023-10-12T00:56:42.014 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.015 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.015 DEBUG:boto:Params: {} 2023-10-12T00:56:42.016 DEBUG:boto:Token: None 2023-10-12T00:56:42.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000000 2023-10-12T00:56:42.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wSaxFwD6Tiymf2FYIjpOaBbyamE= 2023-10-12T00:56:42.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wSaxFwD6Tiymf2FYIjpOaBbyamE=', 'Content-Length': '0'} 2023-10-12T00:56:42.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44e6ff461f99260-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.063 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.065 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.065 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.066 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.066 DEBUG:boto:Method: GET 2023-10-12T00:56:42.066 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.067 DEBUG:boto:Data: 2023-10-12T00:56:42.067 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.068 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.069 DEBUG:boto:Params: {} 2023-10-12T00:56:42.069 DEBUG:boto:Token: None 2023-10-12T00:56:42.070 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325dee3ca01089f2-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.075 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.075 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.076 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.076 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.077 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.077 DEBUG:boto:Method: GET 2023-10-12T00:56:42.078 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000000 2023-10-12T00:56:42.078 DEBUG:boto:Data: 2023-10-12T00:56:42.078 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.079 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.080 DEBUG:boto:Params: {} 2023-10-12T00:56:42.080 DEBUG:boto:Token: None 2023-10-12T00:56:42.081 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c08666846da410-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.087 DEBUG:boto:b'wafmoa-58obj000000000000000000001000falseobj000000000000000000002023-10-12T00:56:36.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:42.088 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.088 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.089 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.089 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.089 DEBUG:boto:Method: GET 2023-10-12T00:56:42.090 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.090 DEBUG:boto:Data: 2023-10-12T00:56:42.091 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.092 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.092 DEBUG:boto:Params: {} 2023-10-12T00:56:42.092 DEBUG:boto:Token: None 2023-10-12T00:56:42.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4= 2023-10-12T00:56:42.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4=', 'Content-Length': '0'} 2023-10-12T00:56:42.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b97dc770fcc35c-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.142 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.143 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.143 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.144 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.144 DEBUG:boto:Method: GET 2023-10-12T00:56:42.145 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.145 DEBUG:boto:Data: 2023-10-12T00:56:42.145 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.146 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.147 DEBUG:boto:Params: {} 2023-10-12T00:56:42.147 DEBUG:boto:Token: None 2023-10-12T00:56:42.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4= 2023-10-12T00:56:42.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4=', 'Content-Length': '0'} 2023-10-12T00:56:42.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbaf41e0af629fde-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.151 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.151 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.152 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.152 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.153 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.153 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.154 DEBUG:boto:Data: 2023-10-12T00:56:42.154 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.155 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.155 DEBUG:boto:Params: {} 2023-10-12T00:56:42.156 DEBUG:boto:Token: None 2023-10-12T00:56:42.156 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kJBTwNJ+PS1fTtRjP31O3MG3XmE= 2023-10-12T00:56:42.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kJBTwNJ+PS1fTtRjP31O3MG3XmE=', 'Content-Length': '0'} 2023-10-12T00:56:42.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bbc229f2a4f86d-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.160 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.161 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.161 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.161 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.162 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.162 DEBUG:boto:Data: 2023-10-12T00:56:42.163 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.164 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.164 DEBUG:boto:Params: {} 2023-10-12T00:56:42.164 DEBUG:boto:Token: None 2023-10-12T00:56:42.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kJBTwNJ+PS1fTtRjP31O3MG3XmE= 2023-10-12T00:56:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kJBTwNJ+PS1fTtRjP31O3MG3XmE=', 'Content-Length': '0'} 2023-10-12T00:56:42.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd6da0931af5a82-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:42.169 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.169 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.170 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.170 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.171 DEBUG:boto:Method: GET 2023-10-12T00:56:42.171 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.172 DEBUG:boto:Data: 2023-10-12T00:56:42.172 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.173 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.173 DEBUG:boto:Params: {} 2023-10-12T00:56:42.174 DEBUG:boto:Token: None 2023-10-12T00:56:42.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4= 2023-10-12T00:56:42.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4=', 'Content-Length': '0'} 2023-10-12T00:56:42.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d32ccf52fc3e4a-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.178 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.178 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.179 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.179 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.180 DEBUG:boto:Method: GET 2023-10-12T00:56:42.180 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.180 DEBUG:boto:Data: 2023-10-12T00:56:42.181 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.182 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.182 DEBUG:boto:Params: {} 2023-10-12T00:56:42.183 DEBUG:boto:Token: None 2023-10-12T00:56:42.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001?versionId=null 2023-10-12T00:56:42.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4= 2023-10-12T00:56:42.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8j0WVE7i9kQg2Ctr0WHyryfSKa4=', 'Content-Length': '0'} 2023-10-12T00:56:42.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e316774c23989e9-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.187 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.187 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.188 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.188 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.189 DEBUG:boto:Method: GET 2023-10-12T00:56:42.189 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.190 DEBUG:boto:Data: 2023-10-12T00:56:42.190 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.191 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.191 DEBUG:boto:Params: {} 2023-10-12T00:56:42.192 DEBUG:boto:Token: None 2023-10-12T00:56:42.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d45e106b85b074b-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.196 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.197 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.198 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.198 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.198 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.199 DEBUG:boto:Method: GET 2023-10-12T00:56:42.199 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.200 DEBUG:boto:Data: 2023-10-12T00:56:42.200 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.201 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.201 DEBUG:boto:Params: {} 2023-10-12T00:56:42.202 DEBUG:boto:Token: None 2023-10-12T00:56:42.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8408b33e9e736ed-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.250 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.251 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.251 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.252 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.252 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.253 DEBUG:boto:Data: 2023-10-12T00:56:42.253 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.254 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.254 DEBUG:boto:Params: {} 2023-10-12T00:56:42.255 DEBUG:boto:Token: None 2023-10-12T00:56:42.255 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0MERB03W54lT3RJWLOaYoQLhgfg= 2023-10-12T00:56:42.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0MERB03W54lT3RJWLOaYoQLhgfg=', 'Content-Length': '0'} 2023-10-12T00:56:42.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4123f12238fdb38-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.259 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.259 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.260 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.260 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.261 DEBUG:boto:Data: 2023-10-12T00:56:42.261 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.262 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.262 DEBUG:boto:Params: {} 2023-10-12T00:56:42.263 DEBUG:boto:Token: None 2023-10-12T00:56:42.263 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0MERB03W54lT3RJWLOaYoQLhgfg= 2023-10-12T00:56:42.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0MERB03W54lT3RJWLOaYoQLhgfg=', 'Content-Length': '0'} 2023-10-12T00:56:42.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008034e3601fa628cc-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-12T00:56:42.267 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.268 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.269 DEBUG:boto:Method: GET 2023-10-12T00:56:42.270 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.270 DEBUG:boto:Data: 2023-10-12T00:56:42.271 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.271 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.272 DEBUG:boto:Params: {} 2023-10-12T00:56:42.272 DEBUG:boto:Token: None 2023-10-12T00:56:42.273 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JYymwIh/6VvWcnaI/nErut4Oj74= 2023-10-12T00:56:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JYymwIh/6VvWcnaI/nErut4Oj74=', 'Content-Length': '0'} 2023-10-12T00:56:42.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07b158bff9ea746-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.318 DEBUG:boto:path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.319 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.319 DEBUG:boto:Method: GET 2023-10-12T00:56:42.320 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.320 DEBUG:boto:Data: 2023-10-12T00:56:42.321 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.322 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.322 DEBUG:boto:Params: {} 2023-10-12T00:56:42.323 DEBUG:boto:Token: None 2023-10-12T00:56:42.323 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000001 2023-10-12T00:56:42.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JYymwIh/6VvWcnaI/nErut4Oj74= 2023-10-12T00:56:42.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JYymwIh/6VvWcnaI/nErut4Oj74=', 'Content-Length': '0'} 2023-10-12T00:56:42.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c716afafa878f3ee-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.327 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.327 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.328 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.328 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.329 DEBUG:boto:Method: GET 2023-10-12T00:56:42.329 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.330 DEBUG:boto:Data: 2023-10-12T00:56:42.330 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.331 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.331 DEBUG:boto:Params: {} 2023-10-12T00:56:42.332 DEBUG:boto:Token: None 2023-10-12T00:56:42.332 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.333 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad3fa361a502dd94-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.336 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.337 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.337 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.338 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.338 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.339 DEBUG:boto:Method: GET 2023-10-12T00:56:42.339 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000001 2023-10-12T00:56:42.340 DEBUG:boto:Data: 2023-10-12T00:56:42.340 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.341 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.341 DEBUG:boto:Params: {} 2023-10-12T00:56:42.342 DEBUG:boto:Token: None 2023-10-12T00:56:42.342 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d61a1a3a3efaf083-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.348 DEBUG:boto:b'wafmoa-58obj000000000000000000011000falseobj000000000000000000012023-10-12T00:56:37.315Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:42.349 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.349 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.350 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.350 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.351 DEBUG:boto:Method: GET 2023-10-12T00:56:42.351 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.351 DEBUG:boto:Data: 2023-10-12T00:56:42.352 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.353 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.353 DEBUG:boto:Params: {} 2023-10-12T00:56:42.354 DEBUG:boto:Token: None 2023-10-12T00:56:42.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc= 2023-10-12T00:56:42.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc=', 'Content-Length': '0'} 2023-10-12T00:56:42.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000672c8ff9208fbb0c-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.358 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.359 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.359 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.360 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.360 DEBUG:boto:Method: GET 2023-10-12T00:56:42.361 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.361 DEBUG:boto:Data: 2023-10-12T00:56:42.362 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.362 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.363 DEBUG:boto:Params: {} 2023-10-12T00:56:42.363 DEBUG:boto:Token: None 2023-10-12T00:56:42.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc= 2023-10-12T00:56:42.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc=', 'Content-Length': '0'} 2023-10-12T00:56:42.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b84dfc8f327a27-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.414 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.415 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.415 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.416 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.416 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.417 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.417 DEBUG:boto:Data: 2023-10-12T00:56:42.418 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.419 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.419 DEBUG:boto:Params: {} 2023-10-12T00:56:42.419 DEBUG:boto:Token: None 2023-10-12T00:56:42.420 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qPZFgf9CjxAfj5vvUCDv07vusmg= 2023-10-12T00:56:42.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qPZFgf9CjxAfj5vvUCDv07vusmg=', 'Content-Length': '0'} 2023-10-12T00:56:42.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aafd2092f4d96012-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.423 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.424 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.424 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.425 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.425 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.426 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.426 DEBUG:boto:Data: 2023-10-12T00:56:42.426 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.427 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.428 DEBUG:boto:Params: {} 2023-10-12T00:56:42.428 DEBUG:boto:Token: None 2023-10-12T00:56:42.429 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qPZFgf9CjxAfj5vvUCDv07vusmg= 2023-10-12T00:56:42.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qPZFgf9CjxAfj5vvUCDv07vusmg=', 'Content-Length': '0'} 2023-10-12T00:56:42.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332e1c14e3b394dd-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:42.432 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.433 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.433 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.434 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.434 DEBUG:boto:Method: GET 2023-10-12T00:56:42.435 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.435 DEBUG:boto:Data: 2023-10-12T00:56:42.435 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.436 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.437 DEBUG:boto:Params: {} 2023-10-12T00:56:42.437 DEBUG:boto:Token: None 2023-10-12T00:56:42.438 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc= 2023-10-12T00:56:42.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc=', 'Content-Length': '0'} 2023-10-12T00:56:42.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051620557af12abdd-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.482 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.483 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.484 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.484 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.484 DEBUG:boto:Method: GET 2023-10-12T00:56:42.485 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.485 DEBUG:boto:Data: 2023-10-12T00:56:42.486 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.487 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.487 DEBUG:boto:Params: {} 2023-10-12T00:56:42.488 DEBUG:boto:Token: None 2023-10-12T00:56:42.488 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002?versionId=null 2023-10-12T00:56:42.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc= 2023-10-12T00:56:42.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HR4P6qkItitt/Ak5Y0gdSgWriRc=', 'Content-Length': '0'} 2023-10-12T00:56:42.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223fa008f5494261-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.492 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.493 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.493 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.493 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.494 DEBUG:boto:Method: GET 2023-10-12T00:56:42.494 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.495 DEBUG:boto:Data: 2023-10-12T00:56:42.495 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.496 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.497 DEBUG:boto:Params: {} 2023-10-12T00:56:42.497 DEBUG:boto:Token: None 2023-10-12T00:56:42.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a4f0424a9fcf7f-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.501 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.502 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.503 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.503 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.503 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.504 DEBUG:boto:Method: GET 2023-10-12T00:56:42.504 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.505 DEBUG:boto:Data: 2023-10-12T00:56:42.505 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.506 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.506 DEBUG:boto:Params: {} 2023-10-12T00:56:42.507 DEBUG:boto:Token: None 2023-10-12T00:56:42.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d548f1bbfd65b3-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.513 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.513 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.514 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.514 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.515 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.515 DEBUG:boto:Data: 2023-10-12T00:56:42.515 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.516 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.517 DEBUG:boto:Params: {} 2023-10-12T00:56:42.517 DEBUG:boto:Token: None 2023-10-12T00:56:42.518 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/SZ1hgDXMZmULf2TYCwaaj9sG28= 2023-10-12T00:56:42.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/SZ1hgDXMZmULf2TYCwaaj9sG28=', 'Content-Length': '0'} 2023-10-12T00:56:42.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035021347627bbb9-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.521 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.522 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.522 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.522 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.523 DEBUG:boto:Data: 2023-10-12T00:56:42.523 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.524 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.524 DEBUG:boto:Params: {} 2023-10-12T00:56:42.525 DEBUG:boto:Token: None 2023-10-12T00:56:42.525 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/SZ1hgDXMZmULf2TYCwaaj9sG28= 2023-10-12T00:56:42.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/SZ1hgDXMZmULf2TYCwaaj9sG28=', 'Content-Length': '0'} 2023-10-12T00:56:42.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefe608660f09212-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-12T00:56:42.529 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.529 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.530 DEBUG:boto:Method: GET 2023-10-12T00:56:42.530 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.531 DEBUG:boto:Data: 2023-10-12T00:56:42.531 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.532 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.533 DEBUG:boto:Params: {} 2023-10-12T00:56:42.533 DEBUG:boto:Token: None 2023-10-12T00:56:42.534 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zmQ37YYnUKMmhyryCqlM7UsDrtk= 2023-10-12T00:56:42.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zmQ37YYnUKMmhyryCqlM7UsDrtk=', 'Content-Length': '0'} 2023-10-12T00:56:42.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d594c4b8ddd0cda-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.537 DEBUG:boto:path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.538 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.538 DEBUG:boto:Method: GET 2023-10-12T00:56:42.539 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.539 DEBUG:boto:Data: 2023-10-12T00:56:42.539 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.540 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.541 DEBUG:boto:Params: {} 2023-10-12T00:56:42.541 DEBUG:boto:Token: None 2023-10-12T00:56:42.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000002 2023-10-12T00:56:42.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zmQ37YYnUKMmhyryCqlM7UsDrtk= 2023-10-12T00:56:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zmQ37YYnUKMmhyryCqlM7UsDrtk=', 'Content-Length': '0'} 2023-10-12T00:56:42.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d35a58f453c64b8-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.545 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.546 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.546 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.547 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.547 DEBUG:boto:Method: GET 2023-10-12T00:56:42.548 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.548 DEBUG:boto:Data: 2023-10-12T00:56:42.548 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.549 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.550 DEBUG:boto:Params: {} 2023-10-12T00:56:42.550 DEBUG:boto:Token: None 2023-10-12T00:56:42.551 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678126f382458f4c-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.555 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.555 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.556 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.556 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.557 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.557 DEBUG:boto:Method: GET 2023-10-12T00:56:42.558 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000002 2023-10-12T00:56:42.558 DEBUG:boto:Data: 2023-10-12T00:56:42.558 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.559 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.560 DEBUG:boto:Params: {} 2023-10-12T00:56:42.560 DEBUG:boto:Token: None 2023-10-12T00:56:42.561 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.561 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a124682735d4c2c0-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.566 DEBUG:boto:b'wafmoa-58obj000000000000000000021000falseobj000000000000000000022023-10-12T00:56:37.236Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:42.567 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.567 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.568 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.568 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.569 DEBUG:boto:Method: GET 2023-10-12T00:56:42.569 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.570 DEBUG:boto:Data: 2023-10-12T00:56:42.570 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.571 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.571 DEBUG:boto:Params: {} 2023-10-12T00:56:42.572 DEBUG:boto:Token: None 2023-10-12T00:56:42.572 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.573 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE= 2023-10-12T00:56:42.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE=', 'Content-Length': '0'} 2023-10-12T00:56:42.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cb3e37979679ec5-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.577 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.577 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.577 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.578 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.578 DEBUG:boto:Method: GET 2023-10-12T00:56:42.579 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.579 DEBUG:boto:Data: 2023-10-12T00:56:42.580 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.580 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.581 DEBUG:boto:Params: {} 2023-10-12T00:56:42.581 DEBUG:boto:Token: None 2023-10-12T00:56:42.582 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.582 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE= 2023-10-12T00:56:42.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE=', 'Content-Length': '0'} 2023-10-12T00:56:42.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31b7f9c5843c43d-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.585 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.586 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.586 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.587 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.587 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.588 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.588 DEBUG:boto:Data: 2023-10-12T00:56:42.588 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.589 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.590 DEBUG:boto:Params: {} 2023-10-12T00:56:42.590 DEBUG:boto:Token: None 2023-10-12T00:56:42.590 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAc2/8Z/5OTdxevM7iXmJe7J6Og= 2023-10-12T00:56:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAc2/8Z/5OTdxevM7iXmJe7J6Og=', 'Content-Length': '0'} 2023-10-12T00:56:42.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee59b3dc4cfb99dc-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.594 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.595 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.595 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.596 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.596 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.596 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.597 DEBUG:boto:Data: 2023-10-12T00:56:42.597 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.598 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.598 DEBUG:boto:Params: {} 2023-10-12T00:56:42.599 DEBUG:boto:Token: None 2023-10-12T00:56:42.599 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAc2/8Z/5OTdxevM7iXmJe7J6Og= 2023-10-12T00:56:42.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAc2/8Z/5OTdxevM7iXmJe7J6Og=', 'Content-Length': '0'} 2023-10-12T00:56:42.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f12127b1ab0a6d9-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:42.603 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.604 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.604 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.604 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.605 DEBUG:boto:Method: GET 2023-10-12T00:56:42.605 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.606 DEBUG:boto:Data: 2023-10-12T00:56:42.606 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.607 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.608 DEBUG:boto:Params: {} 2023-10-12T00:56:42.608 DEBUG:boto:Token: None 2023-10-12T00:56:42.609 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE= 2023-10-12T00:56:42.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE=', 'Content-Length': '0'} 2023-10-12T00:56:42.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcee10d26f6f6354-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.654 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.655 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.655 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.656 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.656 DEBUG:boto:Method: GET 2023-10-12T00:56:42.657 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.657 DEBUG:boto:Data: 2023-10-12T00:56:42.658 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.658 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.659 DEBUG:boto:Params: {} 2023-10-12T00:56:42.659 DEBUG:boto:Token: None 2023-10-12T00:56:42.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003?versionId=null 2023-10-12T00:56:42.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE= 2023-10-12T00:56:42.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/P7dCu92KheNRRVW4LkutzD2+ZE=', 'Content-Length': '0'} 2023-10-12T00:56:42.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e79c7178014e3ca-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.663 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.664 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.664 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.665 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.665 DEBUG:boto:Method: GET 2023-10-12T00:56:42.666 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.666 DEBUG:boto:Data: 2023-10-12T00:56:42.666 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.667 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.668 DEBUG:boto:Params: {} 2023-10-12T00:56:42.671 DEBUG:boto:Token: None 2023-10-12T00:56:42.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b051b834c8e5205e-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.675 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.676 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.676 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.677 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.677 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.678 DEBUG:boto:Method: GET 2023-10-12T00:56:42.678 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.679 DEBUG:boto:Data: 2023-10-12T00:56:42.679 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.680 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.680 DEBUG:boto:Params: {} 2023-10-12T00:56:42.681 DEBUG:boto:Token: None 2023-10-12T00:56:42.682 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0853d95d50aa3ad-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.730 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.731 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.732 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.732 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.733 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.733 DEBUG:boto:Data: 2023-10-12T00:56:42.733 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.734 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.735 DEBUG:boto:Params: {} 2023-10-12T00:56:42.735 DEBUG:boto:Token: None 2023-10-12T00:56:42.736 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tUHDfSh3UvYYpwF8ZFBaxXJ+h8I= 2023-10-12T00:56:42.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tUHDfSh3UvYYpwF8ZFBaxXJ+h8I=', 'Content-Length': '0'} 2023-10-12T00:56:42.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca37b1e774264354-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.739 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.740 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.740 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.741 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.741 DEBUG:boto:Data: 2023-10-12T00:56:42.741 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.742 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.743 DEBUG:boto:Params: {} 2023-10-12T00:56:42.743 DEBUG:boto:Token: None 2023-10-12T00:56:42.744 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tUHDfSh3UvYYpwF8ZFBaxXJ+h8I= 2023-10-12T00:56:42.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tUHDfSh3UvYYpwF8ZFBaxXJ+h8I=', 'Content-Length': '0'} 2023-10-12T00:56:42.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdeec16233fc700c-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-12T00:56:42.747 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.748 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.748 DEBUG:boto:Method: GET 2023-10-12T00:56:42.749 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.749 DEBUG:boto:Data: 2023-10-12T00:56:42.750 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.751 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.751 DEBUG:boto:Params: {} 2023-10-12T00:56:42.752 DEBUG:boto:Token: None 2023-10-12T00:56:42.752 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bA91FBuhzoGSZihDFu8uIWR6ppc= 2023-10-12T00:56:42.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bA91FBuhzoGSZihDFu8uIWR6ppc=', 'Content-Length': '0'} 2023-10-12T00:56:42.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e288fea797dbfa-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.755 DEBUG:boto:path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.756 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.756 DEBUG:boto:Method: GET 2023-10-12T00:56:42.757 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.757 DEBUG:boto:Data: 2023-10-12T00:56:42.758 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.759 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.759 DEBUG:boto:Params: {} 2023-10-12T00:56:42.759 DEBUG:boto:Token: None 2023-10-12T00:56:42.760 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000003 2023-10-12T00:56:42.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bA91FBuhzoGSZihDFu8uIWR6ppc= 2023-10-12T00:56:42.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bA91FBuhzoGSZihDFu8uIWR6ppc=', 'Content-Length': '0'} 2023-10-12T00:56:42.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc7ffd929bca08b-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.767 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.768 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.768 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.769 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.769 DEBUG:boto:Method: GET 2023-10-12T00:56:42.769 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.770 DEBUG:boto:Data: 2023-10-12T00:56:42.770 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.771 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.772 DEBUG:boto:Params: {} 2023-10-12T00:56:42.772 DEBUG:boto:Token: None 2023-10-12T00:56:42.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.773 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ac020470c34bf8-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.822 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.823 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.824 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.824 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.825 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.825 DEBUG:boto:Method: GET 2023-10-12T00:56:42.826 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000003 2023-10-12T00:56:42.827 DEBUG:boto:Data: 2023-10-12T00:56:42.827 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.828 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.828 DEBUG:boto:Params: {} 2023-10-12T00:56:42.829 DEBUG:boto:Token: None 2023-10-12T00:56:42.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006869c4d1ce4bcc66-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.835 DEBUG:boto:b'wafmoa-58obj000000000000000000031000falseobj000000000000000000032023-10-12T00:56:34.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:42.836 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.837 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.837 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.838 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.838 DEBUG:boto:Method: GET 2023-10-12T00:56:42.839 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.839 DEBUG:boto:Data: 2023-10-12T00:56:42.839 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.840 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.840 DEBUG:boto:Params: {} 2023-10-12T00:56:42.841 DEBUG:boto:Token: None 2023-10-12T00:56:42.841 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc= 2023-10-12T00:56:42.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc=', 'Content-Length': '0'} 2023-10-12T00:56:42.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082182cb1f9de5270-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.846 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.847 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.847 DEBUG:boto:Method: GET 2023-10-12T00:56:42.848 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.848 DEBUG:boto:Data: 2023-10-12T00:56:42.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.849 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.850 DEBUG:boto:Params: {} 2023-10-12T00:56:42.850 DEBUG:boto:Token: None 2023-10-12T00:56:42.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc= 2023-10-12T00:56:42.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc=', 'Content-Length': '0'} 2023-10-12T00:56:42.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca28005ce7e6f88-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.895 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.895 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.896 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.896 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.897 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.897 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.897 DEBUG:boto:Data: 2023-10-12T00:56:42.898 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.899 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.899 DEBUG:boto:Params: {} 2023-10-12T00:56:42.899 DEBUG:boto:Token: None 2023-10-12T00:56:42.900 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SPVxj4O0PsxrZ45gTrrDrskdxNc= 2023-10-12T00:56:42.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SPVxj4O0PsxrZ45gTrrDrskdxNc=', 'Content-Length': '0'} 2023-10-12T00:56:42.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d958d5bd4ba792-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.903 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.904 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.904 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.905 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.905 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.905 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.906 DEBUG:boto:Data: 2023-10-12T00:56:42.906 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.907 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.907 DEBUG:boto:Params: {} 2023-10-12T00:56:42.908 DEBUG:boto:Token: None 2023-10-12T00:56:42.908 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.909 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SPVxj4O0PsxrZ45gTrrDrskdxNc= 2023-10-12T00:56:42.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SPVxj4O0PsxrZ45gTrrDrskdxNc=', 'Content-Length': '0'} 2023-10-12T00:56:42.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048cd348b8d35e418-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:42.912 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.912 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.913 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.913 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.913 DEBUG:boto:Method: GET 2023-10-12T00:56:42.914 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.914 DEBUG:boto:Data: 2023-10-12T00:56:42.915 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.915 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.916 DEBUG:boto:Params: {} 2023-10-12T00:56:42.916 DEBUG:boto:Token: None 2023-10-12T00:56:42.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc= 2023-10-12T00:56:42.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc=', 'Content-Length': '0'} 2023-10-12T00:56:42.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007464ecc04a54b0c7-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.920 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.920 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.921 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.921 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.921 DEBUG:boto:Method: GET 2023-10-12T00:56:42.922 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.922 DEBUG:boto:Data: 2023-10-12T00:56:42.923 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.923 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.924 DEBUG:boto:Params: {} 2023-10-12T00:56:42.924 DEBUG:boto:Token: None 2023-10-12T00:56:42.925 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004?versionId=null 2023-10-12T00:56:42.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc= 2023-10-12T00:56:42.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pVjIrYkPTe4a48+wPcZuH5CbYc=', 'Content-Length': '0'} 2023-10-12T00:56:42.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d411da076fbe46af-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.928 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.928 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.929 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.929 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.929 DEBUG:boto:Method: GET 2023-10-12T00:56:42.930 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.930 DEBUG:boto:Data: 2023-10-12T00:56:42.931 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.932 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.932 DEBUG:boto:Params: {} 2023-10-12T00:56:42.932 DEBUG:boto:Token: None 2023-10-12T00:56:42.933 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004432af59a662c8eb-006527444a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.937 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.937 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:42.938 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:42.938 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.939 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.939 DEBUG:boto:Method: GET 2023-10-12T00:56:42.939 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:42.940 DEBUG:boto:Data: 2023-10-12T00:56:42.940 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.941 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.941 DEBUG:boto:Params: {} 2023-10-12T00:56:42.942 DEBUG:boto:Token: None 2023-10-12T00:56:42.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/ 2023-10-12T00:56:42.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk= 2023-10-12T00:56:42.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q6KU8jjI7hY66LV7j1NVWcUraKk=', 'Content-Length': '0'} 2023-10-12T00:56:42.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc5c7fca0824c0e9-006527444a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.947 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:42.948 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.948 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.949 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.949 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.949 DEBUG:boto:Data: 2023-10-12T00:56:42.950 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.951 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.951 DEBUG:boto:Params: {} 2023-10-12T00:56:42.951 DEBUG:boto:Token: None 2023-10-12T00:56:42.952 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WrGimtX6RQlqedIAuL6PmXrwVZs= 2023-10-12T00:56:42.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WrGimtX6RQlqedIAuL6PmXrwVZs=', 'Content-Length': '0'} 2023-10-12T00:56:42.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001892a734a5f4cf2d-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.955 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.956 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.956 DEBUG:boto:Method: HEAD 2023-10-12T00:56:42.956 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.957 DEBUG:boto:Data: 2023-10-12T00:56:42.957 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:42.958 DEBUG:boto:Port: 8000 2023-10-12T00:56:42.959 DEBUG:boto:Params: {} 2023-10-12T00:56:42.959 DEBUG:boto:Token: None 2023-10-12T00:56:42.959 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WrGimtX6RQlqedIAuL6PmXrwVZs= 2023-10-12T00:56:42.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WrGimtX6RQlqedIAuL6PmXrwVZs=', 'Content-Length': '0'} 2023-10-12T00:56:42.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a444384e648dfa0-006527444a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:42.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-12T00:56:42.963 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.964 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.964 DEBUG:boto:Method: GET 2023-10-12T00:56:42.964 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.965 DEBUG:boto:Data: 2023-10-12T00:56:42.965 DEBUG:boto:Headers: {} 2023-10-12T00:56:42.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:42.966 DEBUG:boto:Port: 8001 2023-10-12T00:56:42.966 DEBUG:boto:Params: {} 2023-10-12T00:56:42.967 DEBUG:boto:Token: None 2023-10-12T00:56:42.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:42 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:42.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pS0ci1sU9fDwvBH4kh+k9lIGh7I= 2023-10-12T00:56:42.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pS0ci1sU9fDwvBH4kh+k9lIGh7I=', 'Content-Length': '0'} 2023-10-12T00:56:42.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9480840cb30bd8-006527444a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.010 DEBUG:boto:path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:43.011 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000004 2023-10-12T00:56:43.012 DEBUG:boto:Method: GET 2023-10-12T00:56:43.012 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:43.012 DEBUG:boto:Data: 2023-10-12T00:56:43.013 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.014 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.014 DEBUG:boto:Params: {} 2023-10-12T00:56:43.015 DEBUG:boto:Token: None 2023-10-12T00:56:43.015 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000004 2023-10-12T00:56:43.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mpmEc+y1h1bQ6TMTOw7x73xbwJI= 2023-10-12T00:56:43.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mpmEc+y1h1bQ6TMTOw7x73xbwJI=', 'Content-Length': '0'} 2023-10-12T00:56:43.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2de3e41ee55360-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.019 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.019 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.020 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.020 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.021 DEBUG:boto:Method: GET 2023-10-12T00:56:43.021 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.022 DEBUG:boto:Data: 2023-10-12T00:56:43.022 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.023 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.023 DEBUG:boto:Params: {} 2023-10-12T00:56:43.024 DEBUG:boto:Token: None 2023-10-12T00:56:43.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c62be401be81a0a-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.029 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.029 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.030 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.030 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.031 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.031 DEBUG:boto:Method: GET 2023-10-12T00:56:43.032 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000004 2023-10-12T00:56:43.032 DEBUG:boto:Data: 2023-10-12T00:56:43.032 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.033 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.034 DEBUG:boto:Params: {} 2023-10-12T00:56:43.034 DEBUG:boto:Token: None 2023-10-12T00:56:43.035 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.035 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003181666006cb6dd3-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.040 DEBUG:boto:b'wafmoa-58obj000000000000000000041000falseobj000000000000000000042023-10-12T00:56:36.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:43.041 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.041 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.042 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.042 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.043 DEBUG:boto:Method: GET 2023-10-12T00:56:43.043 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.044 DEBUG:boto:Data: 2023-10-12T00:56:43.044 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.045 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.045 DEBUG:boto:Params: {} 2023-10-12T00:56:43.046 DEBUG:boto:Token: None 2023-10-12T00:56:43.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8= 2023-10-12T00:56:43.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8=', 'Content-Length': '0'} 2023-10-12T00:56:43.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d7c62e72c916d5-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.050 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.051 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.051 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.052 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.052 DEBUG:boto:Method: GET 2023-10-12T00:56:43.053 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.053 DEBUG:boto:Data: 2023-10-12T00:56:43.054 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.054 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.055 DEBUG:boto:Params: {} 2023-10-12T00:56:43.055 DEBUG:boto:Token: None 2023-10-12T00:56:43.056 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8= 2023-10-12T00:56:43.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8=', 'Content-Length': '0'} 2023-10-12T00:56:43.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae019f1920ec395-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.059 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.060 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.060 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.061 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.061 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.061 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.062 DEBUG:boto:Data: 2023-10-12T00:56:43.062 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.063 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.063 DEBUG:boto:Params: {} 2023-10-12T00:56:43.064 DEBUG:boto:Token: None 2023-10-12T00:56:43.064 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pxEVg739Ksts8CUi+ig9Ge6vls= 2023-10-12T00:56:43.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pxEVg739Ksts8CUi+ig9Ge6vls=', 'Content-Length': '0'} 2023-10-12T00:56:43.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9495d18ba051ae-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.068 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.068 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.069 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.069 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.069 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.070 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.070 DEBUG:boto:Data: 2023-10-12T00:56:43.071 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.072 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.072 DEBUG:boto:Params: {} 2023-10-12T00:56:43.072 DEBUG:boto:Token: None 2023-10-12T00:56:43.073 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pxEVg739Ksts8CUi+ig9Ge6vls= 2023-10-12T00:56:43.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pxEVg739Ksts8CUi+ig9Ge6vls=', 'Content-Length': '0'} 2023-10-12T00:56:43.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce11c640a9ac6c5-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:43.077 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.077 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.078 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.078 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.078 DEBUG:boto:Method: GET 2023-10-12T00:56:43.079 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.079 DEBUG:boto:Data: 2023-10-12T00:56:43.080 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.081 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.081 DEBUG:boto:Params: {} 2023-10-12T00:56:43.081 DEBUG:boto:Token: None 2023-10-12T00:56:43.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8= 2023-10-12T00:56:43.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8=', 'Content-Length': '0'} 2023-10-12T00:56:43.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6d2c626b45ebd0-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.085 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.086 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.086 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.087 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.087 DEBUG:boto:Method: GET 2023-10-12T00:56:43.087 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.088 DEBUG:boto:Data: 2023-10-12T00:56:43.088 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.089 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.090 DEBUG:boto:Params: {} 2023-10-12T00:56:43.090 DEBUG:boto:Token: None 2023-10-12T00:56:43.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005?versionId=null 2023-10-12T00:56:43.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8= 2023-10-12T00:56:43.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B+z9Y1s3XRtClLpMBzT3CIlKVF8=', 'Content-Length': '0'} 2023-10-12T00:56:43.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38fa6d269e7bc50-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.138 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.139 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.139 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.140 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.140 DEBUG:boto:Method: GET 2023-10-12T00:56:43.141 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.141 DEBUG:boto:Data: 2023-10-12T00:56:43.142 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.142 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.143 DEBUG:boto:Params: {} 2023-10-12T00:56:43.143 DEBUG:boto:Token: None 2023-10-12T00:56:43.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eaa6c98b2d72756-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.148 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.149 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.149 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.150 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.150 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.151 DEBUG:boto:Method: GET 2023-10-12T00:56:43.151 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.151 DEBUG:boto:Data: 2023-10-12T00:56:43.152 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.153 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.153 DEBUG:boto:Params: {} 2023-10-12T00:56:43.153 DEBUG:boto:Token: None 2023-10-12T00:56:43.154 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.154 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f72d65b8ae30e227-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.159 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.160 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.161 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.161 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.162 DEBUG:boto:Data: 2023-10-12T00:56:43.162 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.163 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.164 DEBUG:boto:Params: {} 2023-10-12T00:56:43.164 DEBUG:boto:Token: None 2023-10-12T00:56:43.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iEUOzItIROzYG2VreX1dUeb24rA= 2023-10-12T00:56:43.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iEUOzItIROzYG2VreX1dUeb24rA=', 'Content-Length': '0'} 2023-10-12T00:56:43.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7979c20e6bb722b-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.168 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.169 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.169 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.170 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.170 DEBUG:boto:Data: 2023-10-12T00:56:43.171 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.172 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.172 DEBUG:boto:Params: {} 2023-10-12T00:56:43.172 DEBUG:boto:Token: None 2023-10-12T00:56:43.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iEUOzItIROzYG2VreX1dUeb24rA= 2023-10-12T00:56:43.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iEUOzItIROzYG2VreX1dUeb24rA=', 'Content-Length': '0'} 2023-10-12T00:56:43.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec09ef16d03ba5e6-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-12T00:56:43.177 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.178 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.178 DEBUG:boto:Method: GET 2023-10-12T00:56:43.179 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.179 DEBUG:boto:Data: 2023-10-12T00:56:43.179 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.180 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.181 DEBUG:boto:Params: {} 2023-10-12T00:56:43.181 DEBUG:boto:Token: None 2023-10-12T00:56:43.182 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oTpb8E/CHWNekZoSI4B80VZ0RG8= 2023-10-12T00:56:43.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oTpb8E/CHWNekZoSI4B80VZ0RG8=', 'Content-Length': '0'} 2023-10-12T00:56:43.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a42a971f3fc169a-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.185 DEBUG:boto:path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.186 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.186 DEBUG:boto:Method: GET 2023-10-12T00:56:43.187 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.187 DEBUG:boto:Data: 2023-10-12T00:56:43.187 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.188 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.189 DEBUG:boto:Params: {} 2023-10-12T00:56:43.189 DEBUG:boto:Token: None 2023-10-12T00:56:43.190 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000005 2023-10-12T00:56:43.190 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oTpb8E/CHWNekZoSI4B80VZ0RG8= 2023-10-12T00:56:43.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oTpb8E/CHWNekZoSI4B80VZ0RG8=', 'Content-Length': '0'} 2023-10-12T00:56:43.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb42e4973c65512-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.194 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.194 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.195 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.195 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.195 DEBUG:boto:Method: GET 2023-10-12T00:56:43.196 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.196 DEBUG:boto:Data: 2023-10-12T00:56:43.197 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.198 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.198 DEBUG:boto:Params: {} 2023-10-12T00:56:43.199 DEBUG:boto:Token: None 2023-10-12T00:56:43.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7a0a80aca77ea3-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.203 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.204 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.204 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.205 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.205 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.205 DEBUG:boto:Method: GET 2023-10-12T00:56:43.206 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000005 2023-10-12T00:56:43.206 DEBUG:boto:Data: 2023-10-12T00:56:43.207 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.207 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.208 DEBUG:boto:Params: {} 2023-10-12T00:56:43.208 DEBUG:boto:Token: None 2023-10-12T00:56:43.209 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91a1f491f9316ed-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.214 DEBUG:boto:b'wafmoa-58obj000000000000000000051000falseobj000000000000000000052023-10-12T00:56:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:43.216 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.216 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.216 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.217 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.217 DEBUG:boto:Method: GET 2023-10-12T00:56:43.218 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.218 DEBUG:boto:Data: 2023-10-12T00:56:43.219 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.219 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.220 DEBUG:boto:Params: {} 2023-10-12T00:56:43.220 DEBUG:boto:Token: None 2023-10-12T00:56:43.221 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4= 2023-10-12T00:56:43.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4=', 'Content-Length': '0'} 2023-10-12T00:56:43.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bac90284da8d06e-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.266 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.267 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.267 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.268 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.268 DEBUG:boto:Method: GET 2023-10-12T00:56:43.269 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.269 DEBUG:boto:Data: 2023-10-12T00:56:43.270 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.271 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.271 DEBUG:boto:Params: {} 2023-10-12T00:56:43.272 DEBUG:boto:Token: None 2023-10-12T00:56:43.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4= 2023-10-12T00:56:43.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4=', 'Content-Length': '0'} 2023-10-12T00:56:43.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ab5e31f6c0289f-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.277 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.277 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.278 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.279 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.280 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.280 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.281 DEBUG:boto:Data: 2023-10-12T00:56:43.282 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.284 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.284 DEBUG:boto:Params: {} 2023-10-12T00:56:43.285 DEBUG:boto:Token: None 2023-10-12T00:56:43.286 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tlUrmC/wGU4rDFF+FaU9EBaw63A= 2023-10-12T00:56:43.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tlUrmC/wGU4rDFF+FaU9EBaw63A=', 'Content-Length': '0'} 2023-10-12T00:56:43.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324ec31658999ee7-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.290 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.290 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.291 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.291 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.292 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.292 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.292 DEBUG:boto:Data: 2023-10-12T00:56:43.293 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.294 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.294 DEBUG:boto:Params: {} 2023-10-12T00:56:43.295 DEBUG:boto:Token: None 2023-10-12T00:56:43.295 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tlUrmC/wGU4rDFF+FaU9EBaw63A= 2023-10-12T00:56:43.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tlUrmC/wGU4rDFF+FaU9EBaw63A=', 'Content-Length': '0'} 2023-10-12T00:56:43.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1826263a817b91-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:43.302 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.302 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.303 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.303 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.304 DEBUG:boto:Method: GET 2023-10-12T00:56:43.305 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.305 DEBUG:boto:Data: 2023-10-12T00:56:43.306 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.307 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.307 DEBUG:boto:Params: {} 2023-10-12T00:56:43.308 DEBUG:boto:Token: None 2023-10-12T00:56:43.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4= 2023-10-12T00:56:43.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4=', 'Content-Length': '0'} 2023-10-12T00:56:43.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a356af6312d11059-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.312 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.313 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.313 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.313 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.314 DEBUG:boto:Method: GET 2023-10-12T00:56:43.314 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.315 DEBUG:boto:Data: 2023-10-12T00:56:43.316 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.318 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.319 DEBUG:boto:Params: {} 2023-10-12T00:56:43.319 DEBUG:boto:Token: None 2023-10-12T00:56:43.320 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006?versionId=null 2023-10-12T00:56:43.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4= 2023-10-12T00:56:43.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgULiLveBUVnP84Jrf/2ojp/zq4=', 'Content-Length': '0'} 2023-10-12T00:56:43.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a2024ff604504e-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.325 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.326 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.326 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.327 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.327 DEBUG:boto:Method: GET 2023-10-12T00:56:43.328 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.328 DEBUG:boto:Data: 2023-10-12T00:56:43.329 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.330 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.330 DEBUG:boto:Params: {} 2023-10-12T00:56:43.331 DEBUG:boto:Token: None 2023-10-12T00:56:43.331 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078dfa9eee49e3050-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.336 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.336 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.337 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.337 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.338 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.338 DEBUG:boto:Method: GET 2023-10-12T00:56:43.339 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.340 DEBUG:boto:Data: 2023-10-12T00:56:43.340 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.341 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.342 DEBUG:boto:Params: {} 2023-10-12T00:56:43.342 DEBUG:boto:Token: None 2023-10-12T00:56:43.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a46124af39f0ddf-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.349 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.349 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.350 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.351 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.351 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.352 DEBUG:boto:Data: 2023-10-12T00:56:43.352 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.353 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.353 DEBUG:boto:Params: {} 2023-10-12T00:56:43.354 DEBUG:boto:Token: None 2023-10-12T00:56:43.354 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8t5vq/dbpO9+VfuO2AZHbCwejFw= 2023-10-12T00:56:43.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8t5vq/dbpO9+VfuO2AZHbCwejFw=', 'Content-Length': '0'} 2023-10-12T00:56:43.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096b79b228577e59c-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.359 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.360 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.361 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.361 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.362 DEBUG:boto:Data: 2023-10-12T00:56:43.363 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.364 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.365 DEBUG:boto:Params: {} 2023-10-12T00:56:43.366 DEBUG:boto:Token: None 2023-10-12T00:56:43.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8t5vq/dbpO9+VfuO2AZHbCwejFw= 2023-10-12T00:56:43.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8t5vq/dbpO9+VfuO2AZHbCwejFw=', 'Content-Length': '0'} 2023-10-12T00:56:43.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332023823976c2f6-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-12T00:56:43.371 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.372 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.373 DEBUG:boto:Method: GET 2023-10-12T00:56:43.374 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.375 DEBUG:boto:Data: 2023-10-12T00:56:43.376 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.377 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.378 DEBUG:boto:Params: {} 2023-10-12T00:56:43.379 DEBUG:boto:Token: None 2023-10-12T00:56:43.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sOP6Di5xnm5Eq8zsZLsObKx+Fwk= 2023-10-12T00:56:43.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sOP6Di5xnm5Eq8zsZLsObKx+Fwk=', 'Content-Length': '0'} 2023-10-12T00:56:43.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073aee4c7efa238b7-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.384 DEBUG:boto:path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.385 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.385 DEBUG:boto:Method: GET 2023-10-12T00:56:43.386 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.386 DEBUG:boto:Data: 2023-10-12T00:56:43.387 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.388 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.388 DEBUG:boto:Params: {} 2023-10-12T00:56:43.389 DEBUG:boto:Token: None 2023-10-12T00:56:43.390 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000006 2023-10-12T00:56:43.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sOP6Di5xnm5Eq8zsZLsObKx+Fwk= 2023-10-12T00:56:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sOP6Di5xnm5Eq8zsZLsObKx+Fwk=', 'Content-Length': '0'} 2023-10-12T00:56:43.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc58739ef4bf3b77-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.434 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.435 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.435 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.436 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.436 DEBUG:boto:Method: GET 2023-10-12T00:56:43.437 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.437 DEBUG:boto:Data: 2023-10-12T00:56:43.438 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.439 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.439 DEBUG:boto:Params: {} 2023-10-12T00:56:43.440 DEBUG:boto:Token: None 2023-10-12T00:56:43.441 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964dcd6e429f5328-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.445 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.446 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.446 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.447 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.447 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.448 DEBUG:boto:Method: GET 2023-10-12T00:56:43.448 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000006 2023-10-12T00:56:43.449 DEBUG:boto:Data: 2023-10-12T00:56:43.449 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.450 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.451 DEBUG:boto:Params: {} 2023-10-12T00:56:43.451 DEBUG:boto:Token: None 2023-10-12T00:56:43.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad336e9162a28d2-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.458 DEBUG:boto:b'wafmoa-58obj000000000000000000061000falseobj000000000000000000062023-10-12T00:56:37.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-12T00:56:43.459 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.459 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.460 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.460 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.460 DEBUG:boto:Method: GET 2023-10-12T00:56:43.461 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.461 DEBUG:boto:Data: 2023-10-12T00:56:43.462 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.462 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.463 DEBUG:boto:Params: {} 2023-10-12T00:56:43.463 DEBUG:boto:Token: None 2023-10-12T00:56:43.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk= 2023-10-12T00:56:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk=', 'Content-Length': '0'} 2023-10-12T00:56:43.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e7f18d3baaa647-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.468 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.468 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.469 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.469 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.470 DEBUG:boto:Method: GET 2023-10-12T00:56:43.470 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.471 DEBUG:boto:Data: 2023-10-12T00:56:43.471 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.472 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.472 DEBUG:boto:Params: {} 2023-10-12T00:56:43.473 DEBUG:boto:Token: None 2023-10-12T00:56:43.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk= 2023-10-12T00:56:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk=', 'Content-Length': '0'} 2023-10-12T00:56:43.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abe84b7b8fce245-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.478 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.478 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.479 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.479 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.480 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.480 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.481 DEBUG:boto:Data: 2023-10-12T00:56:43.481 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.482 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.482 DEBUG:boto:Params: {} 2023-10-12T00:56:43.483 DEBUG:boto:Token: None 2023-10-12T00:56:43.483 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:41tntnLJL4z4281+MPfWfRhAZ1o= 2023-10-12T00:56:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:41tntnLJL4z4281+MPfWfRhAZ1o=', 'Content-Length': '0'} 2023-10-12T00:56:43.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094274e89708f73c1-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.487 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.487 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.488 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.488 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.488 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.489 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.489 DEBUG:boto:Data: 2023-10-12T00:56:43.489 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.490 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.491 DEBUG:boto:Params: {} 2023-10-12T00:56:43.491 DEBUG:boto:Token: None 2023-10-12T00:56:43.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:41tntnLJL4z4281+MPfWfRhAZ1o= 2023-10-12T00:56:43.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:41tntnLJL4z4281+MPfWfRhAZ1o=', 'Content-Length': '0'} 2023-10-12T00:56:43.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084aa20f7680e4004-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-12T00:56:43.496 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.496 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.497 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.497 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.497 DEBUG:boto:Method: GET 2023-10-12T00:56:43.498 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.498 DEBUG:boto:Data: 2023-10-12T00:56:43.499 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.500 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.500 DEBUG:boto:Params: {} 2023-10-12T00:56:43.500 DEBUG:boto:Token: None 2023-10-12T00:56:43.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.502 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk= 2023-10-12T00:56:43.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk=', 'Content-Length': '0'} 2023-10-12T00:56:43.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed2df2b4127e0973-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.546 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.547 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.547 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.548 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.548 DEBUG:boto:Method: GET 2023-10-12T00:56:43.549 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.549 DEBUG:boto:Data: 2023-10-12T00:56:43.550 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.551 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.551 DEBUG:boto:Params: {} 2023-10-12T00:56:43.551 DEBUG:boto:Token: None 2023-10-12T00:56:43.552 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007?versionId=null 2023-10-12T00:56:43.552 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk= 2023-10-12T00:56:43.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXkoC99vRj0TDC0+R0Wp8jy4Ymk=', 'Content-Length': '0'} 2023-10-12T00:56:43.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5efd7bf52bf2a42-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.556 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.556 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.557 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.557 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.558 DEBUG:boto:Method: GET 2023-10-12T00:56:43.558 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.559 DEBUG:boto:Data: 2023-10-12T00:56:43.559 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.560 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.561 DEBUG:boto:Params: {} 2023-10-12T00:56:43.561 DEBUG:boto:Token: None 2023-10-12T00:56:43.561 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.562 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0090ee18555462-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.566 DEBUG:boto:b'wafmoa-58obj000000000000000000071000falseobj000000000000000000072023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.566 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.567 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.567 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.568 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.568 DEBUG:boto:Method: GET 2023-10-12T00:56:43.569 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.569 DEBUG:boto:Data: 2023-10-12T00:56:43.570 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.571 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.571 DEBUG:boto:Params: {} 2023-10-12T00:56:43.572 DEBUG:boto:Token: None 2023-10-12T00:56:43.572 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.573 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089f0b55c8c651b45-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.618 DEBUG:boto:b'wafmoa-58obj000000000000000000071000falseobj000000000000000000072023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.619 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.620 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.620 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.621 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.621 DEBUG:boto:Data: 2023-10-12T00:56:43.622 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.623 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.623 DEBUG:boto:Params: {} 2023-10-12T00:56:43.624 DEBUG:boto:Token: None 2023-10-12T00:56:43.624 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4xc/Ve7lqd2swlDccEOPwkJ9Mt4= 2023-10-12T00:56:43.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4xc/Ve7lqd2swlDccEOPwkJ9Mt4=', 'Content-Length': '0'} 2023-10-12T00:56:43.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bcfb89d159105c-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.628 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.628 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.629 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.629 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.629 DEBUG:boto:Data: 2023-10-12T00:56:43.630 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.631 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.631 DEBUG:boto:Params: {} 2023-10-12T00:56:43.632 DEBUG:boto:Token: None 2023-10-12T00:56:43.632 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4xc/Ve7lqd2swlDccEOPwkJ9Mt4= 2023-10-12T00:56:43.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4xc/Ve7lqd2swlDccEOPwkJ9Mt4=', 'Content-Length': '0'} 2023-10-12T00:56:43.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419f03f857005797-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-12T00:56:43.636 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.636 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.637 DEBUG:boto:Method: GET 2023-10-12T00:56:43.637 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.638 DEBUG:boto:Data: 2023-10-12T00:56:43.638 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.639 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.640 DEBUG:boto:Params: {} 2023-10-12T00:56:43.640 DEBUG:boto:Token: None 2023-10-12T00:56:43.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b2LrJsqxreWp2B5F3cmRfPx/zb0= 2023-10-12T00:56:43.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b2LrJsqxreWp2B5F3cmRfPx/zb0=', 'Content-Length': '0'} 2023-10-12T00:56:43.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be58c1aa70cbf7b-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.644 DEBUG:boto:path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.645 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.645 DEBUG:boto:Method: GET 2023-10-12T00:56:43.645 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.646 DEBUG:boto:Data: 2023-10-12T00:56:43.646 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.647 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.648 DEBUG:boto:Params: {} 2023-10-12T00:56:43.648 DEBUG:boto:Token: None 2023-10-12T00:56:43.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000007 2023-10-12T00:56:43.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:b2LrJsqxreWp2B5F3cmRfPx/zb0= 2023-10-12T00:56:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:b2LrJsqxreWp2B5F3cmRfPx/zb0=', 'Content-Length': '0'} 2023-10-12T00:56:43.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842e5d60ef31b284-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.652 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.653 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.653 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.654 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.654 DEBUG:boto:Method: GET 2023-10-12T00:56:43.654 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.655 DEBUG:boto:Data: 2023-10-12T00:56:43.655 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.656 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.657 DEBUG:boto:Params: {} 2023-10-12T00:56:43.657 DEBUG:boto:Token: None 2023-10-12T00:56:43.658 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.658 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034da66b489b0c72f-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.662 DEBUG:boto:b'wafmoa-58obj000000000000000000071000falseobj000000000000000000072023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.663 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.663 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.664 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.664 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.665 DEBUG:boto:Method: GET 2023-10-12T00:56:43.665 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000007 2023-10-12T00:56:43.666 DEBUG:boto:Data: 2023-10-12T00:56:43.666 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.667 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.667 DEBUG:boto:Params: {} 2023-10-12T00:56:43.668 DEBUG:boto:Token: None 2023-10-12T00:56:43.668 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016cdb509db1d9e1-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.675 DEBUG:boto:b'wafmoa-58obj000000000000000000071000falseobj000000000000000000072023-10-12T00:56:35.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-12T00:56:43.676 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.677 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.677 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.678 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.678 DEBUG:boto:Method: GET 2023-10-12T00:56:43.679 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.679 DEBUG:boto:Data: 2023-10-12T00:56:43.679 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.680 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.681 DEBUG:boto:Params: {} 2023-10-12T00:56:43.681 DEBUG:boto:Token: None 2023-10-12T00:56:43.682 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE= 2023-10-12T00:56:43.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE=', 'Content-Length': '0'} 2023-10-12T00:56:43.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719a54d0d005342f-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.686 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.686 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.687 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.688 DEBUG:boto:Method: GET 2023-10-12T00:56:43.688 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.689 DEBUG:boto:Data: 2023-10-12T00:56:43.689 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.690 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.690 DEBUG:boto:Params: {} 2023-10-12T00:56:43.691 DEBUG:boto:Token: None 2023-10-12T00:56:43.691 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE= 2023-10-12T00:56:43.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE=', 'Content-Length': '0'} 2023-10-12T00:56:43.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6940dea69edbd57-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.738 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.739 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.739 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.740 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.740 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.741 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.741 DEBUG:boto:Data: 2023-10-12T00:56:43.742 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.742 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.743 DEBUG:boto:Params: {} 2023-10-12T00:56:43.743 DEBUG:boto:Token: None 2023-10-12T00:56:43.744 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tlbwxVO9XvEFHmH0rURconG/8wg= 2023-10-12T00:56:43.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tlbwxVO9XvEFHmH0rURconG/8wg=', 'Content-Length': '0'} 2023-10-12T00:56:43.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed4e714fbcb3e67-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.748 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.748 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.749 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.749 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.749 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.750 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.750 DEBUG:boto:Data: 2023-10-12T00:56:43.751 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.752 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.752 DEBUG:boto:Params: {} 2023-10-12T00:56:43.752 DEBUG:boto:Token: None 2023-10-12T00:56:43.753 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tlbwxVO9XvEFHmH0rURconG/8wg= 2023-10-12T00:56:43.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tlbwxVO9XvEFHmH0rURconG/8wg=', 'Content-Length': '0'} 2023-10-12T00:56:43.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d5cec1c29dd2cf-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-12T00:56:43.757 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.757 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.758 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.758 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.759 DEBUG:boto:Method: GET 2023-10-12T00:56:43.759 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.759 DEBUG:boto:Data: 2023-10-12T00:56:43.760 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.761 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.761 DEBUG:boto:Params: {} 2023-10-12T00:56:43.762 DEBUG:boto:Token: None 2023-10-12T00:56:43.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE= 2023-10-12T00:56:43.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE=', 'Content-Length': '0'} 2023-10-12T00:56:43.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec31ff63dc4cf25-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.806 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.807 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.807 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.808 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.808 DEBUG:boto:Method: GET 2023-10-12T00:56:43.809 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.809 DEBUG:boto:Data: 2023-10-12T00:56:43.810 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.811 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.811 DEBUG:boto:Params: {} 2023-10-12T00:56:43.812 DEBUG:boto:Token: None 2023-10-12T00:56:43.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008?versionId=null 2023-10-12T00:56:43.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE= 2023-10-12T00:56:43.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2bhxA9nrF4521uiK7papy21WDsE=', 'Content-Length': '0'} 2023-10-12T00:56:43.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008039fbcfbca0478c-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.816 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.816 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.817 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.817 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.817 DEBUG:boto:Method: GET 2023-10-12T00:56:43.818 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.818 DEBUG:boto:Data: 2023-10-12T00:56:43.819 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.820 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.820 DEBUG:boto:Params: {} 2023-10-12T00:56:43.821 DEBUG:boto:Token: None 2023-10-12T00:56:43.821 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005587baf42de88681-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.825 DEBUG:boto:b'wafmoa-58obj000000000000000000081000falseobj000000000000000000082023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.826 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.827 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.827 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.827 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.828 DEBUG:boto:Method: GET 2023-10-12T00:56:43.828 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.829 DEBUG:boto:Data: 2023-10-12T00:56:43.829 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.830 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.831 DEBUG:boto:Params: {} 2023-10-12T00:56:43.831 DEBUG:boto:Token: None 2023-10-12T00:56:43.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005235f28b5eb6d9bf-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.837 DEBUG:boto:b'wafmoa-58obj000000000000000000081000falseobj000000000000000000082023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.838 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.838 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.839 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.839 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.840 DEBUG:boto:Data: 2023-10-12T00:56:43.840 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.841 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.841 DEBUG:boto:Params: {} 2023-10-12T00:56:43.841 DEBUG:boto:Token: None 2023-10-12T00:56:43.842 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tW96uXHY9AupLpK9/1erjP1xZkw= 2023-10-12T00:56:43.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tW96uXHY9AupLpK9/1erjP1xZkw=', 'Content-Length': '0'} 2023-10-12T00:56:43.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ed4259c58924599-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.846 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.847 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.847 DEBUG:boto:Data: 2023-10-12T00:56:43.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.848 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.849 DEBUG:boto:Params: {} 2023-10-12T00:56:43.849 DEBUG:boto:Token: None 2023-10-12T00:56:43.849 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tW96uXHY9AupLpK9/1erjP1xZkw= 2023-10-12T00:56:43.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tW96uXHY9AupLpK9/1erjP1xZkw=', 'Content-Length': '0'} 2023-10-12T00:56:43.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d92c10de53c5ec-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-12T00:56:43.853 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.854 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.854 DEBUG:boto:Method: GET 2023-10-12T00:56:43.855 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.855 DEBUG:boto:Data: 2023-10-12T00:56:43.856 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.856 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.857 DEBUG:boto:Params: {} 2023-10-12T00:56:43.857 DEBUG:boto:Token: None 2023-10-12T00:56:43.858 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tgXiStE0ZwGyTiY4Kalaj/nipRs= 2023-10-12T00:56:43.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tgXiStE0ZwGyTiY4Kalaj/nipRs=', 'Content-Length': '0'} 2023-10-12T00:56:43.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a8a096d6ee16ba-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.861 DEBUG:boto:path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.862 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.862 DEBUG:boto:Method: GET 2023-10-12T00:56:43.863 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.863 DEBUG:boto:Data: 2023-10-12T00:56:43.864 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.865 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.865 DEBUG:boto:Params: {} 2023-10-12T00:56:43.865 DEBUG:boto:Token: None 2023-10-12T00:56:43.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000008 2023-10-12T00:56:43.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tgXiStE0ZwGyTiY4Kalaj/nipRs= 2023-10-12T00:56:43.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tgXiStE0ZwGyTiY4Kalaj/nipRs=', 'Content-Length': '0'} 2023-10-12T00:56:43.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d4d8f6308fa16e-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.914 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.915 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.915 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.916 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.916 DEBUG:boto:Method: GET 2023-10-12T00:56:43.917 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.917 DEBUG:boto:Data: 2023-10-12T00:56:43.918 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.919 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.919 DEBUG:boto:Params: {} 2023-10-12T00:56:43.920 DEBUG:boto:Token: None 2023-10-12T00:56:43.920 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170b328e641be1fa-006527444b-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.925 DEBUG:boto:b'wafmoa-58obj000000000000000000081000falseobj000000000000000000082023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.925 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:43.926 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:43.926 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.927 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.927 DEBUG:boto:Method: GET 2023-10-12T00:56:43.928 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000008 2023-10-12T00:56:43.928 DEBUG:boto:Data: 2023-10-12T00:56:43.928 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.929 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.930 DEBUG:boto:Params: {} 2023-10-12T00:56:43.930 DEBUG:boto:Token: None 2023-10-12T00:56:43.931 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/ 2023-10-12T00:56:43.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80= 2023-10-12T00:56:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1oH0BqJ5ZbhUAsLd2gEJkF3zM80=', 'Content-Length': '0'} 2023-10-12T00:56:43.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018f8afd779de760e-006527444b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.937 DEBUG:boto:b'wafmoa-58obj000000000000000000081000falseobj000000000000000000082023-10-12T00:56:37.088Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:43.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-12T00:56:43.938 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.939 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.939 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.939 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.940 DEBUG:boto:Method: GET 2023-10-12T00:56:43.940 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.941 DEBUG:boto:Data: 2023-10-12T00:56:43.941 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.942 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.942 DEBUG:boto:Params: {} 2023-10-12T00:56:43.943 DEBUG:boto:Token: None 2023-10-12T00:56:43.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00= 2023-10-12T00:56:43.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00=', 'Content-Length': '0'} 2023-10-12T00:56:43.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82776e2ed42574b-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.947 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.947 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.948 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.948 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.949 DEBUG:boto:Method: GET 2023-10-12T00:56:43.949 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.950 DEBUG:boto:Data: 2023-10-12T00:56:43.950 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.951 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.951 DEBUG:boto:Params: {} 2023-10-12T00:56:43.952 DEBUG:boto:Token: None 2023-10-12T00:56:43.952 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00= 2023-10-12T00:56:43.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00=', 'Content-Length': '0'} 2023-10-12T00:56:43.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26228072adee15e-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.956 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.957 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.957 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.957 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.958 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.958 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.959 DEBUG:boto:Data: 2023-10-12T00:56:43.959 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.960 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.960 DEBUG:boto:Params: {} 2023-10-12T00:56:43.961 DEBUG:boto:Token: None 2023-10-12T00:56:43.961 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Qq+0vIeuIxACdE2HVUVyFdmgCM= 2023-10-12T00:56:43.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Qq+0vIeuIxACdE2HVUVyFdmgCM=', 'Content-Length': '0'} 2023-10-12T00:56:43.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1c669af2b85fe8-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.964 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.965 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.965 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.966 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.966 DEBUG:boto:Method: HEAD 2023-10-12T00:56:43.967 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.967 DEBUG:boto:Data: 2023-10-12T00:56:43.967 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:43.968 DEBUG:boto:Port: 8000 2023-10-12T00:56:43.969 DEBUG:boto:Params: {} 2023-10-12T00:56:43.969 DEBUG:boto:Token: None 2023-10-12T00:56:43.970 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Qq+0vIeuIxACdE2HVUVyFdmgCM= 2023-10-12T00:56:43.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Qq+0vIeuIxACdE2HVUVyFdmgCM=', 'Content-Length': '0'} 2023-10-12T00:56:43.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c3256ac4915ca3-006527444b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:43.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-12T00:56:43.973 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.974 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:43.974 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.975 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.975 DEBUG:boto:Method: GET 2023-10-12T00:56:43.976 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.976 DEBUG:boto:Data: 2023-10-12T00:56:43.977 DEBUG:boto:Headers: {} 2023-10-12T00:56:43.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:43.977 DEBUG:boto:Port: 8001 2023-10-12T00:56:43.978 DEBUG:boto:Params: {} 2023-10-12T00:56:43.978 DEBUG:boto:Token: None 2023-10-12T00:56:43.979 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:43 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:43.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00= 2023-10-12T00:56:43.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f8lvM8ef96C68MFtzzZqWf2qZ00=', 'Content-Length': '0'} 2023-10-12T00:56:43.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449f9bdb02e56b4a-006527444b-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.022 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.023 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.023 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:44.024 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:44.024 DEBUG:boto:Method: GET 2023-10-12T00:56:44.025 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:44.025 DEBUG:boto:Data: 2023-10-12T00:56:44.026 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.026 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.027 DEBUG:boto:Params: {} 2023-10-12T00:56:44.027 DEBUG:boto:Token: None 2023-10-12T00:56:44.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000009?versionId=null 2023-10-12T00:56:44.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vrLp4jHcdLJqA+Ulon9djejz3ho= 2023-10-12T00:56:44.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vrLp4jHcdLJqA+Ulon9djejz3ho=', 'Content-Length': '0'} 2023-10-12T00:56:44.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e0729619843765-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.032 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.033 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.033 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.033 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.034 DEBUG:boto:Method: GET 2023-10-12T00:56:44.034 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.035 DEBUG:boto:Data: 2023-10-12T00:56:44.035 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.036 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.037 DEBUG:boto:Params: {} 2023-10-12T00:56:44.037 DEBUG:boto:Token: None 2023-10-12T00:56:44.038 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673d90bad8dcf78c-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.042 DEBUG:boto:b'wafmoa-58obj000000000000000000091000falseobj000000000000000000092023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.043 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.043 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.044 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.044 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.045 DEBUG:boto:Method: GET 2023-10-12T00:56:44.045 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.046 DEBUG:boto:Data: 2023-10-12T00:56:44.046 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.047 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.047 DEBUG:boto:Params: {} 2023-10-12T00:56:44.048 DEBUG:boto:Token: None 2023-10-12T00:56:44.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59eeb8a327a6af6-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.054 DEBUG:boto:b'wafmoa-58obj000000000000000000091000falseobj000000000000000000092023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.055 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.055 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.056 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.056 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.057 DEBUG:boto:Data: 2023-10-12T00:56:44.057 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.058 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.058 DEBUG:boto:Params: {} 2023-10-12T00:56:44.059 DEBUG:boto:Token: None 2023-10-12T00:56:44.059 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cs4CBdhLRQwp+F5IfWqn13Rgpsw= 2023-10-12T00:56:44.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cs4CBdhLRQwp+F5IfWqn13Rgpsw=', 'Content-Length': '0'} 2023-10-12T00:56:44.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a964aa0b7a1fc4c4-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.063 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.063 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.064 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.064 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.064 DEBUG:boto:Data: 2023-10-12T00:56:44.065 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.066 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.066 DEBUG:boto:Params: {} 2023-10-12T00:56:44.067 DEBUG:boto:Token: None 2023-10-12T00:56:44.067 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cs4CBdhLRQwp+F5IfWqn13Rgpsw= 2023-10-12T00:56:44.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cs4CBdhLRQwp+F5IfWqn13Rgpsw=', 'Content-Length': '0'} 2023-10-12T00:56:44.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91f798e1e50e05d-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-12T00:56:44.071 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.071 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.072 DEBUG:boto:Method: GET 2023-10-12T00:56:44.072 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.073 DEBUG:boto:Data: 2023-10-12T00:56:44.073 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.074 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.074 DEBUG:boto:Params: {} 2023-10-12T00:56:44.075 DEBUG:boto:Token: None 2023-10-12T00:56:44.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RnbueKP0OfWfFyIqxUWGMvP0dt4= 2023-10-12T00:56:44.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RnbueKP0OfWfFyIqxUWGMvP0dt4=', 'Content-Length': '0'} 2023-10-12T00:56:44.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ece538c1c657dc6-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.079 DEBUG:boto:path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.079 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.080 DEBUG:boto:Method: GET 2023-10-12T00:56:44.080 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.081 DEBUG:boto:Data: 2023-10-12T00:56:44.081 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.082 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.082 DEBUG:boto:Params: {} 2023-10-12T00:56:44.083 DEBUG:boto:Token: None 2023-10-12T00:56:44.083 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000009 2023-10-12T00:56:44.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RnbueKP0OfWfFyIqxUWGMvP0dt4= 2023-10-12T00:56:44.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RnbueKP0OfWfFyIqxUWGMvP0dt4=', 'Content-Length': '0'} 2023-10-12T00:56:44.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309967e30c81e5c3-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.087 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.087 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.088 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.088 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.088 DEBUG:boto:Method: GET 2023-10-12T00:56:44.089 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.089 DEBUG:boto:Data: 2023-10-12T00:56:44.090 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.091 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.091 DEBUG:boto:Params: {} 2023-10-12T00:56:44.092 DEBUG:boto:Token: None 2023-10-12T00:56:44.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048af771a4cac0caa-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.138 DEBUG:boto:b'wafmoa-58obj000000000000000000091000falseobj000000000000000000092023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.139 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.139 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.140 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.140 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.141 DEBUG:boto:Method: GET 2023-10-12T00:56:44.141 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000009 2023-10-12T00:56:44.142 DEBUG:boto:Data: 2023-10-12T00:56:44.142 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.143 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.143 DEBUG:boto:Params: {} 2023-10-12T00:56:44.144 DEBUG:boto:Token: None 2023-10-12T00:56:44.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c2f351fadd6d4f0-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.150 DEBUG:boto:b'wafmoa-58obj000000000000000000091000falseobj000000000000000000092023-10-12T00:56:36.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-12T00:56:44.151 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.151 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.152 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.152 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.153 DEBUG:boto:Method: GET 2023-10-12T00:56:44.153 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.153 DEBUG:boto:Data: 2023-10-12T00:56:44.154 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.155 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.155 DEBUG:boto:Params: {} 2023-10-12T00:56:44.155 DEBUG:boto:Token: None 2023-10-12T00:56:44.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM= 2023-10-12T00:56:44.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM=', 'Content-Length': '0'} 2023-10-12T00:56:44.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b901b4e45a120282-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.161 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.161 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.161 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.162 DEBUG:boto:Method: GET 2023-10-12T00:56:44.162 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.163 DEBUG:boto:Data: 2023-10-12T00:56:44.163 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.164 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.164 DEBUG:boto:Params: {} 2023-10-12T00:56:44.165 DEBUG:boto:Token: None 2023-10-12T00:56:44.165 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM= 2023-10-12T00:56:44.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM=', 'Content-Length': '0'} 2023-10-12T00:56:44.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae5df24efa2937e1-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.214 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.215 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.215 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.216 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.216 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.217 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.217 DEBUG:boto:Data: 2023-10-12T00:56:44.217 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.218 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.219 DEBUG:boto:Params: {} 2023-10-12T00:56:44.219 DEBUG:boto:Token: None 2023-10-12T00:56:44.220 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K0Vh/mXnJbsiJvM0oCKvQqRBwkA= 2023-10-12T00:56:44.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K0Vh/mXnJbsiJvM0oCKvQqRBwkA=', 'Content-Length': '0'} 2023-10-12T00:56:44.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e56f7e8460a456-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.224 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.224 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.225 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.225 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.225 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.226 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.226 DEBUG:boto:Data: 2023-10-12T00:56:44.227 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.227 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.228 DEBUG:boto:Params: {} 2023-10-12T00:56:44.228 DEBUG:boto:Token: None 2023-10-12T00:56:44.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K0Vh/mXnJbsiJvM0oCKvQqRBwkA= 2023-10-12T00:56:44.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K0Vh/mXnJbsiJvM0oCKvQqRBwkA=', 'Content-Length': '0'} 2023-10-12T00:56:44.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f8171b358a7f35-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-12T00:56:44.233 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.233 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.234 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.234 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.235 DEBUG:boto:Method: GET 2023-10-12T00:56:44.235 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.236 DEBUG:boto:Data: 2023-10-12T00:56:44.236 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.237 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.237 DEBUG:boto:Params: {} 2023-10-12T00:56:44.238 DEBUG:boto:Token: None 2023-10-12T00:56:44.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM= 2023-10-12T00:56:44.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM=', 'Content-Length': '0'} 2023-10-12T00:56:44.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031a8e73a49267ca5-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.242 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.242 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.243 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.243 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.244 DEBUG:boto:Method: GET 2023-10-12T00:56:44.244 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.245 DEBUG:boto:Data: 2023-10-12T00:56:44.245 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.246 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.247 DEBUG:boto:Params: {} 2023-10-12T00:56:44.247 DEBUG:boto:Token: None 2023-10-12T00:56:44.248 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010?versionId=null 2023-10-12T00:56:44.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM= 2023-10-12T00:56:44.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CCtPyLxIq7Arh5sP22CNoZ5NaUM=', 'Content-Length': '0'} 2023-10-12T00:56:44.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005350b9aa645946d7-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.251 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.252 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.252 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.253 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.253 DEBUG:boto:Method: GET 2023-10-12T00:56:44.254 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.254 DEBUG:boto:Data: 2023-10-12T00:56:44.255 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.255 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.256 DEBUG:boto:Params: {} 2023-10-12T00:56:44.256 DEBUG:boto:Token: None 2023-10-12T00:56:44.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064f9ed7014c0a231-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.261 DEBUG:boto:b'wafmoa-58obj000000000000000000101000falseobj000000000000000000102023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.262 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.262 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.263 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.263 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.264 DEBUG:boto:Method: GET 2023-10-12T00:56:44.264 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.265 DEBUG:boto:Data: 2023-10-12T00:56:44.265 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.266 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.266 DEBUG:boto:Params: {} 2023-10-12T00:56:44.267 DEBUG:boto:Token: None 2023-10-12T00:56:44.267 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df4a79d5a7f72ff-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.273 DEBUG:boto:b'wafmoa-58obj000000000000000000101000falseobj000000000000000000102023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.274 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.274 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.275 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.275 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.275 DEBUG:boto:Data: 2023-10-12T00:56:44.276 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.277 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.277 DEBUG:boto:Params: {} 2023-10-12T00:56:44.277 DEBUG:boto:Token: None 2023-10-12T00:56:44.278 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFn7QiCHsd7IBsCo7zbz37VLmr8= 2023-10-12T00:56:44.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFn7QiCHsd7IBsCo7zbz37VLmr8=', 'Content-Length': '0'} 2023-10-12T00:56:44.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002768436c3ea46cef-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.281 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.282 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.282 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.283 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.283 DEBUG:boto:Data: 2023-10-12T00:56:44.284 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.284 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.285 DEBUG:boto:Params: {} 2023-10-12T00:56:44.285 DEBUG:boto:Token: None 2023-10-12T00:56:44.286 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFn7QiCHsd7IBsCo7zbz37VLmr8= 2023-10-12T00:56:44.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFn7QiCHsd7IBsCo7zbz37VLmr8=', 'Content-Length': '0'} 2023-10-12T00:56:44.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241096f5ea0bebc0-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-12T00:56:44.290 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.290 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.291 DEBUG:boto:Method: GET 2023-10-12T00:56:44.291 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.292 DEBUG:boto:Data: 2023-10-12T00:56:44.292 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.293 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.293 DEBUG:boto:Params: {} 2023-10-12T00:56:44.294 DEBUG:boto:Token: None 2023-10-12T00:56:44.294 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.295 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xce5gArcNRLXq81/dSwc7lv/GWc= 2023-10-12T00:56:44.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xce5gArcNRLXq81/dSwc7lv/GWc=', 'Content-Length': '0'} 2023-10-12T00:56:44.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c485bb7a3d9af955-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.298 DEBUG:boto:path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.298 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.299 DEBUG:boto:Method: GET 2023-10-12T00:56:44.299 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.299 DEBUG:boto:Data: 2023-10-12T00:56:44.300 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.301 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.301 DEBUG:boto:Params: {} 2023-10-12T00:56:44.302 DEBUG:boto:Token: None 2023-10-12T00:56:44.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000010 2023-10-12T00:56:44.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xce5gArcNRLXq81/dSwc7lv/GWc= 2023-10-12T00:56:44.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xce5gArcNRLXq81/dSwc7lv/GWc=', 'Content-Length': '0'} 2023-10-12T00:56:44.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186625a652cb704e-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.346 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.347 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.347 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.348 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.348 DEBUG:boto:Method: GET 2023-10-12T00:56:44.349 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.349 DEBUG:boto:Data: 2023-10-12T00:56:44.350 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.351 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.351 DEBUG:boto:Params: {} 2023-10-12T00:56:44.352 DEBUG:boto:Token: None 2023-10-12T00:56:44.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c36c7fe302d69a8-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.356 DEBUG:boto:b'wafmoa-58obj000000000000000000101000falseobj000000000000000000102023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.357 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.357 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.358 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.358 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.359 DEBUG:boto:Method: GET 2023-10-12T00:56:44.359 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000010 2023-10-12T00:56:44.360 DEBUG:boto:Data: 2023-10-12T00:56:44.360 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.361 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.361 DEBUG:boto:Params: {} 2023-10-12T00:56:44.362 DEBUG:boto:Token: None 2023-10-12T00:56:44.362 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6cb574cc477207b-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.368 DEBUG:boto:b'wafmoa-58obj000000000000000000101000falseobj000000000000000000102023-10-12T00:56:37.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:44.369 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.369 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.370 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.370 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.371 DEBUG:boto:Method: GET 2023-10-12T00:56:44.371 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.372 DEBUG:boto:Data: 2023-10-12T00:56:44.372 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.373 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.373 DEBUG:boto:Params: {} 2023-10-12T00:56:44.374 DEBUG:boto:Token: None 2023-10-12T00:56:44.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI= 2023-10-12T00:56:44.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI=', 'Content-Length': '0'} 2023-10-12T00:56:44.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226931f4c6f934c4-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.418 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.419 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.419 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.420 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.420 DEBUG:boto:Method: GET 2023-10-12T00:56:44.420 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.421 DEBUG:boto:Data: 2023-10-12T00:56:44.421 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.422 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.423 DEBUG:boto:Params: {} 2023-10-12T00:56:44.423 DEBUG:boto:Token: None 2023-10-12T00:56:44.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI= 2023-10-12T00:56:44.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI=', 'Content-Length': '0'} 2023-10-12T00:56:44.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3d6bfbe2069e44-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.428 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.428 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.429 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.429 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.429 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.430 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.430 DEBUG:boto:Data: 2023-10-12T00:56:44.430 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.431 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.432 DEBUG:boto:Params: {} 2023-10-12T00:56:44.432 DEBUG:boto:Token: None 2023-10-12T00:56:44.433 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cusgZo2aVcf2PaVZzHNAcyyW6LE= 2023-10-12T00:56:44.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cusgZo2aVcf2PaVZzHNAcyyW6LE=', 'Content-Length': '0'} 2023-10-12T00:56:44.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa23d92d869bb1d-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.436 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.436 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.437 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.437 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.438 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.438 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.439 DEBUG:boto:Data: 2023-10-12T00:56:44.439 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.440 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.440 DEBUG:boto:Params: {} 2023-10-12T00:56:44.441 DEBUG:boto:Token: None 2023-10-12T00:56:44.441 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cusgZo2aVcf2PaVZzHNAcyyW6LE= 2023-10-12T00:56:44.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cusgZo2aVcf2PaVZzHNAcyyW6LE=', 'Content-Length': '0'} 2023-10-12T00:56:44.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dad7fca5cc0c3b-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:44.445 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.445 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.446 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.446 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.447 DEBUG:boto:Method: GET 2023-10-12T00:56:44.447 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.447 DEBUG:boto:Data: 2023-10-12T00:56:44.448 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.449 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.449 DEBUG:boto:Params: {} 2023-10-12T00:56:44.449 DEBUG:boto:Token: None 2023-10-12T00:56:44.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI= 2023-10-12T00:56:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI=', 'Content-Length': '0'} 2023-10-12T00:56:44.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e983fde98b19a3a8-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.453 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.454 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.454 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.455 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.455 DEBUG:boto:Method: GET 2023-10-12T00:56:44.456 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.456 DEBUG:boto:Data: 2023-10-12T00:56:44.457 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.457 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.458 DEBUG:boto:Params: {} 2023-10-12T00:56:44.458 DEBUG:boto:Token: None 2023-10-12T00:56:44.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011?versionId=null 2023-10-12T00:56:44.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI= 2023-10-12T00:56:44.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p3kmC3p+//jMeWYtZZkufnIVyGI=', 'Content-Length': '0'} 2023-10-12T00:56:44.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814cb0511e66ba95-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.462 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.463 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.463 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.464 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.464 DEBUG:boto:Method: GET 2023-10-12T00:56:44.464 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.465 DEBUG:boto:Data: 2023-10-12T00:56:44.465 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.466 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.466 DEBUG:boto:Params: {} 2023-10-12T00:56:44.467 DEBUG:boto:Token: None 2023-10-12T00:56:44.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf25babf8f7b0cc2-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.471 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.472 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.473 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.473 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.473 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.474 DEBUG:boto:Method: GET 2023-10-12T00:56:44.474 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.475 DEBUG:boto:Data: 2023-10-12T00:56:44.475 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.476 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.476 DEBUG:boto:Params: {} 2023-10-12T00:56:44.477 DEBUG:boto:Token: None 2023-10-12T00:56:44.477 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071127c0ba4d37a0c-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.483 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.483 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.484 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.484 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.485 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.485 DEBUG:boto:Data: 2023-10-12T00:56:44.486 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.486 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.487 DEBUG:boto:Params: {} 2023-10-12T00:56:44.487 DEBUG:boto:Token: None 2023-10-12T00:56:44.488 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JqqJzsjkAPm8Go9bQyaxeSOSDcY= 2023-10-12T00:56:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JqqJzsjkAPm8Go9bQyaxeSOSDcY=', 'Content-Length': '0'} 2023-10-12T00:56:44.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b25f99647b28680-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.491 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.491 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.492 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.492 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.493 DEBUG:boto:Data: 2023-10-12T00:56:44.493 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.494 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.494 DEBUG:boto:Params: {} 2023-10-12T00:56:44.495 DEBUG:boto:Token: None 2023-10-12T00:56:44.495 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JqqJzsjkAPm8Go9bQyaxeSOSDcY= 2023-10-12T00:56:44.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JqqJzsjkAPm8Go9bQyaxeSOSDcY=', 'Content-Length': '0'} 2023-10-12T00:56:44.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a025c0499c532b7-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-12T00:56:44.499 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.499 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.500 DEBUG:boto:Method: GET 2023-10-12T00:56:44.500 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.501 DEBUG:boto:Data: 2023-10-12T00:56:44.501 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.502 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.502 DEBUG:boto:Params: {} 2023-10-12T00:56:44.503 DEBUG:boto:Token: None 2023-10-12T00:56:44.503 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZHYZYBLDP5mdPAPgTCxIsiXr9mg= 2023-10-12T00:56:44.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZHYZYBLDP5mdPAPgTCxIsiXr9mg=', 'Content-Length': '0'} 2023-10-12T00:56:44.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61dc013fde71d7c-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.550 DEBUG:boto:path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.551 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.551 DEBUG:boto:Method: GET 2023-10-12T00:56:44.552 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.552 DEBUG:boto:Data: 2023-10-12T00:56:44.553 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.554 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.554 DEBUG:boto:Params: {} 2023-10-12T00:56:44.555 DEBUG:boto:Token: None 2023-10-12T00:56:44.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000011 2023-10-12T00:56:44.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZHYZYBLDP5mdPAPgTCxIsiXr9mg= 2023-10-12T00:56:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZHYZYBLDP5mdPAPgTCxIsiXr9mg=', 'Content-Length': '0'} 2023-10-12T00:56:44.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167f5e342e067167-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.559 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.560 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.560 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.561 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.561 DEBUG:boto:Method: GET 2023-10-12T00:56:44.561 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.562 DEBUG:boto:Data: 2023-10-12T00:56:44.562 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.563 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.564 DEBUG:boto:Params: {} 2023-10-12T00:56:44.564 DEBUG:boto:Token: None 2023-10-12T00:56:44.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000394ba345c46ac24d-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.569 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.570 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.570 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.571 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.571 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.572 DEBUG:boto:Method: GET 2023-10-12T00:56:44.572 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000011 2023-10-12T00:56:44.573 DEBUG:boto:Data: 2023-10-12T00:56:44.573 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.574 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.574 DEBUG:boto:Params: {} 2023-10-12T00:56:44.575 DEBUG:boto:Token: None 2023-10-12T00:56:44.576 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ff224b984c9dc20-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.582 DEBUG:boto:b'wafmoa-58obj000000000000000000111000falseobj000000000000000000112023-10-12T00:56:37.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:44.583 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.583 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.584 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.584 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.584 DEBUG:boto:Method: GET 2023-10-12T00:56:44.585 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.585 DEBUG:boto:Data: 2023-10-12T00:56:44.586 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.587 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.587 DEBUG:boto:Params: {} 2023-10-12T00:56:44.588 DEBUG:boto:Token: None 2023-10-12T00:56:44.588 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc= 2023-10-12T00:56:44.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc=', 'Content-Length': '0'} 2023-10-12T00:56:44.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49927fed3c07745-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.592 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.593 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.593 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.594 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.594 DEBUG:boto:Method: GET 2023-10-12T00:56:44.595 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.595 DEBUG:boto:Data: 2023-10-12T00:56:44.595 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.596 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.597 DEBUG:boto:Params: {} 2023-10-12T00:56:44.597 DEBUG:boto:Token: None 2023-10-12T00:56:44.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc= 2023-10-12T00:56:44.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc=', 'Content-Length': '0'} 2023-10-12T00:56:44.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f6b2b6c0c45ba3-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.601 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.602 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.602 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.603 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.603 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.603 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.604 DEBUG:boto:Data: 2023-10-12T00:56:44.604 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.605 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.606 DEBUG:boto:Params: {} 2023-10-12T00:56:44.606 DEBUG:boto:Token: None 2023-10-12T00:56:44.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s706OHta2zlqp2TvmL8O2FuTzB4= 2023-10-12T00:56:44.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s706OHta2zlqp2TvmL8O2FuTzB4=', 'Content-Length': '0'} 2023-10-12T00:56:44.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6c321a0aea89c3-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.610 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.611 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.611 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.611 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.612 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.612 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.613 DEBUG:boto:Data: 2023-10-12T00:56:44.613 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.614 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.614 DEBUG:boto:Params: {} 2023-10-12T00:56:44.615 DEBUG:boto:Token: None 2023-10-12T00:56:44.615 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s706OHta2zlqp2TvmL8O2FuTzB4= 2023-10-12T00:56:44.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s706OHta2zlqp2TvmL8O2FuTzB4=', 'Content-Length': '0'} 2023-10-12T00:56:44.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7ffac7f8542df4-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:44.619 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.620 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.620 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.621 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.621 DEBUG:boto:Method: GET 2023-10-12T00:56:44.622 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.622 DEBUG:boto:Data: 2023-10-12T00:56:44.623 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.624 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.624 DEBUG:boto:Params: {} 2023-10-12T00:56:44.624 DEBUG:boto:Token: None 2023-10-12T00:56:44.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc= 2023-10-12T00:56:44.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc=', 'Content-Length': '0'} 2023-10-12T00:56:44.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049457626398aeabc-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.628 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.629 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.629 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.630 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.630 DEBUG:boto:Method: GET 2023-10-12T00:56:44.631 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.631 DEBUG:boto:Data: 2023-10-12T00:56:44.632 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.633 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.633 DEBUG:boto:Params: {} 2023-10-12T00:56:44.634 DEBUG:boto:Token: None 2023-10-12T00:56:44.634 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012?versionId=null 2023-10-12T00:56:44.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc= 2023-10-12T00:56:44.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eDe/6pTffE5MyM9SDKvSNt/jfIc=', 'Content-Length': '0'} 2023-10-12T00:56:44.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518a9a367efbd17d-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.678 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.679 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.679 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.680 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.680 DEBUG:boto:Method: GET 2023-10-12T00:56:44.681 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.681 DEBUG:boto:Data: 2023-10-12T00:56:44.682 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.682 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.683 DEBUG:boto:Params: {} 2023-10-12T00:56:44.683 DEBUG:boto:Token: None 2023-10-12T00:56:44.684 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b4c1181b0f2a12b-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.688 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.688 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.689 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.689 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.690 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.690 DEBUG:boto:Method: GET 2023-10-12T00:56:44.691 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.691 DEBUG:boto:Data: 2023-10-12T00:56:44.691 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.692 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.693 DEBUG:boto:Params: {} 2023-10-12T00:56:44.693 DEBUG:boto:Token: None 2023-10-12T00:56:44.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab3320dda69b9552-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.699 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.700 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.700 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.701 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.701 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.702 DEBUG:boto:Data: 2023-10-12T00:56:44.702 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.703 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.703 DEBUG:boto:Params: {} 2023-10-12T00:56:44.704 DEBUG:boto:Token: None 2023-10-12T00:56:44.704 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ncmniGIy5ew9jedk9mGvye5mma8= 2023-10-12T00:56:44.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ncmniGIy5ew9jedk9mGvye5mma8=', 'Content-Length': '0'} 2023-10-12T00:56:44.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25821a6a6b5b858-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.708 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.708 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.708 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.709 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.709 DEBUG:boto:Data: 2023-10-12T00:56:44.710 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.710 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.711 DEBUG:boto:Params: {} 2023-10-12T00:56:44.711 DEBUG:boto:Token: None 2023-10-12T00:56:44.712 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ncmniGIy5ew9jedk9mGvye5mma8= 2023-10-12T00:56:44.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ncmniGIy5ew9jedk9mGvye5mma8=', 'Content-Length': '0'} 2023-10-12T00:56:44.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a99e30c2fd44af-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-12T00:56:44.716 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.716 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.716 DEBUG:boto:Method: GET 2023-10-12T00:56:44.717 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.717 DEBUG:boto:Data: 2023-10-12T00:56:44.718 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.719 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.719 DEBUG:boto:Params: {} 2023-10-12T00:56:44.720 DEBUG:boto:Token: None 2023-10-12T00:56:44.720 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gQS2k6gGCHBiMa0ar0kmEzQGNM4= 2023-10-12T00:56:44.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gQS2k6gGCHBiMa0ar0kmEzQGNM4=', 'Content-Length': '0'} 2023-10-12T00:56:44.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba6ebc5a8e739b8-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.766 DEBUG:boto:path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.767 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.767 DEBUG:boto:Method: GET 2023-10-12T00:56:44.768 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.768 DEBUG:boto:Data: 2023-10-12T00:56:44.769 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.769 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.770 DEBUG:boto:Params: {} 2023-10-12T00:56:44.770 DEBUG:boto:Token: None 2023-10-12T00:56:44.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000012 2023-10-12T00:56:44.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gQS2k6gGCHBiMa0ar0kmEzQGNM4= 2023-10-12T00:56:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gQS2k6gGCHBiMa0ar0kmEzQGNM4=', 'Content-Length': '0'} 2023-10-12T00:56:44.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acae2c440b0255f8-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.775 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.775 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.775 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.776 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.776 DEBUG:boto:Method: GET 2023-10-12T00:56:44.777 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.777 DEBUG:boto:Data: 2023-10-12T00:56:44.778 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.779 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.779 DEBUG:boto:Params: {} 2023-10-12T00:56:44.779 DEBUG:boto:Token: None 2023-10-12T00:56:44.780 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070604859c0dcb387-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.784 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.785 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.785 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.786 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.786 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.787 DEBUG:boto:Method: GET 2023-10-12T00:56:44.787 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000012 2023-10-12T00:56:44.788 DEBUG:boto:Data: 2023-10-12T00:56:44.788 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.789 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.790 DEBUG:boto:Params: {} 2023-10-12T00:56:44.790 DEBUG:boto:Token: None 2023-10-12T00:56:44.790 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d27b196dd5a4c43-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.796 DEBUG:boto:b'wafmoa-58obj000000000000000000121000falseobj000000000000000000122023-10-12T00:56:36.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:44.797 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.798 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.798 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.798 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.799 DEBUG:boto:Method: GET 2023-10-12T00:56:44.799 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.800 DEBUG:boto:Data: 2023-10-12T00:56:44.800 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.801 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.801 DEBUG:boto:Params: {} 2023-10-12T00:56:44.802 DEBUG:boto:Token: None 2023-10-12T00:56:44.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q= 2023-10-12T00:56:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q=', 'Content-Length': '0'} 2023-10-12T00:56:44.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042bb46021ef68d9f-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.806 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.807 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.807 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.808 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.808 DEBUG:boto:Method: GET 2023-10-12T00:56:44.809 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.809 DEBUG:boto:Data: 2023-10-12T00:56:44.809 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.810 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.811 DEBUG:boto:Params: {} 2023-10-12T00:56:44.811 DEBUG:boto:Token: None 2023-10-12T00:56:44.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q= 2023-10-12T00:56:44.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q=', 'Content-Length': '0'} 2023-10-12T00:56:44.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062967bcc80db1bfb-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.858 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.859 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.860 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.860 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.861 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.861 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.861 DEBUG:boto:Data: 2023-10-12T00:56:44.862 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.863 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.863 DEBUG:boto:Params: {} 2023-10-12T00:56:44.864 DEBUG:boto:Token: None 2023-10-12T00:56:44.864 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YJohwSLJ/hBYciAzfw9TwIPKmhE= 2023-10-12T00:56:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YJohwSLJ/hBYciAzfw9TwIPKmhE=', 'Content-Length': '0'} 2023-10-12T00:56:44.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ec0593f11722a0-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.869 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.870 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.870 DEBUG:boto:Data: 2023-10-12T00:56:44.871 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.872 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.872 DEBUG:boto:Params: {} 2023-10-12T00:56:44.872 DEBUG:boto:Token: None 2023-10-12T00:56:44.873 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YJohwSLJ/hBYciAzfw9TwIPKmhE= 2023-10-12T00:56:44.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YJohwSLJ/hBYciAzfw9TwIPKmhE=', 'Content-Length': '0'} 2023-10-12T00:56:44.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa2b647684fa944-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:44.877 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.878 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.878 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.878 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.879 DEBUG:boto:Method: GET 2023-10-12T00:56:44.880 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.880 DEBUG:boto:Data: 2023-10-12T00:56:44.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.881 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.882 DEBUG:boto:Params: {} 2023-10-12T00:56:44.883 DEBUG:boto:Token: None 2023-10-12T00:56:44.883 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q= 2023-10-12T00:56:44.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q=', 'Content-Length': '0'} 2023-10-12T00:56:44.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e422489b7ec4a6-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.934 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.935 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.937 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.937 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.938 DEBUG:boto:Method: GET 2023-10-12T00:56:44.938 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.939 DEBUG:boto:Data: 2023-10-12T00:56:44.939 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.940 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.940 DEBUG:boto:Params: {} 2023-10-12T00:56:44.941 DEBUG:boto:Token: None 2023-10-12T00:56:44.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013?versionId=null 2023-10-12T00:56:44.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q= 2023-10-12T00:56:44.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i6nh7mz5UrNhHZtTjLcUnzuB5Q=', 'Content-Length': '0'} 2023-10-12T00:56:44.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21793ac4b655d5a-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.946 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.946 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.947 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.947 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.948 DEBUG:boto:Method: GET 2023-10-12T00:56:44.948 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.948 DEBUG:boto:Data: 2023-10-12T00:56:44.949 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.950 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.950 DEBUG:boto:Params: {} 2023-10-12T00:56:44.950 DEBUG:boto:Token: None 2023-10-12T00:56:44.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaefa28989751b5d-006527444c-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.955 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.956 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:44.957 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:44.957 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.957 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.958 DEBUG:boto:Method: GET 2023-10-12T00:56:44.958 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:44.959 DEBUG:boto:Data: 2023-10-12T00:56:44.959 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.960 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.960 DEBUG:boto:Params: {} 2023-10-12T00:56:44.961 DEBUG:boto:Token: None 2023-10-12T00:56:44.961 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/ 2023-10-12T00:56:44.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4= 2023-10-12T00:56:44.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+vJQcKSofE+1TUD31JCW1wKIs4=', 'Content-Length': '0'} 2023-10-12T00:56:44.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000696661985b8af3f0-006527444c-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.967 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:44.968 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.968 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.969 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.969 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.970 DEBUG:boto:Data: 2023-10-12T00:56:44.970 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.971 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.971 DEBUG:boto:Params: {} 2023-10-12T00:56:44.972 DEBUG:boto:Token: None 2023-10-12T00:56:44.972 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L04STCGH30RpbmRJr95uYLGHW4g= 2023-10-12T00:56:44.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L04STCGH30RpbmRJr95uYLGHW4g=', 'Content-Length': '0'} 2023-10-12T00:56:44.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007688a11e2df2f025-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.976 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.976 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.977 DEBUG:boto:Method: HEAD 2023-10-12T00:56:44.977 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.977 DEBUG:boto:Data: 2023-10-12T00:56:44.978 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:44.979 DEBUG:boto:Port: 8000 2023-10-12T00:56:44.979 DEBUG:boto:Params: {} 2023-10-12T00:56:44.979 DEBUG:boto:Token: None 2023-10-12T00:56:44.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L04STCGH30RpbmRJr95uYLGHW4g= 2023-10-12T00:56:44.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L04STCGH30RpbmRJr95uYLGHW4g=', 'Content-Length': '0'} 2023-10-12T00:56:44.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c7afbe1cbdece-006527444c-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:44.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-12T00:56:44.984 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.984 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.985 DEBUG:boto:Method: GET 2023-10-12T00:56:44.985 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.986 DEBUG:boto:Data: 2023-10-12T00:56:44.986 DEBUG:boto:Headers: {} 2023-10-12T00:56:44.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:44.987 DEBUG:boto:Port: 8001 2023-10-12T00:56:44.987 DEBUG:boto:Params: {} 2023-10-12T00:56:44.988 DEBUG:boto:Token: None 2023-10-12T00:56:44.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:44 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:44.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8kzJGlQcc2aL7QQdp3p4VWXz1DQ= 2023-10-12T00:56:44.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8kzJGlQcc2aL7QQdp3p4VWXz1DQ=', 'Content-Length': '0'} 2023-10-12T00:56:44.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d564aa7836260e-006527444c-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.034 DEBUG:boto:path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:45.035 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000013 2023-10-12T00:56:45.035 DEBUG:boto:Method: GET 2023-10-12T00:56:45.036 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:45.036 DEBUG:boto:Data: 2023-10-12T00:56:45.037 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.037 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.038 DEBUG:boto:Params: {} 2023-10-12T00:56:45.038 DEBUG:boto:Token: None 2023-10-12T00:56:45.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000013 2023-10-12T00:56:45.039 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XZPmAlZQlS6PwT5i9wT7AnUm0Xg= 2023-10-12T00:56:45.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XZPmAlZQlS6PwT5i9wT7AnUm0Xg=', 'Content-Length': '0'} 2023-10-12T00:56:45.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a41bb01b1b216e53-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.043 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.043 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.044 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.044 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.044 DEBUG:boto:Method: GET 2023-10-12T00:56:45.045 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.045 DEBUG:boto:Data: 2023-10-12T00:56:45.046 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.046 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.047 DEBUG:boto:Params: {} 2023-10-12T00:56:45.047 DEBUG:boto:Token: None 2023-10-12T00:56:45.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca20d2c739655da-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.052 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.053 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.054 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.054 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.054 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.055 DEBUG:boto:Method: GET 2023-10-12T00:56:45.055 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000013 2023-10-12T00:56:45.056 DEBUG:boto:Data: 2023-10-12T00:56:45.056 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.057 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.058 DEBUG:boto:Params: {} 2023-10-12T00:56:45.058 DEBUG:boto:Token: None 2023-10-12T00:56:45.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070909118040af4a4-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.064 DEBUG:boto:b'wafmoa-58obj000000000000000000131000falseobj000000000000000000132023-10-12T00:56:37.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-12T00:56:45.065 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.065 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.066 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.066 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.067 DEBUG:boto:Method: GET 2023-10-12T00:56:45.067 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.067 DEBUG:boto:Data: 2023-10-12T00:56:45.068 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.069 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.069 DEBUG:boto:Params: {} 2023-10-12T00:56:45.070 DEBUG:boto:Token: None 2023-10-12T00:56:45.070 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI= 2023-10-12T00:56:45.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI=', 'Content-Length': '0'} 2023-10-12T00:56:45.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e08f2d822db0a7b-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.074 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.075 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.075 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.076 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.076 DEBUG:boto:Method: GET 2023-10-12T00:56:45.077 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.077 DEBUG:boto:Data: 2023-10-12T00:56:45.077 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.078 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.079 DEBUG:boto:Params: {} 2023-10-12T00:56:45.079 DEBUG:boto:Token: None 2023-10-12T00:56:45.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI= 2023-10-12T00:56:45.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI=', 'Content-Length': '0'} 2023-10-12T00:56:45.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e431f8418b04ba-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.130 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.131 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.131 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.132 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.132 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.133 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.133 DEBUG:boto:Data: 2023-10-12T00:56:45.134 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.134 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.135 DEBUG:boto:Params: {} 2023-10-12T00:56:45.135 DEBUG:boto:Token: None 2023-10-12T00:56:45.136 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC5edOVyexZ0eyNmCyNNZLHXmwM= 2023-10-12T00:56:45.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC5edOVyexZ0eyNmCyNNZLHXmwM=', 'Content-Length': '0'} 2023-10-12T00:56:45.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3bfb5724f6da9e-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.140 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.140 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.141 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.141 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.142 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.142 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.142 DEBUG:boto:Data: 2023-10-12T00:56:45.143 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.144 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.144 DEBUG:boto:Params: {} 2023-10-12T00:56:45.145 DEBUG:boto:Token: None 2023-10-12T00:56:45.145 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yC5edOVyexZ0eyNmCyNNZLHXmwM= 2023-10-12T00:56:45.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yC5edOVyexZ0eyNmCyNNZLHXmwM=', 'Content-Length': '0'} 2023-10-12T00:56:45.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066f8600a5cf292c0-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-12T00:56:45.149 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.150 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.150 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.151 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.151 DEBUG:boto:Method: GET 2023-10-12T00:56:45.152 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.152 DEBUG:boto:Data: 2023-10-12T00:56:45.153 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.153 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.154 DEBUG:boto:Params: {} 2023-10-12T00:56:45.154 DEBUG:boto:Token: None 2023-10-12T00:56:45.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI= 2023-10-12T00:56:45.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI=', 'Content-Length': '0'} 2023-10-12T00:56:45.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c678fdd548e0baa3-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.202 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.203 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.203 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.204 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.204 DEBUG:boto:Method: GET 2023-10-12T00:56:45.205 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.205 DEBUG:boto:Data: 2023-10-12T00:56:45.206 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.206 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.207 DEBUG:boto:Params: {} 2023-10-12T00:56:45.207 DEBUG:boto:Token: None 2023-10-12T00:56:45.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014?versionId=null 2023-10-12T00:56:45.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI= 2023-10-12T00:56:45.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DDYMxHu2wtPhL0ukQdr6vFYxSAI=', 'Content-Length': '0'} 2023-10-12T00:56:45.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008305cf6958d30149-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.212 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.212 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.213 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.213 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.214 DEBUG:boto:Method: GET 2023-10-12T00:56:45.214 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.215 DEBUG:boto:Data: 2023-10-12T00:56:45.215 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.216 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.216 DEBUG:boto:Params: {} 2023-10-12T00:56:45.217 DEBUG:boto:Token: None 2023-10-12T00:56:45.217 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec1d99e8e6547cae-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.221 DEBUG:boto:b'wafmoa-58obj000000000000000000141000falseobj000000000000000000142023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.222 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.223 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.223 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.224 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.224 DEBUG:boto:Method: GET 2023-10-12T00:56:45.224 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.225 DEBUG:boto:Data: 2023-10-12T00:56:45.225 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.226 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.227 DEBUG:boto:Params: {} 2023-10-12T00:56:45.227 DEBUG:boto:Token: None 2023-10-12T00:56:45.228 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c07c4f6740a6756-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.233 DEBUG:boto:b'wafmoa-58obj000000000000000000141000falseobj000000000000000000142023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.234 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.234 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.235 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.235 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.236 DEBUG:boto:Data: 2023-10-12T00:56:45.236 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.237 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.238 DEBUG:boto:Params: {} 2023-10-12T00:56:45.238 DEBUG:boto:Token: None 2023-10-12T00:56:45.239 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D/TlxJoIIzC439htfM5V1m3qsRo= 2023-10-12T00:56:45.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D/TlxJoIIzC439htfM5V1m3qsRo=', 'Content-Length': '0'} 2023-10-12T00:56:45.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48c7563b0af335b-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.242 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.243 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.243 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.243 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.244 DEBUG:boto:Data: 2023-10-12T00:56:45.244 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.245 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.245 DEBUG:boto:Params: {} 2023-10-12T00:56:45.246 DEBUG:boto:Token: None 2023-10-12T00:56:45.247 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D/TlxJoIIzC439htfM5V1m3qsRo= 2023-10-12T00:56:45.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D/TlxJoIIzC439htfM5V1m3qsRo=', 'Content-Length': '0'} 2023-10-12T00:56:45.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b472f341fbb7dd5c-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-12T00:56:45.250 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.251 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.251 DEBUG:boto:Method: GET 2023-10-12T00:56:45.252 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.252 DEBUG:boto:Data: 2023-10-12T00:56:45.253 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.253 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.254 DEBUG:boto:Params: {} 2023-10-12T00:56:45.254 DEBUG:boto:Token: None 2023-10-12T00:56:45.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vteFn+La6O3oK5+zQ6GnqYWObwY= 2023-10-12T00:56:45.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vteFn+La6O3oK5+zQ6GnqYWObwY=', 'Content-Length': '0'} 2023-10-12T00:56:45.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d78a91f4392557-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.259 DEBUG:boto:path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.259 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.260 DEBUG:boto:Method: GET 2023-10-12T00:56:45.260 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.260 DEBUG:boto:Data: 2023-10-12T00:56:45.261 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.262 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.262 DEBUG:boto:Params: {} 2023-10-12T00:56:45.263 DEBUG:boto:Token: None 2023-10-12T00:56:45.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000014 2023-10-12T00:56:45.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vteFn+La6O3oK5+zQ6GnqYWObwY= 2023-10-12T00:56:45.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vteFn+La6O3oK5+zQ6GnqYWObwY=', 'Content-Length': '0'} 2023-10-12T00:56:45.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006905a2ec3e02552-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.306 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.307 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.307 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.308 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.308 DEBUG:boto:Method: GET 2023-10-12T00:56:45.308 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.309 DEBUG:boto:Data: 2023-10-12T00:56:45.309 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.310 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.310 DEBUG:boto:Params: {} 2023-10-12T00:56:45.311 DEBUG:boto:Token: None 2023-10-12T00:56:45.311 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808e3c3d71f2f009-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.315 DEBUG:boto:b'wafmoa-58obj000000000000000000141000falseobj000000000000000000142023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.316 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.316 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.317 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.317 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.318 DEBUG:boto:Method: GET 2023-10-12T00:56:45.318 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000014 2023-10-12T00:56:45.319 DEBUG:boto:Data: 2023-10-12T00:56:45.319 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.320 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.320 DEBUG:boto:Params: {} 2023-10-12T00:56:45.321 DEBUG:boto:Token: None 2023-10-12T00:56:45.321 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030c5c33dcc13df3f-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.328 DEBUG:boto:b'wafmoa-58obj000000000000000000141000falseobj000000000000000000142023-10-12T00:56:36.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-12T00:56:45.329 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.329 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.330 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.330 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.330 DEBUG:boto:Method: GET 2023-10-12T00:56:45.331 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.331 DEBUG:boto:Data: 2023-10-12T00:56:45.332 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.333 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.333 DEBUG:boto:Params: {} 2023-10-12T00:56:45.333 DEBUG:boto:Token: None 2023-10-12T00:56:45.334 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M= 2023-10-12T00:56:45.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M=', 'Content-Length': '0'} 2023-10-12T00:56:45.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a78441ca19e81b68-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.338 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.338 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.339 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.339 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.340 DEBUG:boto:Method: GET 2023-10-12T00:56:45.340 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.340 DEBUG:boto:Data: 2023-10-12T00:56:45.341 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.342 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.342 DEBUG:boto:Params: {} 2023-10-12T00:56:45.343 DEBUG:boto:Token: None 2023-10-12T00:56:45.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M= 2023-10-12T00:56:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M=', 'Content-Length': '0'} 2023-10-12T00:56:45.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae8dd8bb0b17b61-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.347 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.347 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.348 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.348 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.348 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.349 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.349 DEBUG:boto:Data: 2023-10-12T00:56:45.349 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.350 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.351 DEBUG:boto:Params: {} 2023-10-12T00:56:45.351 DEBUG:boto:Token: None 2023-10-12T00:56:45.351 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QznA+buBCBzWqVUDcGEF61VfrcQ= 2023-10-12T00:56:45.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QznA+buBCBzWqVUDcGEF61VfrcQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cffc7a6451b4f9e-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.355 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.355 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.356 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.356 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.356 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.357 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.357 DEBUG:boto:Data: 2023-10-12T00:56:45.358 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.358 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.359 DEBUG:boto:Params: {} 2023-10-12T00:56:45.359 DEBUG:boto:Token: None 2023-10-12T00:56:45.360 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QznA+buBCBzWqVUDcGEF61VfrcQ= 2023-10-12T00:56:45.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QznA+buBCBzWqVUDcGEF61VfrcQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec2b24e05871dfd-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-12T00:56:45.363 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.364 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.364 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.364 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.365 DEBUG:boto:Method: GET 2023-10-12T00:56:45.365 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.366 DEBUG:boto:Data: 2023-10-12T00:56:45.366 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.367 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.367 DEBUG:boto:Params: {} 2023-10-12T00:56:45.368 DEBUG:boto:Token: None 2023-10-12T00:56:45.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M= 2023-10-12T00:56:45.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M=', 'Content-Length': '0'} 2023-10-12T00:56:45.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f0eafa8de5724e-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.414 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.415 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.415 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.416 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.416 DEBUG:boto:Method: GET 2023-10-12T00:56:45.417 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.417 DEBUG:boto:Data: 2023-10-12T00:56:45.418 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.418 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.419 DEBUG:boto:Params: {} 2023-10-12T00:56:45.419 DEBUG:boto:Token: None 2023-10-12T00:56:45.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015?versionId=null 2023-10-12T00:56:45.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M= 2023-10-12T00:56:45.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eZXDHLATM3KH7WBzXNSJNQZJ71M=', 'Content-Length': '0'} 2023-10-12T00:56:45.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337b681b87f2e28d-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.424 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.424 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.425 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.425 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.426 DEBUG:boto:Method: GET 2023-10-12T00:56:45.426 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.426 DEBUG:boto:Data: 2023-10-12T00:56:45.427 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.428 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.428 DEBUG:boto:Params: {} 2023-10-12T00:56:45.429 DEBUG:boto:Token: None 2023-10-12T00:56:45.429 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.430 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006830e8f3a559a400-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.433 DEBUG:boto:b'wafmoa-58obj000000000000000000151000falseobj000000000000000000152023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.434 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.435 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.435 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.435 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.436 DEBUG:boto:Method: GET 2023-10-12T00:56:45.436 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.437 DEBUG:boto:Data: 2023-10-12T00:56:45.437 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.438 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.439 DEBUG:boto:Params: {} 2023-10-12T00:56:45.439 DEBUG:boto:Token: None 2023-10-12T00:56:45.439 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa1ad094647f0419-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.445 DEBUG:boto:b'wafmoa-58obj000000000000000000151000falseobj000000000000000000152023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.446 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.446 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.447 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.447 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.448 DEBUG:boto:Data: 2023-10-12T00:56:45.448 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.449 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.449 DEBUG:boto:Params: {} 2023-10-12T00:56:45.450 DEBUG:boto:Token: None 2023-10-12T00:56:45.450 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xado8AJF7rsjy8L27DIuVmAf55c= 2023-10-12T00:56:45.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xado8AJF7rsjy8L27DIuVmAf55c=', 'Content-Length': '0'} 2023-10-12T00:56:45.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683e26ae6fc42d1c-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.454 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.454 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.455 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.455 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.455 DEBUG:boto:Data: 2023-10-12T00:56:45.456 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.457 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.457 DEBUG:boto:Params: {} 2023-10-12T00:56:45.458 DEBUG:boto:Token: None 2023-10-12T00:56:45.458 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xado8AJF7rsjy8L27DIuVmAf55c= 2023-10-12T00:56:45.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xado8AJF7rsjy8L27DIuVmAf55c=', 'Content-Length': '0'} 2023-10-12T00:56:45.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc1c3d8953b6c6c-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-12T00:56:45.463 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.463 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.463 DEBUG:boto:Method: GET 2023-10-12T00:56:45.464 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.464 DEBUG:boto:Data: 2023-10-12T00:56:45.465 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.466 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.466 DEBUG:boto:Params: {} 2023-10-12T00:56:45.467 DEBUG:boto:Token: None 2023-10-12T00:56:45.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xGlGbV46vY+jfzzUh2U9tVkCyz8= 2023-10-12T00:56:45.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xGlGbV46vY+jfzzUh2U9tVkCyz8=', 'Content-Length': '0'} 2023-10-12T00:56:45.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06e966116eee3cb-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.471 DEBUG:boto:path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.472 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.472 DEBUG:boto:Method: GET 2023-10-12T00:56:45.473 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.473 DEBUG:boto:Data: 2023-10-12T00:56:45.474 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.475 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.475 DEBUG:boto:Params: {} 2023-10-12T00:56:45.476 DEBUG:boto:Token: None 2023-10-12T00:56:45.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000015 2023-10-12T00:56:45.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xGlGbV46vY+jfzzUh2U9tVkCyz8= 2023-10-12T00:56:45.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xGlGbV46vY+jfzzUh2U9tVkCyz8=', 'Content-Length': '0'} 2023-10-12T00:56:45.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012edbf08a17ae705-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.480 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.481 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.481 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.482 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.482 DEBUG:boto:Method: GET 2023-10-12T00:56:45.483 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.483 DEBUG:boto:Data: 2023-10-12T00:56:45.483 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.484 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.485 DEBUG:boto:Params: {} 2023-10-12T00:56:45.485 DEBUG:boto:Token: None 2023-10-12T00:56:45.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e32b305ec7654a-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.490 DEBUG:boto:b'wafmoa-58obj000000000000000000151000falseobj000000000000000000152023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.491 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.491 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.492 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.492 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.492 DEBUG:boto:Method: GET 2023-10-12T00:56:45.493 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000015 2023-10-12T00:56:45.493 DEBUG:boto:Data: 2023-10-12T00:56:45.494 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.495 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.495 DEBUG:boto:Params: {} 2023-10-12T00:56:45.495 DEBUG:boto:Token: None 2023-10-12T00:56:45.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a86ef830f9178b1f-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.502 DEBUG:boto:b'wafmoa-58obj000000000000000000151000falseobj000000000000000000152023-10-12T00:56:34.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-12T00:56:45.503 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.503 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.504 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.504 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.504 DEBUG:boto:Method: GET 2023-10-12T00:56:45.505 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.505 DEBUG:boto:Data: 2023-10-12T00:56:45.506 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.507 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.507 DEBUG:boto:Params: {} 2023-10-12T00:56:45.508 DEBUG:boto:Token: None 2023-10-12T00:56:45.508 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE= 2023-10-12T00:56:45.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE=', 'Content-Length': '0'} 2023-10-12T00:56:45.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab6b22255fae85d-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.512 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.513 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.513 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.514 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.514 DEBUG:boto:Method: GET 2023-10-12T00:56:45.514 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.515 DEBUG:boto:Data: 2023-10-12T00:56:45.515 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.516 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.517 DEBUG:boto:Params: {} 2023-10-12T00:56:45.517 DEBUG:boto:Token: None 2023-10-12T00:56:45.518 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE= 2023-10-12T00:56:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE=', 'Content-Length': '0'} 2023-10-12T00:56:45.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005381c10b894c0c13-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.562 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.563 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.564 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.564 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.565 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.565 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.566 DEBUG:boto:Data: 2023-10-12T00:56:45.566 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.567 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.567 DEBUG:boto:Params: {} 2023-10-12T00:56:45.568 DEBUG:boto:Token: None 2023-10-12T00:56:45.568 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iK2bBfjtYnJe4uWBw7qINu3DBcc= 2023-10-12T00:56:45.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iK2bBfjtYnJe4uWBw7qINu3DBcc=', 'Content-Length': '0'} 2023-10-12T00:56:45.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eeaf46c4afee049-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.572 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.572 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.572 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.573 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.573 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.573 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.574 DEBUG:boto:Data: 2023-10-12T00:56:45.574 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.575 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.575 DEBUG:boto:Params: {} 2023-10-12T00:56:45.576 DEBUG:boto:Token: None 2023-10-12T00:56:45.576 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iK2bBfjtYnJe4uWBw7qINu3DBcc= 2023-10-12T00:56:45.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iK2bBfjtYnJe4uWBw7qINu3DBcc=', 'Content-Length': '0'} 2023-10-12T00:56:45.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7015f169a2a2c1-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-12T00:56:45.580 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.581 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.581 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.581 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.582 DEBUG:boto:Method: GET 2023-10-12T00:56:45.582 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.583 DEBUG:boto:Data: 2023-10-12T00:56:45.583 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.584 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.584 DEBUG:boto:Params: {} 2023-10-12T00:56:45.585 DEBUG:boto:Token: None 2023-10-12T00:56:45.585 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE= 2023-10-12T00:56:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE=', 'Content-Length': '0'} 2023-10-12T00:56:45.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc98c93b332772ef-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.634 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.635 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.636 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.636 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.636 DEBUG:boto:Method: GET 2023-10-12T00:56:45.637 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.637 DEBUG:boto:Data: 2023-10-12T00:56:45.638 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.639 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.639 DEBUG:boto:Params: {} 2023-10-12T00:56:45.639 DEBUG:boto:Token: None 2023-10-12T00:56:45.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016?versionId=null 2023-10-12T00:56:45.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE= 2023-10-12T00:56:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gA9YKhIeX1HibWtafzS0sSo75gE=', 'Content-Length': '0'} 2023-10-12T00:56:45.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b15c63e58be9d2e-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.644 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.645 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.645 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.646 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.646 DEBUG:boto:Method: GET 2023-10-12T00:56:45.647 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.647 DEBUG:boto:Data: 2023-10-12T00:56:45.648 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.648 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.649 DEBUG:boto:Params: {} 2023-10-12T00:56:45.649 DEBUG:boto:Token: None 2023-10-12T00:56:45.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b41595cb1a0849e-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.654 DEBUG:boto:b'wafmoa-58obj000000000000000000161000falseobj000000000000000000162023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.655 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.655 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.656 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.656 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.657 DEBUG:boto:Method: GET 2023-10-12T00:56:45.657 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.658 DEBUG:boto:Data: 2023-10-12T00:56:45.658 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.659 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.660 DEBUG:boto:Params: {} 2023-10-12T00:56:45.660 DEBUG:boto:Token: None 2023-10-12T00:56:45.661 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d69bbdf7bb3fc83-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.667 DEBUG:boto:b'wafmoa-58obj000000000000000000161000falseobj000000000000000000162023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.667 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.668 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.668 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.668 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.669 DEBUG:boto:Data: 2023-10-12T00:56:45.669 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.670 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.670 DEBUG:boto:Params: {} 2023-10-12T00:56:45.671 DEBUG:boto:Token: None 2023-10-12T00:56:45.671 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PfKos89iZY9NiJ4JNTxyYH4aisQ= 2023-10-12T00:56:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PfKos89iZY9NiJ4JNTxyYH4aisQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004138230c942ad09b-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.675 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.675 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.676 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.676 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.676 DEBUG:boto:Data: 2023-10-12T00:56:45.677 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.678 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.678 DEBUG:boto:Params: {} 2023-10-12T00:56:45.679 DEBUG:boto:Token: None 2023-10-12T00:56:45.679 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PfKos89iZY9NiJ4JNTxyYH4aisQ= 2023-10-12T00:56:45.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PfKos89iZY9NiJ4JNTxyYH4aisQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2148ed35f10a04-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-12T00:56:45.683 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.683 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.683 DEBUG:boto:Method: GET 2023-10-12T00:56:45.684 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.684 DEBUG:boto:Data: 2023-10-12T00:56:45.684 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.685 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.686 DEBUG:boto:Params: {} 2023-10-12T00:56:45.686 DEBUG:boto:Token: None 2023-10-12T00:56:45.687 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HA0fFLBAzEhLSZxI3EH3wN5MoBU= 2023-10-12T00:56:45.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HA0fFLBAzEhLSZxI3EH3wN5MoBU=', 'Content-Length': '0'} 2023-10-12T00:56:45.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cba93eec7cf4fa-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.690 DEBUG:boto:path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.691 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.691 DEBUG:boto:Method: GET 2023-10-12T00:56:45.691 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.692 DEBUG:boto:Data: 2023-10-12T00:56:45.692 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.693 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.693 DEBUG:boto:Params: {} 2023-10-12T00:56:45.694 DEBUG:boto:Token: None 2023-10-12T00:56:45.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000016 2023-10-12T00:56:45.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HA0fFLBAzEhLSZxI3EH3wN5MoBU= 2023-10-12T00:56:45.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HA0fFLBAzEhLSZxI3EH3wN5MoBU=', 'Content-Length': '0'} 2023-10-12T00:56:45.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097dd4591a0923985-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.738 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.739 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.739 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.740 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.740 DEBUG:boto:Method: GET 2023-10-12T00:56:45.741 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.741 DEBUG:boto:Data: 2023-10-12T00:56:45.741 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.742 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.743 DEBUG:boto:Params: {} 2023-10-12T00:56:45.743 DEBUG:boto:Token: None 2023-10-12T00:56:45.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a7d0ba6a537a07-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.748 DEBUG:boto:b'wafmoa-58obj000000000000000000161000falseobj000000000000000000162023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.749 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.749 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.750 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.750 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.751 DEBUG:boto:Method: GET 2023-10-12T00:56:45.751 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000016 2023-10-12T00:56:45.753 DEBUG:boto:Data: 2023-10-12T00:56:45.754 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.755 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.755 DEBUG:boto:Params: {} 2023-10-12T00:56:45.755 DEBUG:boto:Token: None 2023-10-12T00:56:45.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f15f479b87dbcb-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.762 DEBUG:boto:b'wafmoa-58obj000000000000000000161000falseobj000000000000000000162023-10-12T00:56:36.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-12T00:56:45.763 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.763 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.764 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.764 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.764 DEBUG:boto:Method: GET 2023-10-12T00:56:45.765 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.765 DEBUG:boto:Data: 2023-10-12T00:56:45.766 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.766 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.767 DEBUG:boto:Params: {} 2023-10-12T00:56:45.767 DEBUG:boto:Token: None 2023-10-12T00:56:45.768 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY= 2023-10-12T00:56:45.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY=', 'Content-Length': '0'} 2023-10-12T00:56:45.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc4136c86a07ecb-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.772 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.772 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.773 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.773 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.774 DEBUG:boto:Method: GET 2023-10-12T00:56:45.774 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.775 DEBUG:boto:Data: 2023-10-12T00:56:45.775 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.776 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.776 DEBUG:boto:Params: {} 2023-10-12T00:56:45.777 DEBUG:boto:Token: None 2023-10-12T00:56:45.777 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.778 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY= 2023-10-12T00:56:45.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY=', 'Content-Length': '0'} 2023-10-12T00:56:45.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606aaa21f3e5fd52-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.781 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.781 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.782 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.782 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.783 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.783 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.784 DEBUG:boto:Data: 2023-10-12T00:56:45.784 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.785 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.786 DEBUG:boto:Params: {} 2023-10-12T00:56:45.786 DEBUG:boto:Token: None 2023-10-12T00:56:45.787 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mk2iHSTRMbuPi5Tf1FycG7N2pEk= 2023-10-12T00:56:45.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mk2iHSTRMbuPi5Tf1FycG7N2pEk=', 'Content-Length': '0'} 2023-10-12T00:56:45.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa54c9669fadca3-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.790 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.790 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.791 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.791 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.792 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.792 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.792 DEBUG:boto:Data: 2023-10-12T00:56:45.793 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.794 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.794 DEBUG:boto:Params: {} 2023-10-12T00:56:45.795 DEBUG:boto:Token: None 2023-10-12T00:56:45.795 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.796 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mk2iHSTRMbuPi5Tf1FycG7N2pEk= 2023-10-12T00:56:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mk2iHSTRMbuPi5Tf1FycG7N2pEk=', 'Content-Length': '0'} 2023-10-12T00:56:45.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3fded15f7bc96c4-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-12T00:56:45.799 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.800 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.800 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.800 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.801 DEBUG:boto:Method: GET 2023-10-12T00:56:45.801 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.802 DEBUG:boto:Data: 2023-10-12T00:56:45.802 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.803 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.803 DEBUG:boto:Params: {} 2023-10-12T00:56:45.804 DEBUG:boto:Token: None 2023-10-12T00:56:45.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.805 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY= 2023-10-12T00:56:45.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY=', 'Content-Length': '0'} 2023-10-12T00:56:45.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c569a43e9f98d08c-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.850 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.851 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.851 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.851 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.852 DEBUG:boto:Method: GET 2023-10-12T00:56:45.852 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.853 DEBUG:boto:Data: 2023-10-12T00:56:45.853 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.854 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.854 DEBUG:boto:Params: {} 2023-10-12T00:56:45.855 DEBUG:boto:Token: None 2023-10-12T00:56:45.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017?versionId=null 2023-10-12T00:56:45.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY= 2023-10-12T00:56:45.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bTP7IrnaHab4qDxYuAmnjoWltEY=', 'Content-Length': '0'} 2023-10-12T00:56:45.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1084ddc6c42209c-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.859 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.860 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.860 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.861 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.861 DEBUG:boto:Method: GET 2023-10-12T00:56:45.862 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.862 DEBUG:boto:Data: 2023-10-12T00:56:45.862 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.863 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.864 DEBUG:boto:Params: {} 2023-10-12T00:56:45.864 DEBUG:boto:Token: None 2023-10-12T00:56:45.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027752f97df3af40f-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.869 DEBUG:boto:b'wafmoa-58obj000000000000000000171000falseobj000000000000000000172023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.870 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.870 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.871 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.871 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.871 DEBUG:boto:Method: GET 2023-10-12T00:56:45.872 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.872 DEBUG:boto:Data: 2023-10-12T00:56:45.873 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.874 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.874 DEBUG:boto:Params: {} 2023-10-12T00:56:45.874 DEBUG:boto:Token: None 2023-10-12T00:56:45.875 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a7c9526686175f-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.922 DEBUG:boto:b'wafmoa-58obj000000000000000000171000falseobj000000000000000000172023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.923 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.923 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.924 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.924 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.925 DEBUG:boto:Data: 2023-10-12T00:56:45.925 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.926 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.926 DEBUG:boto:Params: {} 2023-10-12T00:56:45.927 DEBUG:boto:Token: None 2023-10-12T00:56:45.927 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EvZVR9VW4cIgRaKuxBQWn+2SOM8= 2023-10-12T00:56:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EvZVR9VW4cIgRaKuxBQWn+2SOM8=', 'Content-Length': '0'} 2023-10-12T00:56:45.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5696d8cc38b46f8-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.930 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.931 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.931 DEBUG:boto:Method: HEAD 2023-10-12T00:56:45.932 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.932 DEBUG:boto:Data: 2023-10-12T00:56:45.933 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.933 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.934 DEBUG:boto:Params: {} 2023-10-12T00:56:45.934 DEBUG:boto:Token: None 2023-10-12T00:56:45.935 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.935 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EvZVR9VW4cIgRaKuxBQWn+2SOM8= 2023-10-12T00:56:45.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EvZVR9VW4cIgRaKuxBQWn+2SOM8=', 'Content-Length': '0'} 2023-10-12T00:56:45.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bb3b7d05937f50-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-12T00:56:45.938 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.939 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.939 DEBUG:boto:Method: GET 2023-10-12T00:56:45.940 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.940 DEBUG:boto:Data: 2023-10-12T00:56:45.941 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.941 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.942 DEBUG:boto:Params: {} 2023-10-12T00:56:45.942 DEBUG:boto:Token: None 2023-10-12T00:56:45.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51YVFjEVLXNisWM2Jm6suus5kaY= 2023-10-12T00:56:45.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51YVFjEVLXNisWM2Jm6suus5kaY=', 'Content-Length': '0'} 2023-10-12T00:56:45.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014551bdd6ddcbd08-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.947 DEBUG:boto:path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.947 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.948 DEBUG:boto:Method: GET 2023-10-12T00:56:45.948 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.948 DEBUG:boto:Data: 2023-10-12T00:56:45.949 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.950 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.950 DEBUG:boto:Params: {} 2023-10-12T00:56:45.951 DEBUG:boto:Token: None 2023-10-12T00:56:45.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000017 2023-10-12T00:56:45.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51YVFjEVLXNisWM2Jm6suus5kaY= 2023-10-12T00:56:45.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51YVFjEVLXNisWM2Jm6suus5kaY=', 'Content-Length': '0'} 2023-10-12T00:56:45.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054576f879f59f0c6-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.955 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.955 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.956 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.956 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.957 DEBUG:boto:Method: GET 2023-10-12T00:56:45.957 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.957 DEBUG:boto:Data: 2023-10-12T00:56:45.958 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.959 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.959 DEBUG:boto:Params: {} 2023-10-12T00:56:45.960 DEBUG:boto:Token: None 2023-10-12T00:56:45.960 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b57d7dcbada8885-006527444d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.964 DEBUG:boto:b'wafmoa-58obj000000000000000000171000falseobj000000000000000000172023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.965 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:45.966 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:45.966 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.966 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.967 DEBUG:boto:Method: GET 2023-10-12T00:56:45.967 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000017 2023-10-12T00:56:45.968 DEBUG:boto:Data: 2023-10-12T00:56:45.968 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.969 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.969 DEBUG:boto:Params: {} 2023-10-12T00:56:45.970 DEBUG:boto:Token: None 2023-10-12T00:56:45.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/ 2023-10-12T00:56:45.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs= 2023-10-12T00:56:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zi5wc1UM5pYiOMIt//v4m3j9VDs=', 'Content-Length': '0'} 2023-10-12T00:56:45.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008f81a618d3ae35-006527444d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.976 DEBUG:boto:b'wafmoa-58obj000000000000000000171000falseobj000000000000000000172023-10-12T00:56:36.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:45.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-12T00:56:45.977 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:45.978 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:45.978 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.979 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.979 DEBUG:boto:Method: GET 2023-10-12T00:56:45.980 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.980 DEBUG:boto:Data: 2023-10-12T00:56:45.980 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:45.981 DEBUG:boto:Port: 8001 2023-10-12T00:56:45.982 DEBUG:boto:Params: {} 2023-10-12T00:56:45.982 DEBUG:boto:Token: None 2023-10-12T00:56:45.983 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.983 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vTGbjxwBQgQikiear0lB8m2uAfQ= 2023-10-12T00:56:45.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vTGbjxwBQgQikiear0lB8m2uAfQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008258469159a2ea27-006527444d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:45.986 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:45.987 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:45.987 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.988 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.988 DEBUG:boto:Method: GET 2023-10-12T00:56:45.989 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.989 DEBUG:boto:Data: 2023-10-12T00:56:45.989 DEBUG:boto:Headers: {} 2023-10-12T00:56:45.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:45.990 DEBUG:boto:Port: 8000 2023-10-12T00:56:45.991 DEBUG:boto:Params: {} 2023-10-12T00:56:45.991 DEBUG:boto:Token: None 2023-10-12T00:56:45.992 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:45 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:45.992 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vTGbjxwBQgQikiear0lB8m2uAfQ= 2023-10-12T00:56:45.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vTGbjxwBQgQikiear0lB8m2uAfQ=', 'Content-Length': '0'} 2023-10-12T00:56:45.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2f428fa0b9feab-006527444d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.042 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.043 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.044 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.044 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.044 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.045 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.045 DEBUG:boto:Data: 2023-10-12T00:56:46.046 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.047 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.047 DEBUG:boto:Params: {} 2023-10-12T00:56:46.048 DEBUG:boto:Token: None 2023-10-12T00:56:46.048 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:apm5J2WycQ0tphjTdEcBLeTF9k4= 2023-10-12T00:56:46.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:apm5J2WycQ0tphjTdEcBLeTF9k4=', 'Content-Length': '0'} 2023-10-12T00:56:46.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303dc71a8ebed1f0-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.052 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.053 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.053 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.054 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.054 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.055 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.055 DEBUG:boto:Data: 2023-10-12T00:56:46.056 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.056 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.057 DEBUG:boto:Params: {} 2023-10-12T00:56:46.058 DEBUG:boto:Token: None 2023-10-12T00:56:46.058 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:apm5J2WycQ0tphjTdEcBLeTF9k4= 2023-10-12T00:56:46.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:apm5J2WycQ0tphjTdEcBLeTF9k4=', 'Content-Length': '0'} 2023-10-12T00:56:46.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f175ddb11c8e5e7-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-12T00:56:46.062 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.063 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.063 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.064 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.064 DEBUG:boto:Method: GET 2023-10-12T00:56:46.065 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.065 DEBUG:boto:Data: 2023-10-12T00:56:46.066 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.067 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.067 DEBUG:boto:Params: {} 2023-10-12T00:56:46.068 DEBUG:boto:Token: None 2023-10-12T00:56:46.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:70xM0JLd3BNO7ce5qWNXJueB5Zc= 2023-10-12T00:56:46.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:70xM0JLd3BNO7ce5qWNXJueB5Zc=', 'Content-Length': '0'} 2023-10-12T00:56:46.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067485f93e3d60ef7-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.072 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.073 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.073 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.074 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.074 DEBUG:boto:Method: GET 2023-10-12T00:56:46.075 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.075 DEBUG:boto:Data: 2023-10-12T00:56:46.076 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.077 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.077 DEBUG:boto:Params: {} 2023-10-12T00:56:46.078 DEBUG:boto:Token: None 2023-10-12T00:56:46.078 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018?versionId=null 2023-10-12T00:56:46.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:70xM0JLd3BNO7ce5qWNXJueB5Zc= 2023-10-12T00:56:46.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:70xM0JLd3BNO7ce5qWNXJueB5Zc=', 'Content-Length': '0'} 2023-10-12T00:56:46.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539a38161548d4b5-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.082 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.083 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.083 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.083 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.084 DEBUG:boto:Method: GET 2023-10-12T00:56:46.084 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.085 DEBUG:boto:Data: 2023-10-12T00:56:46.085 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.086 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.087 DEBUG:boto:Params: {} 2023-10-12T00:56:46.087 DEBUG:boto:Token: None 2023-10-12T00:56:46.088 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025dd0d258fa0a779-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.092 DEBUG:boto:b'wafmoa-58obj000000000000000000181000falseobj000000000000000000182023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.093 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.094 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.094 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.096 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.097 DEBUG:boto:Method: GET 2023-10-12T00:56:46.097 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.098 DEBUG:boto:Data: 2023-10-12T00:56:46.098 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.099 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.099 DEBUG:boto:Params: {} 2023-10-12T00:56:46.100 DEBUG:boto:Token: None 2023-10-12T00:56:46.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009b1f050db6c8b7-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.106 DEBUG:boto:b'wafmoa-58obj000000000000000000181000falseobj000000000000000000182023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.107 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.107 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.108 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.109 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.109 DEBUG:boto:Data: 2023-10-12T00:56:46.110 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.111 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.112 DEBUG:boto:Params: {} 2023-10-12T00:56:46.113 DEBUG:boto:Token: None 2023-10-12T00:56:46.114 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wBDVbD8XHjLmbW6ITec7XNPz9Qw= 2023-10-12T00:56:46.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wBDVbD8XHjLmbW6ITec7XNPz9Qw=', 'Content-Length': '0'} 2023-10-12T00:56:46.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003431ddde1af37501-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.118 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.119 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.119 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.119 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.120 DEBUG:boto:Data: 2023-10-12T00:56:46.120 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.121 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.121 DEBUG:boto:Params: {} 2023-10-12T00:56:46.122 DEBUG:boto:Token: None 2023-10-12T00:56:46.122 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.123 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wBDVbD8XHjLmbW6ITec7XNPz9Qw= 2023-10-12T00:56:46.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wBDVbD8XHjLmbW6ITec7XNPz9Qw=', 'Content-Length': '0'} 2023-10-12T00:56:46.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049fbadb12ca31aa5-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-12T00:56:46.126 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.127 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.127 DEBUG:boto:Method: GET 2023-10-12T00:56:46.128 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.128 DEBUG:boto:Data: 2023-10-12T00:56:46.129 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.129 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.130 DEBUG:boto:Params: {} 2023-10-12T00:56:46.130 DEBUG:boto:Token: None 2023-10-12T00:56:46.131 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q/AcH9ghJaNm8swnoBS9J2YdkKU= 2023-10-12T00:56:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q/AcH9ghJaNm8swnoBS9J2YdkKU=', 'Content-Length': '0'} 2023-10-12T00:56:46.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8358b6ee5d78959-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.174 DEBUG:boto:path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.176 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.176 DEBUG:boto:Method: GET 2023-10-12T00:56:46.177 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.177 DEBUG:boto:Data: 2023-10-12T00:56:46.177 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.178 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.179 DEBUG:boto:Params: {} 2023-10-12T00:56:46.179 DEBUG:boto:Token: None 2023-10-12T00:56:46.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000018 2023-10-12T00:56:46.180 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q/AcH9ghJaNm8swnoBS9J2YdkKU= 2023-10-12T00:56:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q/AcH9ghJaNm8swnoBS9J2YdkKU=', 'Content-Length': '0'} 2023-10-12T00:56:46.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001534eb25b5a0c263-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.184 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.184 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.184 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.185 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.185 DEBUG:boto:Method: GET 2023-10-12T00:56:46.186 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.186 DEBUG:boto:Data: 2023-10-12T00:56:46.187 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.187 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.188 DEBUG:boto:Params: {} 2023-10-12T00:56:46.188 DEBUG:boto:Token: None 2023-10-12T00:56:46.189 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e347d3b8741e5819-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.193 DEBUG:boto:b'wafmoa-58obj000000000000000000181000falseobj000000000000000000182023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.194 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.194 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.195 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.195 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.196 DEBUG:boto:Method: GET 2023-10-12T00:56:46.196 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000018 2023-10-12T00:56:46.197 DEBUG:boto:Data: 2023-10-12T00:56:46.197 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.198 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.198 DEBUG:boto:Params: {} 2023-10-12T00:56:46.199 DEBUG:boto:Token: None 2023-10-12T00:56:46.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f03b98abf06d92d-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.246 DEBUG:boto:b'wafmoa-58obj000000000000000000181000falseobj000000000000000000182023-10-12T00:56:37.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-12T00:56:46.248 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.249 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.249 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.250 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.251 DEBUG:boto:Method: GET 2023-10-12T00:56:46.251 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.252 DEBUG:boto:Data: 2023-10-12T00:56:46.253 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.253 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.254 DEBUG:boto:Params: {} 2023-10-12T00:56:46.254 DEBUG:boto:Token: None 2023-10-12T00:56:46.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4= 2023-10-12T00:56:46.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4=', 'Content-Length': '0'} 2023-10-12T00:56:46.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002794a9e803eb80f5-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.259 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.259 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.260 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.260 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.261 DEBUG:boto:Method: GET 2023-10-12T00:56:46.261 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.262 DEBUG:boto:Data: 2023-10-12T00:56:46.262 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.263 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.263 DEBUG:boto:Params: {} 2023-10-12T00:56:46.264 DEBUG:boto:Token: None 2023-10-12T00:56:46.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4= 2023-10-12T00:56:46.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4=', 'Content-Length': '0'} 2023-10-12T00:56:46.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cecdbc570eeed17-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.268 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.269 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.269 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.270 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.271 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.272 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.272 DEBUG:boto:Data: 2023-10-12T00:56:46.273 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.273 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.274 DEBUG:boto:Params: {} 2023-10-12T00:56:46.274 DEBUG:boto:Token: None 2023-10-12T00:56:46.275 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuKD90fIFsUse5ih/5FKnpP/oYY= 2023-10-12T00:56:46.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuKD90fIFsUse5ih/5FKnpP/oYY=', 'Content-Length': '0'} 2023-10-12T00:56:46.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bbe306a77c2350b-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.278 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.279 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.279 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.280 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.280 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.280 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.281 DEBUG:boto:Data: 2023-10-12T00:56:46.281 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.282 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.283 DEBUG:boto:Params: {} 2023-10-12T00:56:46.283 DEBUG:boto:Token: None 2023-10-12T00:56:46.283 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuKD90fIFsUse5ih/5FKnpP/oYY= 2023-10-12T00:56:46.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuKD90fIFsUse5ih/5FKnpP/oYY=', 'Content-Length': '0'} 2023-10-12T00:56:46.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000530b90856c4d72de-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-12T00:56:46.287 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.288 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.288 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.289 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.289 DEBUG:boto:Method: GET 2023-10-12T00:56:46.290 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.290 DEBUG:boto:Data: 2023-10-12T00:56:46.291 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.291 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.292 DEBUG:boto:Params: {} 2023-10-12T00:56:46.292 DEBUG:boto:Token: None 2023-10-12T00:56:46.293 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4= 2023-10-12T00:56:46.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4=', 'Content-Length': '0'} 2023-10-12T00:56:46.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f261b0080eafbb-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.297 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.298 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.299 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.300 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.301 DEBUG:boto:Method: GET 2023-10-12T00:56:46.301 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.302 DEBUG:boto:Data: 2023-10-12T00:56:46.303 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.305 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.305 DEBUG:boto:Params: {} 2023-10-12T00:56:46.306 DEBUG:boto:Token: None 2023-10-12T00:56:46.306 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019?versionId=null 2023-10-12T00:56:46.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4= 2023-10-12T00:56:46.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LuK1/tLWrgSts+f2Mn6tDRoqDD4=', 'Content-Length': '0'} 2023-10-12T00:56:46.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb3cef042e205e2-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.350 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.351 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.351 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.352 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.352 DEBUG:boto:Method: GET 2023-10-12T00:56:46.353 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.353 DEBUG:boto:Data: 2023-10-12T00:56:46.354 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.355 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.356 DEBUG:boto:Params: {} 2023-10-12T00:56:46.356 DEBUG:boto:Token: None 2023-10-12T00:56:46.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ed86e170b0c1e2-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.362 DEBUG:boto:b'wafmoa-58obj000000000000000000191000falseobj000000000000000000192023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.363 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.365 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.365 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.366 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.366 DEBUG:boto:Method: GET 2023-10-12T00:56:46.367 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.367 DEBUG:boto:Data: 2023-10-12T00:56:46.367 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.368 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.369 DEBUG:boto:Params: {} 2023-10-12T00:56:46.369 DEBUG:boto:Token: None 2023-10-12T00:56:46.370 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.370 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9225b0a1888409-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.376 DEBUG:boto:b'wafmoa-58obj000000000000000000191000falseobj000000000000000000192023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.376 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.377 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.377 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.378 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.378 DEBUG:boto:Data: 2023-10-12T00:56:46.378 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.379 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.380 DEBUG:boto:Params: {} 2023-10-12T00:56:46.380 DEBUG:boto:Token: None 2023-10-12T00:56:46.381 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4qeNEWdXKnr9tNAmC71yuB3GHxA= 2023-10-12T00:56:46.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4qeNEWdXKnr9tNAmC71yuB3GHxA=', 'Content-Length': '0'} 2023-10-12T00:56:46.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3b59b64610a387-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.385 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.385 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.385 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.386 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.386 DEBUG:boto:Data: 2023-10-12T00:56:46.387 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.388 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.388 DEBUG:boto:Params: {} 2023-10-12T00:56:46.389 DEBUG:boto:Token: None 2023-10-12T00:56:46.390 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4qeNEWdXKnr9tNAmC71yuB3GHxA= 2023-10-12T00:56:46.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4qeNEWdXKnr9tNAmC71yuB3GHxA=', 'Content-Length': '0'} 2023-10-12T00:56:46.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621dab1d8b253be9-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-12T00:56:46.396 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.397 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.397 DEBUG:boto:Method: GET 2023-10-12T00:56:46.398 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.398 DEBUG:boto:Data: 2023-10-12T00:56:46.399 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.400 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.400 DEBUG:boto:Params: {} 2023-10-12T00:56:46.400 DEBUG:boto:Token: None 2023-10-12T00:56:46.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yg5s2kJa7LF2SxcVbeba7mPWbEY= 2023-10-12T00:56:46.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yg5s2kJa7LF2SxcVbeba7mPWbEY=', 'Content-Length': '0'} 2023-10-12T00:56:46.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614aa153d02208ac-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.405 DEBUG:boto:path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.405 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.406 DEBUG:boto:Method: GET 2023-10-12T00:56:46.407 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.407 DEBUG:boto:Data: 2023-10-12T00:56:46.407 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.408 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.409 DEBUG:boto:Params: {} 2023-10-12T00:56:46.409 DEBUG:boto:Token: None 2023-10-12T00:56:46.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000019 2023-10-12T00:56:46.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yg5s2kJa7LF2SxcVbeba7mPWbEY= 2023-10-12T00:56:46.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yg5s2kJa7LF2SxcVbeba7mPWbEY=', 'Content-Length': '0'} 2023-10-12T00:56:46.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efdc119872a1e7c0-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.413 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.414 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.414 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.415 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.415 DEBUG:boto:Method: GET 2023-10-12T00:56:46.415 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.416 DEBUG:boto:Data: 2023-10-12T00:56:46.416 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.417 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.418 DEBUG:boto:Params: {} 2023-10-12T00:56:46.419 DEBUG:boto:Token: None 2023-10-12T00:56:46.419 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3112e614db6b5ac-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.423 DEBUG:boto:b'wafmoa-58obj000000000000000000191000falseobj000000000000000000192023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.424 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.425 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.425 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.426 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.427 DEBUG:boto:Method: GET 2023-10-12T00:56:46.427 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000019 2023-10-12T00:56:46.428 DEBUG:boto:Data: 2023-10-12T00:56:46.429 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.430 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.431 DEBUG:boto:Params: {} 2023-10-12T00:56:46.432 DEBUG:boto:Token: None 2023-10-12T00:56:46.433 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8aead9cc5737fb-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.439 DEBUG:boto:b'wafmoa-58obj000000000000000000191000falseobj000000000000000000192023-10-12T00:56:36.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-12T00:56:46.440 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.441 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.441 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.441 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.442 DEBUG:boto:Method: GET 2023-10-12T00:56:46.442 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.443 DEBUG:boto:Data: 2023-10-12T00:56:46.443 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.444 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.444 DEBUG:boto:Params: {} 2023-10-12T00:56:46.445 DEBUG:boto:Token: None 2023-10-12T00:56:46.445 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM= 2023-10-12T00:56:46.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM=', 'Content-Length': '0'} 2023-10-12T00:56:46.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102951dc50d5d177-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.494 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.495 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.496 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.497 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.498 DEBUG:boto:Method: GET 2023-10-12T00:56:46.498 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.499 DEBUG:boto:Data: 2023-10-12T00:56:46.500 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.501 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.502 DEBUG:boto:Params: {} 2023-10-12T00:56:46.502 DEBUG:boto:Token: None 2023-10-12T00:56:46.503 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM= 2023-10-12T00:56:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM=', 'Content-Length': '0'} 2023-10-12T00:56:46.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54eca7a3a6704ab-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.507 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.507 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.508 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.508 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.509 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.509 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.509 DEBUG:boto:Data: 2023-10-12T00:56:46.510 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.511 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.511 DEBUG:boto:Params: {} 2023-10-12T00:56:46.511 DEBUG:boto:Token: None 2023-10-12T00:56:46.512 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t7fyoVHDu2cDlBjkaEqYCuDF16A= 2023-10-12T00:56:46.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t7fyoVHDu2cDlBjkaEqYCuDF16A=', 'Content-Length': '0'} 2023-10-12T00:56:46.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afec64a751d9908f-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.515 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.516 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.516 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.517 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.517 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.517 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.518 DEBUG:boto:Data: 2023-10-12T00:56:46.518 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.519 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.520 DEBUG:boto:Params: {} 2023-10-12T00:56:46.521 DEBUG:boto:Token: None 2023-10-12T00:56:46.521 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t7fyoVHDu2cDlBjkaEqYCuDF16A= 2023-10-12T00:56:46.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t7fyoVHDu2cDlBjkaEqYCuDF16A=', 'Content-Length': '0'} 2023-10-12T00:56:46.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008984e3ea410cf83d-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-12T00:56:46.526 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.526 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.527 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.528 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.529 DEBUG:boto:Method: GET 2023-10-12T00:56:46.529 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.530 DEBUG:boto:Data: 2023-10-12T00:56:46.531 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.533 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.533 DEBUG:boto:Params: {} 2023-10-12T00:56:46.534 DEBUG:boto:Token: None 2023-10-12T00:56:46.535 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM= 2023-10-12T00:56:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM=', 'Content-Length': '0'} 2023-10-12T00:56:46.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045408acc8bb0123e-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.538 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.539 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.539 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.540 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.540 DEBUG:boto:Method: GET 2023-10-12T00:56:46.540 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.541 DEBUG:boto:Data: 2023-10-12T00:56:46.541 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.542 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.543 DEBUG:boto:Params: {} 2023-10-12T00:56:46.543 DEBUG:boto:Token: None 2023-10-12T00:56:46.544 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020?versionId=null 2023-10-12T00:56:46.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM= 2023-10-12T00:56:46.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:blLQvTfMKDNLxBu0LH9PW8ryofM=', 'Content-Length': '0'} 2023-10-12T00:56:46.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2e412fe6bcc22e-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.590 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.591 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.592 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.592 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.593 DEBUG:boto:Method: GET 2023-10-12T00:56:46.593 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.593 DEBUG:boto:Data: 2023-10-12T00:56:46.594 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.595 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.595 DEBUG:boto:Params: {} 2023-10-12T00:56:46.596 DEBUG:boto:Token: None 2023-10-12T00:56:46.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e183b35194d84383-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.601 DEBUG:boto:b'wafmoa-58obj000000000000000000201000falseobj000000000000000000202023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.602 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.603 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.604 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.604 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.605 DEBUG:boto:Method: GET 2023-10-12T00:56:46.606 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.607 DEBUG:boto:Data: 2023-10-12T00:56:46.607 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.609 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.609 DEBUG:boto:Params: {} 2023-10-12T00:56:46.610 DEBUG:boto:Token: None 2023-10-12T00:56:46.610 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc43082b58c81ce-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.616 DEBUG:boto:b'wafmoa-58obj000000000000000000201000falseobj000000000000000000202023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.617 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.617 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.618 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.618 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.618 DEBUG:boto:Data: 2023-10-12T00:56:46.619 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.620 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.620 DEBUG:boto:Params: {} 2023-10-12T00:56:46.620 DEBUG:boto:Token: None 2023-10-12T00:56:46.621 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0HQCZrCMzU308IDIMR/f+v/KJH8= 2023-10-12T00:56:46.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0HQCZrCMzU308IDIMR/f+v/KJH8=', 'Content-Length': '0'} 2023-10-12T00:56:46.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29159ec7f0bcecf-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.625 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.625 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.625 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.626 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.627 DEBUG:boto:Data: 2023-10-12T00:56:46.627 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.628 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.628 DEBUG:boto:Params: {} 2023-10-12T00:56:46.629 DEBUG:boto:Token: None 2023-10-12T00:56:46.629 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.630 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0HQCZrCMzU308IDIMR/f+v/KJH8= 2023-10-12T00:56:46.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0HQCZrCMzU308IDIMR/f+v/KJH8=', 'Content-Length': '0'} 2023-10-12T00:56:46.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9468a95075108e4-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-12T00:56:46.634 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.634 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.634 DEBUG:boto:Method: GET 2023-10-12T00:56:46.635 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.635 DEBUG:boto:Data: 2023-10-12T00:56:46.636 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.637 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.637 DEBUG:boto:Params: {} 2023-10-12T00:56:46.638 DEBUG:boto:Token: None 2023-10-12T00:56:46.638 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UtCQjmjnMbJSua/TqaLwjJHbcPU= 2023-10-12T00:56:46.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UtCQjmjnMbJSua/TqaLwjJHbcPU=', 'Content-Length': '0'} 2023-10-12T00:56:46.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdec6ab780f62341-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.642 DEBUG:boto:path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.642 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.643 DEBUG:boto:Method: GET 2023-10-12T00:56:46.643 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.644 DEBUG:boto:Data: 2023-10-12T00:56:46.644 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.645 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.645 DEBUG:boto:Params: {} 2023-10-12T00:56:46.646 DEBUG:boto:Token: None 2023-10-12T00:56:46.646 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000020 2023-10-12T00:56:46.647 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UtCQjmjnMbJSua/TqaLwjJHbcPU= 2023-10-12T00:56:46.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UtCQjmjnMbJSua/TqaLwjJHbcPU=', 'Content-Length': '0'} 2023-10-12T00:56:46.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f56546c9c2a4b6e-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.650 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.651 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.651 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.652 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.652 DEBUG:boto:Method: GET 2023-10-12T00:56:46.652 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.653 DEBUG:boto:Data: 2023-10-12T00:56:46.654 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.655 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.655 DEBUG:boto:Params: {} 2023-10-12T00:56:46.656 DEBUG:boto:Token: None 2023-10-12T00:56:46.656 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001df4881cca57d80-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.661 DEBUG:boto:b'wafmoa-58obj000000000000000000201000falseobj000000000000000000202023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.662 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.663 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.664 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.664 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.665 DEBUG:boto:Method: GET 2023-10-12T00:56:46.665 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000020 2023-10-12T00:56:46.666 DEBUG:boto:Data: 2023-10-12T00:56:46.667 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.668 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.669 DEBUG:boto:Params: {} 2023-10-12T00:56:46.669 DEBUG:boto:Token: None 2023-10-12T00:56:46.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.670 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c770f206f282e816-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.676 DEBUG:boto:b'wafmoa-58obj000000000000000000201000falseobj000000000000000000202023-10-12T00:56:37.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-12T00:56:46.678 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.679 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.679 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.680 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.680 DEBUG:boto:Method: GET 2023-10-12T00:56:46.680 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.681 DEBUG:boto:Data: 2023-10-12T00:56:46.681 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.682 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.683 DEBUG:boto:Params: {} 2023-10-12T00:56:46.683 DEBUG:boto:Token: None 2023-10-12T00:56:46.684 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM= 2023-10-12T00:56:46.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM=', 'Content-Length': '0'} 2023-10-12T00:56:46.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef825c1f70d6876a-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.688 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.688 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.689 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.689 DEBUG:boto:Method: GET 2023-10-12T00:56:46.690 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.690 DEBUG:boto:Data: 2023-10-12T00:56:46.690 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.691 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.692 DEBUG:boto:Params: {} 2023-10-12T00:56:46.692 DEBUG:boto:Token: None 2023-10-12T00:56:46.693 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM= 2023-10-12T00:56:46.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM=', 'Content-Length': '0'} 2023-10-12T00:56:46.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fea7c55578a3f4-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.742 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.743 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.743 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.744 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.744 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.744 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.745 DEBUG:boto:Data: 2023-10-12T00:56:46.745 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.746 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.746 DEBUG:boto:Params: {} 2023-10-12T00:56:46.747 DEBUG:boto:Token: None 2023-10-12T00:56:46.747 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Otiv8AQUZ7VB7HX5a/1N8RS0BlI= 2023-10-12T00:56:46.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Otiv8AQUZ7VB7HX5a/1N8RS0BlI=', 'Content-Length': '0'} 2023-10-12T00:56:46.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae7a7d819da44c17-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.751 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.752 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.752 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.752 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.753 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.753 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.754 DEBUG:boto:Data: 2023-10-12T00:56:46.754 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.755 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.755 DEBUG:boto:Params: {} 2023-10-12T00:56:46.756 DEBUG:boto:Token: None 2023-10-12T00:56:46.756 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Otiv8AQUZ7VB7HX5a/1N8RS0BlI= 2023-10-12T00:56:46.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Otiv8AQUZ7VB7HX5a/1N8RS0BlI=', 'Content-Length': '0'} 2023-10-12T00:56:46.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d265fb2101060a6-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-12T00:56:46.760 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.760 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.761 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.761 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.762 DEBUG:boto:Method: GET 2023-10-12T00:56:46.762 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.763 DEBUG:boto:Data: 2023-10-12T00:56:46.763 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.764 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.764 DEBUG:boto:Params: {} 2023-10-12T00:56:46.765 DEBUG:boto:Token: None 2023-10-12T00:56:46.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM= 2023-10-12T00:56:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM=', 'Content-Length': '0'} 2023-10-12T00:56:46.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369c1facad576a52-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.814 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.815 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.815 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.816 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.816 DEBUG:boto:Method: GET 2023-10-12T00:56:46.817 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.817 DEBUG:boto:Data: 2023-10-12T00:56:46.817 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.818 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.819 DEBUG:boto:Params: {} 2023-10-12T00:56:46.819 DEBUG:boto:Token: None 2023-10-12T00:56:46.820 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021?versionId=null 2023-10-12T00:56:46.820 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM= 2023-10-12T00:56:46.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PisOqDs0duktdcUb7213kb3RlYM=', 'Content-Length': '0'} 2023-10-12T00:56:46.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4521e0ba51f57f9-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.824 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.824 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.825 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.825 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.825 DEBUG:boto:Method: GET 2023-10-12T00:56:46.826 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.826 DEBUG:boto:Data: 2023-10-12T00:56:46.827 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.828 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.828 DEBUG:boto:Params: {} 2023-10-12T00:56:46.829 DEBUG:boto:Token: None 2023-10-12T00:56:46.829 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e166501282b64a2-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.833 DEBUG:boto:b'wafmoa-58obj000000000000000000211000falseobj000000000000000000212023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.834 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.834 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.835 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.835 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.836 DEBUG:boto:Method: GET 2023-10-12T00:56:46.836 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.836 DEBUG:boto:Data: 2023-10-12T00:56:46.837 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.838 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.838 DEBUG:boto:Params: {} 2023-10-12T00:56:46.839 DEBUG:boto:Token: None 2023-10-12T00:56:46.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2e23bf97a51739-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.845 DEBUG:boto:b'wafmoa-58obj000000000000000000211000falseobj000000000000000000212023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.846 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.847 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.847 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.848 DEBUG:boto:Data: 2023-10-12T00:56:46.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.849 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.849 DEBUG:boto:Params: {} 2023-10-12T00:56:46.850 DEBUG:boto:Token: None 2023-10-12T00:56:46.850 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yFqzIJ5cNUzNPh89xna1mkTqJxY= 2023-10-12T00:56:46.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yFqzIJ5cNUzNPh89xna1mkTqJxY=', 'Content-Length': '0'} 2023-10-12T00:56:46.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005cc8f3e5a6e6984-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.854 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.854 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.854 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.855 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.855 DEBUG:boto:Data: 2023-10-12T00:56:46.856 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.856 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.857 DEBUG:boto:Params: {} 2023-10-12T00:56:46.857 DEBUG:boto:Token: None 2023-10-12T00:56:46.857 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yFqzIJ5cNUzNPh89xna1mkTqJxY= 2023-10-12T00:56:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yFqzIJ5cNUzNPh89xna1mkTqJxY=', 'Content-Length': '0'} 2023-10-12T00:56:46.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b872d6a818b659-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-12T00:56:46.861 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.862 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.862 DEBUG:boto:Method: GET 2023-10-12T00:56:46.863 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.863 DEBUG:boto:Data: 2023-10-12T00:56:46.864 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.864 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.865 DEBUG:boto:Params: {} 2023-10-12T00:56:46.865 DEBUG:boto:Token: None 2023-10-12T00:56:46.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LFMGjseHRL8NU1YNh7b027frG+8= 2023-10-12T00:56:46.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LFMGjseHRL8NU1YNh7b027frG+8=', 'Content-Length': '0'} 2023-10-12T00:56:46.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd42a7b27154fac-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.869 DEBUG:boto:path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.870 DEBUG:boto:Method: GET 2023-10-12T00:56:46.870 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.871 DEBUG:boto:Data: 2023-10-12T00:56:46.871 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.872 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.872 DEBUG:boto:Params: {} 2023-10-12T00:56:46.873 DEBUG:boto:Token: None 2023-10-12T00:56:46.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000021 2023-10-12T00:56:46.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LFMGjseHRL8NU1YNh7b027frG+8= 2023-10-12T00:56:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LFMGjseHRL8NU1YNh7b027frG+8=', 'Content-Length': '0'} 2023-10-12T00:56:46.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8e330b5b3c60fc-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.922 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.923 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.924 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.925 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.926 DEBUG:boto:Method: GET 2023-10-12T00:56:46.926 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.927 DEBUG:boto:Data: 2023-10-12T00:56:46.928 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.929 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.930 DEBUG:boto:Params: {} 2023-10-12T00:56:46.931 DEBUG:boto:Token: None 2023-10-12T00:56:46.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a6acf264b0768a-006527444e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.936 DEBUG:boto:b'wafmoa-58obj000000000000000000211000falseobj000000000000000000212023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.937 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:46.937 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:46.938 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.938 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.939 DEBUG:boto:Method: GET 2023-10-12T00:56:46.939 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000021 2023-10-12T00:56:46.940 DEBUG:boto:Data: 2023-10-12T00:56:46.940 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.941 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.942 DEBUG:boto:Params: {} 2023-10-12T00:56:46.942 DEBUG:boto:Token: None 2023-10-12T00:56:46.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/ 2023-10-12T00:56:46.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ= 2023-10-12T00:56:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dy3cf2fMNXqS54ksD8oYezOoioQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075bf7b1c855aecc2-006527444e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.948 DEBUG:boto:b'wafmoa-58obj000000000000000000211000falseobj000000000000000000212023-10-12T00:56:36.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:46.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-12T00:56:46.949 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.950 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.950 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.951 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.951 DEBUG:boto:Method: GET 2023-10-12T00:56:46.952 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.953 DEBUG:boto:Data: 2023-10-12T00:56:46.953 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.955 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.956 DEBUG:boto:Params: {} 2023-10-12T00:56:46.957 DEBUG:boto:Token: None 2023-10-12T00:56:46.958 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ= 2023-10-12T00:56:46.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe27c111cf847b2-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.962 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.963 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.963 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.964 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.964 DEBUG:boto:Method: GET 2023-10-12T00:56:46.965 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.965 DEBUG:boto:Data: 2023-10-12T00:56:46.966 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.966 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.967 DEBUG:boto:Params: {} 2023-10-12T00:56:46.967 DEBUG:boto:Token: None 2023-10-12T00:56:46.968 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ= 2023-10-12T00:56:46.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ=', 'Content-Length': '0'} 2023-10-12T00:56:46.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c5fc913f7961d01-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.972 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.972 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.973 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.973 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.974 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.974 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.974 DEBUG:boto:Data: 2023-10-12T00:56:46.975 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.976 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.977 DEBUG:boto:Params: {} 2023-10-12T00:56:46.977 DEBUG:boto:Token: None 2023-10-12T00:56:46.978 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E8vwOU+e3x6gjcGDCc/ZpuWvHXA= 2023-10-12T00:56:46.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E8vwOU+e3x6gjcGDCc/ZpuWvHXA=', 'Content-Length': '0'} 2023-10-12T00:56:46.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b00899fb4961c9a-006527444e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.982 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.982 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.983 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.983 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.983 DEBUG:boto:Method: HEAD 2023-10-12T00:56:46.984 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.984 DEBUG:boto:Data: 2023-10-12T00:56:46.985 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:46.985 DEBUG:boto:Port: 8000 2023-10-12T00:56:46.986 DEBUG:boto:Params: {} 2023-10-12T00:56:46.987 DEBUG:boto:Token: None 2023-10-12T00:56:46.987 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E8vwOU+e3x6gjcGDCc/ZpuWvHXA= 2023-10-12T00:56:46.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E8vwOU+e3x6gjcGDCc/ZpuWvHXA=', 'Content-Length': '0'} 2023-10-12T00:56:46.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cabc39b3ebf32a79-006527444e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:46.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-12T00:56:46.993 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.994 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:46.994 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.995 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.995 DEBUG:boto:Method: GET 2023-10-12T00:56:46.995 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.996 DEBUG:boto:Data: 2023-10-12T00:56:46.997 DEBUG:boto:Headers: {} 2023-10-12T00:56:46.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:46.998 DEBUG:boto:Port: 8001 2023-10-12T00:56:46.998 DEBUG:boto:Params: {} 2023-10-12T00:56:46.999 DEBUG:boto:Token: None 2023-10-12T00:56:46.999 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:46 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:46.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ= 2023-10-12T00:56:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xdKJS8uwcfsLx5C7o2tGNUD1TAQ=', 'Content-Length': '0'} 2023-10-12T00:56:47.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b035bc406ef369b-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.003 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.003 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.004 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:47.004 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:47.004 DEBUG:boto:Method: GET 2023-10-12T00:56:47.005 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:47.005 DEBUG:boto:Data: 2023-10-12T00:56:47.006 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.007 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.007 DEBUG:boto:Params: {} 2023-10-12T00:56:47.007 DEBUG:boto:Token: None 2023-10-12T00:56:47.008 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000022?versionId=null 2023-10-12T00:56:47.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uMlUo/cHtvevYSnxF9G3UAtpb50= 2023-10-12T00:56:47.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uMlUo/cHtvevYSnxF9G3UAtpb50=', 'Content-Length': '0'} 2023-10-12T00:56:47.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58eb6c482cb3050-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.058 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.059 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.060 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.060 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.061 DEBUG:boto:Method: GET 2023-10-12T00:56:47.061 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.061 DEBUG:boto:Data: 2023-10-12T00:56:47.062 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.063 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.063 DEBUG:boto:Params: {} 2023-10-12T00:56:47.064 DEBUG:boto:Token: None 2023-10-12T00:56:47.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a5e5c28488533f-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.068 DEBUG:boto:b'wafmoa-58obj000000000000000000221000falseobj000000000000000000222023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.069 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.069 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.070 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.070 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.071 DEBUG:boto:Method: GET 2023-10-12T00:56:47.071 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.072 DEBUG:boto:Data: 2023-10-12T00:56:47.072 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.073 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.073 DEBUG:boto:Params: {} 2023-10-12T00:56:47.074 DEBUG:boto:Token: None 2023-10-12T00:56:47.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004050ea047f253b5e-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.080 DEBUG:boto:b'wafmoa-58obj000000000000000000221000falseobj000000000000000000222023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.080 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.081 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.081 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.082 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.082 DEBUG:boto:Data: 2023-10-12T00:56:47.082 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.083 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.084 DEBUG:boto:Params: {} 2023-10-12T00:56:47.084 DEBUG:boto:Token: None 2023-10-12T00:56:47.085 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ts1TfiQZPyZ7SfXdP8q5VtAvDpM= 2023-10-12T00:56:47.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ts1TfiQZPyZ7SfXdP8q5VtAvDpM=', 'Content-Length': '0'} 2023-10-12T00:56:47.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751ae5f19d72ccab-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.088 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.089 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.089 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.089 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.090 DEBUG:boto:Data: 2023-10-12T00:56:47.090 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.091 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.091 DEBUG:boto:Params: {} 2023-10-12T00:56:47.092 DEBUG:boto:Token: None 2023-10-12T00:56:47.092 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ts1TfiQZPyZ7SfXdP8q5VtAvDpM= 2023-10-12T00:56:47.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ts1TfiQZPyZ7SfXdP8q5VtAvDpM=', 'Content-Length': '0'} 2023-10-12T00:56:47.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072508adfc25a8240-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-12T00:56:47.096 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.097 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.097 DEBUG:boto:Method: GET 2023-10-12T00:56:47.098 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.098 DEBUG:boto:Data: 2023-10-12T00:56:47.099 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.099 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.100 DEBUG:boto:Params: {} 2023-10-12T00:56:47.100 DEBUG:boto:Token: None 2023-10-12T00:56:47.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1DvpZBmYAb5d0max2w2xw8KJ9rE= 2023-10-12T00:56:47.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1DvpZBmYAb5d0max2w2xw8KJ9rE=', 'Content-Length': '0'} 2023-10-12T00:56:47.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15efc4f6094609b-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.146 DEBUG:boto:path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.147 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.148 DEBUG:boto:Method: GET 2023-10-12T00:56:47.148 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.148 DEBUG:boto:Data: 2023-10-12T00:56:47.149 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.150 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.150 DEBUG:boto:Params: {} 2023-10-12T00:56:47.151 DEBUG:boto:Token: None 2023-10-12T00:56:47.151 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000022 2023-10-12T00:56:47.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1DvpZBmYAb5d0max2w2xw8KJ9rE= 2023-10-12T00:56:47.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1DvpZBmYAb5d0max2w2xw8KJ9rE=', 'Content-Length': '0'} 2023-10-12T00:56:47.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9d3826052f001d-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.155 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.156 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.156 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.157 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.157 DEBUG:boto:Method: GET 2023-10-12T00:56:47.157 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.158 DEBUG:boto:Data: 2023-10-12T00:56:47.158 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.159 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.160 DEBUG:boto:Params: {} 2023-10-12T00:56:47.160 DEBUG:boto:Token: None 2023-10-12T00:56:47.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9518b958c178b5-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.165 DEBUG:boto:b'wafmoa-58obj000000000000000000221000falseobj000000000000000000222023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.166 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.166 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.166 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.167 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.167 DEBUG:boto:Method: GET 2023-10-12T00:56:47.168 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000022 2023-10-12T00:56:47.168 DEBUG:boto:Data: 2023-10-12T00:56:47.169 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.170 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.170 DEBUG:boto:Params: {} 2023-10-12T00:56:47.171 DEBUG:boto:Token: None 2023-10-12T00:56:47.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091969df0285dd840-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.218 DEBUG:boto:b'wafmoa-58obj000000000000000000221000falseobj000000000000000000222023-10-12T00:56:35.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-12T00:56:47.220 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.220 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.221 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.221 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.222 DEBUG:boto:Method: GET 2023-10-12T00:56:47.222 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.223 DEBUG:boto:Data: 2023-10-12T00:56:47.223 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.224 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.225 DEBUG:boto:Params: {} 2023-10-12T00:56:47.225 DEBUG:boto:Token: None 2023-10-12T00:56:47.226 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato= 2023-10-12T00:56:47.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato=', 'Content-Length': '0'} 2023-10-12T00:56:47.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f1f3b411a5b1e03-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.230 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.231 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.231 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.232 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.232 DEBUG:boto:Method: GET 2023-10-12T00:56:47.233 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.234 DEBUG:boto:Data: 2023-10-12T00:56:47.234 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.236 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.236 DEBUG:boto:Params: {} 2023-10-12T00:56:47.237 DEBUG:boto:Token: None 2023-10-12T00:56:47.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato= 2023-10-12T00:56:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato=', 'Content-Length': '0'} 2023-10-12T00:56:47.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01aab2f8d5292a6-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.242 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.242 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.243 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.244 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.244 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.245 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.246 DEBUG:boto:Data: 2023-10-12T00:56:47.247 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.248 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.249 DEBUG:boto:Params: {} 2023-10-12T00:56:47.249 DEBUG:boto:Token: None 2023-10-12T00:56:47.249 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q2WWBoiRHaJdXDqDLSx3ozotitw= 2023-10-12T00:56:47.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q2WWBoiRHaJdXDqDLSx3ozotitw=', 'Content-Length': '0'} 2023-10-12T00:56:47.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a3594785ea1721-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.253 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.254 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.255 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.255 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.255 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.256 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.256 DEBUG:boto:Data: 2023-10-12T00:56:47.257 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.257 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.258 DEBUG:boto:Params: {} 2023-10-12T00:56:47.259 DEBUG:boto:Token: None 2023-10-12T00:56:47.259 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q2WWBoiRHaJdXDqDLSx3ozotitw= 2023-10-12T00:56:47.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q2WWBoiRHaJdXDqDLSx3ozotitw=', 'Content-Length': '0'} 2023-10-12T00:56:47.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff699ffee16e66f-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-12T00:56:47.265 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.265 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.266 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.267 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.268 DEBUG:boto:Method: GET 2023-10-12T00:56:47.268 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.269 DEBUG:boto:Data: 2023-10-12T00:56:47.270 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.271 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.272 DEBUG:boto:Params: {} 2023-10-12T00:56:47.273 DEBUG:boto:Token: None 2023-10-12T00:56:47.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.274 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato= 2023-10-12T00:56:47.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato=', 'Content-Length': '0'} 2023-10-12T00:56:47.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc009db5a8e58d3c-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.278 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.279 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.279 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.279 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.280 DEBUG:boto:Method: GET 2023-10-12T00:56:47.280 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.281 DEBUG:boto:Data: 2023-10-12T00:56:47.281 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.282 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.283 DEBUG:boto:Params: {} 2023-10-12T00:56:47.284 DEBUG:boto:Token: None 2023-10-12T00:56:47.284 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023?versionId=null 2023-10-12T00:56:47.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato= 2023-10-12T00:56:47.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IX/DX6DvIHwMND95v2VjHJXlato=', 'Content-Length': '0'} 2023-10-12T00:56:47.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f885797fa3c5700e-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.289 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.289 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.289 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.290 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.291 DEBUG:boto:Method: GET 2023-10-12T00:56:47.291 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.292 DEBUG:boto:Data: 2023-10-12T00:56:47.292 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.293 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.293 DEBUG:boto:Params: {} 2023-10-12T00:56:47.294 DEBUG:boto:Token: None 2023-10-12T00:56:47.294 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.295 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388248c180e4c9b1-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.299 DEBUG:boto:b'wafmoa-58obj000000000000000000231000falseobj000000000000000000232023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.300 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.300 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.300 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.301 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.301 DEBUG:boto:Method: GET 2023-10-12T00:56:47.302 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.302 DEBUG:boto:Data: 2023-10-12T00:56:47.303 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.304 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.304 DEBUG:boto:Params: {} 2023-10-12T00:56:47.305 DEBUG:boto:Token: None 2023-10-12T00:56:47.305 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.306 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e77ff5dbe5d06530-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.311 DEBUG:boto:b'wafmoa-58obj000000000000000000231000falseobj000000000000000000232023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.312 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.312 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.312 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.313 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.313 DEBUG:boto:Data: 2023-10-12T00:56:47.314 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.315 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.315 DEBUG:boto:Params: {} 2023-10-12T00:56:47.315 DEBUG:boto:Token: None 2023-10-12T00:56:47.316 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pV9pONhpnkD7ppXaJac7urxhNto= 2023-10-12T00:56:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pV9pONhpnkD7ppXaJac7urxhNto=', 'Content-Length': '0'} 2023-10-12T00:56:47.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46b469d6c277638-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.319 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.320 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.320 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.320 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.321 DEBUG:boto:Data: 2023-10-12T00:56:47.321 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.322 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.323 DEBUG:boto:Params: {} 2023-10-12T00:56:47.323 DEBUG:boto:Token: None 2023-10-12T00:56:47.323 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pV9pONhpnkD7ppXaJac7urxhNto= 2023-10-12T00:56:47.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pV9pONhpnkD7ppXaJac7urxhNto=', 'Content-Length': '0'} 2023-10-12T00:56:47.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8d7d77dc1c6654-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-12T00:56:47.327 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.328 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.328 DEBUG:boto:Method: GET 2023-10-12T00:56:47.329 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.329 DEBUG:boto:Data: 2023-10-12T00:56:47.329 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.330 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.331 DEBUG:boto:Params: {} 2023-10-12T00:56:47.331 DEBUG:boto:Token: None 2023-10-12T00:56:47.332 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AtWjotmRkmrvBG9FcqBgQbTLhwM= 2023-10-12T00:56:47.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AtWjotmRkmrvBG9FcqBgQbTLhwM=', 'Content-Length': '0'} 2023-10-12T00:56:47.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027dfe40c186abbaa-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.378 DEBUG:boto:path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.379 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.380 DEBUG:boto:Method: GET 2023-10-12T00:56:47.380 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.380 DEBUG:boto:Data: 2023-10-12T00:56:47.381 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.382 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.382 DEBUG:boto:Params: {} 2023-10-12T00:56:47.382 DEBUG:boto:Token: None 2023-10-12T00:56:47.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000023 2023-10-12T00:56:47.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AtWjotmRkmrvBG9FcqBgQbTLhwM= 2023-10-12T00:56:47.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AtWjotmRkmrvBG9FcqBgQbTLhwM=', 'Content-Length': '0'} 2023-10-12T00:56:47.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea861332e6986870-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.389 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.389 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.390 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.390 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.391 DEBUG:boto:Method: GET 2023-10-12T00:56:47.391 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.392 DEBUG:boto:Data: 2023-10-12T00:56:47.392 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.393 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.393 DEBUG:boto:Params: {} 2023-10-12T00:56:47.394 DEBUG:boto:Token: None 2023-10-12T00:56:47.394 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.395 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a4e1adc328aeda-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.399 DEBUG:boto:b'wafmoa-58obj000000000000000000231000falseobj000000000000000000232023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.399 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.400 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.400 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.401 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.401 DEBUG:boto:Method: GET 2023-10-12T00:56:47.401 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000023 2023-10-12T00:56:47.402 DEBUG:boto:Data: 2023-10-12T00:56:47.402 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.403 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.403 DEBUG:boto:Params: {} 2023-10-12T00:56:47.404 DEBUG:boto:Token: None 2023-10-12T00:56:47.404 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001820674a8e39556e-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.410 DEBUG:boto:b'wafmoa-58obj000000000000000000231000falseobj000000000000000000232023-10-12T00:56:35.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-12T00:56:47.411 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.411 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.412 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.412 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.413 DEBUG:boto:Method: GET 2023-10-12T00:56:47.413 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.413 DEBUG:boto:Data: 2023-10-12T00:56:47.414 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.415 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.415 DEBUG:boto:Params: {} 2023-10-12T00:56:47.415 DEBUG:boto:Token: None 2023-10-12T00:56:47.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A= 2023-10-12T00:56:47.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A=', 'Content-Length': '0'} 2023-10-12T00:56:47.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10bd265fa186a66-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.420 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.420 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.421 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.421 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.421 DEBUG:boto:Method: GET 2023-10-12T00:56:47.422 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.422 DEBUG:boto:Data: 2023-10-12T00:56:47.423 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.423 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.424 DEBUG:boto:Params: {} 2023-10-12T00:56:47.424 DEBUG:boto:Token: None 2023-10-12T00:56:47.425 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A= 2023-10-12T00:56:47.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A=', 'Content-Length': '0'} 2023-10-12T00:56:47.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3155effb3cd3ae2-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.429 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.429 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.429 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.430 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.430 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.431 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.431 DEBUG:boto:Data: 2023-10-12T00:56:47.431 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.432 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.433 DEBUG:boto:Params: {} 2023-10-12T00:56:47.433 DEBUG:boto:Token: None 2023-10-12T00:56:47.433 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HHoLMGqVttGdxCwUG+Y4u8NQJaA= 2023-10-12T00:56:47.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HHoLMGqVttGdxCwUG+Y4u8NQJaA=', 'Content-Length': '0'} 2023-10-12T00:56:47.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446dd5c5fc256a05-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.437 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.437 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.438 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.438 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.439 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.439 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.439 DEBUG:boto:Data: 2023-10-12T00:56:47.440 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.441 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.441 DEBUG:boto:Params: {} 2023-10-12T00:56:47.442 DEBUG:boto:Token: None 2023-10-12T00:56:47.442 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HHoLMGqVttGdxCwUG+Y4u8NQJaA= 2023-10-12T00:56:47.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HHoLMGqVttGdxCwUG+Y4u8NQJaA=', 'Content-Length': '0'} 2023-10-12T00:56:47.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007855d15f32f427c8-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-12T00:56:47.446 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.447 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.447 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.447 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.448 DEBUG:boto:Method: GET 2023-10-12T00:56:47.448 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.449 DEBUG:boto:Data: 2023-10-12T00:56:47.449 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.450 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.450 DEBUG:boto:Params: {} 2023-10-12T00:56:47.451 DEBUG:boto:Token: None 2023-10-12T00:56:47.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A= 2023-10-12T00:56:47.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A=', 'Content-Length': '0'} 2023-10-12T00:56:47.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91fe07db31ade09-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.455 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.455 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.456 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.456 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.456 DEBUG:boto:Method: GET 2023-10-12T00:56:47.457 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.457 DEBUG:boto:Data: 2023-10-12T00:56:47.458 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.458 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.459 DEBUG:boto:Params: {} 2023-10-12T00:56:47.459 DEBUG:boto:Token: None 2023-10-12T00:56:47.460 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024?versionId=null 2023-10-12T00:56:47.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A= 2023-10-12T00:56:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LeE6/19S7g/PvsVJxFC6YkNwA4A=', 'Content-Length': '0'} 2023-10-12T00:56:47.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a102672ac580091-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.463 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.464 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.464 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.465 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.465 DEBUG:boto:Method: GET 2023-10-12T00:56:47.465 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.466 DEBUG:boto:Data: 2023-10-12T00:56:47.466 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.467 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.468 DEBUG:boto:Params: {} 2023-10-12T00:56:47.468 DEBUG:boto:Token: None 2023-10-12T00:56:47.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000836046fbbe29897f-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.518 DEBUG:boto:b'wafmoa-58obj000000000000000000241000falseobj000000000000000000242023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.519 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.520 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.520 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.521 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.521 DEBUG:boto:Method: GET 2023-10-12T00:56:47.522 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.522 DEBUG:boto:Data: 2023-10-12T00:56:47.522 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.523 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.524 DEBUG:boto:Params: {} 2023-10-12T00:56:47.524 DEBUG:boto:Token: None 2023-10-12T00:56:47.525 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7b24063b07c4076-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.531 DEBUG:boto:b'wafmoa-58obj000000000000000000241000falseobj000000000000000000242023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.531 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.532 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.532 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.533 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.533 DEBUG:boto:Data: 2023-10-12T00:56:47.534 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.534 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.535 DEBUG:boto:Params: {} 2023-10-12T00:56:47.535 DEBUG:boto:Token: None 2023-10-12T00:56:47.536 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.536 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y/wAcOffVB8+Lf63RICf+6WcLKc= 2023-10-12T00:56:47.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y/wAcOffVB8+Lf63RICf+6WcLKc=', 'Content-Length': '0'} 2023-10-12T00:56:47.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084123e90034c18a3-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.539 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.540 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.540 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.541 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.541 DEBUG:boto:Data: 2023-10-12T00:56:47.541 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.542 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.543 DEBUG:boto:Params: {} 2023-10-12T00:56:47.543 DEBUG:boto:Token: None 2023-10-12T00:56:47.544 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y/wAcOffVB8+Lf63RICf+6WcLKc= 2023-10-12T00:56:47.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y/wAcOffVB8+Lf63RICf+6WcLKc=', 'Content-Length': '0'} 2023-10-12T00:56:47.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001579d9418f65a91c-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-12T00:56:47.548 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.549 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.549 DEBUG:boto:Method: GET 2023-10-12T00:56:47.550 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.550 DEBUG:boto:Data: 2023-10-12T00:56:47.551 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.551 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.552 DEBUG:boto:Params: {} 2023-10-12T00:56:47.552 DEBUG:boto:Token: None 2023-10-12T00:56:47.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hOmhYGyEdjko56M1Kj2BTF9fvi0= 2023-10-12T00:56:47.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hOmhYGyEdjko56M1Kj2BTF9fvi0=', 'Content-Length': '0'} 2023-10-12T00:56:47.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113e0e7fbd5c0456-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.557 DEBUG:boto:path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.558 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.558 DEBUG:boto:Method: GET 2023-10-12T00:56:47.559 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.560 DEBUG:boto:Data: 2023-10-12T00:56:47.560 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.562 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.562 DEBUG:boto:Params: {} 2023-10-12T00:56:47.563 DEBUG:boto:Token: None 2023-10-12T00:56:47.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000024 2023-10-12T00:56:47.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hOmhYGyEdjko56M1Kj2BTF9fvi0= 2023-10-12T00:56:47.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hOmhYGyEdjko56M1Kj2BTF9fvi0=', 'Content-Length': '0'} 2023-10-12T00:56:47.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eff5c2201f8660c-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.610 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.611 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.611 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.612 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.613 DEBUG:boto:Method: GET 2023-10-12T00:56:47.613 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.613 DEBUG:boto:Data: 2023-10-12T00:56:47.614 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.615 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.615 DEBUG:boto:Params: {} 2023-10-12T00:56:47.616 DEBUG:boto:Token: None 2023-10-12T00:56:47.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009057a2a51191d24b-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.621 DEBUG:boto:b'wafmoa-58obj000000000000000000241000falseobj000000000000000000242023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.621 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.622 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.622 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.623 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.623 DEBUG:boto:Method: GET 2023-10-12T00:56:47.624 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000024 2023-10-12T00:56:47.624 DEBUG:boto:Data: 2023-10-12T00:56:47.624 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.625 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.626 DEBUG:boto:Params: {} 2023-10-12T00:56:47.626 DEBUG:boto:Token: None 2023-10-12T00:56:47.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59bc9ca574e0d92-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.633 DEBUG:boto:b'wafmoa-58obj000000000000000000241000falseobj000000000000000000242023-10-12T00:56:36.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-12T00:56:47.634 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.634 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.635 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.635 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.635 DEBUG:boto:Method: GET 2023-10-12T00:56:47.636 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.636 DEBUG:boto:Data: 2023-10-12T00:56:47.637 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.638 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.638 DEBUG:boto:Params: {} 2023-10-12T00:56:47.639 DEBUG:boto:Token: None 2023-10-12T00:56:47.639 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss= 2023-10-12T00:56:47.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss=', 'Content-Length': '0'} 2023-10-12T00:56:47.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fcc486a1a1d8bae-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.643 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.643 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.644 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.644 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.645 DEBUG:boto:Method: GET 2023-10-12T00:56:47.645 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.645 DEBUG:boto:Data: 2023-10-12T00:56:47.646 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.647 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.647 DEBUG:boto:Params: {} 2023-10-12T00:56:47.648 DEBUG:boto:Token: None 2023-10-12T00:56:47.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss= 2023-10-12T00:56:47.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss=', 'Content-Length': '0'} 2023-10-12T00:56:47.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cba0ff123f2a66e-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.652 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.652 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.653 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.653 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.653 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.654 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.654 DEBUG:boto:Data: 2023-10-12T00:56:47.655 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.655 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.656 DEBUG:boto:Params: {} 2023-10-12T00:56:47.656 DEBUG:boto:Token: None 2023-10-12T00:56:47.657 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NyLn3d1achFMS++03gajEHr1J5k= 2023-10-12T00:56:47.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NyLn3d1achFMS++03gajEHr1J5k=', 'Content-Length': '0'} 2023-10-12T00:56:47.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f47d95baefef845-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.660 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.661 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.661 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.662 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.662 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.663 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.663 DEBUG:boto:Data: 2023-10-12T00:56:47.663 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.664 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.665 DEBUG:boto:Params: {} 2023-10-12T00:56:47.665 DEBUG:boto:Token: None 2023-10-12T00:56:47.666 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.666 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NyLn3d1achFMS++03gajEHr1J5k= 2023-10-12T00:56:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NyLn3d1achFMS++03gajEHr1J5k=', 'Content-Length': '0'} 2023-10-12T00:56:47.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228099ef47f272c5-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-12T00:56:47.670 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.670 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.671 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.671 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.672 DEBUG:boto:Method: GET 2023-10-12T00:56:47.672 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.673 DEBUG:boto:Data: 2023-10-12T00:56:47.673 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.674 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.674 DEBUG:boto:Params: {} 2023-10-12T00:56:47.675 DEBUG:boto:Token: None 2023-10-12T00:56:47.675 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss= 2023-10-12T00:56:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss=', 'Content-Length': '0'} 2023-10-12T00:56:47.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a076785d0553f2-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.722 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.723 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.723 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.724 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.724 DEBUG:boto:Method: GET 2023-10-12T00:56:47.725 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.725 DEBUG:boto:Data: 2023-10-12T00:56:47.726 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.727 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.727 DEBUG:boto:Params: {} 2023-10-12T00:56:47.728 DEBUG:boto:Token: None 2023-10-12T00:56:47.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025?versionId=null 2023-10-12T00:56:47.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss= 2023-10-12T00:56:47.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7NmR86PCPUuMX0ozb7ysHHne5Ss=', 'Content-Length': '0'} 2023-10-12T00:56:47.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ffd95adc30ccc2-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.732 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.733 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.733 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.733 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.734 DEBUG:boto:Method: GET 2023-10-12T00:56:47.735 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.735 DEBUG:boto:Data: 2023-10-12T00:56:47.736 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.736 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.737 DEBUG:boto:Params: {} 2023-10-12T00:56:47.737 DEBUG:boto:Token: None 2023-10-12T00:56:47.738 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b4ac84ce81f13f7-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.742 DEBUG:boto:b'wafmoa-58obj000000000000000000251000falseobj000000000000000000252023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.743 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.743 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.744 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.744 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.745 DEBUG:boto:Method: GET 2023-10-12T00:56:47.745 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.746 DEBUG:boto:Data: 2023-10-12T00:56:47.746 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.747 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.747 DEBUG:boto:Params: {} 2023-10-12T00:56:47.748 DEBUG:boto:Token: None 2023-10-12T00:56:47.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd24f7fd37ad065-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.754 DEBUG:boto:b'wafmoa-58obj000000000000000000251000falseobj000000000000000000252023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.755 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.755 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.756 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.756 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.757 DEBUG:boto:Data: 2023-10-12T00:56:47.757 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.758 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.758 DEBUG:boto:Params: {} 2023-10-12T00:56:47.759 DEBUG:boto:Token: None 2023-10-12T00:56:47.759 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Npfs548+ozPYRtf1YqwH8rw9nOM= 2023-10-12T00:56:47.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Npfs548+ozPYRtf1YqwH8rw9nOM=', 'Content-Length': '0'} 2023-10-12T00:56:47.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d04332de994017-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.763 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.763 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.764 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.764 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.765 DEBUG:boto:Data: 2023-10-12T00:56:47.765 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.766 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.767 DEBUG:boto:Params: {} 2023-10-12T00:56:47.768 DEBUG:boto:Token: None 2023-10-12T00:56:47.769 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Npfs548+ozPYRtf1YqwH8rw9nOM= 2023-10-12T00:56:47.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Npfs548+ozPYRtf1YqwH8rw9nOM=', 'Content-Length': '0'} 2023-10-12T00:56:47.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004813081892a6311b-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-12T00:56:47.774 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.775 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.775 DEBUG:boto:Method: GET 2023-10-12T00:56:47.776 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.776 DEBUG:boto:Data: 2023-10-12T00:56:47.776 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.777 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.778 DEBUG:boto:Params: {} 2023-10-12T00:56:47.778 DEBUG:boto:Token: None 2023-10-12T00:56:47.779 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jvs7+pVVoxpJl4cRBhr1yLa5miM= 2023-10-12T00:56:47.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jvs7+pVVoxpJl4cRBhr1yLa5miM=', 'Content-Length': '0'} 2023-10-12T00:56:47.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc1e3fe47b8eb47-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.783 DEBUG:boto:path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.783 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.783 DEBUG:boto:Method: GET 2023-10-12T00:56:47.784 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.784 DEBUG:boto:Data: 2023-10-12T00:56:47.785 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.786 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.786 DEBUG:boto:Params: {} 2023-10-12T00:56:47.787 DEBUG:boto:Token: None 2023-10-12T00:56:47.787 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000025 2023-10-12T00:56:47.788 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jvs7+pVVoxpJl4cRBhr1yLa5miM= 2023-10-12T00:56:47.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jvs7+pVVoxpJl4cRBhr1yLa5miM=', 'Content-Length': '0'} 2023-10-12T00:56:47.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c342ef7c31e9069e-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.791 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.792 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.792 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.793 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.794 DEBUG:boto:Method: GET 2023-10-12T00:56:47.794 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.795 DEBUG:boto:Data: 2023-10-12T00:56:47.796 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.797 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.797 DEBUG:boto:Params: {} 2023-10-12T00:56:47.798 DEBUG:boto:Token: None 2023-10-12T00:56:47.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009765d15ddd940602-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.802 DEBUG:boto:b'wafmoa-58obj000000000000000000251000falseobj000000000000000000252023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.803 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.803 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.806 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.806 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.807 DEBUG:boto:Method: GET 2023-10-12T00:56:47.807 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000025 2023-10-12T00:56:47.808 DEBUG:boto:Data: 2023-10-12T00:56:47.808 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.809 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.809 DEBUG:boto:Params: {} 2023-10-12T00:56:47.810 DEBUG:boto:Token: None 2023-10-12T00:56:47.810 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004332b34a5e67fce9-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.816 DEBUG:boto:b'wafmoa-58obj000000000000000000251000falseobj000000000000000000252023-10-12T00:56:33.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-12T00:56:47.818 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.818 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.818 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.819 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.819 DEBUG:boto:Method: GET 2023-10-12T00:56:47.820 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.820 DEBUG:boto:Data: 2023-10-12T00:56:47.821 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.821 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.822 DEBUG:boto:Params: {} 2023-10-12T00:56:47.822 DEBUG:boto:Token: None 2023-10-12T00:56:47.823 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew= 2023-10-12T00:56:47.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew=', 'Content-Length': '0'} 2023-10-12T00:56:47.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021cb15d730e1fb1b-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.827 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.827 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.828 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.828 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.829 DEBUG:boto:Method: GET 2023-10-12T00:56:47.829 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.830 DEBUG:boto:Data: 2023-10-12T00:56:47.830 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.831 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.831 DEBUG:boto:Params: {} 2023-10-12T00:56:47.832 DEBUG:boto:Token: None 2023-10-12T00:56:47.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew= 2023-10-12T00:56:47.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew=', 'Content-Length': '0'} 2023-10-12T00:56:47.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e412ee339e921b-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.836 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.836 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.837 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.837 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.838 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.838 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.839 DEBUG:boto:Data: 2023-10-12T00:56:47.839 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.840 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.840 DEBUG:boto:Params: {} 2023-10-12T00:56:47.841 DEBUG:boto:Token: None 2023-10-12T00:56:47.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h8jpoDefbwgi1EOQ19mtOJHWzXI= 2023-10-12T00:56:47.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h8jpoDefbwgi1EOQ19mtOJHWzXI=', 'Content-Length': '0'} 2023-10-12T00:56:47.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1e18a4a7520ac9-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.845 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.846 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.846 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.847 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.847 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.848 DEBUG:boto:Data: 2023-10-12T00:56:47.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.849 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.850 DEBUG:boto:Params: {} 2023-10-12T00:56:47.850 DEBUG:boto:Token: None 2023-10-12T00:56:47.851 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h8jpoDefbwgi1EOQ19mtOJHWzXI= 2023-10-12T00:56:47.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h8jpoDefbwgi1EOQ19mtOJHWzXI=', 'Content-Length': '0'} 2023-10-12T00:56:47.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c345df84339680f1-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-12T00:56:47.856 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.857 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.857 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.858 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.859 DEBUG:boto:Method: GET 2023-10-12T00:56:47.859 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.860 DEBUG:boto:Data: 2023-10-12T00:56:47.861 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.862 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.862 DEBUG:boto:Params: {} 2023-10-12T00:56:47.863 DEBUG:boto:Token: None 2023-10-12T00:56:47.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew= 2023-10-12T00:56:47.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew=', 'Content-Length': '0'} 2023-10-12T00:56:47.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e3a81bfcd534cf-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.869 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.870 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.870 DEBUG:boto:Method: GET 2023-10-12T00:56:47.871 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.872 DEBUG:boto:Data: 2023-10-12T00:56:47.873 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.874 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.875 DEBUG:boto:Params: {} 2023-10-12T00:56:47.876 DEBUG:boto:Token: None 2023-10-12T00:56:47.877 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026?versionId=null 2023-10-12T00:56:47.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew= 2023-10-12T00:56:47.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AhfFAn96c0p+IrvQKAlVrgB37ew=', 'Content-Length': '0'} 2023-10-12T00:56:47.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000758e3beec3ca60e9-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.922 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.923 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.924 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.924 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.924 DEBUG:boto:Method: GET 2023-10-12T00:56:47.925 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.925 DEBUG:boto:Data: 2023-10-12T00:56:47.926 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.927 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.927 DEBUG:boto:Params: {} 2023-10-12T00:56:47.927 DEBUG:boto:Token: None 2023-10-12T00:56:47.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31842c4af309f31-006527444f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.932 DEBUG:boto:b'wafmoa-58obj000000000000000000261000falseobj000000000000000000262023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.933 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:47.934 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:47.934 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.935 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.935 DEBUG:boto:Method: GET 2023-10-12T00:56:47.935 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:47.936 DEBUG:boto:Data: 2023-10-12T00:56:47.936 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.937 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.938 DEBUG:boto:Params: {} 2023-10-12T00:56:47.938 DEBUG:boto:Token: None 2023-10-12T00:56:47.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/ 2023-10-12T00:56:47.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8= 2023-10-12T00:56:47.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ydDseYCCF7X+0i3xkABKNMR3mX8=', 'Content-Length': '0'} 2023-10-12T00:56:47.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004744635c189e1c4a-006527444f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.944 DEBUG:boto:b'wafmoa-58obj000000000000000000261000falseobj000000000000000000262023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:47.945 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.946 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.946 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.947 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.947 DEBUG:boto:Data: 2023-10-12T00:56:47.947 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.948 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.949 DEBUG:boto:Params: {} 2023-10-12T00:56:47.949 DEBUG:boto:Token: None 2023-10-12T00:56:47.949 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J4p7cnW5bQsB1PrreySFCzDhfTA= 2023-10-12T00:56:47.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J4p7cnW5bQsB1PrreySFCzDhfTA=', 'Content-Length': '0'} 2023-10-12T00:56:47.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea7e56b66f21a82-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.953 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.954 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.954 DEBUG:boto:Method: HEAD 2023-10-12T00:56:47.954 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.955 DEBUG:boto:Data: 2023-10-12T00:56:47.955 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:47.956 DEBUG:boto:Port: 8000 2023-10-12T00:56:47.957 DEBUG:boto:Params: {} 2023-10-12T00:56:47.957 DEBUG:boto:Token: None 2023-10-12T00:56:47.958 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J4p7cnW5bQsB1PrreySFCzDhfTA= 2023-10-12T00:56:47.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J4p7cnW5bQsB1PrreySFCzDhfTA=', 'Content-Length': '0'} 2023-10-12T00:56:47.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6001f3d8c0837bd-006527444f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:47.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-12T00:56:47.962 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.962 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.963 DEBUG:boto:Method: GET 2023-10-12T00:56:47.963 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.964 DEBUG:boto:Data: 2023-10-12T00:56:47.964 DEBUG:boto:Headers: {} 2023-10-12T00:56:47.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:47.965 DEBUG:boto:Port: 8001 2023-10-12T00:56:47.965 DEBUG:boto:Params: {} 2023-10-12T00:56:47.966 DEBUG:boto:Token: None 2023-10-12T00:56:47.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:47 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:47.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gAg8qmWRIjBtaPl5wV0wezjIfxU= 2023-10-12T00:56:47.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gAg8qmWRIjBtaPl5wV0wezjIfxU=', 'Content-Length': '0'} 2023-10-12T00:56:47.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000084875bff93393a5-006527444f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.014 DEBUG:boto:path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:48.015 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000026 2023-10-12T00:56:48.015 DEBUG:boto:Method: GET 2023-10-12T00:56:48.016 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:48.016 DEBUG:boto:Data: 2023-10-12T00:56:48.017 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.018 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.018 DEBUG:boto:Params: {} 2023-10-12T00:56:48.019 DEBUG:boto:Token: None 2023-10-12T00:56:48.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000026 2023-10-12T00:56:48.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kT4br/pZHX/pQCKmkkBstPXEKeo= 2023-10-12T00:56:48.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kT4br/pZHX/pQCKmkkBstPXEKeo=', 'Content-Length': '0'} 2023-10-12T00:56:48.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbdb0f6ae63a0a67-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.023 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.023 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.024 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.024 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.025 DEBUG:boto:Method: GET 2023-10-12T00:56:48.025 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.026 DEBUG:boto:Data: 2023-10-12T00:56:48.026 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.027 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.028 DEBUG:boto:Params: {} 2023-10-12T00:56:48.028 DEBUG:boto:Token: None 2023-10-12T00:56:48.029 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000543c3d16acad8900-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.033 DEBUG:boto:b'wafmoa-58obj000000000000000000261000falseobj000000000000000000262023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.034 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.034 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.035 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.035 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.036 DEBUG:boto:Method: GET 2023-10-12T00:56:48.036 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000026 2023-10-12T00:56:48.037 DEBUG:boto:Data: 2023-10-12T00:56:48.038 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.038 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.039 DEBUG:boto:Params: {} 2023-10-12T00:56:48.039 DEBUG:boto:Token: None 2023-10-12T00:56:48.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c6be37deccbde5-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.086 DEBUG:boto:b'wafmoa-58obj000000000000000000261000falseobj000000000000000000262023-10-12T00:56:36.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-12T00:56:48.088 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.088 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.089 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.089 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.090 DEBUG:boto:Method: GET 2023-10-12T00:56:48.090 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.091 DEBUG:boto:Data: 2023-10-12T00:56:48.091 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.092 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.092 DEBUG:boto:Params: {} 2023-10-12T00:56:48.093 DEBUG:boto:Token: None 2023-10-12T00:56:48.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg= 2023-10-12T00:56:48.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg=', 'Content-Length': '0'} 2023-10-12T00:56:48.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5d95df76beff70-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.098 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.098 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.099 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.099 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.100 DEBUG:boto:Method: GET 2023-10-12T00:56:48.101 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.101 DEBUG:boto:Data: 2023-10-12T00:56:48.102 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.104 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.104 DEBUG:boto:Params: {} 2023-10-12T00:56:48.105 DEBUG:boto:Token: None 2023-10-12T00:56:48.105 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg= 2023-10-12T00:56:48.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg=', 'Content-Length': '0'} 2023-10-12T00:56:48.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea1b25cd10349686-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.109 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.110 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.110 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.111 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.111 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.112 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.112 DEBUG:boto:Data: 2023-10-12T00:56:48.112 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.113 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.114 DEBUG:boto:Params: {} 2023-10-12T00:56:48.114 DEBUG:boto:Token: None 2023-10-12T00:56:48.115 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.115 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:llG/2Z4KE2VNw1zwl0ChBKVZOjU= 2023-10-12T00:56:48.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:llG/2Z4KE2VNw1zwl0ChBKVZOjU=', 'Content-Length': '0'} 2023-10-12T00:56:48.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000addf4d8f873f9984-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.118 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.119 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.119 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.120 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.120 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.120 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.121 DEBUG:boto:Data: 2023-10-12T00:56:48.122 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.123 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.124 DEBUG:boto:Params: {} 2023-10-12T00:56:48.125 DEBUG:boto:Token: None 2023-10-12T00:56:48.125 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:llG/2Z4KE2VNw1zwl0ChBKVZOjU= 2023-10-12T00:56:48.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:llG/2Z4KE2VNw1zwl0ChBKVZOjU=', 'Content-Length': '0'} 2023-10-12T00:56:48.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cafce41115d2009d-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-12T00:56:48.130 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.130 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.131 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.131 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.132 DEBUG:boto:Method: GET 2023-10-12T00:56:48.132 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.133 DEBUG:boto:Data: 2023-10-12T00:56:48.133 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.134 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.135 DEBUG:boto:Params: {} 2023-10-12T00:56:48.135 DEBUG:boto:Token: None 2023-10-12T00:56:48.136 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg= 2023-10-12T00:56:48.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg=', 'Content-Length': '0'} 2023-10-12T00:56:48.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe913433e3be0dd-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.140 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.141 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.141 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.142 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.142 DEBUG:boto:Method: GET 2023-10-12T00:56:48.142 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.143 DEBUG:boto:Data: 2023-10-12T00:56:48.143 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.144 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.145 DEBUG:boto:Params: {} 2023-10-12T00:56:48.145 DEBUG:boto:Token: None 2023-10-12T00:56:48.146 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027?versionId=null 2023-10-12T00:56:48.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg= 2023-10-12T00:56:48.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pkw0wI1mVxUlFh4mIbIso+8PDvg=', 'Content-Length': '0'} 2023-10-12T00:56:48.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9590d4225c1d10-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.149 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.150 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.151 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.151 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.152 DEBUG:boto:Method: GET 2023-10-12T00:56:48.153 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.153 DEBUG:boto:Data: 2023-10-12T00:56:48.154 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.155 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.156 DEBUG:boto:Params: {} 2023-10-12T00:56:48.157 DEBUG:boto:Token: None 2023-10-12T00:56:48.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d7b9e9c47e75539-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.162 DEBUG:boto:b'wafmoa-58obj000000000000000000271000falseobj000000000000000000272023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.163 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.163 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.163 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.164 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.164 DEBUG:boto:Method: GET 2023-10-12T00:56:48.165 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.165 DEBUG:boto:Data: 2023-10-12T00:56:48.166 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.167 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.167 DEBUG:boto:Params: {} 2023-10-12T00:56:48.168 DEBUG:boto:Token: None 2023-10-12T00:56:48.169 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dfda0f9f4b1dd45-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.175 DEBUG:boto:b'wafmoa-58obj000000000000000000271000falseobj000000000000000000272023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.176 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.176 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.177 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.177 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.177 DEBUG:boto:Data: 2023-10-12T00:56:48.178 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.179 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.179 DEBUG:boto:Params: {} 2023-10-12T00:56:48.180 DEBUG:boto:Token: None 2023-10-12T00:56:48.180 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bfvjLPp4vhqMjEPX5QkHdsudJYY= 2023-10-12T00:56:48.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bfvjLPp4vhqMjEPX5QkHdsudJYY=', 'Content-Length': '0'} 2023-10-12T00:56:48.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac289dd88c8627dc-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.184 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.185 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.185 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.185 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.186 DEBUG:boto:Data: 2023-10-12T00:56:48.186 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.187 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.188 DEBUG:boto:Params: {} 2023-10-12T00:56:48.188 DEBUG:boto:Token: None 2023-10-12T00:56:48.189 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bfvjLPp4vhqMjEPX5QkHdsudJYY= 2023-10-12T00:56:48.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bfvjLPp4vhqMjEPX5QkHdsudJYY=', 'Content-Length': '0'} 2023-10-12T00:56:48.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029067de62e4dc3dc-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-12T00:56:48.193 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.193 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.193 DEBUG:boto:Method: GET 2023-10-12T00:56:48.194 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.194 DEBUG:boto:Data: 2023-10-12T00:56:48.195 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.196 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.196 DEBUG:boto:Params: {} 2023-10-12T00:56:48.197 DEBUG:boto:Token: None 2023-10-12T00:56:48.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.198 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iNXawiV6xWfXkixH7SnoD9W60Pc= 2023-10-12T00:56:48.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iNXawiV6xWfXkixH7SnoD9W60Pc=', 'Content-Length': '0'} 2023-10-12T00:56:48.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e7045d0a048dba-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.201 DEBUG:boto:path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.201 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.202 DEBUG:boto:Method: GET 2023-10-12T00:56:48.202 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.203 DEBUG:boto:Data: 2023-10-12T00:56:48.203 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.204 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.204 DEBUG:boto:Params: {} 2023-10-12T00:56:48.205 DEBUG:boto:Token: None 2023-10-12T00:56:48.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000027 2023-10-12T00:56:48.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iNXawiV6xWfXkixH7SnoD9W60Pc= 2023-10-12T00:56:48.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iNXawiV6xWfXkixH7SnoD9W60Pc=', 'Content-Length': '0'} 2023-10-12T00:56:48.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2e63cc8c7bd460-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.209 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.210 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.210 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.210 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.211 DEBUG:boto:Method: GET 2023-10-12T00:56:48.211 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.212 DEBUG:boto:Data: 2023-10-12T00:56:48.212 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.213 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.213 DEBUG:boto:Params: {} 2023-10-12T00:56:48.214 DEBUG:boto:Token: None 2023-10-12T00:56:48.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ba4ba39c90ece2-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.219 DEBUG:boto:b'wafmoa-58obj000000000000000000271000falseobj000000000000000000272023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.219 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.220 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.220 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.220 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.221 DEBUG:boto:Method: GET 2023-10-12T00:56:48.221 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000027 2023-10-12T00:56:48.222 DEBUG:boto:Data: 2023-10-12T00:56:48.222 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.223 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.224 DEBUG:boto:Params: {} 2023-10-12T00:56:48.224 DEBUG:boto:Token: None 2023-10-12T00:56:48.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2c31f6e25b9ac9-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.230 DEBUG:boto:b'wafmoa-58obj000000000000000000271000falseobj000000000000000000272023-10-12T00:56:37.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-12T00:56:48.231 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.232 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.232 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.233 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.233 DEBUG:boto:Method: GET 2023-10-12T00:56:48.233 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.234 DEBUG:boto:Data: 2023-10-12T00:56:48.234 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.235 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.236 DEBUG:boto:Params: {} 2023-10-12T00:56:48.236 DEBUG:boto:Token: None 2023-10-12T00:56:48.237 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao= 2023-10-12T00:56:48.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao=', 'Content-Length': '0'} 2023-10-12T00:56:48.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748cd8c4c3a537bd-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.241 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.241 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.241 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.242 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.242 DEBUG:boto:Method: GET 2023-10-12T00:56:48.243 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.243 DEBUG:boto:Data: 2023-10-12T00:56:48.244 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.244 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.245 DEBUG:boto:Params: {} 2023-10-12T00:56:48.245 DEBUG:boto:Token: None 2023-10-12T00:56:48.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao= 2023-10-12T00:56:48.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao=', 'Content-Length': '0'} 2023-10-12T00:56:48.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea42281b91a8e99-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.249 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.250 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.250 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.250 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.251 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.251 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.252 DEBUG:boto:Data: 2023-10-12T00:56:48.252 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.253 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.253 DEBUG:boto:Params: {} 2023-10-12T00:56:48.253 DEBUG:boto:Token: None 2023-10-12T00:56:48.254 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v1K9/YXt+tdFmT00mUQuRRyApWs= 2023-10-12T00:56:48.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v1K9/YXt+tdFmT00mUQuRRyApWs=', 'Content-Length': '0'} 2023-10-12T00:56:48.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890d31c55e4acbb6-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.257 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.258 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.258 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.259 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.259 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.259 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.260 DEBUG:boto:Data: 2023-10-12T00:56:48.260 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.261 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.261 DEBUG:boto:Params: {} 2023-10-12T00:56:48.262 DEBUG:boto:Token: None 2023-10-12T00:56:48.262 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v1K9/YXt+tdFmT00mUQuRRyApWs= 2023-10-12T00:56:48.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v1K9/YXt+tdFmT00mUQuRRyApWs=', 'Content-Length': '0'} 2023-10-12T00:56:48.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa5694fa8b45fbd2-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-12T00:56:48.266 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.267 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.267 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.267 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.268 DEBUG:boto:Method: GET 2023-10-12T00:56:48.268 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.269 DEBUG:boto:Data: 2023-10-12T00:56:48.269 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.270 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.270 DEBUG:boto:Params: {} 2023-10-12T00:56:48.271 DEBUG:boto:Token: None 2023-10-12T00:56:48.271 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao= 2023-10-12T00:56:48.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao=', 'Content-Length': '0'} 2023-10-12T00:56:48.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf3c6d52939351c-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.275 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.275 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.275 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.276 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.276 DEBUG:boto:Method: GET 2023-10-12T00:56:48.277 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.277 DEBUG:boto:Data: 2023-10-12T00:56:48.278 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.278 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.279 DEBUG:boto:Params: {} 2023-10-12T00:56:48.279 DEBUG:boto:Token: None 2023-10-12T00:56:48.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028?versionId=null 2023-10-12T00:56:48.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao= 2023-10-12T00:56:48.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iYQUmFCvXh4b6wvqr7yKwZ0OAao=', 'Content-Length': '0'} 2023-10-12T00:56:48.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1292e26e888613f-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.283 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.284 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.284 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.285 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.285 DEBUG:boto:Method: GET 2023-10-12T00:56:48.286 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.286 DEBUG:boto:Data: 2023-10-12T00:56:48.286 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.287 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.288 DEBUG:boto:Params: {} 2023-10-12T00:56:48.288 DEBUG:boto:Token: None 2023-10-12T00:56:48.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c465e7336ac747ae-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.293 DEBUG:boto:b'wafmoa-58obj000000000000000000281000falseobj000000000000000000282023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.294 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.294 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.295 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.295 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.296 DEBUG:boto:Method: GET 2023-10-12T00:56:48.296 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.296 DEBUG:boto:Data: 2023-10-12T00:56:48.297 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.298 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.298 DEBUG:boto:Params: {} 2023-10-12T00:56:48.299 DEBUG:boto:Token: None 2023-10-12T00:56:48.299 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.299 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f93847cbd71bc2d-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.346 DEBUG:boto:b'wafmoa-58obj000000000000000000281000falseobj000000000000000000282023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.347 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.348 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.348 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.348 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.349 DEBUG:boto:Data: 2023-10-12T00:56:48.349 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.350 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.350 DEBUG:boto:Params: {} 2023-10-12T00:56:48.351 DEBUG:boto:Token: None 2023-10-12T00:56:48.351 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nz4Nyfq/RlNBJey4pgEuK4FwLkI= 2023-10-12T00:56:48.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nz4Nyfq/RlNBJey4pgEuK4FwLkI=', 'Content-Length': '0'} 2023-10-12T00:56:48.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cffeef41942f20b-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.355 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.355 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.356 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.356 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.356 DEBUG:boto:Data: 2023-10-12T00:56:48.357 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.358 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.358 DEBUG:boto:Params: {} 2023-10-12T00:56:48.359 DEBUG:boto:Token: None 2023-10-12T00:56:48.359 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nz4Nyfq/RlNBJey4pgEuK4FwLkI= 2023-10-12T00:56:48.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nz4Nyfq/RlNBJey4pgEuK4FwLkI=', 'Content-Length': '0'} 2023-10-12T00:56:48.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134cf04c7334f10f-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-12T00:56:48.363 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.363 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.364 DEBUG:boto:Method: GET 2023-10-12T00:56:48.364 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.364 DEBUG:boto:Data: 2023-10-12T00:56:48.365 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.366 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.366 DEBUG:boto:Params: {} 2023-10-12T00:56:48.367 DEBUG:boto:Token: None 2023-10-12T00:56:48.367 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bAVh2CV2f88+okRWL88tEnHJJW4= 2023-10-12T00:56:48.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bAVh2CV2f88+okRWL88tEnHJJW4=', 'Content-Length': '0'} 2023-10-12T00:56:48.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37db39cd54259bf-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.418 DEBUG:boto:path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.419 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.420 DEBUG:boto:Method: GET 2023-10-12T00:56:48.421 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.421 DEBUG:boto:Data: 2023-10-12T00:56:48.421 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.422 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.423 DEBUG:boto:Params: {} 2023-10-12T00:56:48.423 DEBUG:boto:Token: None 2023-10-12T00:56:48.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000028 2023-10-12T00:56:48.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bAVh2CV2f88+okRWL88tEnHJJW4= 2023-10-12T00:56:48.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bAVh2CV2f88+okRWL88tEnHJJW4=', 'Content-Length': '0'} 2023-10-12T00:56:48.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c33a6b3f553f03-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.428 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.429 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.429 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.429 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.430 DEBUG:boto:Method: GET 2023-10-12T00:56:48.430 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.431 DEBUG:boto:Data: 2023-10-12T00:56:48.431 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.432 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.433 DEBUG:boto:Params: {} 2023-10-12T00:56:48.433 DEBUG:boto:Token: None 2023-10-12T00:56:48.434 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005dd595c76b671e7-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.438 DEBUG:boto:b'wafmoa-58obj000000000000000000281000falseobj000000000000000000282023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.438 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.439 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.439 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.440 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.440 DEBUG:boto:Method: GET 2023-10-12T00:56:48.441 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000028 2023-10-12T00:56:48.441 DEBUG:boto:Data: 2023-10-12T00:56:48.442 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.442 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.443 DEBUG:boto:Params: {} 2023-10-12T00:56:48.443 DEBUG:boto:Token: None 2023-10-12T00:56:48.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.444 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f36fa82181538a05-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.449 DEBUG:boto:b'wafmoa-58obj000000000000000000281000falseobj000000000000000000282023-10-12T00:56:30.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-12T00:56:48.450 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.451 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.451 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.452 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.452 DEBUG:boto:Method: GET 2023-10-12T00:56:48.452 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.453 DEBUG:boto:Data: 2023-10-12T00:56:48.453 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.454 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.454 DEBUG:boto:Params: {} 2023-10-12T00:56:48.455 DEBUG:boto:Token: None 2023-10-12T00:56:48.456 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss= 2023-10-12T00:56:48.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss=', 'Content-Length': '0'} 2023-10-12T00:56:48.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc81a67775d673f-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.459 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.460 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.460 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.461 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.461 DEBUG:boto:Method: GET 2023-10-12T00:56:48.462 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.462 DEBUG:boto:Data: 2023-10-12T00:56:48.462 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.463 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.464 DEBUG:boto:Params: {} 2023-10-12T00:56:48.464 DEBUG:boto:Token: None 2023-10-12T00:56:48.465 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.465 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss= 2023-10-12T00:56:48.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss=', 'Content-Length': '0'} 2023-10-12T00:56:48.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26972db28449c35-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.469 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.469 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.470 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.470 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.470 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.471 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.471 DEBUG:boto:Data: 2023-10-12T00:56:48.472 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.473 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.473 DEBUG:boto:Params: {} 2023-10-12T00:56:48.473 DEBUG:boto:Token: None 2023-10-12T00:56:48.474 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkk+HTRot9EsylUB6Ixu3tAvKEM= 2023-10-12T00:56:48.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkk+HTRot9EsylUB6Ixu3tAvKEM=', 'Content-Length': '0'} 2023-10-12T00:56:48.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b926530821942510-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.477 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.478 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.478 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.479 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.479 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.480 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.480 DEBUG:boto:Data: 2023-10-12T00:56:48.480 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.481 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.482 DEBUG:boto:Params: {} 2023-10-12T00:56:48.482 DEBUG:boto:Token: None 2023-10-12T00:56:48.483 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.483 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkk+HTRot9EsylUB6Ixu3tAvKEM= 2023-10-12T00:56:48.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkk+HTRot9EsylUB6Ixu3tAvKEM=', 'Content-Length': '0'} 2023-10-12T00:56:48.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d62ea39fc77cc8-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-12T00:56:48.486 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.487 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.487 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.488 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.488 DEBUG:boto:Method: GET 2023-10-12T00:56:48.489 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.489 DEBUG:boto:Data: 2023-10-12T00:56:48.489 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.490 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.491 DEBUG:boto:Params: {} 2023-10-12T00:56:48.491 DEBUG:boto:Token: None 2023-10-12T00:56:48.492 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss= 2023-10-12T00:56:48.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss=', 'Content-Length': '0'} 2023-10-12T00:56:48.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0d95de4f9eeeb6-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.495 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.496 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.496 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.497 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.497 DEBUG:boto:Method: GET 2023-10-12T00:56:48.498 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.498 DEBUG:boto:Data: 2023-10-12T00:56:48.498 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.499 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.500 DEBUG:boto:Params: {} 2023-10-12T00:56:48.500 DEBUG:boto:Token: None 2023-10-12T00:56:48.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029?versionId=null 2023-10-12T00:56:48.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss= 2023-10-12T00:56:48.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+LyRhKTHz9YLzewxE+2qlD7pss=', 'Content-Length': '0'} 2023-10-12T00:56:48.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cac6646222c135a-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.504 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.505 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.505 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.506 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.506 DEBUG:boto:Method: GET 2023-10-12T00:56:48.507 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.507 DEBUG:boto:Data: 2023-10-12T00:56:48.507 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.508 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.509 DEBUG:boto:Params: {} 2023-10-12T00:56:48.509 DEBUG:boto:Token: None 2023-10-12T00:56:48.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002998320e679c91a7-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.558 DEBUG:boto:b'wafmoa-58obj000000000000000000291000falseobj000000000000000000292023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.559 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.560 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.560 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.560 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.561 DEBUG:boto:Method: GET 2023-10-12T00:56:48.561 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.561 DEBUG:boto:Data: 2023-10-12T00:56:48.562 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.563 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.563 DEBUG:boto:Params: {} 2023-10-12T00:56:48.564 DEBUG:boto:Token: None 2023-10-12T00:56:48.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd84bcadb4f79f3-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.570 DEBUG:boto:b'wafmoa-58obj000000000000000000291000falseobj000000000000000000292023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.571 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.571 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.571 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.572 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.572 DEBUG:boto:Data: 2023-10-12T00:56:48.573 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.573 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.574 DEBUG:boto:Params: {} 2023-10-12T00:56:48.574 DEBUG:boto:Token: None 2023-10-12T00:56:48.575 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RlZUDzzQYs+LiVsJuPjbkIQjlFU= 2023-10-12T00:56:48.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RlZUDzzQYs+LiVsJuPjbkIQjlFU=', 'Content-Length': '0'} 2023-10-12T00:56:48.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1680708c3b6390-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.579 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.579 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.579 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.580 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.580 DEBUG:boto:Data: 2023-10-12T00:56:48.581 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.581 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.582 DEBUG:boto:Params: {} 2023-10-12T00:56:48.582 DEBUG:boto:Token: None 2023-10-12T00:56:48.583 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RlZUDzzQYs+LiVsJuPjbkIQjlFU= 2023-10-12T00:56:48.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RlZUDzzQYs+LiVsJuPjbkIQjlFU=', 'Content-Length': '0'} 2023-10-12T00:56:48.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033858398b6f00d5f-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-12T00:56:48.587 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.587 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.588 DEBUG:boto:Method: GET 2023-10-12T00:56:48.588 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.588 DEBUG:boto:Data: 2023-10-12T00:56:48.589 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.590 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.590 DEBUG:boto:Params: {} 2023-10-12T00:56:48.591 DEBUG:boto:Token: None 2023-10-12T00:56:48.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IG2tuSZNnBL+Mx1CpQ88SV73K5Q= 2023-10-12T00:56:48.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IG2tuSZNnBL+Mx1CpQ88SV73K5Q=', 'Content-Length': '0'} 2023-10-12T00:56:48.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000766f6558bd119874-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.595 DEBUG:boto:path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.595 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.596 DEBUG:boto:Method: GET 2023-10-12T00:56:48.596 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.596 DEBUG:boto:Data: 2023-10-12T00:56:48.597 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.598 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.598 DEBUG:boto:Params: {} 2023-10-12T00:56:48.598 DEBUG:boto:Token: None 2023-10-12T00:56:48.599 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000029 2023-10-12T00:56:48.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IG2tuSZNnBL+Mx1CpQ88SV73K5Q= 2023-10-12T00:56:48.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IG2tuSZNnBL+Mx1CpQ88SV73K5Q=', 'Content-Length': '0'} 2023-10-12T00:56:48.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9cd523fc69d609-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.642 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.643 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.643 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.644 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.644 DEBUG:boto:Method: GET 2023-10-12T00:56:48.645 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.645 DEBUG:boto:Data: 2023-10-12T00:56:48.646 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.647 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.647 DEBUG:boto:Params: {} 2023-10-12T00:56:48.647 DEBUG:boto:Token: None 2023-10-12T00:56:48.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb0c6edd4e5abed-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.652 DEBUG:boto:b'wafmoa-58obj000000000000000000291000falseobj000000000000000000292023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.653 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.653 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.654 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.654 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.655 DEBUG:boto:Method: GET 2023-10-12T00:56:48.655 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000029 2023-10-12T00:56:48.656 DEBUG:boto:Data: 2023-10-12T00:56:48.656 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.657 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.657 DEBUG:boto:Params: {} 2023-10-12T00:56:48.658 DEBUG:boto:Token: None 2023-10-12T00:56:48.658 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fbed2c3d31bf77-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.664 DEBUG:boto:b'wafmoa-58obj000000000000000000291000falseobj000000000000000000292023-10-12T00:56:35.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-12T00:56:48.665 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.666 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.666 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.666 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.667 DEBUG:boto:Method: GET 2023-10-12T00:56:48.667 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.668 DEBUG:boto:Data: 2023-10-12T00:56:48.668 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.669 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.669 DEBUG:boto:Params: {} 2023-10-12T00:56:48.670 DEBUG:boto:Token: None 2023-10-12T00:56:48.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI= 2023-10-12T00:56:48.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI=', 'Content-Length': '0'} 2023-10-12T00:56:48.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b9c8fd738de4fc5-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.675 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.675 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.675 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.676 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.676 DEBUG:boto:Method: GET 2023-10-12T00:56:48.677 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.677 DEBUG:boto:Data: 2023-10-12T00:56:48.678 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.678 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.679 DEBUG:boto:Params: {} 2023-10-12T00:56:48.679 DEBUG:boto:Token: None 2023-10-12T00:56:48.680 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI= 2023-10-12T00:56:48.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI=', 'Content-Length': '0'} 2023-10-12T00:56:48.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028bb45771aaea679-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.684 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.684 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.684 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.685 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.685 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.686 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.686 DEBUG:boto:Data: 2023-10-12T00:56:48.686 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.687 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.688 DEBUG:boto:Params: {} 2023-10-12T00:56:48.688 DEBUG:boto:Token: None 2023-10-12T00:56:48.689 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fdGIKRwBH7bqvt1VX1Vi0NXAiT4= 2023-10-12T00:56:48.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fdGIKRwBH7bqvt1VX1Vi0NXAiT4=', 'Content-Length': '0'} 2023-10-12T00:56:48.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b72879f4a8292e1-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.692 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.692 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.693 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.693 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.694 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.694 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.695 DEBUG:boto:Data: 2023-10-12T00:56:48.695 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.696 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.696 DEBUG:boto:Params: {} 2023-10-12T00:56:48.697 DEBUG:boto:Token: None 2023-10-12T00:56:48.697 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fdGIKRwBH7bqvt1VX1Vi0NXAiT4= 2023-10-12T00:56:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fdGIKRwBH7bqvt1VX1Vi0NXAiT4=', 'Content-Length': '0'} 2023-10-12T00:56:48.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9d73ff8ddf7b4ca-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-12T00:56:48.701 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.701 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.702 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.702 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.703 DEBUG:boto:Method: GET 2023-10-12T00:56:48.703 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.703 DEBUG:boto:Data: 2023-10-12T00:56:48.704 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.705 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.705 DEBUG:boto:Params: {} 2023-10-12T00:56:48.706 DEBUG:boto:Token: None 2023-10-12T00:56:48.706 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI= 2023-10-12T00:56:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI=', 'Content-Length': '0'} 2023-10-12T00:56:48.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ceb8b12a459879-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.710 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.710 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.711 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.711 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.711 DEBUG:boto:Method: GET 2023-10-12T00:56:48.712 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.712 DEBUG:boto:Data: 2023-10-12T00:56:48.713 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.714 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.714 DEBUG:boto:Params: {} 2023-10-12T00:56:48.714 DEBUG:boto:Token: None 2023-10-12T00:56:48.715 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030?versionId=null 2023-10-12T00:56:48.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI= 2023-10-12T00:56:48.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CenMYypp5Y3YTuxBhdJpHrDNcjI=', 'Content-Length': '0'} 2023-10-12T00:56:48.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c3f7ce6875bb25-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.758 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.759 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.759 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.760 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.760 DEBUG:boto:Method: GET 2023-10-12T00:56:48.761 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.761 DEBUG:boto:Data: 2023-10-12T00:56:48.761 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.762 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.763 DEBUG:boto:Params: {} 2023-10-12T00:56:48.763 DEBUG:boto:Token: None 2023-10-12T00:56:48.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be067bbf3f3593f6-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.768 DEBUG:boto:b'wafmoa-58obj000000000000000000301000falseobj000000000000000000302023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.769 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.769 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.770 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.770 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.771 DEBUG:boto:Method: GET 2023-10-12T00:56:48.771 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.771 DEBUG:boto:Data: 2023-10-12T00:56:48.772 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.773 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.773 DEBUG:boto:Params: {} 2023-10-12T00:56:48.773 DEBUG:boto:Token: None 2023-10-12T00:56:48.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003deeba78106e5328-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.782 DEBUG:boto:b'wafmoa-58obj000000000000000000301000falseobj000000000000000000302023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.782 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.783 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.783 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.784 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.784 DEBUG:boto:Data: 2023-10-12T00:56:48.784 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.785 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.786 DEBUG:boto:Params: {} 2023-10-12T00:56:48.786 DEBUG:boto:Token: None 2023-10-12T00:56:48.786 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnMbWjdr7FjZAcYbA7E1N6a+rTg= 2023-10-12T00:56:48.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnMbWjdr7FjZAcYbA7E1N6a+rTg=', 'Content-Length': '0'} 2023-10-12T00:56:48.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ecc932216975d6-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.790 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.791 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.791 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.791 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.792 DEBUG:boto:Data: 2023-10-12T00:56:48.792 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.793 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.793 DEBUG:boto:Params: {} 2023-10-12T00:56:48.794 DEBUG:boto:Token: None 2023-10-12T00:56:48.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qnMbWjdr7FjZAcYbA7E1N6a+rTg= 2023-10-12T00:56:48.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qnMbWjdr7FjZAcYbA7E1N6a+rTg=', 'Content-Length': '0'} 2023-10-12T00:56:48.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2311ac702e84614-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-12T00:56:48.798 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.798 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.799 DEBUG:boto:Method: GET 2023-10-12T00:56:48.799 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.800 DEBUG:boto:Data: 2023-10-12T00:56:48.800 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.801 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.802 DEBUG:boto:Params: {} 2023-10-12T00:56:48.802 DEBUG:boto:Token: None 2023-10-12T00:56:48.803 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIeFM3Fv+BeqeoBPE4nZ/FobfnU= 2023-10-12T00:56:48.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIeFM3Fv+BeqeoBPE4nZ/FobfnU=', 'Content-Length': '0'} 2023-10-12T00:56:48.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089bbcfa86b8f9907-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.846 DEBUG:boto:path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.847 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.847 DEBUG:boto:Method: GET 2023-10-12T00:56:48.848 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.848 DEBUG:boto:Data: 2023-10-12T00:56:48.848 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.849 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.850 DEBUG:boto:Params: {} 2023-10-12T00:56:48.850 DEBUG:boto:Token: None 2023-10-12T00:56:48.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000030 2023-10-12T00:56:48.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NIeFM3Fv+BeqeoBPE4nZ/FobfnU= 2023-10-12T00:56:48.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NIeFM3Fv+BeqeoBPE4nZ/FobfnU=', 'Content-Length': '0'} 2023-10-12T00:56:48.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016115a7bc251f421-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.854 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.855 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.855 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.855 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.856 DEBUG:boto:Method: GET 2023-10-12T00:56:48.856 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.857 DEBUG:boto:Data: 2023-10-12T00:56:48.857 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.858 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.858 DEBUG:boto:Params: {} 2023-10-12T00:56:48.859 DEBUG:boto:Token: None 2023-10-12T00:56:48.859 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8690e5c90ba4695-0065274450-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.863 DEBUG:boto:b'wafmoa-58obj000000000000000000301000falseobj000000000000000000302023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.864 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:48.864 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:48.865 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.865 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.866 DEBUG:boto:Method: GET 2023-10-12T00:56:48.866 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000030 2023-10-12T00:56:48.867 DEBUG:boto:Data: 2023-10-12T00:56:48.867 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.868 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.868 DEBUG:boto:Params: {} 2023-10-12T00:56:48.869 DEBUG:boto:Token: None 2023-10-12T00:56:48.869 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/ 2023-10-12T00:56:48.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ= 2023-10-12T00:56:48.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8lrTT1aHf7mlzBIQhdi366VnkQQ=', 'Content-Length': '0'} 2023-10-12T00:56:48.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad0a74f85f17c6a-0065274450-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.875 DEBUG:boto:b'wafmoa-58obj000000000000000000301000falseobj000000000000000000302023-10-12T00:56:36.209Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:48.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-12T00:56:48.876 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.876 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.877 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.877 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.877 DEBUG:boto:Method: GET 2023-10-12T00:56:48.878 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.878 DEBUG:boto:Data: 2023-10-12T00:56:48.879 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.879 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.880 DEBUG:boto:Params: {} 2023-10-12T00:56:48.880 DEBUG:boto:Token: None 2023-10-12T00:56:48.881 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.881 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig= 2023-10-12T00:56:48.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig=', 'Content-Length': '0'} 2023-10-12T00:56:48.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040849bf58224fe3d-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.884 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.885 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.885 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.886 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.886 DEBUG:boto:Method: GET 2023-10-12T00:56:48.887 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.887 DEBUG:boto:Data: 2023-10-12T00:56:48.887 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.888 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.889 DEBUG:boto:Params: {} 2023-10-12T00:56:48.889 DEBUG:boto:Token: None 2023-10-12T00:56:48.889 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig= 2023-10-12T00:56:48.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig=', 'Content-Length': '0'} 2023-10-12T00:56:48.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4e20e51f6dc028-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.934 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.935 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.935 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.936 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.936 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.936 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.937 DEBUG:boto:Data: 2023-10-12T00:56:48.937 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.938 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.938 DEBUG:boto:Params: {} 2023-10-12T00:56:48.939 DEBUG:boto:Token: None 2023-10-12T00:56:48.939 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcxYyGIyOKtNLq1os1AqsLM99bU= 2023-10-12T00:56:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcxYyGIyOKtNLq1os1AqsLM99bU=', 'Content-Length': '0'} 2023-10-12T00:56:48.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eed99abc6533a28-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.943 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.943 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.944 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.944 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.945 DEBUG:boto:Method: HEAD 2023-10-12T00:56:48.945 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.946 DEBUG:boto:Data: 2023-10-12T00:56:48.946 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:48.947 DEBUG:boto:Port: 8000 2023-10-12T00:56:48.947 DEBUG:boto:Params: {} 2023-10-12T00:56:48.948 DEBUG:boto:Token: None 2023-10-12T00:56:48.948 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcxYyGIyOKtNLq1os1AqsLM99bU= 2023-10-12T00:56:48.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcxYyGIyOKtNLq1os1AqsLM99bU=', 'Content-Length': '0'} 2023-10-12T00:56:48.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006839f393c8679ce-0065274450-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:48.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-12T00:56:48.952 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.952 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:48.953 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.953 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.953 DEBUG:boto:Method: GET 2023-10-12T00:56:48.954 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.954 DEBUG:boto:Data: 2023-10-12T00:56:48.955 DEBUG:boto:Headers: {} 2023-10-12T00:56:48.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:48.956 DEBUG:boto:Port: 8001 2023-10-12T00:56:48.956 DEBUG:boto:Params: {} 2023-10-12T00:56:48.957 DEBUG:boto:Token: None 2023-10-12T00:56:48.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:48 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:48.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig= 2023-10-12T00:56:48.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lBKYHsgU1+VFzoNLzHHl+fifqig=', 'Content-Length': '0'} 2023-10-12T00:56:48.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6fa8a4d994e93e-0065274450-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.002 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.003 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.004 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:49.005 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:49.006 DEBUG:boto:Method: GET 2023-10-12T00:56:49.006 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:49.007 DEBUG:boto:Data: 2023-10-12T00:56:49.007 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.008 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.009 DEBUG:boto:Params: {} 2023-10-12T00:56:49.009 DEBUG:boto:Token: None 2023-10-12T00:56:49.010 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000031?versionId=null 2023-10-12T00:56:49.010 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxPbK7BVJnEeoLCoMXPZGUVNMI0= 2023-10-12T00:56:49.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxPbK7BVJnEeoLCoMXPZGUVNMI0=', 'Content-Length': '0'} 2023-10-12T00:56:49.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749dd173f7cea764-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.014 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.014 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.014 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.015 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.015 DEBUG:boto:Method: GET 2023-10-12T00:56:49.016 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.016 DEBUG:boto:Data: 2023-10-12T00:56:49.017 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.018 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.018 DEBUG:boto:Params: {} 2023-10-12T00:56:49.019 DEBUG:boto:Token: None 2023-10-12T00:56:49.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df27a133c8f5aa6b-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.023 DEBUG:boto:b'wafmoa-58obj000000000000000000311000falseobj000000000000000000312023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.024 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.024 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.025 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.025 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.026 DEBUG:boto:Method: GET 2023-10-12T00:56:49.026 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.027 DEBUG:boto:Data: 2023-10-12T00:56:49.027 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.030 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.030 DEBUG:boto:Params: {} 2023-10-12T00:56:49.031 DEBUG:boto:Token: None 2023-10-12T00:56:49.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d655d0d4deaec8e9-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.038 DEBUG:boto:b'wafmoa-58obj000000000000000000311000falseobj000000000000000000312023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.039 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.040 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.040 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.041 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.041 DEBUG:boto:Data: 2023-10-12T00:56:49.041 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.042 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.043 DEBUG:boto:Params: {} 2023-10-12T00:56:49.043 DEBUG:boto:Token: None 2023-10-12T00:56:49.044 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CVdFR8DLdbHUCRLlgwSTdVWabdI= 2023-10-12T00:56:49.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CVdFR8DLdbHUCRLlgwSTdVWabdI=', 'Content-Length': '0'} 2023-10-12T00:56:49.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd46828a8b3250a4-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.047 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.048 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.048 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.049 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.049 DEBUG:boto:Data: 2023-10-12T00:56:49.050 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.051 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.052 DEBUG:boto:Params: {} 2023-10-12T00:56:49.053 DEBUG:boto:Token: None 2023-10-12T00:56:49.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CVdFR8DLdbHUCRLlgwSTdVWabdI= 2023-10-12T00:56:49.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CVdFR8DLdbHUCRLlgwSTdVWabdI=', 'Content-Length': '0'} 2023-10-12T00:56:49.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d3184a2a76a03d-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-12T00:56:49.059 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.059 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.060 DEBUG:boto:Method: GET 2023-10-12T00:56:49.060 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.061 DEBUG:boto:Data: 2023-10-12T00:56:49.061 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.062 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.062 DEBUG:boto:Params: {} 2023-10-12T00:56:49.063 DEBUG:boto:Token: None 2023-10-12T00:56:49.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ft2/wNzwbxy/AgFNi4myVUyg/cg= 2023-10-12T00:56:49.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ft2/wNzwbxy/AgFNi4myVUyg/cg=', 'Content-Length': '0'} 2023-10-12T00:56:49.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007728d4915695a907-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.110 DEBUG:boto:path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.111 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.111 DEBUG:boto:Method: GET 2023-10-12T00:56:49.112 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.112 DEBUG:boto:Data: 2023-10-12T00:56:49.113 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.114 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.114 DEBUG:boto:Params: {} 2023-10-12T00:56:49.115 DEBUG:boto:Token: None 2023-10-12T00:56:49.115 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000031 2023-10-12T00:56:49.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ft2/wNzwbxy/AgFNi4myVUyg/cg= 2023-10-12T00:56:49.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ft2/wNzwbxy/AgFNi4myVUyg/cg=', 'Content-Length': '0'} 2023-10-12T00:56:49.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2b462e076f3849-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.119 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.119 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.120 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.120 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.121 DEBUG:boto:Method: GET 2023-10-12T00:56:49.121 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.122 DEBUG:boto:Data: 2023-10-12T00:56:49.123 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.124 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.124 DEBUG:boto:Params: {} 2023-10-12T00:56:49.125 DEBUG:boto:Token: None 2023-10-12T00:56:49.126 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004811aaaa5f71a3a8-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.131 DEBUG:boto:b'wafmoa-58obj000000000000000000311000falseobj000000000000000000312023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.132 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.133 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.133 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.133 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.134 DEBUG:boto:Method: GET 2023-10-12T00:56:49.134 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000031 2023-10-12T00:56:49.135 DEBUG:boto:Data: 2023-10-12T00:56:49.135 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.136 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.136 DEBUG:boto:Params: {} 2023-10-12T00:56:49.137 DEBUG:boto:Token: None 2023-10-12T00:56:49.137 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df86b48c8b5be7e-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.186 DEBUG:boto:b'wafmoa-58obj000000000000000000311000falseobj000000000000000000312023-10-12T00:56:34.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-12T00:56:49.187 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.188 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.188 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.189 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.189 DEBUG:boto:Method: GET 2023-10-12T00:56:49.189 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.190 DEBUG:boto:Data: 2023-10-12T00:56:49.190 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.191 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.191 DEBUG:boto:Params: {} 2023-10-12T00:56:49.192 DEBUG:boto:Token: None 2023-10-12T00:56:49.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30= 2023-10-12T00:56:49.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30=', 'Content-Length': '0'} 2023-10-12T00:56:49.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007111c4b446ccf5e6-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.196 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.197 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.197 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.198 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.198 DEBUG:boto:Method: GET 2023-10-12T00:56:49.199 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.199 DEBUG:boto:Data: 2023-10-12T00:56:49.199 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.200 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.201 DEBUG:boto:Params: {} 2023-10-12T00:56:49.201 DEBUG:boto:Token: None 2023-10-12T00:56:49.201 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.202 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30= 2023-10-12T00:56:49.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30=', 'Content-Length': '0'} 2023-10-12T00:56:49.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007baab53a9a175b26-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.205 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.205 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.206 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.206 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.207 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.207 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.208 DEBUG:boto:Data: 2023-10-12T00:56:49.208 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.209 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.209 DEBUG:boto:Params: {} 2023-10-12T00:56:49.210 DEBUG:boto:Token: None 2023-10-12T00:56:49.210 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X4AJ6mwK2KdhDE7igxIKxl1NfaA= 2023-10-12T00:56:49.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X4AJ6mwK2KdhDE7igxIKxl1NfaA=', 'Content-Length': '0'} 2023-10-12T00:56:49.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972156c5bf0ccdcc-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.214 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.214 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.215 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.215 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.215 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.216 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.216 DEBUG:boto:Data: 2023-10-12T00:56:49.217 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.217 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.218 DEBUG:boto:Params: {} 2023-10-12T00:56:49.218 DEBUG:boto:Token: None 2023-10-12T00:56:49.219 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.219 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X4AJ6mwK2KdhDE7igxIKxl1NfaA= 2023-10-12T00:56:49.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X4AJ6mwK2KdhDE7igxIKxl1NfaA=', 'Content-Length': '0'} 2023-10-12T00:56:49.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f8e5a6e179f235-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-12T00:56:49.223 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.223 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.224 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.224 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.225 DEBUG:boto:Method: GET 2023-10-12T00:56:49.226 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.227 DEBUG:boto:Data: 2023-10-12T00:56:49.228 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.229 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.230 DEBUG:boto:Params: {} 2023-10-12T00:56:49.231 DEBUG:boto:Token: None 2023-10-12T00:56:49.231 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30= 2023-10-12T00:56:49.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30=', 'Content-Length': '0'} 2023-10-12T00:56:49.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bde2a009db6b377-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.238 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.238 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.239 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.239 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.240 DEBUG:boto:Method: GET 2023-10-12T00:56:49.241 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.241 DEBUG:boto:Data: 2023-10-12T00:56:49.242 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.244 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.244 DEBUG:boto:Params: {} 2023-10-12T00:56:49.245 DEBUG:boto:Token: None 2023-10-12T00:56:49.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032?versionId=null 2023-10-12T00:56:49.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30= 2023-10-12T00:56:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MKSvXEmAaDb/78RYBp6u8qbMZ30=', 'Content-Length': '0'} 2023-10-12T00:56:49.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200ff2426d947142-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.251 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.252 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.253 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.254 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.254 DEBUG:boto:Method: GET 2023-10-12T00:56:49.255 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.256 DEBUG:boto:Data: 2023-10-12T00:56:49.257 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.258 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.259 DEBUG:boto:Params: {} 2023-10-12T00:56:49.260 DEBUG:boto:Token: None 2023-10-12T00:56:49.261 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.262 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d742286f175159-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.267 DEBUG:boto:b'wafmoa-58obj000000000000000000321000falseobj000000000000000000322023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.268 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.268 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.269 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.270 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.270 DEBUG:boto:Method: GET 2023-10-12T00:56:49.271 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.271 DEBUG:boto:Data: 2023-10-12T00:56:49.272 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.273 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.273 DEBUG:boto:Params: {} 2023-10-12T00:56:49.273 DEBUG:boto:Token: None 2023-10-12T00:56:49.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000087267bcfc29463-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.280 DEBUG:boto:b'wafmoa-58obj000000000000000000321000falseobj000000000000000000322023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.281 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.282 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.283 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.284 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.284 DEBUG:boto:Data: 2023-10-12T00:56:49.285 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.286 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.287 DEBUG:boto:Params: {} 2023-10-12T00:56:49.287 DEBUG:boto:Token: None 2023-10-12T00:56:49.288 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xGNp3H6WEW62JsA4Tvzz26j8GGQ= 2023-10-12T00:56:49.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xGNp3H6WEW62JsA4Tvzz26j8GGQ=', 'Content-Length': '0'} 2023-10-12T00:56:49.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000affc62d9058de95-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.292 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.292 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.293 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.293 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.293 DEBUG:boto:Data: 2023-10-12T00:56:49.294 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.295 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.295 DEBUG:boto:Params: {} 2023-10-12T00:56:49.295 DEBUG:boto:Token: None 2023-10-12T00:56:49.296 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xGNp3H6WEW62JsA4Tvzz26j8GGQ= 2023-10-12T00:56:49.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xGNp3H6WEW62JsA4Tvzz26j8GGQ=', 'Content-Length': '0'} 2023-10-12T00:56:49.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84ceebba46b072c-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-12T00:56:49.300 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.300 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.301 DEBUG:boto:Method: GET 2023-10-12T00:56:49.301 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.302 DEBUG:boto:Data: 2023-10-12T00:56:49.302 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.303 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.303 DEBUG:boto:Params: {} 2023-10-12T00:56:49.304 DEBUG:boto:Token: None 2023-10-12T00:56:49.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FonAwSe3eZHMfgYvh+zENMdgrXc= 2023-10-12T00:56:49.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FonAwSe3eZHMfgYvh+zENMdgrXc=', 'Content-Length': '0'} 2023-10-12T00:56:49.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3a5134346bd4ee-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.350 DEBUG:boto:path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.351 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.351 DEBUG:boto:Method: GET 2023-10-12T00:56:49.352 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.352 DEBUG:boto:Data: 2023-10-12T00:56:49.353 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.353 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.354 DEBUG:boto:Params: {} 2023-10-12T00:56:49.355 DEBUG:boto:Token: None 2023-10-12T00:56:49.355 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000032 2023-10-12T00:56:49.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FonAwSe3eZHMfgYvh+zENMdgrXc= 2023-10-12T00:56:49.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FonAwSe3eZHMfgYvh+zENMdgrXc=', 'Content-Length': '0'} 2023-10-12T00:56:49.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc611af0c4de3d25-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.359 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.360 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.360 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.361 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.361 DEBUG:boto:Method: GET 2023-10-12T00:56:49.362 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.362 DEBUG:boto:Data: 2023-10-12T00:56:49.362 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.363 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.364 DEBUG:boto:Params: {} 2023-10-12T00:56:49.364 DEBUG:boto:Token: None 2023-10-12T00:56:49.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63c90250b4857fe-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.369 DEBUG:boto:b'wafmoa-58obj000000000000000000321000falseobj000000000000000000322023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.370 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.370 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.371 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.371 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.372 DEBUG:boto:Method: GET 2023-10-12T00:56:49.372 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000032 2023-10-12T00:56:49.372 DEBUG:boto:Data: 2023-10-12T00:56:49.373 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.374 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.374 DEBUG:boto:Params: {} 2023-10-12T00:56:49.375 DEBUG:boto:Token: None 2023-10-12T00:56:49.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000002cb9eef0138f9-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.381 DEBUG:boto:b'wafmoa-58obj000000000000000000321000falseobj000000000000000000322023-10-12T00:56:32.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-12T00:56:49.382 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.382 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.383 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.383 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.384 DEBUG:boto:Method: GET 2023-10-12T00:56:49.384 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.384 DEBUG:boto:Data: 2023-10-12T00:56:49.385 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.386 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.386 DEBUG:boto:Params: {} 2023-10-12T00:56:49.387 DEBUG:boto:Token: None 2023-10-12T00:56:49.387 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0= 2023-10-12T00:56:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0=', 'Content-Length': '0'} 2023-10-12T00:56:49.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b8e4396e3c208c-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.391 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.391 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.392 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.392 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.393 DEBUG:boto:Method: GET 2023-10-12T00:56:49.393 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.393 DEBUG:boto:Data: 2023-10-12T00:56:49.394 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.395 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.395 DEBUG:boto:Params: {} 2023-10-12T00:56:49.396 DEBUG:boto:Token: None 2023-10-12T00:56:49.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0= 2023-10-12T00:56:49.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0=', 'Content-Length': '0'} 2023-10-12T00:56:49.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cbe9da719c70a86-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.400 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.400 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.401 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.401 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.401 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.402 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.402 DEBUG:boto:Data: 2023-10-12T00:56:49.403 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.403 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.404 DEBUG:boto:Params: {} 2023-10-12T00:56:49.404 DEBUG:boto:Token: None 2023-10-12T00:56:49.405 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zKPlsTYFlSK7O/anpyYdUzxCO94= 2023-10-12T00:56:49.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zKPlsTYFlSK7O/anpyYdUzxCO94=', 'Content-Length': '0'} 2023-10-12T00:56:49.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b4f8fde1bb51ee-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.408 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.409 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.409 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.410 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.410 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.410 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.411 DEBUG:boto:Data: 2023-10-12T00:56:49.411 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.412 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.413 DEBUG:boto:Params: {} 2023-10-12T00:56:49.413 DEBUG:boto:Token: None 2023-10-12T00:56:49.413 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zKPlsTYFlSK7O/anpyYdUzxCO94= 2023-10-12T00:56:49.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zKPlsTYFlSK7O/anpyYdUzxCO94=', 'Content-Length': '0'} 2023-10-12T00:56:49.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055365b9772965e4b-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-12T00:56:49.418 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.418 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.419 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.419 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.419 DEBUG:boto:Method: GET 2023-10-12T00:56:49.420 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.420 DEBUG:boto:Data: 2023-10-12T00:56:49.421 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.422 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.422 DEBUG:boto:Params: {} 2023-10-12T00:56:49.422 DEBUG:boto:Token: None 2023-10-12T00:56:49.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0= 2023-10-12T00:56:49.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0=', 'Content-Length': '0'} 2023-10-12T00:56:49.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc3da5c095b9857-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.427 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.427 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.428 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.428 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.428 DEBUG:boto:Method: GET 2023-10-12T00:56:49.429 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.429 DEBUG:boto:Data: 2023-10-12T00:56:49.430 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.431 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.431 DEBUG:boto:Params: {} 2023-10-12T00:56:49.431 DEBUG:boto:Token: None 2023-10-12T00:56:49.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033?versionId=null 2023-10-12T00:56:49.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0= 2023-10-12T00:56:49.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Ub75ZAyRwoLC1AQC3ErT8uPpi0=', 'Content-Length': '0'} 2023-10-12T00:56:49.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5944304bb2a6a6c-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.478 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.479 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.480 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.480 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.480 DEBUG:boto:Method: GET 2023-10-12T00:56:49.481 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.481 DEBUG:boto:Data: 2023-10-12T00:56:49.482 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.483 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.483 DEBUG:boto:Params: {} 2023-10-12T00:56:49.484 DEBUG:boto:Token: None 2023-10-12T00:56:49.484 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042475e75adf8b90b-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.488 DEBUG:boto:b'wafmoa-58obj000000000000000000331000falseobj000000000000000000332023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.489 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.489 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.490 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.490 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.491 DEBUG:boto:Method: GET 2023-10-12T00:56:49.491 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.492 DEBUG:boto:Data: 2023-10-12T00:56:49.492 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.493 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.493 DEBUG:boto:Params: {} 2023-10-12T00:56:49.494 DEBUG:boto:Token: None 2023-10-12T00:56:49.494 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7b749f9e5d919e6-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.500 DEBUG:boto:b'wafmoa-58obj000000000000000000331000falseobj000000000000000000332023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.500 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.501 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.501 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.502 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.502 DEBUG:boto:Data: 2023-10-12T00:56:49.503 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.503 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.504 DEBUG:boto:Params: {} 2023-10-12T00:56:49.504 DEBUG:boto:Token: None 2023-10-12T00:56:49.505 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4I7MsYVK7Q99XsKhomIqC8tA8+4= 2023-10-12T00:56:49.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4I7MsYVK7Q99XsKhomIqC8tA8+4=', 'Content-Length': '0'} 2023-10-12T00:56:49.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000012de09ad134cf8-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.509 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.509 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.509 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.510 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.510 DEBUG:boto:Data: 2023-10-12T00:56:49.511 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.511 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.512 DEBUG:boto:Params: {} 2023-10-12T00:56:49.512 DEBUG:boto:Token: None 2023-10-12T00:56:49.513 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4I7MsYVK7Q99XsKhomIqC8tA8+4= 2023-10-12T00:56:49.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4I7MsYVK7Q99XsKhomIqC8tA8+4=', 'Content-Length': '0'} 2023-10-12T00:56:49.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbce3a17718e417-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-12T00:56:49.517 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.518 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.518 DEBUG:boto:Method: GET 2023-10-12T00:56:49.518 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.519 DEBUG:boto:Data: 2023-10-12T00:56:49.519 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.520 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.521 DEBUG:boto:Params: {} 2023-10-12T00:56:49.521 DEBUG:boto:Token: None 2023-10-12T00:56:49.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8n6XXSKzwvorlnyYQ8VvaKEhp30= 2023-10-12T00:56:49.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8n6XXSKzwvorlnyYQ8VvaKEhp30=', 'Content-Length': '0'} 2023-10-12T00:56:49.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000734d4cbf629fec5e-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.570 DEBUG:boto:path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.571 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.571 DEBUG:boto:Method: GET 2023-10-12T00:56:49.572 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.572 DEBUG:boto:Data: 2023-10-12T00:56:49.573 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.574 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.574 DEBUG:boto:Params: {} 2023-10-12T00:56:49.574 DEBUG:boto:Token: None 2023-10-12T00:56:49.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000033 2023-10-12T00:56:49.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8n6XXSKzwvorlnyYQ8VvaKEhp30= 2023-10-12T00:56:49.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8n6XXSKzwvorlnyYQ8VvaKEhp30=', 'Content-Length': '0'} 2023-10-12T00:56:49.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2b54cee4edd806-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.579 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.579 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.580 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.580 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.581 DEBUG:boto:Method: GET 2023-10-12T00:56:49.581 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.582 DEBUG:boto:Data: 2023-10-12T00:56:49.582 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.583 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.583 DEBUG:boto:Params: {} 2023-10-12T00:56:49.584 DEBUG:boto:Token: None 2023-10-12T00:56:49.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058dc2cbe5a8bcc4f-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.589 DEBUG:boto:b'wafmoa-58obj000000000000000000331000falseobj000000000000000000332023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.589 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.590 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.590 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.591 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.591 DEBUG:boto:Method: GET 2023-10-12T00:56:49.592 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000033 2023-10-12T00:56:49.592 DEBUG:boto:Data: 2023-10-12T00:56:49.592 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.593 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.594 DEBUG:boto:Params: {} 2023-10-12T00:56:49.594 DEBUG:boto:Token: None 2023-10-12T00:56:49.595 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5acda1157b3d68-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.642 DEBUG:boto:b'wafmoa-58obj000000000000000000331000falseobj000000000000000000332023-10-12T00:56:36.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-12T00:56:49.644 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.644 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.645 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.645 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.645 DEBUG:boto:Method: GET 2023-10-12T00:56:49.646 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.646 DEBUG:boto:Data: 2023-10-12T00:56:49.647 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.648 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.648 DEBUG:boto:Params: {} 2023-10-12T00:56:49.649 DEBUG:boto:Token: None 2023-10-12T00:56:49.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98= 2023-10-12T00:56:49.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98=', 'Content-Length': '0'} 2023-10-12T00:56:49.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887e79c2ace01145-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.653 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.654 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.654 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.655 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.655 DEBUG:boto:Method: GET 2023-10-12T00:56:49.656 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.656 DEBUG:boto:Data: 2023-10-12T00:56:49.656 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.657 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.658 DEBUG:boto:Params: {} 2023-10-12T00:56:49.658 DEBUG:boto:Token: None 2023-10-12T00:56:49.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98= 2023-10-12T00:56:49.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98=', 'Content-Length': '0'} 2023-10-12T00:56:49.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9574b7c7edff6e9-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.662 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.663 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.663 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.664 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.664 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.664 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.665 DEBUG:boto:Data: 2023-10-12T00:56:49.665 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.666 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.667 DEBUG:boto:Params: {} 2023-10-12T00:56:49.667 DEBUG:boto:Token: None 2023-10-12T00:56:49.668 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BsxJqz41N8ImaEJCQmDPAA60uG0= 2023-10-12T00:56:49.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BsxJqz41N8ImaEJCQmDPAA60uG0=', 'Content-Length': '0'} 2023-10-12T00:56:49.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c3afe8a99507cda-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.671 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.671 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.672 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.672 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.673 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.673 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.673 DEBUG:boto:Data: 2023-10-12T00:56:49.674 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.675 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.675 DEBUG:boto:Params: {} 2023-10-12T00:56:49.676 DEBUG:boto:Token: None 2023-10-12T00:56:49.676 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.676 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BsxJqz41N8ImaEJCQmDPAA60uG0= 2023-10-12T00:56:49.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BsxJqz41N8ImaEJCQmDPAA60uG0=', 'Content-Length': '0'} 2023-10-12T00:56:49.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e8dd2c807eff97-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-12T00:56:49.680 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.680 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.681 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.681 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.681 DEBUG:boto:Method: GET 2023-10-12T00:56:49.682 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.682 DEBUG:boto:Data: 2023-10-12T00:56:49.683 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.684 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.684 DEBUG:boto:Params: {} 2023-10-12T00:56:49.685 DEBUG:boto:Token: None 2023-10-12T00:56:49.685 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98= 2023-10-12T00:56:49.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98=', 'Content-Length': '0'} 2023-10-12T00:56:49.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720667ea3d08c67d-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.689 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.690 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.690 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.691 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.691 DEBUG:boto:Method: GET 2023-10-12T00:56:49.692 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.692 DEBUG:boto:Data: 2023-10-12T00:56:49.693 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.694 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.694 DEBUG:boto:Params: {} 2023-10-12T00:56:49.695 DEBUG:boto:Token: None 2023-10-12T00:56:49.695 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036?versionId=null 2023-10-12T00:56:49.696 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98= 2023-10-12T00:56:49.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SDdYl8703AEklrXkmK4eMVnx98=', 'Content-Length': '0'} 2023-10-12T00:56:49.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e52e4f04f48837-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.699 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.699 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.700 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.700 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.700 DEBUG:boto:Method: GET 2023-10-12T00:56:49.701 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.701 DEBUG:boto:Data: 2023-10-12T00:56:49.701 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.702 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.703 DEBUG:boto:Params: {} 2023-10-12T00:56:49.703 DEBUG:boto:Token: None 2023-10-12T00:56:49.704 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907aed152adca6cf-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.708 DEBUG:boto:b'wafmoa-58obj000000000000000000361000falseobj000000000000000000362023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.708 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.709 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.709 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.710 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.710 DEBUG:boto:Method: GET 2023-10-12T00:56:49.710 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.711 DEBUG:boto:Data: 2023-10-12T00:56:49.711 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.712 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.712 DEBUG:boto:Params: {} 2023-10-12T00:56:49.713 DEBUG:boto:Token: None 2023-10-12T00:56:49.713 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188b5953e6fe291b-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.762 DEBUG:boto:b'wafmoa-58obj000000000000000000361000falseobj000000000000000000362023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.763 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.763 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.764 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.764 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.765 DEBUG:boto:Data: 2023-10-12T00:56:49.765 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.766 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.766 DEBUG:boto:Params: {} 2023-10-12T00:56:49.767 DEBUG:boto:Token: None 2023-10-12T00:56:49.767 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wQ0cmPxsZbXQSd4x1+vlKSGYsdY= 2023-10-12T00:56:49.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wQ0cmPxsZbXQSd4x1+vlKSGYsdY=', 'Content-Length': '0'} 2023-10-12T00:56:49.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f4b36fc88e3062-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.771 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.771 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.772 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.772 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.773 DEBUG:boto:Data: 2023-10-12T00:56:49.773 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.774 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.774 DEBUG:boto:Params: {} 2023-10-12T00:56:49.775 DEBUG:boto:Token: None 2023-10-12T00:56:49.775 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wQ0cmPxsZbXQSd4x1+vlKSGYsdY= 2023-10-12T00:56:49.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wQ0cmPxsZbXQSd4x1+vlKSGYsdY=', 'Content-Length': '0'} 2023-10-12T00:56:49.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029da4506ca6af59a-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-12T00:56:49.779 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.779 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.780 DEBUG:boto:Method: GET 2023-10-12T00:56:49.780 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.781 DEBUG:boto:Data: 2023-10-12T00:56:49.781 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.782 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.782 DEBUG:boto:Params: {} 2023-10-12T00:56:49.783 DEBUG:boto:Token: None 2023-10-12T00:56:49.783 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Eecufyc/4wKs9AWY+qnChbY3Q4= 2023-10-12T00:56:49.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Eecufyc/4wKs9AWY+qnChbY3Q4=', 'Content-Length': '0'} 2023-10-12T00:56:49.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3bbce71aef11a91-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.826 DEBUG:boto:path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.827 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.827 DEBUG:boto:Method: GET 2023-10-12T00:56:49.828 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.828 DEBUG:boto:Data: 2023-10-12T00:56:49.829 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.829 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.830 DEBUG:boto:Params: {} 2023-10-12T00:56:49.830 DEBUG:boto:Token: None 2023-10-12T00:56:49.831 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000036 2023-10-12T00:56:49.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Eecufyc/4wKs9AWY+qnChbY3Q4= 2023-10-12T00:56:49.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Eecufyc/4wKs9AWY+qnChbY3Q4=', 'Content-Length': '0'} 2023-10-12T00:56:49.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607c1cef66b0554b-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.834 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.835 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.835 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.836 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.836 DEBUG:boto:Method: GET 2023-10-12T00:56:49.837 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.837 DEBUG:boto:Data: 2023-10-12T00:56:49.837 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.838 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.839 DEBUG:boto:Params: {} 2023-10-12T00:56:49.839 DEBUG:boto:Token: None 2023-10-12T00:56:49.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf39e657e7b8139-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.844 DEBUG:boto:b'wafmoa-58obj000000000000000000361000falseobj000000000000000000362023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.844 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.845 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.845 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.846 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.846 DEBUG:boto:Method: GET 2023-10-12T00:56:49.847 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000036 2023-10-12T00:56:49.847 DEBUG:boto:Data: 2023-10-12T00:56:49.847 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.850 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.850 DEBUG:boto:Params: {} 2023-10-12T00:56:49.851 DEBUG:boto:Token: None 2023-10-12T00:56:49.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e425dc6f52dba46-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.857 DEBUG:boto:b'wafmoa-58obj000000000000000000361000falseobj000000000000000000362023-10-12T00:56:37.501Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-12T00:56:49.858 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.859 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.859 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.860 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.860 DEBUG:boto:Method: GET 2023-10-12T00:56:49.860 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.861 DEBUG:boto:Data: 2023-10-12T00:56:49.861 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.862 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.863 DEBUG:boto:Params: {} 2023-10-12T00:56:49.863 DEBUG:boto:Token: None 2023-10-12T00:56:49.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs= 2023-10-12T00:56:49.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs=', 'Content-Length': '0'} 2023-10-12T00:56:49.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b5d776c7d33be7-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.869 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.869 DEBUG:boto:Method: GET 2023-10-12T00:56:49.870 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.870 DEBUG:boto:Data: 2023-10-12T00:56:49.871 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.872 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.872 DEBUG:boto:Params: {} 2023-10-12T00:56:49.873 DEBUG:boto:Token: None 2023-10-12T00:56:49.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs= 2023-10-12T00:56:49.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs=', 'Content-Length': '0'} 2023-10-12T00:56:49.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be94debf452b5af-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.877 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.877 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.878 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.878 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.879 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.879 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.879 DEBUG:boto:Data: 2023-10-12T00:56:49.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.881 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.881 DEBUG:boto:Params: {} 2023-10-12T00:56:49.882 DEBUG:boto:Token: None 2023-10-12T00:56:49.882 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xYacWo56o/8pDu7tB4gRXYu2YbY= 2023-10-12T00:56:49.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xYacWo56o/8pDu7tB4gRXYu2YbY=', 'Content-Length': '0'} 2023-10-12T00:56:49.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840caf732974e33b-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.886 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.886 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.887 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.887 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.887 DEBUG:boto:Method: HEAD 2023-10-12T00:56:49.888 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.888 DEBUG:boto:Data: 2023-10-12T00:56:49.889 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.890 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.890 DEBUG:boto:Params: {} 2023-10-12T00:56:49.890 DEBUG:boto:Token: None 2023-10-12T00:56:49.891 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xYacWo56o/8pDu7tB4gRXYu2YbY= 2023-10-12T00:56:49.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xYacWo56o/8pDu7tB4gRXYu2YbY=', 'Content-Length': '0'} 2023-10-12T00:56:49.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010d454318475fdb8-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-12T00:56:49.895 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.895 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.896 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.896 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.897 DEBUG:boto:Method: GET 2023-10-12T00:56:49.897 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.898 DEBUG:boto:Data: 2023-10-12T00:56:49.898 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.899 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.899 DEBUG:boto:Params: {} 2023-10-12T00:56:49.900 DEBUG:boto:Token: None 2023-10-12T00:56:49.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.901 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs= 2023-10-12T00:56:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs=', 'Content-Length': '0'} 2023-10-12T00:56:49.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763dba29441cce06-0065274451-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.950 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.951 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:49.951 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.952 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.952 DEBUG:boto:Method: GET 2023-10-12T00:56:49.953 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.953 DEBUG:boto:Data: 2023-10-12T00:56:49.954 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.954 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.955 DEBUG:boto:Params: {} 2023-10-12T00:56:49.955 DEBUG:boto:Token: None 2023-10-12T00:56:49.956 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/obj00000000000000000037?versionId=null 2023-10-12T00:56:49.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs= 2023-10-12T00:56:49.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CMtW6NvjnJOTVTToxiJn8K6RcIs=', 'Content-Length': '0'} 2023-10-12T00:56:49.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea752379b70ddb1-0065274451-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.960 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.960 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.961 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.961 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.962 DEBUG:boto:Method: GET 2023-10-12T00:56:49.962 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.962 DEBUG:boto:Data: 2023-10-12T00:56:49.963 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:49.964 DEBUG:boto:Port: 8001 2023-10-12T00:56:49.964 DEBUG:boto:Params: {} 2023-10-12T00:56:49.965 DEBUG:boto:Token: None 2023-10-12T00:56:49.965 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fcd1021a0bcc26-0065274451-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:49.969 DEBUG:boto:b'wafmoa-58obj000000000000000000371000falseobj000000000000000000372023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:49.970 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:49.970 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:49.971 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.971 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.972 DEBUG:boto:Method: GET 2023-10-12T00:56:49.972 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:49.972 DEBUG:boto:Data: 2023-10-12T00:56:49.973 DEBUG:boto:Headers: {} 2023-10-12T00:56:49.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:49.974 DEBUG:boto:Port: 8000 2023-10-12T00:56:49.974 DEBUG:boto:Params: {} 2023-10-12T00:56:49.975 DEBUG:boto:Token: None 2023-10-12T00:56:49.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:49 GMT /wafmoa-58/ 2023-10-12T00:56:49.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY= 2023-10-12T00:56:49.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BA8MMK9Qr4B+D4/XKtDiY+WLhVY=', 'Content-Length': '0'} 2023-10-12T00:56:49.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733ca3c9d75974d6-0065274451-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.022 DEBUG:boto:b'wafmoa-58obj000000000000000000371000falseobj000000000000000000372023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.023 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.024 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.024 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.024 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.025 DEBUG:boto:Data: 2023-10-12T00:56:50.025 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.026 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.027 DEBUG:boto:Params: {} 2023-10-12T00:56:50.027 DEBUG:boto:Token: None 2023-10-12T00:56:50.028 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8mri7If4+NHTCv7ggFDsl252OOg= 2023-10-12T00:56:50.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8mri7If4+NHTCv7ggFDsl252OOg=', 'Content-Length': '0'} 2023-10-12T00:56:50.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c8a5e446e6a413-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.031 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.032 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.032 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.032 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.033 DEBUG:boto:Data: 2023-10-12T00:56:50.033 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.034 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.035 DEBUG:boto:Params: {} 2023-10-12T00:56:50.035 DEBUG:boto:Token: None 2023-10-12T00:56:50.035 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8mri7If4+NHTCv7ggFDsl252OOg= 2023-10-12T00:56:50.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8mri7If4+NHTCv7ggFDsl252OOg=', 'Content-Length': '0'} 2023-10-12T00:56:50.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37b596f406f5225-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-12T00:56:50.039 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.040 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.040 DEBUG:boto:Method: GET 2023-10-12T00:56:50.041 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.041 DEBUG:boto:Data: 2023-10-12T00:56:50.042 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.043 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.043 DEBUG:boto:Params: {} 2023-10-12T00:56:50.043 DEBUG:boto:Token: None 2023-10-12T00:56:50.044 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEbeh76rxbokeDs6puSTSFLpuy4= 2023-10-12T00:56:50.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEbeh76rxbokeDs6puSTSFLpuy4=', 'Content-Length': '0'} 2023-10-12T00:56:50.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c745b00243394377-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.050 DEBUG:boto:path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.050 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.051 DEBUG:boto:Method: GET 2023-10-12T00:56:50.051 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.052 DEBUG:boto:Data: 2023-10-12T00:56:50.052 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.053 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.053 DEBUG:boto:Params: {} 2023-10-12T00:56:50.054 DEBUG:boto:Token: None 2023-10-12T00:56:50.054 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000037 2023-10-12T00:56:50.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mEbeh76rxbokeDs6puSTSFLpuy4= 2023-10-12T00:56:50.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mEbeh76rxbokeDs6puSTSFLpuy4=', 'Content-Length': '0'} 2023-10-12T00:56:50.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a7302e8ffe8075-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.058 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.058 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.059 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.059 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.059 DEBUG:boto:Method: GET 2023-10-12T00:56:50.060 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.060 DEBUG:boto:Data: 2023-10-12T00:56:50.061 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.061 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.062 DEBUG:boto:Params: {} 2023-10-12T00:56:50.062 DEBUG:boto:Token: None 2023-10-12T00:56:50.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c3e85a968b356c-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.110 DEBUG:boto:b'wafmoa-58obj000000000000000000371000falseobj000000000000000000372023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.111 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.111 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.112 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.112 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.113 DEBUG:boto:Method: GET 2023-10-12T00:56:50.113 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000037 2023-10-12T00:56:50.114 DEBUG:boto:Data: 2023-10-12T00:56:50.114 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.115 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.115 DEBUG:boto:Params: {} 2023-10-12T00:56:50.116 DEBUG:boto:Token: None 2023-10-12T00:56:50.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab638ad67ad4a27e-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.122 DEBUG:boto:b'wafmoa-58obj000000000000000000371000falseobj000000000000000000372023-10-12T00:56:33.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-12T00:56:50.123 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.124 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.124 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.125 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.125 DEBUG:boto:Method: GET 2023-10-12T00:56:50.125 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.126 DEBUG:boto:Data: 2023-10-12T00:56:50.126 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.127 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.128 DEBUG:boto:Params: {} 2023-10-12T00:56:50.128 DEBUG:boto:Token: None 2023-10-12T00:56:50.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk= 2023-10-12T00:56:50.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk=', 'Content-Length': '0'} 2023-10-12T00:56:50.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2495228ee403373-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.132 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.133 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.133 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.134 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.134 DEBUG:boto:Method: GET 2023-10-12T00:56:50.135 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.135 DEBUG:boto:Data: 2023-10-12T00:56:50.135 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.136 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.137 DEBUG:boto:Params: {} 2023-10-12T00:56:50.137 DEBUG:boto:Token: None 2023-10-12T00:56:50.138 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk= 2023-10-12T00:56:50.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk=', 'Content-Length': '0'} 2023-10-12T00:56:50.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c5d7d986a76bad-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.142 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.142 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.142 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.143 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.143 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.144 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.144 DEBUG:boto:Data: 2023-10-12T00:56:50.144 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.145 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.146 DEBUG:boto:Params: {} 2023-10-12T00:56:50.146 DEBUG:boto:Token: None 2023-10-12T00:56:50.147 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iZ3glMSDqU7utPCkj8mWUBg366w= 2023-10-12T00:56:50.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iZ3glMSDqU7utPCkj8mWUBg366w=', 'Content-Length': '0'} 2023-10-12T00:56:50.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ded716f47b05015-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.150 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.150 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.151 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.151 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.152 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.152 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.152 DEBUG:boto:Data: 2023-10-12T00:56:50.153 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.154 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.154 DEBUG:boto:Params: {} 2023-10-12T00:56:50.154 DEBUG:boto:Token: None 2023-10-12T00:56:50.155 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iZ3glMSDqU7utPCkj8mWUBg366w= 2023-10-12T00:56:50.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iZ3glMSDqU7utPCkj8mWUBg366w=', 'Content-Length': '0'} 2023-10-12T00:56:50.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002238a9e9e3544064-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-12T00:56:50.159 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.159 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.160 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.161 DEBUG:boto:Method: GET 2023-10-12T00:56:50.161 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.162 DEBUG:boto:Data: 2023-10-12T00:56:50.162 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.163 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.163 DEBUG:boto:Params: {} 2023-10-12T00:56:50.164 DEBUG:boto:Token: None 2023-10-12T00:56:50.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk= 2023-10-12T00:56:50.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk=', 'Content-Length': '0'} 2023-10-12T00:56:50.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a23b32b9301cc01-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.168 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.168 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.169 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.169 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.170 DEBUG:boto:Method: GET 2023-10-12T00:56:50.170 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.171 DEBUG:boto:Data: 2023-10-12T00:56:50.171 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.172 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.172 DEBUG:boto:Params: {} 2023-10-12T00:56:50.173 DEBUG:boto:Token: None 2023-10-12T00:56:50.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039?versionId=null 2023-10-12T00:56:50.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk= 2023-10-12T00:56:50.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iKQ2TQ8deAnhDkvH/W4wSLl6Ejk=', 'Content-Length': '0'} 2023-10-12T00:56:50.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003630d425a6ad3e38-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.177 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.177 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.178 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.178 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.178 DEBUG:boto:Method: GET 2023-10-12T00:56:50.179 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.179 DEBUG:boto:Data: 2023-10-12T00:56:50.180 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.181 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.181 DEBUG:boto:Params: {} 2023-10-12T00:56:50.182 DEBUG:boto:Token: None 2023-10-12T00:56:50.182 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f1efff1dc0896b-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.186 DEBUG:boto:b'wafmoa-58obj000000000000000000391000falseobj000000000000000000392023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.187 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.187 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.188 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.188 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.188 DEBUG:boto:Method: GET 2023-10-12T00:56:50.189 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.189 DEBUG:boto:Data: 2023-10-12T00:56:50.190 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.191 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.191 DEBUG:boto:Params: {} 2023-10-12T00:56:50.191 DEBUG:boto:Token: None 2023-10-12T00:56:50.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034bd38e412d1613-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.197 DEBUG:boto:b'wafmoa-58obj000000000000000000391000falseobj000000000000000000392023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.198 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.198 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.199 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.199 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.200 DEBUG:boto:Data: 2023-10-12T00:56:50.200 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.201 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.201 DEBUG:boto:Params: {} 2023-10-12T00:56:50.202 DEBUG:boto:Token: None 2023-10-12T00:56:50.202 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pmBbEP6EV/DGadT0Qz0jql2wqSY= 2023-10-12T00:56:50.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pmBbEP6EV/DGadT0Qz0jql2wqSY=', 'Content-Length': '0'} 2023-10-12T00:56:50.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ade191bcb15594-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.206 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.206 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.207 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.207 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.207 DEBUG:boto:Data: 2023-10-12T00:56:50.208 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.209 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.209 DEBUG:boto:Params: {} 2023-10-12T00:56:50.209 DEBUG:boto:Token: None 2023-10-12T00:56:50.210 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pmBbEP6EV/DGadT0Qz0jql2wqSY= 2023-10-12T00:56:50.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pmBbEP6EV/DGadT0Qz0jql2wqSY=', 'Content-Length': '0'} 2023-10-12T00:56:50.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e487a47d7bba6c-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-12T00:56:50.214 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.214 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.215 DEBUG:boto:Method: GET 2023-10-12T00:56:50.215 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.216 DEBUG:boto:Data: 2023-10-12T00:56:50.216 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.217 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.218 DEBUG:boto:Params: {} 2023-10-12T00:56:50.218 DEBUG:boto:Token: None 2023-10-12T00:56:50.219 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.219 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qZS41tvPSs6xzdh5Tjvve/uIj94= 2023-10-12T00:56:50.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qZS41tvPSs6xzdh5Tjvve/uIj94=', 'Content-Length': '0'} 2023-10-12T00:56:50.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c9e8125e204de9-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.222 DEBUG:boto:path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.223 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.223 DEBUG:boto:Method: GET 2023-10-12T00:56:50.224 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.224 DEBUG:boto:Data: 2023-10-12T00:56:50.224 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.225 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.226 DEBUG:boto:Params: {} 2023-10-12T00:56:50.226 DEBUG:boto:Token: None 2023-10-12T00:56:50.227 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000039 2023-10-12T00:56:50.227 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qZS41tvPSs6xzdh5Tjvve/uIj94= 2023-10-12T00:56:50.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qZS41tvPSs6xzdh5Tjvve/uIj94=', 'Content-Length': '0'} 2023-10-12T00:56:50.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d19b7eb128e12ab-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.274 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.275 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.275 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.276 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.276 DEBUG:boto:Method: GET 2023-10-12T00:56:50.277 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.277 DEBUG:boto:Data: 2023-10-12T00:56:50.277 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.278 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.279 DEBUG:boto:Params: {} 2023-10-12T00:56:50.279 DEBUG:boto:Token: None 2023-10-12T00:56:50.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7150d2291d12c0b-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.284 DEBUG:boto:b'wafmoa-58obj000000000000000000391000falseobj000000000000000000392023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.285 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.285 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.285 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.286 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.286 DEBUG:boto:Method: GET 2023-10-12T00:56:50.287 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000039 2023-10-12T00:56:50.287 DEBUG:boto:Data: 2023-10-12T00:56:50.288 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.288 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.289 DEBUG:boto:Params: {} 2023-10-12T00:56:50.289 DEBUG:boto:Token: None 2023-10-12T00:56:50.290 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b697c1c472840f-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.296 DEBUG:boto:b'wafmoa-58obj000000000000000000391000falseobj000000000000000000392023-10-12T00:56:36.319Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-12T00:56:50.297 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.297 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.298 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.298 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.299 DEBUG:boto:Method: GET 2023-10-12T00:56:50.299 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.299 DEBUG:boto:Data: 2023-10-12T00:56:50.300 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.301 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.301 DEBUG:boto:Params: {} 2023-10-12T00:56:50.301 DEBUG:boto:Token: None 2023-10-12T00:56:50.302 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo= 2023-10-12T00:56:50.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo=', 'Content-Length': '0'} 2023-10-12T00:56:50.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae895648bae5c45-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.346 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.347 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.347 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.348 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.348 DEBUG:boto:Method: GET 2023-10-12T00:56:50.348 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.349 DEBUG:boto:Data: 2023-10-12T00:56:50.349 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.350 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.350 DEBUG:boto:Params: {} 2023-10-12T00:56:50.351 DEBUG:boto:Token: None 2023-10-12T00:56:50.351 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo= 2023-10-12T00:56:50.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo=', 'Content-Length': '0'} 2023-10-12T00:56:50.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc44ff5a5ad1ffd-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.355 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.355 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.356 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.356 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.357 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.357 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.358 DEBUG:boto:Data: 2023-10-12T00:56:50.358 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.359 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.359 DEBUG:boto:Params: {} 2023-10-12T00:56:50.360 DEBUG:boto:Token: None 2023-10-12T00:56:50.360 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rils4srn8YwVbCyrPZo55rmw9VY= 2023-10-12T00:56:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rils4srn8YwVbCyrPZo55rmw9VY=', 'Content-Length': '0'} 2023-10-12T00:56:50.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bc1190a35a3b5c8-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.363 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.364 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.364 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.365 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.365 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.365 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.366 DEBUG:boto:Data: 2023-10-12T00:56:50.366 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.367 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.368 DEBUG:boto:Params: {} 2023-10-12T00:56:50.368 DEBUG:boto:Token: None 2023-10-12T00:56:50.369 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rils4srn8YwVbCyrPZo55rmw9VY= 2023-10-12T00:56:50.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rils4srn8YwVbCyrPZo55rmw9VY=', 'Content-Length': '0'} 2023-10-12T00:56:50.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000168bc1e5089e93f8-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-12T00:56:50.372 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.373 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.373 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.374 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.374 DEBUG:boto:Method: GET 2023-10-12T00:56:50.375 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.375 DEBUG:boto:Data: 2023-10-12T00:56:50.375 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.376 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.377 DEBUG:boto:Params: {} 2023-10-12T00:56:50.377 DEBUG:boto:Token: None 2023-10-12T00:56:50.378 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo= 2023-10-12T00:56:50.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo=', 'Content-Length': '0'} 2023-10-12T00:56:50.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000defed2fcdba6fd2-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.381 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.382 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.382 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.383 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.383 DEBUG:boto:Method: GET 2023-10-12T00:56:50.383 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.384 DEBUG:boto:Data: 2023-10-12T00:56:50.384 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.385 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.386 DEBUG:boto:Params: {} 2023-10-12T00:56:50.386 DEBUG:boto:Token: None 2023-10-12T00:56:50.387 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040?versionId=null 2023-10-12T00:56:50.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo= 2023-10-12T00:56:50.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oKDrNUA4PPyiyHE5wO4OWoMugCo=', 'Content-Length': '0'} 2023-10-12T00:56:50.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f84c8b1ef156b05-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.390 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.391 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.391 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.392 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.392 DEBUG:boto:Method: GET 2023-10-12T00:56:50.393 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.393 DEBUG:boto:Data: 2023-10-12T00:56:50.393 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.394 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.395 DEBUG:boto:Params: {} 2023-10-12T00:56:50.395 DEBUG:boto:Token: None 2023-10-12T00:56:50.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006908efcaef4be5a9-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.400 DEBUG:boto:b'wafmoa-58obj000000000000000000401000falseobj000000000000000000402023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.400 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.401 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.401 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.402 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.402 DEBUG:boto:Method: GET 2023-10-12T00:56:50.403 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.403 DEBUG:boto:Data: 2023-10-12T00:56:50.404 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.404 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.405 DEBUG:boto:Params: {} 2023-10-12T00:56:50.405 DEBUG:boto:Token: None 2023-10-12T00:56:50.406 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201d8122e19c9fb5-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.412 DEBUG:boto:b'wafmoa-58obj000000000000000000401000falseobj000000000000000000402023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.412 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.413 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.413 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.414 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.414 DEBUG:boto:Data: 2023-10-12T00:56:50.414 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.415 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.416 DEBUG:boto:Params: {} 2023-10-12T00:56:50.416 DEBUG:boto:Token: None 2023-10-12T00:56:50.416 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5GM0VIa5vgliv7j9Hz+w4gwaYsA= 2023-10-12T00:56:50.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5GM0VIa5vgliv7j9Hz+w4gwaYsA=', 'Content-Length': '0'} 2023-10-12T00:56:50.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7df73345ea36e8-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.420 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.421 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.421 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.422 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.422 DEBUG:boto:Data: 2023-10-12T00:56:50.423 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.423 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.424 DEBUG:boto:Params: {} 2023-10-12T00:56:50.424 DEBUG:boto:Token: None 2023-10-12T00:56:50.425 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5GM0VIa5vgliv7j9Hz+w4gwaYsA= 2023-10-12T00:56:50.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5GM0VIa5vgliv7j9Hz+w4gwaYsA=', 'Content-Length': '0'} 2023-10-12T00:56:50.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1d3afceba540d7-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-12T00:56:50.429 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.429 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.430 DEBUG:boto:Method: GET 2023-10-12T00:56:50.430 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.431 DEBUG:boto:Data: 2023-10-12T00:56:50.431 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.432 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.432 DEBUG:boto:Params: {} 2023-10-12T00:56:50.433 DEBUG:boto:Token: None 2023-10-12T00:56:50.433 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ee5S1/icEY3/FBKOt1MPtpaseQ0= 2023-10-12T00:56:50.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ee5S1/icEY3/FBKOt1MPtpaseQ0=', 'Content-Length': '0'} 2023-10-12T00:56:50.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4bec166bc1f17e5-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.478 DEBUG:boto:path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.479 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.480 DEBUG:boto:Method: GET 2023-10-12T00:56:50.480 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.481 DEBUG:boto:Data: 2023-10-12T00:56:50.481 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.482 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.482 DEBUG:boto:Params: {} 2023-10-12T00:56:50.483 DEBUG:boto:Token: None 2023-10-12T00:56:50.483 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000040 2023-10-12T00:56:50.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ee5S1/icEY3/FBKOt1MPtpaseQ0= 2023-10-12T00:56:50.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ee5S1/icEY3/FBKOt1MPtpaseQ0=', 'Content-Length': '0'} 2023-10-12T00:56:50.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ec56d1bec206ee-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.488 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.488 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.489 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.489 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.489 DEBUG:boto:Method: GET 2023-10-12T00:56:50.490 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.490 DEBUG:boto:Data: 2023-10-12T00:56:50.491 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.492 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.492 DEBUG:boto:Params: {} 2023-10-12T00:56:50.493 DEBUG:boto:Token: None 2023-10-12T00:56:50.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d567a278b81c4d5d-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.497 DEBUG:boto:b'wafmoa-58obj000000000000000000401000falseobj000000000000000000402023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.498 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.499 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.499 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.499 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.500 DEBUG:boto:Method: GET 2023-10-12T00:56:50.500 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000040 2023-10-12T00:56:50.501 DEBUG:boto:Data: 2023-10-12T00:56:50.501 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.502 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.503 DEBUG:boto:Params: {} 2023-10-12T00:56:50.503 DEBUG:boto:Token: None 2023-10-12T00:56:50.503 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6053d552bf3f7e-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.510 DEBUG:boto:b'wafmoa-58obj000000000000000000401000falseobj000000000000000000402023-10-12T00:56:36.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-12T00:56:50.511 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.511 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.512 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.512 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.513 DEBUG:boto:Method: GET 2023-10-12T00:56:50.513 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.514 DEBUG:boto:Data: 2023-10-12T00:56:50.514 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.515 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.516 DEBUG:boto:Params: {} 2023-10-12T00:56:50.516 DEBUG:boto:Token: None 2023-10-12T00:56:50.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM= 2023-10-12T00:56:50.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM=', 'Content-Length': '0'} 2023-10-12T00:56:50.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6892205d00f5e19-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.521 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.522 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.522 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.522 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.523 DEBUG:boto:Method: GET 2023-10-12T00:56:50.523 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.524 DEBUG:boto:Data: 2023-10-12T00:56:50.524 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.525 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.525 DEBUG:boto:Params: {} 2023-10-12T00:56:50.526 DEBUG:boto:Token: None 2023-10-12T00:56:50.526 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM= 2023-10-12T00:56:50.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM=', 'Content-Length': '0'} 2023-10-12T00:56:50.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060be690d66cdb03b-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.530 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.531 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.531 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.532 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.532 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.532 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.533 DEBUG:boto:Data: 2023-10-12T00:56:50.533 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.534 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.534 DEBUG:boto:Params: {} 2023-10-12T00:56:50.535 DEBUG:boto:Token: None 2023-10-12T00:56:50.535 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.536 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71VBLq0HSG5HvNp7Kqq236pwwGA= 2023-10-12T00:56:50.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71VBLq0HSG5HvNp7Kqq236pwwGA=', 'Content-Length': '0'} 2023-10-12T00:56:50.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000489e3b0675f2b476-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.539 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.539 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.540 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.540 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.541 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.541 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.541 DEBUG:boto:Data: 2023-10-12T00:56:50.542 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.543 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.543 DEBUG:boto:Params: {} 2023-10-12T00:56:50.543 DEBUG:boto:Token: None 2023-10-12T00:56:50.544 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71VBLq0HSG5HvNp7Kqq236pwwGA= 2023-10-12T00:56:50.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71VBLq0HSG5HvNp7Kqq236pwwGA=', 'Content-Length': '0'} 2023-10-12T00:56:50.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b6240de71cffb2-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-12T00:56:50.548 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.548 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.549 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.549 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.550 DEBUG:boto:Method: GET 2023-10-12T00:56:50.550 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.551 DEBUG:boto:Data: 2023-10-12T00:56:50.551 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.552 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.552 DEBUG:boto:Params: {} 2023-10-12T00:56:50.553 DEBUG:boto:Token: None 2023-10-12T00:56:50.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM= 2023-10-12T00:56:50.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM=', 'Content-Length': '0'} 2023-10-12T00:56:50.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408661846ac004ef-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.557 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.557 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.558 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.558 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.559 DEBUG:boto:Method: GET 2023-10-12T00:56:50.559 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.559 DEBUG:boto:Data: 2023-10-12T00:56:50.560 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.561 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.561 DEBUG:boto:Params: {} 2023-10-12T00:56:50.562 DEBUG:boto:Token: None 2023-10-12T00:56:50.562 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042?versionId=null 2023-10-12T00:56:50.563 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM= 2023-10-12T00:56:50.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pfaWWPSG6iu85Ex7OaEWvy4CIdM=', 'Content-Length': '0'} 2023-10-12T00:56:50.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0105f414599c97-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.606 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.607 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.607 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.608 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.608 DEBUG:boto:Method: GET 2023-10-12T00:56:50.608 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.609 DEBUG:boto:Data: 2023-10-12T00:56:50.609 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.610 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.610 DEBUG:boto:Params: {} 2023-10-12T00:56:50.611 DEBUG:boto:Token: None 2023-10-12T00:56:50.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0de65e22e86946-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.615 DEBUG:boto:b'wafmoa-58obj000000000000000000421000falseobj000000000000000000422023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.616 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.616 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.617 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.617 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.618 DEBUG:boto:Method: GET 2023-10-12T00:56:50.618 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.619 DEBUG:boto:Data: 2023-10-12T00:56:50.619 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.620 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.620 DEBUG:boto:Params: {} 2023-10-12T00:56:50.621 DEBUG:boto:Token: None 2023-10-12T00:56:50.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3052d65c5bdcad6-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.627 DEBUG:boto:b'wafmoa-58obj000000000000000000421000falseobj000000000000000000422023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.627 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.628 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.628 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.629 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.629 DEBUG:boto:Data: 2023-10-12T00:56:50.629 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.630 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.631 DEBUG:boto:Params: {} 2023-10-12T00:56:50.631 DEBUG:boto:Token: None 2023-10-12T00:56:50.631 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4UyW7whqODgy5S+gkC52f5TOdgY= 2023-10-12T00:56:50.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4UyW7whqODgy5S+gkC52f5TOdgY=', 'Content-Length': '0'} 2023-10-12T00:56:50.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7b46b8d0d68f1f-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.635 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.635 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.636 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.636 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.636 DEBUG:boto:Data: 2023-10-12T00:56:50.637 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.638 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.638 DEBUG:boto:Params: {} 2023-10-12T00:56:50.639 DEBUG:boto:Token: None 2023-10-12T00:56:50.639 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4UyW7whqODgy5S+gkC52f5TOdgY= 2023-10-12T00:56:50.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4UyW7whqODgy5S+gkC52f5TOdgY=', 'Content-Length': '0'} 2023-10-12T00:56:50.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2a5cfaddd506b2-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-12T00:56:50.643 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.643 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.644 DEBUG:boto:Method: GET 2023-10-12T00:56:50.644 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.644 DEBUG:boto:Data: 2023-10-12T00:56:50.645 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.646 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.646 DEBUG:boto:Params: {} 2023-10-12T00:56:50.647 DEBUG:boto:Token: None 2023-10-12T00:56:50.647 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.647 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SZa7QWoGZIFNfMaQxiB9wzF++rU= 2023-10-12T00:56:50.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SZa7QWoGZIFNfMaQxiB9wzF++rU=', 'Content-Length': '0'} 2023-10-12T00:56:50.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67a83b850d3f323-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.690 DEBUG:boto:path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.691 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.691 DEBUG:boto:Method: GET 2023-10-12T00:56:50.692 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.692 DEBUG:boto:Data: 2023-10-12T00:56:50.693 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.693 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.694 DEBUG:boto:Params: {} 2023-10-12T00:56:50.695 DEBUG:boto:Token: None 2023-10-12T00:56:50.695 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000042 2023-10-12T00:56:50.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SZa7QWoGZIFNfMaQxiB9wzF++rU= 2023-10-12T00:56:50.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SZa7QWoGZIFNfMaQxiB9wzF++rU=', 'Content-Length': '0'} 2023-10-12T00:56:50.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cb2d491ae81b23-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.699 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.699 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.699 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.700 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.700 DEBUG:boto:Method: GET 2023-10-12T00:56:50.701 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.701 DEBUG:boto:Data: 2023-10-12T00:56:50.701 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.702 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.703 DEBUG:boto:Params: {} 2023-10-12T00:56:50.703 DEBUG:boto:Token: None 2023-10-12T00:56:50.704 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccaf1dccc6f7712-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.708 DEBUG:boto:b'wafmoa-58obj000000000000000000421000falseobj000000000000000000422023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.708 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.709 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.709 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.710 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.710 DEBUG:boto:Method: GET 2023-10-12T00:56:50.711 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000042 2023-10-12T00:56:50.711 DEBUG:boto:Data: 2023-10-12T00:56:50.711 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.712 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.713 DEBUG:boto:Params: {} 2023-10-12T00:56:50.713 DEBUG:boto:Token: None 2023-10-12T00:56:50.714 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e00dafe1c6ca21-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.719 DEBUG:boto:b'wafmoa-58obj000000000000000000421000falseobj000000000000000000422023-10-12T00:56:34.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-12T00:56:50.720 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.721 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.721 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.722 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.722 DEBUG:boto:Method: GET 2023-10-12T00:56:50.723 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.723 DEBUG:boto:Data: 2023-10-12T00:56:50.723 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.724 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.725 DEBUG:boto:Params: {} 2023-10-12T00:56:50.725 DEBUG:boto:Token: None 2023-10-12T00:56:50.726 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g= 2023-10-12T00:56:50.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g=', 'Content-Length': '0'} 2023-10-12T00:56:50.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d919aee6e5f032f3-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.729 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.730 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.730 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.731 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.731 DEBUG:boto:Method: GET 2023-10-12T00:56:50.732 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.732 DEBUG:boto:Data: 2023-10-12T00:56:50.733 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.733 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.734 DEBUG:boto:Params: {} 2023-10-12T00:56:50.734 DEBUG:boto:Token: None 2023-10-12T00:56:50.735 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g= 2023-10-12T00:56:50.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g=', 'Content-Length': '0'} 2023-10-12T00:56:50.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438cc9bd11ef5384-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.778 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.779 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.779 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.780 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.780 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.781 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.781 DEBUG:boto:Data: 2023-10-12T00:56:50.781 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.782 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.783 DEBUG:boto:Params: {} 2023-10-12T00:56:50.783 DEBUG:boto:Token: None 2023-10-12T00:56:50.784 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nrRnt3xptmK5Rr8sPJEmR2OCSq4= 2023-10-12T00:56:50.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nrRnt3xptmK5Rr8sPJEmR2OCSq4=', 'Content-Length': '0'} 2023-10-12T00:56:50.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98f72848b13b353-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.787 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.788 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.788 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.789 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.789 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.789 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.790 DEBUG:boto:Data: 2023-10-12T00:56:50.790 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.791 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.791 DEBUG:boto:Params: {} 2023-10-12T00:56:50.792 DEBUG:boto:Token: None 2023-10-12T00:56:50.792 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nrRnt3xptmK5Rr8sPJEmR2OCSq4= 2023-10-12T00:56:50.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nrRnt3xptmK5Rr8sPJEmR2OCSq4=', 'Content-Length': '0'} 2023-10-12T00:56:50.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c041a0d9284be60-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-12T00:56:50.796 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.797 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.797 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.798 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.798 DEBUG:boto:Method: GET 2023-10-12T00:56:50.799 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.799 DEBUG:boto:Data: 2023-10-12T00:56:50.799 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.800 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.801 DEBUG:boto:Params: {} 2023-10-12T00:56:50.801 DEBUG:boto:Token: None 2023-10-12T00:56:50.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.802 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g= 2023-10-12T00:56:50.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g=', 'Content-Length': '0'} 2023-10-12T00:56:50.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de20169ca0f0d89-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.850 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.851 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.851 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.852 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.852 DEBUG:boto:Method: GET 2023-10-12T00:56:50.853 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.853 DEBUG:boto:Data: 2023-10-12T00:56:50.854 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.855 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.855 DEBUG:boto:Params: {} 2023-10-12T00:56:50.855 DEBUG:boto:Token: None 2023-10-12T00:56:50.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045?versionId=null 2023-10-12T00:56:50.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g= 2023-10-12T00:56:50.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY2i6DBaEya5F755v72Ly+hUO1g=', 'Content-Length': '0'} 2023-10-12T00:56:50.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d997c0b790c5db-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.860 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.860 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.861 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.861 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.862 DEBUG:boto:Method: GET 2023-10-12T00:56:50.862 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.863 DEBUG:boto:Data: 2023-10-12T00:56:50.863 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.864 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.865 DEBUG:boto:Params: {} 2023-10-12T00:56:50.865 DEBUG:boto:Token: None 2023-10-12T00:56:50.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c45cd72a45103f5-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.872 DEBUG:boto:b'wafmoa-58obj000000000000000000451000falseobj000000000000000000452023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.872 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.873 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.873 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.874 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.874 DEBUG:boto:Method: GET 2023-10-12T00:56:50.875 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.875 DEBUG:boto:Data: 2023-10-12T00:56:50.876 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.876 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.877 DEBUG:boto:Params: {} 2023-10-12T00:56:50.877 DEBUG:boto:Token: None 2023-10-12T00:56:50.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b09cc5d546a4f81-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.883 DEBUG:boto:b'wafmoa-58obj000000000000000000451000falseobj000000000000000000452023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.884 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.884 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.885 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.885 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.886 DEBUG:boto:Data: 2023-10-12T00:56:50.886 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.887 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.887 DEBUG:boto:Params: {} 2023-10-12T00:56:50.888 DEBUG:boto:Token: None 2023-10-12T00:56:50.888 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wxhO6BrqjKaR1FAc/9xV80O+cwA= 2023-10-12T00:56:50.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wxhO6BrqjKaR1FAc/9xV80O+cwA=', 'Content-Length': '0'} 2023-10-12T00:56:50.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70207f750ac6ae4-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.892 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.892 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.893 DEBUG:boto:Method: HEAD 2023-10-12T00:56:50.893 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.894 DEBUG:boto:Data: 2023-10-12T00:56:50.894 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.895 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.895 DEBUG:boto:Params: {} 2023-10-12T00:56:50.896 DEBUG:boto:Token: None 2023-10-12T00:56:50.896 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wxhO6BrqjKaR1FAc/9xV80O+cwA= 2023-10-12T00:56:50.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wxhO6BrqjKaR1FAc/9xV80O+cwA=', 'Content-Length': '0'} 2023-10-12T00:56:50.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074c07a493e38b5bf-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-12T00:56:50.900 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.901 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.901 DEBUG:boto:Method: GET 2023-10-12T00:56:50.902 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.902 DEBUG:boto:Data: 2023-10-12T00:56:50.903 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.903 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.904 DEBUG:boto:Params: {} 2023-10-12T00:56:50.904 DEBUG:boto:Token: None 2023-10-12T00:56:50.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hLNqU4430gzbwQoA7HlBord0pGI= 2023-10-12T00:56:50.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hLNqU4430gzbwQoA7HlBord0pGI=', 'Content-Length': '0'} 2023-10-12T00:56:50.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8ec8fbd022d4b4-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.909 DEBUG:boto:path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.909 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.910 DEBUG:boto:Method: GET 2023-10-12T00:56:50.910 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.910 DEBUG:boto:Data: 2023-10-12T00:56:50.911 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.912 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.912 DEBUG:boto:Params: {} 2023-10-12T00:56:50.913 DEBUG:boto:Token: None 2023-10-12T00:56:50.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000045 2023-10-12T00:56:50.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hLNqU4430gzbwQoA7HlBord0pGI= 2023-10-12T00:56:50.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hLNqU4430gzbwQoA7HlBord0pGI=', 'Content-Length': '0'} 2023-10-12T00:56:50.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194a7d2e6d975688-0065274452-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.958 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.959 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.960 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.960 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.960 DEBUG:boto:Method: GET 2023-10-12T00:56:50.961 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.961 DEBUG:boto:Data: 2023-10-12T00:56:50.962 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.963 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.963 DEBUG:boto:Params: {} 2023-10-12T00:56:50.963 DEBUG:boto:Token: None 2023-10-12T00:56:50.964 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de7c2787d83a9b8-0065274452-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.968 DEBUG:boto:b'wafmoa-58obj000000000000000000451000falseobj000000000000000000452023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.969 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:50.969 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:50.970 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.970 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.971 DEBUG:boto:Method: GET 2023-10-12T00:56:50.971 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000045 2023-10-12T00:56:50.972 DEBUG:boto:Data: 2023-10-12T00:56:50.972 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:50.973 DEBUG:boto:Port: 8000 2023-10-12T00:56:50.973 DEBUG:boto:Params: {} 2023-10-12T00:56:50.974 DEBUG:boto:Token: None 2023-10-12T00:56:50.974 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/ 2023-10-12T00:56:50.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA= 2023-10-12T00:56:50.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rn8K1qUQmFxT426y8v4AYKp0+PA=', 'Content-Length': '0'} 2023-10-12T00:56:50.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006327d4738b3c31d6-0065274452-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:50.980 DEBUG:boto:b'wafmoa-58obj000000000000000000451000falseobj000000000000000000452023-10-12T00:56:36.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:50.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-12T00:56:50.981 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:50.982 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:50.982 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:50.982 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:50.983 DEBUG:boto:Method: GET 2023-10-12T00:56:50.983 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:50.984 DEBUG:boto:Data: 2023-10-12T00:56:50.984 DEBUG:boto:Headers: {} 2023-10-12T00:56:50.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:50.985 DEBUG:boto:Port: 8001 2023-10-12T00:56:50.985 DEBUG:boto:Params: {} 2023-10-12T00:56:50.986 DEBUG:boto:Token: None 2023-10-12T00:56:50.986 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:50 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:50.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nzq/JHFpyzlw7iO4VOJ63YLO+ho= 2023-10-12T00:56:50.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nzq/JHFpyzlw7iO4VOJ63YLO+ho=', 'Content-Length': '0'} 2023-10-12T00:56:50.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e27ad1d850f8a33-0065274452-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.030 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.031 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.031 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.032 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.032 DEBUG:boto:Method: GET 2023-10-12T00:56:51.033 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.033 DEBUG:boto:Data: 2023-10-12T00:56:51.034 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.035 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.035 DEBUG:boto:Params: {} 2023-10-12T00:56:51.035 DEBUG:boto:Token: None 2023-10-12T00:56:51.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs= 2023-10-12T00:56:51.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs=', 'Content-Length': '0'} 2023-10-12T00:56:51.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a18ca2a3aa09964-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.040 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.040 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.041 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.041 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.041 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.042 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.042 DEBUG:boto:Data: 2023-10-12T00:56:51.043 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.044 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.044 DEBUG:boto:Params: {} 2023-10-12T00:56:51.044 DEBUG:boto:Token: None 2023-10-12T00:56:51.045 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FrCkJ1a4SF2JiDMlMXn+K6aZhzI= 2023-10-12T00:56:51.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FrCkJ1a4SF2JiDMlMXn+K6aZhzI=', 'Content-Length': '0'} 2023-10-12T00:56:51.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024230d3ce1e87159-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.048 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.049 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.049 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.050 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.050 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.051 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.051 DEBUG:boto:Data: 2023-10-12T00:56:51.051 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.052 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.053 DEBUG:boto:Params: {} 2023-10-12T00:56:51.053 DEBUG:boto:Token: None 2023-10-12T00:56:51.053 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FrCkJ1a4SF2JiDMlMXn+K6aZhzI= 2023-10-12T00:56:51.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FrCkJ1a4SF2JiDMlMXn+K6aZhzI=', 'Content-Length': '0'} 2023-10-12T00:56:51.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067638764510f0132-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-12T00:56:51.057 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.058 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.058 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.059 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.059 DEBUG:boto:Method: GET 2023-10-12T00:56:51.059 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.060 DEBUG:boto:Data: 2023-10-12T00:56:51.060 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.061 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.062 DEBUG:boto:Params: {} 2023-10-12T00:56:51.062 DEBUG:boto:Token: None 2023-10-12T00:56:51.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs= 2023-10-12T00:56:51.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs=', 'Content-Length': '0'} 2023-10-12T00:56:51.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c08600f83b950d-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.066 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.067 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.067 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.067 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.068 DEBUG:boto:Method: GET 2023-10-12T00:56:51.068 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.069 DEBUG:boto:Data: 2023-10-12T00:56:51.069 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.070 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.071 DEBUG:boto:Params: {} 2023-10-12T00:56:51.073 DEBUG:boto:Token: None 2023-10-12T00:56:51.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046?versionId=null 2023-10-12T00:56:51.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs= 2023-10-12T00:56:51.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BH4iftbuu22jCibSPH9A/Us6sDs=', 'Content-Length': '0'} 2023-10-12T00:56:51.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5729a9858402503-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.077 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.078 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.078 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.079 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.079 DEBUG:boto:Method: GET 2023-10-12T00:56:51.080 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.080 DEBUG:boto:Data: 2023-10-12T00:56:51.081 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.081 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.082 DEBUG:boto:Params: {} 2023-10-12T00:56:51.082 DEBUG:boto:Token: None 2023-10-12T00:56:51.083 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002988c19d7c9c3ded-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.087 DEBUG:boto:b'wafmoa-58obj000000000000000000461000falseobj000000000000000000462023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.088 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.088 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.089 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.089 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.090 DEBUG:boto:Method: GET 2023-10-12T00:56:51.090 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.090 DEBUG:boto:Data: 2023-10-12T00:56:51.091 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.092 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.092 DEBUG:boto:Params: {} 2023-10-12T00:56:51.093 DEBUG:boto:Token: None 2023-10-12T00:56:51.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3106101246358be-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.099 DEBUG:boto:b'wafmoa-58obj000000000000000000461000falseobj000000000000000000462023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.100 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.100 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.100 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.101 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.101 DEBUG:boto:Data: 2023-10-12T00:56:51.102 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.102 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.103 DEBUG:boto:Params: {} 2023-10-12T00:56:51.103 DEBUG:boto:Token: None 2023-10-12T00:56:51.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qA2syGc0snOOrpFXEr6hsYaJk8c= 2023-10-12T00:56:51.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qA2syGc0snOOrpFXEr6hsYaJk8c=', 'Content-Length': '0'} 2023-10-12T00:56:51.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b2f949894f5dc5-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.108 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.108 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.108 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.109 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.109 DEBUG:boto:Data: 2023-10-12T00:56:51.110 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.111 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.111 DEBUG:boto:Params: {} 2023-10-12T00:56:51.111 DEBUG:boto:Token: None 2023-10-12T00:56:51.112 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qA2syGc0snOOrpFXEr6hsYaJk8c= 2023-10-12T00:56:51.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qA2syGc0snOOrpFXEr6hsYaJk8c=', 'Content-Length': '0'} 2023-10-12T00:56:51.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427c3034c4921777-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-12T00:56:51.116 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.116 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.117 DEBUG:boto:Method: GET 2023-10-12T00:56:51.117 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.117 DEBUG:boto:Data: 2023-10-12T00:56:51.118 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.119 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.119 DEBUG:boto:Params: {} 2023-10-12T00:56:51.120 DEBUG:boto:Token: None 2023-10-12T00:56:51.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZHeK34rk6r3ZifwVs2mgcX7x5E= 2023-10-12T00:56:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZHeK34rk6r3ZifwVs2mgcX7x5E=', 'Content-Length': '0'} 2023-10-12T00:56:51.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45c6ba2edce4c4f-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.166 DEBUG:boto:path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.167 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.168 DEBUG:boto:Method: GET 2023-10-12T00:56:51.168 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.168 DEBUG:boto:Data: 2023-10-12T00:56:51.169 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.170 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.170 DEBUG:boto:Params: {} 2023-10-12T00:56:51.171 DEBUG:boto:Token: None 2023-10-12T00:56:51.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000046 2023-10-12T00:56:51.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZHeK34rk6r3ZifwVs2mgcX7x5E= 2023-10-12T00:56:51.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZHeK34rk6r3ZifwVs2mgcX7x5E=', 'Content-Length': '0'} 2023-10-12T00:56:51.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cffc91b25f7844-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.175 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.175 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.176 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.176 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.177 DEBUG:boto:Method: GET 2023-10-12T00:56:51.177 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.178 DEBUG:boto:Data: 2023-10-12T00:56:51.178 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.179 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.179 DEBUG:boto:Params: {} 2023-10-12T00:56:51.180 DEBUG:boto:Token: None 2023-10-12T00:56:51.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f8eaa67a0ad611-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.184 DEBUG:boto:b'wafmoa-58obj000000000000000000461000falseobj000000000000000000462023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.185 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.185 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.186 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.186 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.187 DEBUG:boto:Method: GET 2023-10-12T00:56:51.187 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000046 2023-10-12T00:56:51.188 DEBUG:boto:Data: 2023-10-12T00:56:51.188 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.189 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.189 DEBUG:boto:Params: {} 2023-10-12T00:56:51.190 DEBUG:boto:Token: None 2023-10-12T00:56:51.190 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.191 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063861d8c65e4dccb-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.198 DEBUG:boto:b'wafmoa-58obj000000000000000000461000falseobj000000000000000000462023-10-12T00:56:32.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-12T00:56:51.199 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.199 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.199 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.200 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.200 DEBUG:boto:Method: GET 2023-10-12T00:56:51.201 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.201 DEBUG:boto:Data: 2023-10-12T00:56:51.202 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.203 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.204 DEBUG:boto:Params: {} 2023-10-12T00:56:51.204 DEBUG:boto:Token: None 2023-10-12T00:56:51.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0= 2023-10-12T00:56:51.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0=', 'Content-Length': '0'} 2023-10-12T00:56:51.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f28dec4cbd19255-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.209 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.210 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.210 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.211 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.211 DEBUG:boto:Method: GET 2023-10-12T00:56:51.212 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.212 DEBUG:boto:Data: 2023-10-12T00:56:51.212 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.213 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.214 DEBUG:boto:Params: {} 2023-10-12T00:56:51.214 DEBUG:boto:Token: None 2023-10-12T00:56:51.215 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0= 2023-10-12T00:56:51.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0=', 'Content-Length': '0'} 2023-10-12T00:56:51.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e4075441ac723c-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.219 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.219 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.220 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.220 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.220 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.221 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.221 DEBUG:boto:Data: 2023-10-12T00:56:51.222 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.222 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.223 DEBUG:boto:Params: {} 2023-10-12T00:56:51.223 DEBUG:boto:Token: None 2023-10-12T00:56:51.224 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:20t6ZcUdmoG21SJP7UqDVogk6KE= 2023-10-12T00:56:51.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:20t6ZcUdmoG21SJP7UqDVogk6KE=', 'Content-Length': '0'} 2023-10-12T00:56:51.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001785aeffb531efc7-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.227 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.228 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.228 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.228 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.229 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.229 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.230 DEBUG:boto:Data: 2023-10-12T00:56:51.230 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.231 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.231 DEBUG:boto:Params: {} 2023-10-12T00:56:51.232 DEBUG:boto:Token: None 2023-10-12T00:56:51.232 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:20t6ZcUdmoG21SJP7UqDVogk6KE= 2023-10-12T00:56:51.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:20t6ZcUdmoG21SJP7UqDVogk6KE=', 'Content-Length': '0'} 2023-10-12T00:56:51.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f7d66d1461950b-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-12T00:56:51.236 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.237 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.237 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.237 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.238 DEBUG:boto:Method: GET 2023-10-12T00:56:51.238 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.239 DEBUG:boto:Data: 2023-10-12T00:56:51.239 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.240 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.240 DEBUG:boto:Params: {} 2023-10-12T00:56:51.241 DEBUG:boto:Token: None 2023-10-12T00:56:51.241 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.242 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0= 2023-10-12T00:56:51.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0=', 'Content-Length': '0'} 2023-10-12T00:56:51.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b667124d782bbb30-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.286 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.287 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.287 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.288 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.288 DEBUG:boto:Method: GET 2023-10-12T00:56:51.289 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.289 DEBUG:boto:Data: 2023-10-12T00:56:51.290 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.291 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.291 DEBUG:boto:Params: {} 2023-10-12T00:56:51.292 DEBUG:boto:Token: None 2023-10-12T00:56:51.292 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048?versionId=null 2023-10-12T00:56:51.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0= 2023-10-12T00:56:51.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9aZM5TpilhNgLYMnDJe5JAeXAa0=', 'Content-Length': '0'} 2023-10-12T00:56:51.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fc0ae93c736ff5-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.296 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.296 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.297 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.297 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.298 DEBUG:boto:Method: GET 2023-10-12T00:56:51.298 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.299 DEBUG:boto:Data: 2023-10-12T00:56:51.299 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.300 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.300 DEBUG:boto:Params: {} 2023-10-12T00:56:51.301 DEBUG:boto:Token: None 2023-10-12T00:56:51.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe63617463b74ba-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.305 DEBUG:boto:b'wafmoa-58obj000000000000000000481000falseobj000000000000000000482023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.306 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.306 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.307 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.307 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.308 DEBUG:boto:Method: GET 2023-10-12T00:56:51.308 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.309 DEBUG:boto:Data: 2023-10-12T00:56:51.309 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.310 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.310 DEBUG:boto:Params: {} 2023-10-12T00:56:51.311 DEBUG:boto:Token: None 2023-10-12T00:56:51.311 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b210a8587ef8b6eb-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.317 DEBUG:boto:b'wafmoa-58obj000000000000000000481000falseobj000000000000000000482023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.317 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.318 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.318 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.319 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.319 DEBUG:boto:Data: 2023-10-12T00:56:51.320 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.320 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.321 DEBUG:boto:Params: {} 2023-10-12T00:56:51.321 DEBUG:boto:Token: None 2023-10-12T00:56:51.322 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wmz44BgPLcpb9V5QA7IYnCPwJRw= 2023-10-12T00:56:51.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wmz44BgPLcpb9V5QA7IYnCPwJRw=', 'Content-Length': '0'} 2023-10-12T00:56:51.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d933387dfeba3db3-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.325 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.326 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.326 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.326 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.327 DEBUG:boto:Data: 2023-10-12T00:56:51.327 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.328 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.328 DEBUG:boto:Params: {} 2023-10-12T00:56:51.329 DEBUG:boto:Token: None 2023-10-12T00:56:51.329 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wmz44BgPLcpb9V5QA7IYnCPwJRw= 2023-10-12T00:56:51.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wmz44BgPLcpb9V5QA7IYnCPwJRw=', 'Content-Length': '0'} 2023-10-12T00:56:51.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af6edbb870a04c8-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-12T00:56:51.333 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.333 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.334 DEBUG:boto:Method: GET 2023-10-12T00:56:51.334 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.335 DEBUG:boto:Data: 2023-10-12T00:56:51.335 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.336 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.337 DEBUG:boto:Params: {} 2023-10-12T00:56:51.337 DEBUG:boto:Token: None 2023-10-12T00:56:51.338 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7ztnNgcV/5rQ5HMyDsKaNlgbTVI= 2023-10-12T00:56:51.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7ztnNgcV/5rQ5HMyDsKaNlgbTVI=', 'Content-Length': '0'} 2023-10-12T00:56:51.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcd25b52b9a2fd4-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.341 DEBUG:boto:path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.342 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.342 DEBUG:boto:Method: GET 2023-10-12T00:56:51.343 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.343 DEBUG:boto:Data: 2023-10-12T00:56:51.344 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.344 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.345 DEBUG:boto:Params: {} 2023-10-12T00:56:51.345 DEBUG:boto:Token: None 2023-10-12T00:56:51.346 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000048 2023-10-12T00:56:51.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7ztnNgcV/5rQ5HMyDsKaNlgbTVI= 2023-10-12T00:56:51.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7ztnNgcV/5rQ5HMyDsKaNlgbTVI=', 'Content-Length': '0'} 2023-10-12T00:56:51.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c92985c8680647-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.349 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.350 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.350 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.351 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.351 DEBUG:boto:Method: GET 2023-10-12T00:56:51.352 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.352 DEBUG:boto:Data: 2023-10-12T00:56:51.352 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.353 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.354 DEBUG:boto:Params: {} 2023-10-12T00:56:51.354 DEBUG:boto:Token: None 2023-10-12T00:56:51.355 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd8b65ffbfaa6a5-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.359 DEBUG:boto:b'wafmoa-58obj000000000000000000481000falseobj000000000000000000482023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.360 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.361 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.361 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.361 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.362 DEBUG:boto:Method: GET 2023-10-12T00:56:51.362 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000048 2023-10-12T00:56:51.363 DEBUG:boto:Data: 2023-10-12T00:56:51.363 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.364 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.364 DEBUG:boto:Params: {} 2023-10-12T00:56:51.365 DEBUG:boto:Token: None 2023-10-12T00:56:51.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f441ad7e392e1c-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.371 DEBUG:boto:b'wafmoa-58obj000000000000000000481000falseobj000000000000000000482023-10-12T00:56:37.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-12T00:56:51.372 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.372 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.373 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.373 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.374 DEBUG:boto:Method: GET 2023-10-12T00:56:51.374 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.374 DEBUG:boto:Data: 2023-10-12T00:56:51.375 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.376 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.376 DEBUG:boto:Params: {} 2023-10-12T00:56:51.377 DEBUG:boto:Token: None 2023-10-12T00:56:51.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo= 2023-10-12T00:56:51.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo=', 'Content-Length': '0'} 2023-10-12T00:56:51.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b19eba37916289-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.381 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.382 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.382 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.382 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.383 DEBUG:boto:Method: GET 2023-10-12T00:56:51.383 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.384 DEBUG:boto:Data: 2023-10-12T00:56:51.384 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.385 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.385 DEBUG:boto:Params: {} 2023-10-12T00:56:51.386 DEBUG:boto:Token: None 2023-10-12T00:56:51.386 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo= 2023-10-12T00:56:51.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo=', 'Content-Length': '0'} 2023-10-12T00:56:51.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a8e83893a8c117-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.430 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.431 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.431 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.432 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.432 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.433 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.433 DEBUG:boto:Data: 2023-10-12T00:56:51.433 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.434 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.435 DEBUG:boto:Params: {} 2023-10-12T00:56:51.435 DEBUG:boto:Token: None 2023-10-12T00:56:51.436 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:24LqqBUZIgk2r1qsdMaDSwJbiu8= 2023-10-12T00:56:51.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:24LqqBUZIgk2r1qsdMaDSwJbiu8=', 'Content-Length': '0'} 2023-10-12T00:56:51.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701f60b677911381-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.439 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.439 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.440 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.440 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.441 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.441 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.441 DEBUG:boto:Data: 2023-10-12T00:56:51.442 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.443 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.443 DEBUG:boto:Params: {} 2023-10-12T00:56:51.444 DEBUG:boto:Token: None 2023-10-12T00:56:51.444 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:24LqqBUZIgk2r1qsdMaDSwJbiu8= 2023-10-12T00:56:51.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:24LqqBUZIgk2r1qsdMaDSwJbiu8=', 'Content-Length': '0'} 2023-10-12T00:56:51.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f8af24fd855476-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-12T00:56:51.448 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.448 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.449 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.449 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.449 DEBUG:boto:Method: GET 2023-10-12T00:56:51.450 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.450 DEBUG:boto:Data: 2023-10-12T00:56:51.451 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.452 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.452 DEBUG:boto:Params: {} 2023-10-12T00:56:51.453 DEBUG:boto:Token: None 2023-10-12T00:56:51.453 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.454 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo= 2023-10-12T00:56:51.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo=', 'Content-Length': '0'} 2023-10-12T00:56:51.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024150883b17068d1-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.502 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.503 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.503 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.504 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.504 DEBUG:boto:Method: GET 2023-10-12T00:56:51.505 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.505 DEBUG:boto:Data: 2023-10-12T00:56:51.506 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.507 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.507 DEBUG:boto:Params: {} 2023-10-12T00:56:51.507 DEBUG:boto:Token: None 2023-10-12T00:56:51.508 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049?versionId=null 2023-10-12T00:56:51.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo= 2023-10-12T00:56:51.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bR1yZvzZhTG01sITGcEydXtU0Oo=', 'Content-Length': '0'} 2023-10-12T00:56:51.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d793d796f8120443-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.511 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.512 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.512 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.513 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.513 DEBUG:boto:Method: GET 2023-10-12T00:56:51.513 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.514 DEBUG:boto:Data: 2023-10-12T00:56:51.514 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.515 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.516 DEBUG:boto:Params: {} 2023-10-12T00:56:51.516 DEBUG:boto:Token: None 2023-10-12T00:56:51.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf0c1f906dd4af1-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.521 DEBUG:boto:b'wafmoa-58obj000000000000000000491000falseobj000000000000000000492023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.521 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.522 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.522 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.523 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.523 DEBUG:boto:Method: GET 2023-10-12T00:56:51.523 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.524 DEBUG:boto:Data: 2023-10-12T00:56:51.524 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.525 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.526 DEBUG:boto:Params: {} 2023-10-12T00:56:51.526 DEBUG:boto:Token: None 2023-10-12T00:56:51.527 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a506370be1fee4e-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.532 DEBUG:boto:b'wafmoa-58obj000000000000000000491000falseobj000000000000000000492023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.533 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.533 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.534 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.534 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.534 DEBUG:boto:Data: 2023-10-12T00:56:51.535 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.536 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.536 DEBUG:boto:Params: {} 2023-10-12T00:56:51.536 DEBUG:boto:Token: None 2023-10-12T00:56:51.537 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6UR+qDcgwtXb9jCHEFmUmM00yu0= 2023-10-12T00:56:51.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6UR+qDcgwtXb9jCHEFmUmM00yu0=', 'Content-Length': '0'} 2023-10-12T00:56:51.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3b45040774c7da-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.540 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.541 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.541 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.542 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.542 DEBUG:boto:Data: 2023-10-12T00:56:51.543 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.543 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.544 DEBUG:boto:Params: {} 2023-10-12T00:56:51.544 DEBUG:boto:Token: None 2023-10-12T00:56:51.545 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.545 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6UR+qDcgwtXb9jCHEFmUmM00yu0= 2023-10-12T00:56:51.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6UR+qDcgwtXb9jCHEFmUmM00yu0=', 'Content-Length': '0'} 2023-10-12T00:56:51.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9dbf7f9097d88ee-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-12T00:56:51.549 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.549 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.549 DEBUG:boto:Method: GET 2023-10-12T00:56:51.550 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.550 DEBUG:boto:Data: 2023-10-12T00:56:51.551 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.551 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.552 DEBUG:boto:Params: {} 2023-10-12T00:56:51.552 DEBUG:boto:Token: None 2023-10-12T00:56:51.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xeIyzK7lsXe9wvXfAlRSVgBfZNQ= 2023-10-12T00:56:51.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xeIyzK7lsXe9wvXfAlRSVgBfZNQ=', 'Content-Length': '0'} 2023-10-12T00:56:51.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000223d0c9723178f1-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.556 DEBUG:boto:path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.557 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.557 DEBUG:boto:Method: GET 2023-10-12T00:56:51.557 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.558 DEBUG:boto:Data: 2023-10-12T00:56:51.558 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.559 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.560 DEBUG:boto:Params: {} 2023-10-12T00:56:51.560 DEBUG:boto:Token: None 2023-10-12T00:56:51.561 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000049 2023-10-12T00:56:51.561 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xeIyzK7lsXe9wvXfAlRSVgBfZNQ= 2023-10-12T00:56:51.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xeIyzK7lsXe9wvXfAlRSVgBfZNQ=', 'Content-Length': '0'} 2023-10-12T00:56:51.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30231587de5e599-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.606 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.607 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.607 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.608 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.608 DEBUG:boto:Method: GET 2023-10-12T00:56:51.608 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.609 DEBUG:boto:Data: 2023-10-12T00:56:51.609 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.610 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.611 DEBUG:boto:Params: {} 2023-10-12T00:56:51.611 DEBUG:boto:Token: None 2023-10-12T00:56:51.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecfcb4c5d1915875-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.615 DEBUG:boto:b'wafmoa-58obj000000000000000000491000falseobj000000000000000000492023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.616 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.616 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.617 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.617 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.618 DEBUG:boto:Method: GET 2023-10-12T00:56:51.618 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000049 2023-10-12T00:56:51.619 DEBUG:boto:Data: 2023-10-12T00:56:51.619 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.620 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.620 DEBUG:boto:Params: {} 2023-10-12T00:56:51.621 DEBUG:boto:Token: None 2023-10-12T00:56:51.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44059cae890504b-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.627 DEBUG:boto:b'wafmoa-58obj000000000000000000491000falseobj000000000000000000492023-10-12T00:56:34.268Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-12T00:56:51.628 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.628 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.629 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.629 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.630 DEBUG:boto:Method: GET 2023-10-12T00:56:51.630 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.631 DEBUG:boto:Data: 2023-10-12T00:56:51.631 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.632 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.632 DEBUG:boto:Params: {} 2023-10-12T00:56:51.633 DEBUG:boto:Token: None 2023-10-12T00:56:51.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds= 2023-10-12T00:56:51.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds=', 'Content-Length': '0'} 2023-10-12T00:56:51.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067db80c6d85a7fa6-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.678 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.679 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.679 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.680 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.680 DEBUG:boto:Method: GET 2023-10-12T00:56:51.681 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.681 DEBUG:boto:Data: 2023-10-12T00:56:51.682 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.682 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.683 DEBUG:boto:Params: {} 2023-10-12T00:56:51.683 DEBUG:boto:Token: None 2023-10-12T00:56:51.684 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds= 2023-10-12T00:56:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds=', 'Content-Length': '0'} 2023-10-12T00:56:51.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584f2499069c4067-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.688 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.688 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.688 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.689 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.689 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.690 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.690 DEBUG:boto:Data: 2023-10-12T00:56:51.691 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.692 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.692 DEBUG:boto:Params: {} 2023-10-12T00:56:51.692 DEBUG:boto:Token: None 2023-10-12T00:56:51.693 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDt/m3VDWoHzMbvYUnUcGFChtaw= 2023-10-12T00:56:51.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDt/m3VDWoHzMbvYUnUcGFChtaw=', 'Content-Length': '0'} 2023-10-12T00:56:51.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b0350090125161-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.696 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.697 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.697 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.698 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.698 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.699 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.699 DEBUG:boto:Data: 2023-10-12T00:56:51.700 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.700 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.701 DEBUG:boto:Params: {} 2023-10-12T00:56:51.701 DEBUG:boto:Token: None 2023-10-12T00:56:51.702 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDt/m3VDWoHzMbvYUnUcGFChtaw= 2023-10-12T00:56:51.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDt/m3VDWoHzMbvYUnUcGFChtaw=', 'Content-Length': '0'} 2023-10-12T00:56:51.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be6594b704797262-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-12T00:56:51.706 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.707 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.707 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.708 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.708 DEBUG:boto:Method: GET 2023-10-12T00:56:51.709 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.709 DEBUG:boto:Data: 2023-10-12T00:56:51.710 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.711 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.711 DEBUG:boto:Params: {} 2023-10-12T00:56:51.711 DEBUG:boto:Token: None 2023-10-12T00:56:51.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds= 2023-10-12T00:56:51.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds=', 'Content-Length': '0'} 2023-10-12T00:56:51.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8baff6af6c45031-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.716 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.716 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.717 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.717 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.718 DEBUG:boto:Method: GET 2023-10-12T00:56:51.718 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.719 DEBUG:boto:Data: 2023-10-12T00:56:51.719 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.720 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.720 DEBUG:boto:Params: {} 2023-10-12T00:56:51.721 DEBUG:boto:Token: None 2023-10-12T00:56:51.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050?versionId=null 2023-10-12T00:56:51.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds= 2023-10-12T00:56:51.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dX0tHPXsCoXhipHs02OLwoXpxds=', 'Content-Length': '0'} 2023-10-12T00:56:51.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2beb79d6319c2a0-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.726 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.726 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.727 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.727 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.728 DEBUG:boto:Method: GET 2023-10-12T00:56:51.728 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.729 DEBUG:boto:Data: 2023-10-12T00:56:51.729 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.730 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.730 DEBUG:boto:Params: {} 2023-10-12T00:56:51.731 DEBUG:boto:Token: None 2023-10-12T00:56:51.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a614c336a26f2d-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.736 DEBUG:boto:b'wafmoa-58obj000000000000000000501000falseobj000000000000000000502023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.736 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.737 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.737 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.738 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.738 DEBUG:boto:Method: GET 2023-10-12T00:56:51.739 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.739 DEBUG:boto:Data: 2023-10-12T00:56:51.740 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.740 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.741 DEBUG:boto:Params: {} 2023-10-12T00:56:51.741 DEBUG:boto:Token: None 2023-10-12T00:56:51.742 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.743 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2ea6c3aed28e2e-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.747 DEBUG:boto:b'wafmoa-58obj000000000000000000501000falseobj000000000000000000502023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.748 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.748 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.749 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.749 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.749 DEBUG:boto:Data: 2023-10-12T00:56:51.750 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.751 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.751 DEBUG:boto:Params: {} 2023-10-12T00:56:51.752 DEBUG:boto:Token: None 2023-10-12T00:56:51.752 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kbItYEkz/uyN3MwjA5a8gwKpg80= 2023-10-12T00:56:51.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kbItYEkz/uyN3MwjA5a8gwKpg80=', 'Content-Length': '0'} 2023-10-12T00:56:51.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fdd3c71cbe32778-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.755 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.756 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.756 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.756 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.757 DEBUG:boto:Data: 2023-10-12T00:56:51.757 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.758 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.758 DEBUG:boto:Params: {} 2023-10-12T00:56:51.759 DEBUG:boto:Token: None 2023-10-12T00:56:51.759 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kbItYEkz/uyN3MwjA5a8gwKpg80= 2023-10-12T00:56:51.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kbItYEkz/uyN3MwjA5a8gwKpg80=', 'Content-Length': '0'} 2023-10-12T00:56:51.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd7d36e25961259-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-12T00:56:51.763 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.763 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.764 DEBUG:boto:Method: GET 2023-10-12T00:56:51.764 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.765 DEBUG:boto:Data: 2023-10-12T00:56:51.765 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.766 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.766 DEBUG:boto:Params: {} 2023-10-12T00:56:51.767 DEBUG:boto:Token: None 2023-10-12T00:56:51.767 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FVWYbGjukx7obyK2GKso8b2fk1A= 2023-10-12T00:56:51.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FVWYbGjukx7obyK2GKso8b2fk1A=', 'Content-Length': '0'} 2023-10-12T00:56:51.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7df087567cfeff8-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.810 DEBUG:boto:path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.811 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.811 DEBUG:boto:Method: GET 2023-10-12T00:56:51.812 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.812 DEBUG:boto:Data: 2023-10-12T00:56:51.812 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.813 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.814 DEBUG:boto:Params: {} 2023-10-12T00:56:51.814 DEBUG:boto:Token: None 2023-10-12T00:56:51.815 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000050 2023-10-12T00:56:51.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FVWYbGjukx7obyK2GKso8b2fk1A= 2023-10-12T00:56:51.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FVWYbGjukx7obyK2GKso8b2fk1A=', 'Content-Length': '0'} 2023-10-12T00:56:51.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00c84926b785dae-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.818 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.819 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.819 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.820 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.820 DEBUG:boto:Method: GET 2023-10-12T00:56:51.821 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.821 DEBUG:boto:Data: 2023-10-12T00:56:51.821 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.822 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.823 DEBUG:boto:Params: {} 2023-10-12T00:56:51.823 DEBUG:boto:Token: None 2023-10-12T00:56:51.824 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073737e6d791652d1-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.828 DEBUG:boto:b'wafmoa-58obj000000000000000000501000falseobj000000000000000000502023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.828 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.829 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.829 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.830 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.830 DEBUG:boto:Method: GET 2023-10-12T00:56:51.831 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000050 2023-10-12T00:56:51.831 DEBUG:boto:Data: 2023-10-12T00:56:51.832 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.832 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.833 DEBUG:boto:Params: {} 2023-10-12T00:56:51.833 DEBUG:boto:Token: None 2023-10-12T00:56:51.834 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c003a474de387c9e-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.840 DEBUG:boto:b'wafmoa-58obj000000000000000000501000falseobj000000000000000000502023-10-12T00:56:35.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-12T00:56:51.841 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.841 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.842 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.842 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.843 DEBUG:boto:Method: GET 2023-10-12T00:56:51.843 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.843 DEBUG:boto:Data: 2023-10-12T00:56:51.844 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.845 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.845 DEBUG:boto:Params: {} 2023-10-12T00:56:51.846 DEBUG:boto:Token: None 2023-10-12T00:56:51.846 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag= 2023-10-12T00:56:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag=', 'Content-Length': '0'} 2023-10-12T00:56:51.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91606dcc5e4e484-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.850 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.850 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.851 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.851 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.852 DEBUG:boto:Method: GET 2023-10-12T00:56:51.852 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.853 DEBUG:boto:Data: 2023-10-12T00:56:51.853 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.854 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.854 DEBUG:boto:Params: {} 2023-10-12T00:56:51.855 DEBUG:boto:Token: None 2023-10-12T00:56:51.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag= 2023-10-12T00:56:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag=', 'Content-Length': '0'} 2023-10-12T00:56:51.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515ccdaf82159d48-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.859 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.859 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.860 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.860 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.861 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.861 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.861 DEBUG:boto:Data: 2023-10-12T00:56:51.862 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.863 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.863 DEBUG:boto:Params: {} 2023-10-12T00:56:51.863 DEBUG:boto:Token: None 2023-10-12T00:56:51.864 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pr81MqSzDGZ7gtmtKsDTyCZ7i+o= 2023-10-12T00:56:51.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pr81MqSzDGZ7gtmtKsDTyCZ7i+o=', 'Content-Length': '0'} 2023-10-12T00:56:51.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8b6f96ae6af4d1-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.868 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.868 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.869 DEBUG:boto:Method: HEAD 2023-10-12T00:56:51.870 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.870 DEBUG:boto:Data: 2023-10-12T00:56:51.870 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.871 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.872 DEBUG:boto:Params: {} 2023-10-12T00:56:51.872 DEBUG:boto:Token: None 2023-10-12T00:56:51.873 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pr81MqSzDGZ7gtmtKsDTyCZ7i+o= 2023-10-12T00:56:51.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pr81MqSzDGZ7gtmtKsDTyCZ7i+o=', 'Content-Length': '0'} 2023-10-12T00:56:51.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928c12ee0a2fa4a8-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-12T00:56:51.877 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.877 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.878 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.878 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.878 DEBUG:boto:Method: GET 2023-10-12T00:56:51.879 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.879 DEBUG:boto:Data: 2023-10-12T00:56:51.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.881 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.881 DEBUG:boto:Params: {} 2023-10-12T00:56:51.882 DEBUG:boto:Token: None 2023-10-12T00:56:51.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag= 2023-10-12T00:56:51.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag=', 'Content-Length': '0'} 2023-10-12T00:56:51.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14f97a85b9f8136-0065274453-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.930 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.931 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:51.931 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.932 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.932 DEBUG:boto:Method: GET 2023-10-12T00:56:51.933 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.933 DEBUG:boto:Data: 2023-10-12T00:56:51.934 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.935 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.935 DEBUG:boto:Params: {} 2023-10-12T00:56:51.935 DEBUG:boto:Token: None 2023-10-12T00:56:51.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/obj00000000000000000053?versionId=null 2023-10-12T00:56:51.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag= 2023-10-12T00:56:51.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nfm37g5nEfQk9Vm4eo+TDCU4Kag=', 'Content-Length': '0'} 2023-10-12T00:56:51.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001feb904fc4f10984-0065274453-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.940 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.940 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.941 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.941 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.942 DEBUG:boto:Method: GET 2023-10-12T00:56:51.942 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.943 DEBUG:boto:Data: 2023-10-12T00:56:51.943 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:51.944 DEBUG:boto:Port: 8001 2023-10-12T00:56:51.944 DEBUG:boto:Params: {} 2023-10-12T00:56:51.945 DEBUG:boto:Token: None 2023-10-12T00:56:51.945 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e47e9ea2d97368-0065274453-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:51.949 DEBUG:boto:b'wafmoa-58obj000000000000000000531000falseobj000000000000000000532023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:51.950 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:51.951 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:51.951 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.951 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.952 DEBUG:boto:Method: GET 2023-10-12T00:56:51.952 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:51.953 DEBUG:boto:Data: 2023-10-12T00:56:51.953 DEBUG:boto:Headers: {} 2023-10-12T00:56:51.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:51.954 DEBUG:boto:Port: 8000 2023-10-12T00:56:51.954 DEBUG:boto:Params: {} 2023-10-12T00:56:51.955 DEBUG:boto:Token: None 2023-10-12T00:56:51.955 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:51 GMT /wafmoa-58/ 2023-10-12T00:56:51.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU= 2023-10-12T00:56:51.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:51 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycZNn77/CzifHSsTxpFaWuJDAwU=', 'Content-Length': '0'} 2023-10-12T00:56:51.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000657d4b04739cad25-0065274453-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.002 DEBUG:boto:b'wafmoa-58obj000000000000000000531000falseobj000000000000000000532023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.003 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.004 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.004 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.004 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.005 DEBUG:boto:Data: 2023-10-12T00:56:52.005 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.006 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.007 DEBUG:boto:Params: {} 2023-10-12T00:56:52.007 DEBUG:boto:Token: None 2023-10-12T00:56:52.008 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UO3DQQ9xnfqlaoaG0hahJzBxAMk= 2023-10-12T00:56:52.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UO3DQQ9xnfqlaoaG0hahJzBxAMk=', 'Content-Length': '0'} 2023-10-12T00:56:52.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f294e2e04f481af-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.011 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.012 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.012 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.013 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.013 DEBUG:boto:Data: 2023-10-12T00:56:52.013 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.014 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.015 DEBUG:boto:Params: {} 2023-10-12T00:56:52.015 DEBUG:boto:Token: None 2023-10-12T00:56:52.016 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UO3DQQ9xnfqlaoaG0hahJzBxAMk= 2023-10-12T00:56:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UO3DQQ9xnfqlaoaG0hahJzBxAMk=', 'Content-Length': '0'} 2023-10-12T00:56:52.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182e56cab3cf5222-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-12T00:56:52.020 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.020 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.021 DEBUG:boto:Method: GET 2023-10-12T00:56:52.021 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.022 DEBUG:boto:Data: 2023-10-12T00:56:52.022 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.023 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.024 DEBUG:boto:Params: {} 2023-10-12T00:56:52.024 DEBUG:boto:Token: None 2023-10-12T00:56:52.025 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KzOuDL+7Qm3DeaNAb8hM+1NJxAM= 2023-10-12T00:56:52.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KzOuDL+7Qm3DeaNAb8hM+1NJxAM=', 'Content-Length': '0'} 2023-10-12T00:56:52.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f1c906f1c60c8d-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.028 DEBUG:boto:path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.029 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.029 DEBUG:boto:Method: GET 2023-10-12T00:56:52.030 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.030 DEBUG:boto:Data: 2023-10-12T00:56:52.031 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.032 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.032 DEBUG:boto:Params: {} 2023-10-12T00:56:52.033 DEBUG:boto:Token: None 2023-10-12T00:56:52.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000053 2023-10-12T00:56:52.034 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KzOuDL+7Qm3DeaNAb8hM+1NJxAM= 2023-10-12T00:56:52.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KzOuDL+7Qm3DeaNAb8hM+1NJxAM=', 'Content-Length': '0'} 2023-10-12T00:56:52.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea3292139361733-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.037 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.038 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.038 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.039 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.039 DEBUG:boto:Method: GET 2023-10-12T00:56:52.039 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.040 DEBUG:boto:Data: 2023-10-12T00:56:52.040 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.041 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.042 DEBUG:boto:Params: {} 2023-10-12T00:56:52.042 DEBUG:boto:Token: None 2023-10-12T00:56:52.043 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.043 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6d61a12e944f7b-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.047 DEBUG:boto:b'wafmoa-58obj000000000000000000531000falseobj000000000000000000532023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.048 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.048 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.048 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.049 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.049 DEBUG:boto:Method: GET 2023-10-12T00:56:52.050 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000053 2023-10-12T00:56:52.050 DEBUG:boto:Data: 2023-10-12T00:56:52.051 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.051 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.052 DEBUG:boto:Params: {} 2023-10-12T00:56:52.052 DEBUG:boto:Token: None 2023-10-12T00:56:52.053 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.053 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004ef4b8760d230bb-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.058 DEBUG:boto:b'wafmoa-58obj000000000000000000531000falseobj000000000000000000532023-10-12T00:56:36.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-12T00:56:52.059 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.060 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.060 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.061 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.061 DEBUG:boto:Method: GET 2023-10-12T00:56:52.061 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.062 DEBUG:boto:Data: 2023-10-12T00:56:52.062 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.063 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.064 DEBUG:boto:Params: {} 2023-10-12T00:56:52.064 DEBUG:boto:Token: None 2023-10-12T00:56:52.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k= 2023-10-12T00:56:52.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k=', 'Content-Length': '0'} 2023-10-12T00:56:52.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000673fdc5d063afc08-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.110 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.150 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.151 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.151 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.152 DEBUG:boto:Method: GET 2023-10-12T00:56:52.152 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.153 DEBUG:boto:Data: 2023-10-12T00:56:52.153 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.154 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.155 DEBUG:boto:Params: {} 2023-10-12T00:56:52.155 DEBUG:boto:Token: None 2023-10-12T00:56:52.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k= 2023-10-12T00:56:52.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k=', 'Content-Length': '0'} 2023-10-12T00:56:52.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eb4ab9a540a0b4e-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.160 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.160 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.161 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.161 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.161 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.162 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.162 DEBUG:boto:Data: 2023-10-12T00:56:52.163 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.163 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.164 DEBUG:boto:Params: {} 2023-10-12T00:56:52.164 DEBUG:boto:Token: None 2023-10-12T00:56:52.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tQuYRmBx+s3oIYt35l4sgH4I6z0= 2023-10-12T00:56:52.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tQuYRmBx+s3oIYt35l4sgH4I6z0=', 'Content-Length': '0'} 2023-10-12T00:56:52.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2782253906378a-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.168 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.169 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.169 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.170 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.170 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.170 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.171 DEBUG:boto:Data: 2023-10-12T00:56:52.171 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.172 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.173 DEBUG:boto:Params: {} 2023-10-12T00:56:52.173 DEBUG:boto:Token: None 2023-10-12T00:56:52.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tQuYRmBx+s3oIYt35l4sgH4I6z0= 2023-10-12T00:56:52.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tQuYRmBx+s3oIYt35l4sgH4I6z0=', 'Content-Length': '0'} 2023-10-12T00:56:52.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068533987be80aaeb-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-12T00:56:52.177 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.178 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.178 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.179 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.179 DEBUG:boto:Method: GET 2023-10-12T00:56:52.180 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.180 DEBUG:boto:Data: 2023-10-12T00:56:52.180 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.181 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.182 DEBUG:boto:Params: {} 2023-10-12T00:56:52.182 DEBUG:boto:Token: None 2023-10-12T00:56:52.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k= 2023-10-12T00:56:52.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k=', 'Content-Length': '0'} 2023-10-12T00:56:52.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf832b731b29cd95-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.187 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.187 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.187 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.188 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.188 DEBUG:boto:Method: GET 2023-10-12T00:56:52.189 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.189 DEBUG:boto:Data: 2023-10-12T00:56:52.190 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.190 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.191 DEBUG:boto:Params: {} 2023-10-12T00:56:52.191 DEBUG:boto:Token: None 2023-10-12T00:56:52.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055?versionId=null 2023-10-12T00:56:52.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k= 2023-10-12T00:56:52.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bX8oGC3RwpAInTTfgvlU7XNnc3k=', 'Content-Length': '0'} 2023-10-12T00:56:52.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b1f59f122175440-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.238 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.239 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.239 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.240 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.240 DEBUG:boto:Method: GET 2023-10-12T00:56:52.241 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.241 DEBUG:boto:Data: 2023-10-12T00:56:52.242 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.242 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.243 DEBUG:boto:Params: {} 2023-10-12T00:56:52.243 DEBUG:boto:Token: None 2023-10-12T00:56:52.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ad850b5b3d63cdc-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.248 DEBUG:boto:b'wafmoa-58obj000000000000000000551000falseobj000000000000000000552023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.249 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.249 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.250 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.250 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.251 DEBUG:boto:Method: GET 2023-10-12T00:56:52.251 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.252 DEBUG:boto:Data: 2023-10-12T00:56:52.252 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.253 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.253 DEBUG:boto:Params: {} 2023-10-12T00:56:52.254 DEBUG:boto:Token: None 2023-10-12T00:56:52.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000085d8f9b4b881e08-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.260 DEBUG:boto:b'wafmoa-58obj000000000000000000551000falseobj000000000000000000552023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.261 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.261 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.262 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.262 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.263 DEBUG:boto:Data: 2023-10-12T00:56:52.263 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.264 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.264 DEBUG:boto:Params: {} 2023-10-12T00:56:52.265 DEBUG:boto:Token: None 2023-10-12T00:56:52.265 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ZRz4yhh45sH71T8uYoPzhqUEeo= 2023-10-12T00:56:52.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ZRz4yhh45sH71T8uYoPzhqUEeo=', 'Content-Length': '0'} 2023-10-12T00:56:52.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c889e3d30774322e-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.269 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.270 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.270 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.271 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.271 DEBUG:boto:Data: 2023-10-12T00:56:52.271 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.272 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.273 DEBUG:boto:Params: {} 2023-10-12T00:56:52.273 DEBUG:boto:Token: None 2023-10-12T00:56:52.273 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.274 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ZRz4yhh45sH71T8uYoPzhqUEeo= 2023-10-12T00:56:52.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ZRz4yhh45sH71T8uYoPzhqUEeo=', 'Content-Length': '0'} 2023-10-12T00:56:52.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e96339ad40da1c-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-12T00:56:52.278 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.278 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.278 DEBUG:boto:Method: GET 2023-10-12T00:56:52.279 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.279 DEBUG:boto:Data: 2023-10-12T00:56:52.280 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.281 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.281 DEBUG:boto:Params: {} 2023-10-12T00:56:52.282 DEBUG:boto:Token: None 2023-10-12T00:56:52.282 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dpFQDoAFGE1nu54BZ5Sg4oikgPE= 2023-10-12T00:56:52.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dpFQDoAFGE1nu54BZ5Sg4oikgPE=', 'Content-Length': '0'} 2023-10-12T00:56:52.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd75c833898108e9-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.286 DEBUG:boto:path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.286 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.287 DEBUG:boto:Method: GET 2023-10-12T00:56:52.287 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.288 DEBUG:boto:Data: 2023-10-12T00:56:52.288 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.289 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.290 DEBUG:boto:Params: {} 2023-10-12T00:56:52.290 DEBUG:boto:Token: None 2023-10-12T00:56:52.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000055 2023-10-12T00:56:52.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dpFQDoAFGE1nu54BZ5Sg4oikgPE= 2023-10-12T00:56:52.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dpFQDoAFGE1nu54BZ5Sg4oikgPE=', 'Content-Length': '0'} 2023-10-12T00:56:52.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870bf653a88444b3-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.295 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.295 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.295 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.296 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.296 DEBUG:boto:Method: GET 2023-10-12T00:56:52.297 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.297 DEBUG:boto:Data: 2023-10-12T00:56:52.298 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.299 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.299 DEBUG:boto:Params: {} 2023-10-12T00:56:52.299 DEBUG:boto:Token: None 2023-10-12T00:56:52.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000baf5fca1eb65c06-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.304 DEBUG:boto:b'wafmoa-58obj000000000000000000551000falseobj000000000000000000552023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.305 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.305 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.306 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.306 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.307 DEBUG:boto:Method: GET 2023-10-12T00:56:52.307 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000055 2023-10-12T00:56:52.307 DEBUG:boto:Data: 2023-10-12T00:56:52.308 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.309 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.309 DEBUG:boto:Params: {} 2023-10-12T00:56:52.310 DEBUG:boto:Token: None 2023-10-12T00:56:52.310 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4becd1a021ef1d0-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.316 DEBUG:boto:b'wafmoa-58obj000000000000000000551000falseobj000000000000000000552023-10-12T00:56:32.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-12T00:56:52.317 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.318 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.318 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.319 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.319 DEBUG:boto:Method: GET 2023-10-12T00:56:52.320 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.320 DEBUG:boto:Data: 2023-10-12T00:56:52.321 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.321 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.322 DEBUG:boto:Params: {} 2023-10-12T00:56:52.322 DEBUG:boto:Token: None 2023-10-12T00:56:52.323 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.323 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o= 2023-10-12T00:56:52.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o=', 'Content-Length': '0'} 2023-10-12T00:56:52.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc55f0626b0471d-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.327 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.327 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.328 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.328 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.328 DEBUG:boto:Method: GET 2023-10-12T00:56:52.329 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.329 DEBUG:boto:Data: 2023-10-12T00:56:52.330 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.331 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.331 DEBUG:boto:Params: {} 2023-10-12T00:56:52.331 DEBUG:boto:Token: None 2023-10-12T00:56:52.332 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o= 2023-10-12T00:56:52.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o=', 'Content-Length': '0'} 2023-10-12T00:56:52.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031a8c3d5b6ff9272-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.382 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.383 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.383 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.384 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.384 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.384 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.385 DEBUG:boto:Data: 2023-10-12T00:56:52.385 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.386 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.386 DEBUG:boto:Params: {} 2023-10-12T00:56:52.387 DEBUG:boto:Token: None 2023-10-12T00:56:52.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yVKY+ZVPQhk7Wfb7KnBE9FaU7s= 2023-10-12T00:56:52.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yVKY+ZVPQhk7Wfb7KnBE9FaU7s=', 'Content-Length': '0'} 2023-10-12T00:56:52.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920aa9442bb51aff-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.391 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.391 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.391 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.392 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.392 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.393 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.393 DEBUG:boto:Data: 2023-10-12T00:56:52.393 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.394 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.395 DEBUG:boto:Params: {} 2023-10-12T00:56:52.395 DEBUG:boto:Token: None 2023-10-12T00:56:52.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3yVKY+ZVPQhk7Wfb7KnBE9FaU7s= 2023-10-12T00:56:52.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3yVKY+ZVPQhk7Wfb7KnBE9FaU7s=', 'Content-Length': '0'} 2023-10-12T00:56:52.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae7b662714f43c6f-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-12T00:56:52.400 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.400 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.401 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.401 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.401 DEBUG:boto:Method: GET 2023-10-12T00:56:52.402 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.402 DEBUG:boto:Data: 2023-10-12T00:56:52.403 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.403 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.404 DEBUG:boto:Params: {} 2023-10-12T00:56:52.404 DEBUG:boto:Token: None 2023-10-12T00:56:52.405 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o= 2023-10-12T00:56:52.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o=', 'Content-Length': '0'} 2023-10-12T00:56:52.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492efb58b23442be-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.408 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.409 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.409 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.409 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.410 DEBUG:boto:Method: GET 2023-10-12T00:56:52.410 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.411 DEBUG:boto:Data: 2023-10-12T00:56:52.411 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.412 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.412 DEBUG:boto:Params: {} 2023-10-12T00:56:52.413 DEBUG:boto:Token: None 2023-10-12T00:56:52.413 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057?versionId=null 2023-10-12T00:56:52.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o= 2023-10-12T00:56:52.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDRYiIyz73BV8KJ2VxB2wV4RN3o=', 'Content-Length': '0'} 2023-10-12T00:56:52.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af842abf5e171b14-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.417 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.418 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.418 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.418 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.419 DEBUG:boto:Method: GET 2023-10-12T00:56:52.419 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.420 DEBUG:boto:Data: 2023-10-12T00:56:52.420 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.421 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.421 DEBUG:boto:Params: {} 2023-10-12T00:56:52.422 DEBUG:boto:Token: None 2023-10-12T00:56:52.422 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e5d634691040e0-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.426 DEBUG:boto:b'wafmoa-58obj000000000000000000571000falseobj000000000000000000572023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.427 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.427 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.428 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.428 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.428 DEBUG:boto:Method: GET 2023-10-12T00:56:52.429 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.429 DEBUG:boto:Data: 2023-10-12T00:56:52.430 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.431 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.431 DEBUG:boto:Params: {} 2023-10-12T00:56:52.431 DEBUG:boto:Token: None 2023-10-12T00:56:52.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003310914cbd251eef-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.437 DEBUG:boto:b'wafmoa-58obj000000000000000000571000falseobj000000000000000000572023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.438 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.439 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.439 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.439 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.440 DEBUG:boto:Data: 2023-10-12T00:56:52.440 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.441 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.441 DEBUG:boto:Params: {} 2023-10-12T00:56:52.442 DEBUG:boto:Token: None 2023-10-12T00:56:52.442 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+KlHndJhqBY2jIizJvNQnGQRfQ= 2023-10-12T00:56:52.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+KlHndJhqBY2jIizJvNQnGQRfQ=', 'Content-Length': '0'} 2023-10-12T00:56:52.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224b2a2b604ed7e6-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.446 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.446 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.447 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.447 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.448 DEBUG:boto:Data: 2023-10-12T00:56:52.448 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.449 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.449 DEBUG:boto:Params: {} 2023-10-12T00:56:52.450 DEBUG:boto:Token: None 2023-10-12T00:56:52.450 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+KlHndJhqBY2jIizJvNQnGQRfQ= 2023-10-12T00:56:52.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+KlHndJhqBY2jIizJvNQnGQRfQ=', 'Content-Length': '0'} 2023-10-12T00:56:52.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc043d0b04a0d3b-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-12T00:56:52.454 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.455 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.455 DEBUG:boto:Method: GET 2023-10-12T00:56:52.456 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.456 DEBUG:boto:Data: 2023-10-12T00:56:52.456 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.457 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.458 DEBUG:boto:Params: {} 2023-10-12T00:56:52.458 DEBUG:boto:Token: None 2023-10-12T00:56:52.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJi5AtkXXWXMUWBAUagWL7dBSa8= 2023-10-12T00:56:52.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJi5AtkXXWXMUWBAUagWL7dBSa8=', 'Content-Length': '0'} 2023-10-12T00:56:52.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb41e7cddaaf3f68-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.462 DEBUG:boto:path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.463 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.463 DEBUG:boto:Method: GET 2023-10-12T00:56:52.464 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.464 DEBUG:boto:Data: 2023-10-12T00:56:52.465 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.465 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.466 DEBUG:boto:Params: {} 2023-10-12T00:56:52.466 DEBUG:boto:Token: None 2023-10-12T00:56:52.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000057 2023-10-12T00:56:52.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJi5AtkXXWXMUWBAUagWL7dBSa8= 2023-10-12T00:56:52.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJi5AtkXXWXMUWBAUagWL7dBSa8=', 'Content-Length': '0'} 2023-10-12T00:56:52.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5691555164d606d-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.510 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.511 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.511 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.512 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.512 DEBUG:boto:Method: GET 2023-10-12T00:56:52.513 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.513 DEBUG:boto:Data: 2023-10-12T00:56:52.513 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.514 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.515 DEBUG:boto:Params: {} 2023-10-12T00:56:52.517 DEBUG:boto:Token: None 2023-10-12T00:56:52.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871b8c08a6a12a15-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.522 DEBUG:boto:b'wafmoa-58obj000000000000000000571000falseobj000000000000000000572023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.523 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.523 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.524 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.524 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.525 DEBUG:boto:Method: GET 2023-10-12T00:56:52.525 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000057 2023-10-12T00:56:52.525 DEBUG:boto:Data: 2023-10-12T00:56:52.526 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.527 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.527 DEBUG:boto:Params: {} 2023-10-12T00:56:52.528 DEBUG:boto:Token: None 2023-10-12T00:56:52.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8af46fab400123f-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.534 DEBUG:boto:b'wafmoa-58obj000000000000000000571000falseobj000000000000000000572023-10-12T00:56:36.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-12T00:56:52.535 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.535 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.536 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.536 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.537 DEBUG:boto:Method: GET 2023-10-12T00:56:52.537 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.537 DEBUG:boto:Data: 2023-10-12T00:56:52.538 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.539 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.539 DEBUG:boto:Params: {} 2023-10-12T00:56:52.540 DEBUG:boto:Token: None 2023-10-12T00:56:52.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I= 2023-10-12T00:56:52.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I=', 'Content-Length': '0'} 2023-10-12T00:56:52.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009760d8e250fcffca-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.544 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.544 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.545 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.545 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.546 DEBUG:boto:Method: GET 2023-10-12T00:56:52.546 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.547 DEBUG:boto:Data: 2023-10-12T00:56:52.547 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.548 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.548 DEBUG:boto:Params: {} 2023-10-12T00:56:52.549 DEBUG:boto:Token: None 2023-10-12T00:56:52.549 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I= 2023-10-12T00:56:52.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I=', 'Content-Length': '0'} 2023-10-12T00:56:52.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000209e55aa739b65e4-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.554 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.554 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.555 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.555 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.556 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.556 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.556 DEBUG:boto:Data: 2023-10-12T00:56:52.557 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.558 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.558 DEBUG:boto:Params: {} 2023-10-12T00:56:52.559 DEBUG:boto:Token: None 2023-10-12T00:56:52.559 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7Qe9UqvhaF/WDwWPXU4ulUoNLQ= 2023-10-12T00:56:52.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7Qe9UqvhaF/WDwWPXU4ulUoNLQ=', 'Content-Length': '0'} 2023-10-12T00:56:52.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075dc86f86ad1ee78-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.563 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.563 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.564 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.564 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.564 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.565 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.565 DEBUG:boto:Data: 2023-10-12T00:56:52.566 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.567 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.567 DEBUG:boto:Params: {} 2023-10-12T00:56:52.568 DEBUG:boto:Token: None 2023-10-12T00:56:52.568 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.569 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7Qe9UqvhaF/WDwWPXU4ulUoNLQ= 2023-10-12T00:56:52.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7Qe9UqvhaF/WDwWPXU4ulUoNLQ=', 'Content-Length': '0'} 2023-10-12T00:56:52.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9caecae0bc8400f-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-12T00:56:52.572 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.572 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.573 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.573 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.574 DEBUG:boto:Method: GET 2023-10-12T00:56:52.574 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.575 DEBUG:boto:Data: 2023-10-12T00:56:52.575 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.576 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.576 DEBUG:boto:Params: {} 2023-10-12T00:56:52.577 DEBUG:boto:Token: None 2023-10-12T00:56:52.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I= 2023-10-12T00:56:52.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I=', 'Content-Length': '0'} 2023-10-12T00:56:52.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03171313b398d14-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.622 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.623 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.623 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.624 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.624 DEBUG:boto:Method: GET 2023-10-12T00:56:52.625 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.625 DEBUG:boto:Data: 2023-10-12T00:56:52.625 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.626 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.627 DEBUG:boto:Params: {} 2023-10-12T00:56:52.627 DEBUG:boto:Token: None 2023-10-12T00:56:52.628 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058?versionId=null 2023-10-12T00:56:52.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I= 2023-10-12T00:56:52.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUlZQ0o8Aghut3nMPmVvaOst1/I=', 'Content-Length': '0'} 2023-10-12T00:56:52.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463fc5c370a383aa-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.632 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.632 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.633 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.633 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.633 DEBUG:boto:Method: GET 2023-10-12T00:56:52.634 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.634 DEBUG:boto:Data: 2023-10-12T00:56:52.635 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.636 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.636 DEBUG:boto:Params: {} 2023-10-12T00:56:52.637 DEBUG:boto:Token: None 2023-10-12T00:56:52.637 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004edf6d2c21746a37-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.641 DEBUG:boto:b'wafmoa-58obj000000000000000000581000falseobj000000000000000000582023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.642 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.642 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.643 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.643 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.644 DEBUG:boto:Method: GET 2023-10-12T00:56:52.644 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.644 DEBUG:boto:Data: 2023-10-12T00:56:52.645 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.646 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.646 DEBUG:boto:Params: {} 2023-10-12T00:56:52.647 DEBUG:boto:Token: None 2023-10-12T00:56:52.647 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c776c61b6fd944e-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.694 DEBUG:boto:b'wafmoa-58obj000000000000000000581000falseobj000000000000000000582023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.695 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.696 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.696 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.697 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.697 DEBUG:boto:Data: 2023-10-12T00:56:52.697 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.698 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.699 DEBUG:boto:Params: {} 2023-10-12T00:56:52.699 DEBUG:boto:Token: None 2023-10-12T00:56:52.699 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+unekJ6nBGiYQMg/0aj+GsYbwn0= 2023-10-12T00:56:52.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+unekJ6nBGiYQMg/0aj+GsYbwn0=', 'Content-Length': '0'} 2023-10-12T00:56:52.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a141cfda4a65559c-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.703 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.703 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.704 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.704 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.704 DEBUG:boto:Data: 2023-10-12T00:56:52.705 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.705 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.706 DEBUG:boto:Params: {} 2023-10-12T00:56:52.706 DEBUG:boto:Token: None 2023-10-12T00:56:52.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+unekJ6nBGiYQMg/0aj+GsYbwn0= 2023-10-12T00:56:52.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+unekJ6nBGiYQMg/0aj+GsYbwn0=', 'Content-Length': '0'} 2023-10-12T00:56:52.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db265ec2fb989ea5-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-12T00:56:52.711 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.711 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.711 DEBUG:boto:Method: GET 2023-10-12T00:56:52.712 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.712 DEBUG:boto:Data: 2023-10-12T00:56:52.713 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.713 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.714 DEBUG:boto:Params: {} 2023-10-12T00:56:52.714 DEBUG:boto:Token: None 2023-10-12T00:56:52.715 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9+7Sm5wwVSDO/8Zre0M6qduYkqk= 2023-10-12T00:56:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9+7Sm5wwVSDO/8Zre0M6qduYkqk=', 'Content-Length': '0'} 2023-10-12T00:56:52.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab2096bb24b1561-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.719 DEBUG:boto:path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.719 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.720 DEBUG:boto:Method: GET 2023-10-12T00:56:52.720 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.720 DEBUG:boto:Data: 2023-10-12T00:56:52.721 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.722 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.722 DEBUG:boto:Params: {} 2023-10-12T00:56:52.723 DEBUG:boto:Token: None 2023-10-12T00:56:52.723 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000058 2023-10-12T00:56:52.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9+7Sm5wwVSDO/8Zre0M6qduYkqk= 2023-10-12T00:56:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9+7Sm5wwVSDO/8Zre0M6qduYkqk=', 'Content-Length': '0'} 2023-10-12T00:56:52.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002754c75d4086e385-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.727 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.727 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.728 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.728 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.729 DEBUG:boto:Method: GET 2023-10-12T00:56:52.729 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.729 DEBUG:boto:Data: 2023-10-12T00:56:52.730 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.731 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.731 DEBUG:boto:Params: {} 2023-10-12T00:56:52.732 DEBUG:boto:Token: None 2023-10-12T00:56:52.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.733 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c21405ee58d5d85b-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.736 DEBUG:boto:b'wafmoa-58obj000000000000000000581000falseobj000000000000000000582023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.737 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.738 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.738 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.739 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.739 DEBUG:boto:Method: GET 2023-10-12T00:56:52.739 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000058 2023-10-12T00:56:52.740 DEBUG:boto:Data: 2023-10-12T00:56:52.740 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.741 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.742 DEBUG:boto:Params: {} 2023-10-12T00:56:52.743 DEBUG:boto:Token: None 2023-10-12T00:56:52.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bf4d95f000322d8-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.749 DEBUG:boto:b'wafmoa-58obj000000000000000000581000falseobj000000000000000000582023-10-12T00:56:32.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-12T00:56:52.750 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.751 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.751 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.752 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.752 DEBUG:boto:Method: GET 2023-10-12T00:56:52.752 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.753 DEBUG:boto:Data: 2023-10-12T00:56:52.753 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.754 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.755 DEBUG:boto:Params: {} 2023-10-12T00:56:52.755 DEBUG:boto:Token: None 2023-10-12T00:56:52.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k= 2023-10-12T00:56:52.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k=', 'Content-Length': '0'} 2023-10-12T00:56:52.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1ccd661b6fe93a-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.759 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.760 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.760 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.761 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.761 DEBUG:boto:Method: GET 2023-10-12T00:56:52.762 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.762 DEBUG:boto:Data: 2023-10-12T00:56:52.762 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.764 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.764 DEBUG:boto:Params: {} 2023-10-12T00:56:52.765 DEBUG:boto:Token: None 2023-10-12T00:56:52.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k= 2023-10-12T00:56:52.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k=', 'Content-Length': '0'} 2023-10-12T00:56:52.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d586cdee345bfdf4-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.810 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.811 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.811 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.812 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.812 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.813 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.813 DEBUG:boto:Data: 2023-10-12T00:56:52.813 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.814 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.815 DEBUG:boto:Params: {} 2023-10-12T00:56:52.815 DEBUG:boto:Token: None 2023-10-12T00:56:52.816 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.816 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DgCuVi5cc/G2DuGJSExn/MjipGc= 2023-10-12T00:56:52.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DgCuVi5cc/G2DuGJSExn/MjipGc=', 'Content-Length': '0'} 2023-10-12T00:56:52.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e81f75e73278fd1-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.819 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.820 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.820 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.821 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.821 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.822 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.822 DEBUG:boto:Data: 2023-10-12T00:56:52.823 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.823 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.824 DEBUG:boto:Params: {} 2023-10-12T00:56:52.824 DEBUG:boto:Token: None 2023-10-12T00:56:52.825 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DgCuVi5cc/G2DuGJSExn/MjipGc= 2023-10-12T00:56:52.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DgCuVi5cc/G2DuGJSExn/MjipGc=', 'Content-Length': '0'} 2023-10-12T00:56:52.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b8f49e3aa758b9-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-12T00:56:52.829 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.829 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.829 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.830 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.830 DEBUG:boto:Method: GET 2023-10-12T00:56:52.831 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.831 DEBUG:boto:Data: 2023-10-12T00:56:52.832 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.832 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.833 DEBUG:boto:Params: {} 2023-10-12T00:56:52.833 DEBUG:boto:Token: None 2023-10-12T00:56:52.834 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k= 2023-10-12T00:56:52.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k=', 'Content-Length': '0'} 2023-10-12T00:56:52.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43302d4b1359d31-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.837 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.838 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.838 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.839 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.839 DEBUG:boto:Method: GET 2023-10-12T00:56:52.840 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.840 DEBUG:boto:Data: 2023-10-12T00:56:52.841 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.841 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.842 DEBUG:boto:Params: {} 2023-10-12T00:56:52.842 DEBUG:boto:Token: None 2023-10-12T00:56:52.843 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059?versionId=null 2023-10-12T00:56:52.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k= 2023-10-12T00:56:52.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1+9lzOQjtPre5rJpz/9tDj7L79k=', 'Content-Length': '0'} 2023-10-12T00:56:52.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ca48b82db2ddb1-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.847 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.847 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.848 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.848 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.849 DEBUG:boto:Method: GET 2023-10-12T00:56:52.849 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.850 DEBUG:boto:Data: 2023-10-12T00:56:52.850 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.851 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.851 DEBUG:boto:Params: {} 2023-10-12T00:56:52.852 DEBUG:boto:Token: None 2023-10-12T00:56:52.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007123859ae1a4ac19-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.856 DEBUG:boto:b'wafmoa-58obj000000000000000000591000falseobj000000000000000000592023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.857 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.858 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.858 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.859 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.859 DEBUG:boto:Method: GET 2023-10-12T00:56:52.859 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.860 DEBUG:boto:Data: 2023-10-12T00:56:52.860 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.861 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.862 DEBUG:boto:Params: {} 2023-10-12T00:56:52.862 DEBUG:boto:Token: None 2023-10-12T00:56:52.863 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736baf4ef7d7d878-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.868 DEBUG:boto:b'wafmoa-58obj000000000000000000591000falseobj000000000000000000592023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.869 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.869 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.870 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.870 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.870 DEBUG:boto:Data: 2023-10-12T00:56:52.871 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.872 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.872 DEBUG:boto:Params: {} 2023-10-12T00:56:52.873 DEBUG:boto:Token: None 2023-10-12T00:56:52.873 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FHJ0kMrCPoZ9pdt1B9LRrf4EMTM= 2023-10-12T00:56:52.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FHJ0kMrCPoZ9pdt1B9LRrf4EMTM=', 'Content-Length': '0'} 2023-10-12T00:56:52.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b88cb59b599ddbf0-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.876 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.877 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.877 DEBUG:boto:Method: HEAD 2023-10-12T00:56:52.878 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.878 DEBUG:boto:Data: 2023-10-12T00:56:52.879 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.879 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.880 DEBUG:boto:Params: {} 2023-10-12T00:56:52.880 DEBUG:boto:Token: None 2023-10-12T00:56:52.881 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.881 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FHJ0kMrCPoZ9pdt1B9LRrf4EMTM= 2023-10-12T00:56:52.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FHJ0kMrCPoZ9pdt1B9LRrf4EMTM=', 'Content-Length': '0'} 2023-10-12T00:56:52.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2d377226740d6a-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-12T00:56:52.885 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.885 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.886 DEBUG:boto:Method: GET 2023-10-12T00:56:52.886 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.887 DEBUG:boto:Data: 2023-10-12T00:56:52.887 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.888 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.888 DEBUG:boto:Params: {} 2023-10-12T00:56:52.889 DEBUG:boto:Token: None 2023-10-12T00:56:52.889 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bsSvhU/kUc6+YWeOvMrzBH9PKrE= 2023-10-12T00:56:52.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bsSvhU/kUc6+YWeOvMrzBH9PKrE=', 'Content-Length': '0'} 2023-10-12T00:56:52.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1c5d4399f0955e-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.893 DEBUG:boto:path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.894 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.894 DEBUG:boto:Method: GET 2023-10-12T00:56:52.895 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.895 DEBUG:boto:Data: 2023-10-12T00:56:52.896 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.897 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.897 DEBUG:boto:Params: {} 2023-10-12T00:56:52.898 DEBUG:boto:Token: None 2023-10-12T00:56:52.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000059 2023-10-12T00:56:52.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bsSvhU/kUc6+YWeOvMrzBH9PKrE= 2023-10-12T00:56:52.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bsSvhU/kUc6+YWeOvMrzBH9PKrE=', 'Content-Length': '0'} 2023-10-12T00:56:52.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ff7bedfa476197-0065274454-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.942 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.943 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.943 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.944 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.944 DEBUG:boto:Method: GET 2023-10-12T00:56:52.945 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.945 DEBUG:boto:Data: 2023-10-12T00:56:52.946 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.947 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.947 DEBUG:boto:Params: {} 2023-10-12T00:56:52.948 DEBUG:boto:Token: None 2023-10-12T00:56:52.948 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e90c3ee4a59075-0065274454-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.952 DEBUG:boto:b'wafmoa-58obj000000000000000000591000falseobj000000000000000000592023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.953 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:52.953 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:52.954 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.954 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.955 DEBUG:boto:Method: GET 2023-10-12T00:56:52.955 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000059 2023-10-12T00:56:52.956 DEBUG:boto:Data: 2023-10-12T00:56:52.956 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:52.957 DEBUG:boto:Port: 8000 2023-10-12T00:56:52.958 DEBUG:boto:Params: {} 2023-10-12T00:56:52.958 DEBUG:boto:Token: None 2023-10-12T00:56:52.959 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/ 2023-10-12T00:56:52.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc= 2023-10-12T00:56:52.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmUGDmxCcXYw7J/CGXAtVupH+Mc=', 'Content-Length': '0'} 2023-10-12T00:56:52.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5882b45c7a812d-0065274454-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:52.964 DEBUG:boto:b'wafmoa-58obj000000000000000000591000falseobj000000000000000000592023-10-12T00:56:33.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:52.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-12T00:56:52.966 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:52.966 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:52.967 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:52.967 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:52.968 DEBUG:boto:Method: GET 2023-10-12T00:56:52.968 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:52.969 DEBUG:boto:Data: 2023-10-12T00:56:52.969 DEBUG:boto:Headers: {} 2023-10-12T00:56:52.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:52.970 DEBUG:boto:Port: 8001 2023-10-12T00:56:52.970 DEBUG:boto:Params: {} 2023-10-12T00:56:52.971 DEBUG:boto:Token: None 2023-10-12T00:56:52.971 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:52 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:52.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Syuu/NmpeDP1c92s07Zc3qTpLfs= 2023-10-12T00:56:52.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Syuu/NmpeDP1c92s07Zc3qTpLfs=', 'Content-Length': '0'} 2023-10-12T00:56:52.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af24277e20aba542-0065274454-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.022 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.023 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.023 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.024 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.024 DEBUG:boto:Method: GET 2023-10-12T00:56:53.024 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.025 DEBUG:boto:Data: 2023-10-12T00:56:53.025 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.026 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.027 DEBUG:boto:Params: {} 2023-10-12T00:56:53.027 DEBUG:boto:Token: None 2023-10-12T00:56:53.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE= 2023-10-12T00:56:53.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE=', 'Content-Length': '0'} 2023-10-12T00:56:53.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed05fa5992d9efa1-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.031 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.032 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.032 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.033 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.033 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.034 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.034 DEBUG:boto:Data: 2023-10-12T00:56:53.034 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.035 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.036 DEBUG:boto:Params: {} 2023-10-12T00:56:53.036 DEBUG:boto:Token: None 2023-10-12T00:56:53.037 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9BjTJbupe2M431YNyss7PO0jVQ= 2023-10-12T00:56:53.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9BjTJbupe2M431YNyss7PO0jVQ=', 'Content-Length': '0'} 2023-10-12T00:56:53.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3e35f31f99af1b-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.040 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.041 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.041 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.042 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.042 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.043 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.043 DEBUG:boto:Data: 2023-10-12T00:56:53.043 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.044 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.045 DEBUG:boto:Params: {} 2023-10-12T00:56:53.045 DEBUG:boto:Token: None 2023-10-12T00:56:53.046 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/9BjTJbupe2M431YNyss7PO0jVQ= 2023-10-12T00:56:53.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/9BjTJbupe2M431YNyss7PO0jVQ=', 'Content-Length': '0'} 2023-10-12T00:56:53.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9402e19e4287351-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-12T00:56:53.050 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.050 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.051 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.051 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.051 DEBUG:boto:Method: GET 2023-10-12T00:56:53.052 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.052 DEBUG:boto:Data: 2023-10-12T00:56:53.053 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.054 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.054 DEBUG:boto:Params: {} 2023-10-12T00:56:53.054 DEBUG:boto:Token: None 2023-10-12T00:56:53.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE= 2023-10-12T00:56:53.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE=', 'Content-Length': '0'} 2023-10-12T00:56:53.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130f5035d87926cf-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.059 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.059 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.060 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.060 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.061 DEBUG:boto:Method: GET 2023-10-12T00:56:53.061 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.062 DEBUG:boto:Data: 2023-10-12T00:56:53.062 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.063 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.063 DEBUG:boto:Params: {} 2023-10-12T00:56:53.064 DEBUG:boto:Token: None 2023-10-12T00:56:53.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060?versionId=null 2023-10-12T00:56:53.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE= 2023-10-12T00:56:53.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0JPIpcBYicR8k52IbRt854WcmFE=', 'Content-Length': '0'} 2023-10-12T00:56:53.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093bd00f8ec24160f-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.068 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.069 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.069 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.070 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.070 DEBUG:boto:Method: GET 2023-10-12T00:56:53.071 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.071 DEBUG:boto:Data: 2023-10-12T00:56:53.072 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.073 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.073 DEBUG:boto:Params: {} 2023-10-12T00:56:53.073 DEBUG:boto:Token: None 2023-10-12T00:56:53.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002017499531632848-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.078 DEBUG:boto:b'wafmoa-58obj000000000000000000601000falseobj000000000000000000602023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.079 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.080 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.080 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.081 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.081 DEBUG:boto:Method: GET 2023-10-12T00:56:53.082 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.082 DEBUG:boto:Data: 2023-10-12T00:56:53.083 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.084 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.084 DEBUG:boto:Params: {} 2023-10-12T00:56:53.085 DEBUG:boto:Token: None 2023-10-12T00:56:53.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.086 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000222274817f976a6-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.091 DEBUG:boto:b'wafmoa-58obj000000000000000000601000falseobj000000000000000000602023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.092 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.092 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.092 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.093 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.093 DEBUG:boto:Data: 2023-10-12T00:56:53.094 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.094 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.095 DEBUG:boto:Params: {} 2023-10-12T00:56:53.095 DEBUG:boto:Token: None 2023-10-12T00:56:53.096 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gqDOYUGnzUFKb5L6d4rdztOjoFY= 2023-10-12T00:56:53.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gqDOYUGnzUFKb5L6d4rdztOjoFY=', 'Content-Length': '0'} 2023-10-12T00:56:53.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a730f38060c6af7-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.099 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.100 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.100 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.101 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.101 DEBUG:boto:Data: 2023-10-12T00:56:53.102 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.102 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.103 DEBUG:boto:Params: {} 2023-10-12T00:56:53.103 DEBUG:boto:Token: None 2023-10-12T00:56:53.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gqDOYUGnzUFKb5L6d4rdztOjoFY= 2023-10-12T00:56:53.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gqDOYUGnzUFKb5L6d4rdztOjoFY=', 'Content-Length': '0'} 2023-10-12T00:56:53.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068db857ea4c605dd-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-12T00:56:53.108 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.108 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.109 DEBUG:boto:Method: GET 2023-10-12T00:56:53.109 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.109 DEBUG:boto:Data: 2023-10-12T00:56:53.110 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.111 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.111 DEBUG:boto:Params: {} 2023-10-12T00:56:53.112 DEBUG:boto:Token: None 2023-10-12T00:56:53.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZdJuoXSGYFuOMP4JOjACS1YXXlk= 2023-10-12T00:56:53.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZdJuoXSGYFuOMP4JOjACS1YXXlk=', 'Content-Length': '0'} 2023-10-12T00:56:53.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7ace1a46f7d622-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.158 DEBUG:boto:path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.159 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.159 DEBUG:boto:Method: GET 2023-10-12T00:56:53.160 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.160 DEBUG:boto:Data: 2023-10-12T00:56:53.161 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.162 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.162 DEBUG:boto:Params: {} 2023-10-12T00:56:53.163 DEBUG:boto:Token: None 2023-10-12T00:56:53.163 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000060 2023-10-12T00:56:53.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZdJuoXSGYFuOMP4JOjACS1YXXlk= 2023-10-12T00:56:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZdJuoXSGYFuOMP4JOjACS1YXXlk=', 'Content-Length': '0'} 2023-10-12T00:56:53.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db4236ba4349c77-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.167 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.168 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.168 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.169 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.169 DEBUG:boto:Method: GET 2023-10-12T00:56:53.170 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.170 DEBUG:boto:Data: 2023-10-12T00:56:53.170 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.171 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.172 DEBUG:boto:Params: {} 2023-10-12T00:56:53.172 DEBUG:boto:Token: None 2023-10-12T00:56:53.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa962eab496f62f-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.177 DEBUG:boto:b'wafmoa-58obj000000000000000000601000falseobj000000000000000000602023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.178 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.178 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.178 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.179 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.179 DEBUG:boto:Method: GET 2023-10-12T00:56:53.180 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000060 2023-10-12T00:56:53.180 DEBUG:boto:Data: 2023-10-12T00:56:53.181 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.182 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.182 DEBUG:boto:Params: {} 2023-10-12T00:56:53.183 DEBUG:boto:Token: None 2023-10-12T00:56:53.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b060c64b8174b66e-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.189 DEBUG:boto:b'wafmoa-58obj000000000000000000601000falseobj000000000000000000602023-10-12T00:56:37.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-12T00:56:53.190 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.190 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.191 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.191 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.192 DEBUG:boto:Method: GET 2023-10-12T00:56:53.192 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.193 DEBUG:boto:Data: 2023-10-12T00:56:53.193 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.194 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.194 DEBUG:boto:Params: {} 2023-10-12T00:56:53.195 DEBUG:boto:Token: None 2023-10-12T00:56:53.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI= 2023-10-12T00:56:53.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI=', 'Content-Length': '0'} 2023-10-12T00:56:53.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f559ce066066af2e-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.200 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.201 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.201 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.201 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.202 DEBUG:boto:Method: GET 2023-10-12T00:56:53.202 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.203 DEBUG:boto:Data: 2023-10-12T00:56:53.203 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.204 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.204 DEBUG:boto:Params: {} 2023-10-12T00:56:53.205 DEBUG:boto:Token: None 2023-10-12T00:56:53.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI= 2023-10-12T00:56:53.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI=', 'Content-Length': '0'} 2023-10-12T00:56:53.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095353f3960c117b6-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.209 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.209 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.210 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.210 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.211 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.211 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.212 DEBUG:boto:Data: 2023-10-12T00:56:53.212 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.213 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.213 DEBUG:boto:Params: {} 2023-10-12T00:56:53.214 DEBUG:boto:Token: None 2023-10-12T00:56:53.214 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.215 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hcgNHpBwKvWdkzJXqq+Tyk5ji+Q= 2023-10-12T00:56:53.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hcgNHpBwKvWdkzJXqq+Tyk5ji+Q=', 'Content-Length': '0'} 2023-10-12T00:56:53.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c1055d438cdd24-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.217 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.218 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.218 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.219 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.219 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.220 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.220 DEBUG:boto:Data: 2023-10-12T00:56:53.220 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.221 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.222 DEBUG:boto:Params: {} 2023-10-12T00:56:53.222 DEBUG:boto:Token: None 2023-10-12T00:56:53.223 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hcgNHpBwKvWdkzJXqq+Tyk5ji+Q= 2023-10-12T00:56:53.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hcgNHpBwKvWdkzJXqq+Tyk5ji+Q=', 'Content-Length': '0'} 2023-10-12T00:56:53.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0057dd690fedb6d-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-12T00:56:53.226 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.227 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.227 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.228 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.228 DEBUG:boto:Method: GET 2023-10-12T00:56:53.229 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.229 DEBUG:boto:Data: 2023-10-12T00:56:53.229 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.230 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.231 DEBUG:boto:Params: {} 2023-10-12T00:56:53.231 DEBUG:boto:Token: None 2023-10-12T00:56:53.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI= 2023-10-12T00:56:53.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI=', 'Content-Length': '0'} 2023-10-12T00:56:53.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb25ef59a2755d3c-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.235 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.236 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.236 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.237 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.237 DEBUG:boto:Method: GET 2023-10-12T00:56:53.237 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.238 DEBUG:boto:Data: 2023-10-12T00:56:53.238 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.239 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.240 DEBUG:boto:Params: {} 2023-10-12T00:56:53.240 DEBUG:boto:Token: None 2023-10-12T00:56:53.240 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061?versionId=null 2023-10-12T00:56:53.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI= 2023-10-12T00:56:53.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NkhQag5O1CBCvDu4zkEvurimxcI=', 'Content-Length': '0'} 2023-10-12T00:56:53.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d3b7a8c6320e15-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.244 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.245 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.245 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.245 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.246 DEBUG:boto:Method: GET 2023-10-12T00:56:53.246 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.247 DEBUG:boto:Data: 2023-10-12T00:56:53.247 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.248 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.248 DEBUG:boto:Params: {} 2023-10-12T00:56:53.249 DEBUG:boto:Token: None 2023-10-12T00:56:53.249 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001146e63c464fd65c-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.294 DEBUG:boto:b'wafmoa-58obj000000000000000000611000falseobj000000000000000000612023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.295 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.296 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.296 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.297 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.297 DEBUG:boto:Method: GET 2023-10-12T00:56:53.297 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.298 DEBUG:boto:Data: 2023-10-12T00:56:53.298 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.299 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.300 DEBUG:boto:Params: {} 2023-10-12T00:56:53.300 DEBUG:boto:Token: None 2023-10-12T00:56:53.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e179537c42d2ab4-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.306 DEBUG:boto:b'wafmoa-58obj000000000000000000611000falseobj000000000000000000612023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.307 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.307 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.308 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.308 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.308 DEBUG:boto:Data: 2023-10-12T00:56:53.309 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.310 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.310 DEBUG:boto:Params: {} 2023-10-12T00:56:53.310 DEBUG:boto:Token: None 2023-10-12T00:56:53.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lM7QMgfWgYfFZclnh5w/6QIbHJg= 2023-10-12T00:56:53.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lM7QMgfWgYfFZclnh5w/6QIbHJg=', 'Content-Length': '0'} 2023-10-12T00:56:53.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9fd398af7ae4a6-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.314 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.315 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.316 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.316 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.316 DEBUG:boto:Data: 2023-10-12T00:56:53.317 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.318 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.318 DEBUG:boto:Params: {} 2023-10-12T00:56:53.318 DEBUG:boto:Token: None 2023-10-12T00:56:53.319 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lM7QMgfWgYfFZclnh5w/6QIbHJg= 2023-10-12T00:56:53.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lM7QMgfWgYfFZclnh5w/6QIbHJg=', 'Content-Length': '0'} 2023-10-12T00:56:53.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d946473b44c7790e-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-12T00:56:53.324 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.324 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.325 DEBUG:boto:Method: GET 2023-10-12T00:56:53.325 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.326 DEBUG:boto:Data: 2023-10-12T00:56:53.326 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.327 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.328 DEBUG:boto:Params: {} 2023-10-12T00:56:53.328 DEBUG:boto:Token: None 2023-10-12T00:56:53.329 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4NYkQ8DoehYE26ZrBfdZRfuF3ow= 2023-10-12T00:56:53.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4NYkQ8DoehYE26ZrBfdZRfuF3ow=', 'Content-Length': '0'} 2023-10-12T00:56:53.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c364c2c70ad28f3-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.332 DEBUG:boto:path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.333 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.333 DEBUG:boto:Method: GET 2023-10-12T00:56:53.334 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.334 DEBUG:boto:Data: 2023-10-12T00:56:53.335 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.335 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.336 DEBUG:boto:Params: {} 2023-10-12T00:56:53.336 DEBUG:boto:Token: None 2023-10-12T00:56:53.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000061 2023-10-12T00:56:53.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4NYkQ8DoehYE26ZrBfdZRfuF3ow= 2023-10-12T00:56:53.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4NYkQ8DoehYE26ZrBfdZRfuF3ow=', 'Content-Length': '0'} 2023-10-12T00:56:53.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cbf54bcf7abf73d-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.386 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.387 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.387 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.388 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.388 DEBUG:boto:Method: GET 2023-10-12T00:56:53.388 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.389 DEBUG:boto:Data: 2023-10-12T00:56:53.389 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.390 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.390 DEBUG:boto:Params: {} 2023-10-12T00:56:53.391 DEBUG:boto:Token: None 2023-10-12T00:56:53.391 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071915dbc36fe0913-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.396 DEBUG:boto:b'wafmoa-58obj000000000000000000611000falseobj000000000000000000612023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.396 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.397 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.397 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.397 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.398 DEBUG:boto:Method: GET 2023-10-12T00:56:53.398 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000061 2023-10-12T00:56:53.399 DEBUG:boto:Data: 2023-10-12T00:56:53.399 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.400 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.400 DEBUG:boto:Params: {} 2023-10-12T00:56:53.401 DEBUG:boto:Token: None 2023-10-12T00:56:53.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeeeaa7bb23d3ad2-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.407 DEBUG:boto:b'wafmoa-58obj000000000000000000611000falseobj000000000000000000612023-10-12T00:56:34.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-12T00:56:53.408 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.409 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.409 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.409 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.410 DEBUG:boto:Method: GET 2023-10-12T00:56:53.410 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.411 DEBUG:boto:Data: 2023-10-12T00:56:53.411 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.412 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.412 DEBUG:boto:Params: {} 2023-10-12T00:56:53.413 DEBUG:boto:Token: None 2023-10-12T00:56:53.413 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI= 2023-10-12T00:56:53.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI=', 'Content-Length': '0'} 2023-10-12T00:56:53.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c3eb0095e5f84d-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.417 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.418 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.418 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.419 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.419 DEBUG:boto:Method: GET 2023-10-12T00:56:53.420 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.420 DEBUG:boto:Data: 2023-10-12T00:56:53.421 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.421 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.422 DEBUG:boto:Params: {} 2023-10-12T00:56:53.422 DEBUG:boto:Token: None 2023-10-12T00:56:53.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI= 2023-10-12T00:56:53.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI=', 'Content-Length': '0'} 2023-10-12T00:56:53.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b164cfe3dff51c-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.426 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.427 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.427 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.428 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.428 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.428 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.429 DEBUG:boto:Data: 2023-10-12T00:56:53.429 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.430 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.431 DEBUG:boto:Params: {} 2023-10-12T00:56:53.431 DEBUG:boto:Token: None 2023-10-12T00:56:53.431 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9/Qb7py8ItQpaWkB33oSsCiDfVo= 2023-10-12T00:56:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9/Qb7py8ItQpaWkB33oSsCiDfVo=', 'Content-Length': '0'} 2023-10-12T00:56:53.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d663bf6506eda0-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.435 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.435 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.436 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.436 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.436 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.437 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.437 DEBUG:boto:Data: 2023-10-12T00:56:53.438 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.438 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.439 DEBUG:boto:Params: {} 2023-10-12T00:56:53.439 DEBUG:boto:Token: None 2023-10-12T00:56:53.440 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9/Qb7py8ItQpaWkB33oSsCiDfVo= 2023-10-12T00:56:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9/Qb7py8ItQpaWkB33oSsCiDfVo=', 'Content-Length': '0'} 2023-10-12T00:56:53.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ff3ad998e6b1b1-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-12T00:56:53.444 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.444 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.445 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.445 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.446 DEBUG:boto:Method: GET 2023-10-12T00:56:53.446 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.446 DEBUG:boto:Data: 2023-10-12T00:56:53.447 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.448 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.448 DEBUG:boto:Params: {} 2023-10-12T00:56:53.449 DEBUG:boto:Token: None 2023-10-12T00:56:53.449 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI= 2023-10-12T00:56:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI=', 'Content-Length': '0'} 2023-10-12T00:56:53.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0717613103a9caa-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.453 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.453 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.454 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.454 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.455 DEBUG:boto:Method: GET 2023-10-12T00:56:53.456 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.456 DEBUG:boto:Data: 2023-10-12T00:56:53.457 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.458 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.458 DEBUG:boto:Params: {} 2023-10-12T00:56:53.459 DEBUG:boto:Token: None 2023-10-12T00:56:53.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062?versionId=null 2023-10-12T00:56:53.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI= 2023-10-12T00:56:53.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XVYtce7H1UpCTv0GYYkPMWh4ZlI=', 'Content-Length': '0'} 2023-10-12T00:56:53.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006715b81de980030-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.506 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.507 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.507 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.508 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.508 DEBUG:boto:Method: GET 2023-10-12T00:56:53.509 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.509 DEBUG:boto:Data: 2023-10-12T00:56:53.509 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.510 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.511 DEBUG:boto:Params: {} 2023-10-12T00:56:53.511 DEBUG:boto:Token: None 2023-10-12T00:56:53.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026932e6cf74ddd5f-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.516 DEBUG:boto:b'wafmoa-58obj000000000000000000621000falseobj000000000000000000622023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.516 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.517 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.517 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.518 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.518 DEBUG:boto:Method: GET 2023-10-12T00:56:53.519 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.519 DEBUG:boto:Data: 2023-10-12T00:56:53.520 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.521 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.521 DEBUG:boto:Params: {} 2023-10-12T00:56:53.522 DEBUG:boto:Token: None 2023-10-12T00:56:53.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846b077a8bc62c47-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.528 DEBUG:boto:b'wafmoa-58obj000000000000000000621000falseobj000000000000000000622023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.529 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.529 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.530 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.530 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.530 DEBUG:boto:Data: 2023-10-12T00:56:53.531 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.531 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.532 DEBUG:boto:Params: {} 2023-10-12T00:56:53.532 DEBUG:boto:Token: None 2023-10-12T00:56:53.533 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1C7SKLbEu947WnopJUhSZOKUbzs= 2023-10-12T00:56:53.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1C7SKLbEu947WnopJUhSZOKUbzs=', 'Content-Length': '0'} 2023-10-12T00:56:53.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0bc5e7e1aeb48b-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.536 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.537 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.537 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.537 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.538 DEBUG:boto:Data: 2023-10-12T00:56:53.538 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.539 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.539 DEBUG:boto:Params: {} 2023-10-12T00:56:53.540 DEBUG:boto:Token: None 2023-10-12T00:56:53.540 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1C7SKLbEu947WnopJUhSZOKUbzs= 2023-10-12T00:56:53.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1C7SKLbEu947WnopJUhSZOKUbzs=', 'Content-Length': '0'} 2023-10-12T00:56:53.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ba8a17149810ab-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-12T00:56:53.546 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.546 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.547 DEBUG:boto:Method: GET 2023-10-12T00:56:53.547 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.548 DEBUG:boto:Data: 2023-10-12T00:56:53.548 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.549 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.549 DEBUG:boto:Params: {} 2023-10-12T00:56:53.550 DEBUG:boto:Token: None 2023-10-12T00:56:53.550 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:puCJKrnjll5CNygIWRarq6JiyNE= 2023-10-12T00:56:53.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:puCJKrnjll5CNygIWRarq6JiyNE=', 'Content-Length': '0'} 2023-10-12T00:56:53.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158af442d5293c1d-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.598 DEBUG:boto:path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.599 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.599 DEBUG:boto:Method: GET 2023-10-12T00:56:53.600 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.600 DEBUG:boto:Data: 2023-10-12T00:56:53.601 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.601 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.602 DEBUG:boto:Params: {} 2023-10-12T00:56:53.602 DEBUG:boto:Token: None 2023-10-12T00:56:53.603 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000062 2023-10-12T00:56:53.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:puCJKrnjll5CNygIWRarq6JiyNE= 2023-10-12T00:56:53.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:puCJKrnjll5CNygIWRarq6JiyNE=', 'Content-Length': '0'} 2023-10-12T00:56:53.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037130d9eab7b08c0-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.606 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.607 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.607 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.608 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.608 DEBUG:boto:Method: GET 2023-10-12T00:56:53.608 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.609 DEBUG:boto:Data: 2023-10-12T00:56:53.609 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.610 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.610 DEBUG:boto:Params: {} 2023-10-12T00:56:53.611 DEBUG:boto:Token: None 2023-10-12T00:56:53.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.612 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f9b4f557cb5a69-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.615 DEBUG:boto:b'wafmoa-58obj000000000000000000621000falseobj000000000000000000622023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.616 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.617 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.617 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.617 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.618 DEBUG:boto:Method: GET 2023-10-12T00:56:53.618 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000062 2023-10-12T00:56:53.619 DEBUG:boto:Data: 2023-10-12T00:56:53.619 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.620 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.620 DEBUG:boto:Params: {} 2023-10-12T00:56:53.621 DEBUG:boto:Token: None 2023-10-12T00:56:53.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000923ddba7ed4d26f-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.627 DEBUG:boto:b'wafmoa-58obj000000000000000000621000falseobj000000000000000000622023-10-12T00:56:36.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-12T00:56:53.628 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.628 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.629 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.629 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.630 DEBUG:boto:Method: GET 2023-10-12T00:56:53.630 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.631 DEBUG:boto:Data: 2023-10-12T00:56:53.631 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.632 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.633 DEBUG:boto:Params: {} 2023-10-12T00:56:53.633 DEBUG:boto:Token: None 2023-10-12T00:56:53.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4= 2023-10-12T00:56:53.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4=', 'Content-Length': '0'} 2023-10-12T00:56:53.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a343c8b6e2e2fa-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.637 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.638 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.638 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.639 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.639 DEBUG:boto:Method: GET 2023-10-12T00:56:53.640 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.640 DEBUG:boto:Data: 2023-10-12T00:56:53.641 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.642 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.642 DEBUG:boto:Params: {} 2023-10-12T00:56:53.642 DEBUG:boto:Token: None 2023-10-12T00:56:53.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.643 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4= 2023-10-12T00:56:53.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4=', 'Content-Length': '0'} 2023-10-12T00:56:53.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aebd88567f906cee-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.686 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.687 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.687 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.688 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.688 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.688 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.689 DEBUG:boto:Data: 2023-10-12T00:56:53.689 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.690 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.690 DEBUG:boto:Params: {} 2023-10-12T00:56:53.691 DEBUG:boto:Token: None 2023-10-12T00:56:53.691 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8NKFNMjVf1ZP5r3thtchZSrQUa8= 2023-10-12T00:56:53.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8NKFNMjVf1ZP5r3thtchZSrQUa8=', 'Content-Length': '0'} 2023-10-12T00:56:53.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d042e33e6da4608-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.695 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.695 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.696 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.696 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.697 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.697 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.697 DEBUG:boto:Data: 2023-10-12T00:56:53.698 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.699 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.699 DEBUG:boto:Params: {} 2023-10-12T00:56:53.699 DEBUG:boto:Token: None 2023-10-12T00:56:53.700 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8NKFNMjVf1ZP5r3thtchZSrQUa8= 2023-10-12T00:56:53.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8NKFNMjVf1ZP5r3thtchZSrQUa8=', 'Content-Length': '0'} 2023-10-12T00:56:53.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc5f05b31365907-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-12T00:56:53.704 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.704 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.705 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.705 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.706 DEBUG:boto:Method: GET 2023-10-12T00:56:53.706 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.707 DEBUG:boto:Data: 2023-10-12T00:56:53.707 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.708 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.708 DEBUG:boto:Params: {} 2023-10-12T00:56:53.709 DEBUG:boto:Token: None 2023-10-12T00:56:53.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4= 2023-10-12T00:56:53.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4=', 'Content-Length': '0'} 2023-10-12T00:56:53.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d744f1d882d08a-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.758 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.759 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.759 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.760 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.760 DEBUG:boto:Method: GET 2023-10-12T00:56:53.761 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.761 DEBUG:boto:Data: 2023-10-12T00:56:53.761 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.762 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.763 DEBUG:boto:Params: {} 2023-10-12T00:56:53.763 DEBUG:boto:Token: None 2023-10-12T00:56:53.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065?versionId=null 2023-10-12T00:56:53.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4= 2023-10-12T00:56:53.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JvDRrwZ25M5KT5qBxtCvUk93Pa4=', 'Content-Length': '0'} 2023-10-12T00:56:53.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b9b8f78559ddde-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.768 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.768 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.769 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.769 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.769 DEBUG:boto:Method: GET 2023-10-12T00:56:53.770 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.770 DEBUG:boto:Data: 2023-10-12T00:56:53.771 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.772 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.772 DEBUG:boto:Params: {} 2023-10-12T00:56:53.773 DEBUG:boto:Token: None 2023-10-12T00:56:53.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.773 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f89ee44b2cd978a-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.777 DEBUG:boto:b'wafmoa-58obj000000000000000000651000falseobj000000000000000000652023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.778 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.779 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.779 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.779 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.780 DEBUG:boto:Method: GET 2023-10-12T00:56:53.781 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.781 DEBUG:boto:Data: 2023-10-12T00:56:53.781 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.782 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.783 DEBUG:boto:Params: {} 2023-10-12T00:56:53.783 DEBUG:boto:Token: None 2023-10-12T00:56:53.784 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c519701cfdd0175-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.790 DEBUG:boto:b'wafmoa-58obj000000000000000000651000falseobj000000000000000000652023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.791 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.791 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.791 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.792 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.792 DEBUG:boto:Data: 2023-10-12T00:56:53.793 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.794 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.794 DEBUG:boto:Params: {} 2023-10-12T00:56:53.794 DEBUG:boto:Token: None 2023-10-12T00:56:53.795 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A6txNalzFUzMPj5YFX4OTfi4atQ= 2023-10-12T00:56:53.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A6txNalzFUzMPj5YFX4OTfi4atQ=', 'Content-Length': '0'} 2023-10-12T00:56:53.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006818cae1c6fdbcb3-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.799 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.799 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.799 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.800 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.800 DEBUG:boto:Data: 2023-10-12T00:56:53.800 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.801 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.802 DEBUG:boto:Params: {} 2023-10-12T00:56:53.802 DEBUG:boto:Token: None 2023-10-12T00:56:53.802 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A6txNalzFUzMPj5YFX4OTfi4atQ= 2023-10-12T00:56:53.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A6txNalzFUzMPj5YFX4OTfi4atQ=', 'Content-Length': '0'} 2023-10-12T00:56:53.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c51c587c7f4b5d-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-12T00:56:53.806 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.806 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.807 DEBUG:boto:Method: GET 2023-10-12T00:56:53.807 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.808 DEBUG:boto:Data: 2023-10-12T00:56:53.808 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.809 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.809 DEBUG:boto:Params: {} 2023-10-12T00:56:53.810 DEBUG:boto:Token: None 2023-10-12T00:56:53.810 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oqmaHb3c2aCY9IaWaeGQbNhLR6I= 2023-10-12T00:56:53.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oqmaHb3c2aCY9IaWaeGQbNhLR6I=', 'Content-Length': '0'} 2023-10-12T00:56:53.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053f4dc069160087-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.814 DEBUG:boto:path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.815 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.815 DEBUG:boto:Method: GET 2023-10-12T00:56:53.815 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.816 DEBUG:boto:Data: 2023-10-12T00:56:53.816 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.817 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.817 DEBUG:boto:Params: {} 2023-10-12T00:56:53.818 DEBUG:boto:Token: None 2023-10-12T00:56:53.818 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000065 2023-10-12T00:56:53.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oqmaHb3c2aCY9IaWaeGQbNhLR6I= 2023-10-12T00:56:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oqmaHb3c2aCY9IaWaeGQbNhLR6I=', 'Content-Length': '0'} 2023-10-12T00:56:53.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970806a8c3bcce50-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.866 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.867 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.867 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.868 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.868 DEBUG:boto:Method: GET 2023-10-12T00:56:53.869 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.869 DEBUG:boto:Data: 2023-10-12T00:56:53.869 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.870 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.871 DEBUG:boto:Params: {} 2023-10-12T00:56:53.871 DEBUG:boto:Token: None 2023-10-12T00:56:53.872 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9e7989f2032e22-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.876 DEBUG:boto:b'wafmoa-58obj000000000000000000651000falseobj000000000000000000652023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.877 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.877 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.878 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.878 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.878 DEBUG:boto:Method: GET 2023-10-12T00:56:53.879 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000065 2023-10-12T00:56:53.879 DEBUG:boto:Data: 2023-10-12T00:56:53.880 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.881 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.881 DEBUG:boto:Params: {} 2023-10-12T00:56:53.881 DEBUG:boto:Token: None 2023-10-12T00:56:53.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007897d976902e43b1-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.887 DEBUG:boto:b'wafmoa-58obj000000000000000000651000falseobj000000000000000000652023-10-12T00:56:36.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-12T00:56:53.888 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.889 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.889 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.890 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.890 DEBUG:boto:Method: GET 2023-10-12T00:56:53.891 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.891 DEBUG:boto:Data: 2023-10-12T00:56:53.891 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.892 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.893 DEBUG:boto:Params: {} 2023-10-12T00:56:53.893 DEBUG:boto:Token: None 2023-10-12T00:56:53.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA= 2023-10-12T00:56:53.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA=', 'Content-Length': '0'} 2023-10-12T00:56:53.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d266e2bf77db273-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.897 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.897 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.898 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.898 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.899 DEBUG:boto:Method: GET 2023-10-12T00:56:53.899 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.899 DEBUG:boto:Data: 2023-10-12T00:56:53.900 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.901 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.901 DEBUG:boto:Params: {} 2023-10-12T00:56:53.902 DEBUG:boto:Token: None 2023-10-12T00:56:53.902 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA= 2023-10-12T00:56:53.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA=', 'Content-Length': '0'} 2023-10-12T00:56:53.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b16e1c50a42ab4-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.906 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.906 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.907 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.907 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.908 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.908 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.908 DEBUG:boto:Data: 2023-10-12T00:56:53.909 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.910 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.910 DEBUG:boto:Params: {} 2023-10-12T00:56:53.910 DEBUG:boto:Token: None 2023-10-12T00:56:53.911 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GU4yfhxVOgrarYH7+nbsDjN5Cbs= 2023-10-12T00:56:53.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GU4yfhxVOgrarYH7+nbsDjN5Cbs=', 'Content-Length': '0'} 2023-10-12T00:56:53.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f41adda04f32050-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.914 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.915 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.915 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.916 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.916 DEBUG:boto:Method: HEAD 2023-10-12T00:56:53.917 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.917 DEBUG:boto:Data: 2023-10-12T00:56:53.917 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.918 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.918 DEBUG:boto:Params: {} 2023-10-12T00:56:53.919 DEBUG:boto:Token: None 2023-10-12T00:56:53.919 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GU4yfhxVOgrarYH7+nbsDjN5Cbs= 2023-10-12T00:56:53.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GU4yfhxVOgrarYH7+nbsDjN5Cbs=', 'Content-Length': '0'} 2023-10-12T00:56:53.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0f781f66f332d0-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-12T00:56:53.923 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.924 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.924 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.924 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.925 DEBUG:boto:Method: GET 2023-10-12T00:56:53.925 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.926 DEBUG:boto:Data: 2023-10-12T00:56:53.926 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.927 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.927 DEBUG:boto:Params: {} 2023-10-12T00:56:53.928 DEBUG:boto:Token: None 2023-10-12T00:56:53.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA= 2023-10-12T00:56:53.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA=', 'Content-Length': '0'} 2023-10-12T00:56:53.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f708b8865ef5f9e8-0065274455-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.932 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.932 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:53.933 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.933 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.933 DEBUG:boto:Method: GET 2023-10-12T00:56:53.934 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.934 DEBUG:boto:Data: 2023-10-12T00:56:53.935 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.936 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.936 DEBUG:boto:Params: {} 2023-10-12T00:56:53.936 DEBUG:boto:Token: None 2023-10-12T00:56:53.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/obj00000000000000000076?versionId=null 2023-10-12T00:56:53.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA= 2023-10-12T00:56:53.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aIkZFmeJEb0594R/Q6GDzZl2jTA=', 'Content-Length': '0'} 2023-10-12T00:56:53.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13eb69cf3963571-0065274455-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.941 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.941 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.942 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.942 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.942 DEBUG:boto:Method: GET 2023-10-12T00:56:53.943 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.943 DEBUG:boto:Data: 2023-10-12T00:56:53.944 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:53.944 DEBUG:boto:Port: 8001 2023-10-12T00:56:53.945 DEBUG:boto:Params: {} 2023-10-12T00:56:53.945 DEBUG:boto:Token: None 2023-10-12T00:56:53.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:53.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b1c4f07b2a1024e-0065274455-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:53.990 DEBUG:boto:b'wafmoa-58obj000000000000000000761000falseobj000000000000000000762023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:53.991 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:53.991 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:53.992 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.992 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.993 DEBUG:boto:Method: GET 2023-10-12T00:56:53.993 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:53.993 DEBUG:boto:Data: 2023-10-12T00:56:53.994 DEBUG:boto:Headers: {} 2023-10-12T00:56:53.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:53.995 DEBUG:boto:Port: 8000 2023-10-12T00:56:53.995 DEBUG:boto:Params: {} 2023-10-12T00:56:53.995 DEBUG:boto:Token: None 2023-10-12T00:56:53.996 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:53 GMT /wafmoa-58/ 2023-10-12T00:56:53.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg= 2023-10-12T00:56:53.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zoNGcoXvGsa79+U575t90mo4fg=', 'Content-Length': '0'} 2023-10-12T00:56:54.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84df6a88bc546a0-0065274455-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.001 DEBUG:boto:b'wafmoa-58obj000000000000000000761000falseobj000000000000000000762023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.002 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.002 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.003 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.003 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.003 DEBUG:boto:Data: 2023-10-12T00:56:54.004 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.005 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.005 DEBUG:boto:Params: {} 2023-10-12T00:56:54.005 DEBUG:boto:Token: None 2023-10-12T00:56:54.006 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wQ1oaIcRnwekq7gOLu0Jmj+Iyso= 2023-10-12T00:56:54.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wQ1oaIcRnwekq7gOLu0Jmj+Iyso=', 'Content-Length': '0'} 2023-10-12T00:56:54.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e6ce1b9a31fc5d-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.009 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.010 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.010 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.010 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.011 DEBUG:boto:Data: 2023-10-12T00:56:54.011 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.012 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.012 DEBUG:boto:Params: {} 2023-10-12T00:56:54.013 DEBUG:boto:Token: None 2023-10-12T00:56:54.013 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.014 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wQ1oaIcRnwekq7gOLu0Jmj+Iyso= 2023-10-12T00:56:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wQ1oaIcRnwekq7gOLu0Jmj+Iyso=', 'Content-Length': '0'} 2023-10-12T00:56:54.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371f4730f0796300-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-12T00:56:54.017 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.017 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.018 DEBUG:boto:Method: GET 2023-10-12T00:56:54.018 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.019 DEBUG:boto:Data: 2023-10-12T00:56:54.019 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.020 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.020 DEBUG:boto:Params: {} 2023-10-12T00:56:54.021 DEBUG:boto:Token: None 2023-10-12T00:56:54.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y01jdtJr5ogx7/yGMEjvk2VGvxM= 2023-10-12T00:56:54.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y01jdtJr5ogx7/yGMEjvk2VGvxM=', 'Content-Length': '0'} 2023-10-12T00:56:54.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58e1690d0a1d265-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.025 DEBUG:boto:path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.025 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.026 DEBUG:boto:Method: GET 2023-10-12T00:56:54.026 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.027 DEBUG:boto:Data: 2023-10-12T00:56:54.027 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.028 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.028 DEBUG:boto:Params: {} 2023-10-12T00:56:54.029 DEBUG:boto:Token: None 2023-10-12T00:56:54.029 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000076 2023-10-12T00:56:54.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y01jdtJr5ogx7/yGMEjvk2VGvxM= 2023-10-12T00:56:54.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y01jdtJr5ogx7/yGMEjvk2VGvxM=', 'Content-Length': '0'} 2023-10-12T00:56:54.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea12916929e6fc8-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.075 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.075 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.076 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.076 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.077 DEBUG:boto:Method: GET 2023-10-12T00:56:54.077 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.077 DEBUG:boto:Data: 2023-10-12T00:56:54.078 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.079 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.079 DEBUG:boto:Params: {} 2023-10-12T00:56:54.079 DEBUG:boto:Token: None 2023-10-12T00:56:54.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb8a95b620af7e68-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.085 DEBUG:boto:b'wafmoa-58obj000000000000000000761000falseobj000000000000000000762023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.085 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.086 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.086 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.087 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.087 DEBUG:boto:Method: GET 2023-10-12T00:56:54.088 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000076 2023-10-12T00:56:54.088 DEBUG:boto:Data: 2023-10-12T00:56:54.089 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.090 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.090 DEBUG:boto:Params: {} 2023-10-12T00:56:54.091 DEBUG:boto:Token: None 2023-10-12T00:56:54.091 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642956b21b51d46f-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.097 DEBUG:boto:b'wafmoa-58obj000000000000000000761000falseobj000000000000000000762023-10-12T00:56:30.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-12T00:56:54.098 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.098 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.099 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.099 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.100 DEBUG:boto:Method: GET 2023-10-12T00:56:54.100 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.101 DEBUG:boto:Data: 2023-10-12T00:56:54.101 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.102 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.102 DEBUG:boto:Params: {} 2023-10-12T00:56:54.103 DEBUG:boto:Token: None 2023-10-12T00:56:54.103 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM= 2023-10-12T00:56:54.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM=', 'Content-Length': '0'} 2023-10-12T00:56:54.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd72eac1b15c43a8-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.107 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.108 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.108 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.109 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.109 DEBUG:boto:Method: GET 2023-10-12T00:56:54.109 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.110 DEBUG:boto:Data: 2023-10-12T00:56:54.110 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.111 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.112 DEBUG:boto:Params: {} 2023-10-12T00:56:54.112 DEBUG:boto:Token: None 2023-10-12T00:56:54.113 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM= 2023-10-12T00:56:54.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM=', 'Content-Length': '0'} 2023-10-12T00:56:54.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005738b1bcb95877b2-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.116 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.117 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.117 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.118 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.118 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.118 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.119 DEBUG:boto:Data: 2023-10-12T00:56:54.119 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.120 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.120 DEBUG:boto:Params: {} 2023-10-12T00:56:54.121 DEBUG:boto:Token: None 2023-10-12T00:56:54.121 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.122 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1WM0v52rk3vt8wQJYwFLHqL4OlY= 2023-10-12T00:56:54.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1WM0v52rk3vt8wQJYwFLHqL4OlY=', 'Content-Length': '0'} 2023-10-12T00:56:54.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f55b93f2cae38e-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.125 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.125 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.125 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.126 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.126 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.127 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.127 DEBUG:boto:Data: 2023-10-12T00:56:54.128 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.128 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.129 DEBUG:boto:Params: {} 2023-10-12T00:56:54.129 DEBUG:boto:Token: None 2023-10-12T00:56:54.130 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.130 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1WM0v52rk3vt8wQJYwFLHqL4OlY= 2023-10-12T00:56:54.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1WM0v52rk3vt8wQJYwFLHqL4OlY=', 'Content-Length': '0'} 2023-10-12T00:56:54.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2329f54ca6942d-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-12T00:56:54.134 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.134 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.135 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.135 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.135 DEBUG:boto:Method: GET 2023-10-12T00:56:54.136 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.136 DEBUG:boto:Data: 2023-10-12T00:56:54.137 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.138 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.138 DEBUG:boto:Params: {} 2023-10-12T00:56:54.138 DEBUG:boto:Token: None 2023-10-12T00:56:54.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM= 2023-10-12T00:56:54.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM=', 'Content-Length': '0'} 2023-10-12T00:56:54.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007620d97c74cb54e6-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.186 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.187 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.187 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.188 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.188 DEBUG:boto:Method: GET 2023-10-12T00:56:54.189 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.189 DEBUG:boto:Data: 2023-10-12T00:56:54.190 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.191 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.191 DEBUG:boto:Params: {} 2023-10-12T00:56:54.192 DEBUG:boto:Token: None 2023-10-12T00:56:54.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133?versionId=null 2023-10-12T00:56:54.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM= 2023-10-12T00:56:54.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PK6TvEIICIFktJ56Ydl1APV7cJM=', 'Content-Length': '0'} 2023-10-12T00:56:54.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb29e8198be5f7a2-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.196 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.196 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.197 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.197 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.198 DEBUG:boto:Method: GET 2023-10-12T00:56:54.198 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.199 DEBUG:boto:Data: 2023-10-12T00:56:54.199 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.200 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.200 DEBUG:boto:Params: {} 2023-10-12T00:56:54.201 DEBUG:boto:Token: None 2023-10-12T00:56:54.201 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.201 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031a48f6c8ef6b114-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.205 DEBUG:boto:b'wafmoa-58obj000000000000000001331000falseobj000000000000000001332023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.206 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.206 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.207 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.207 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.208 DEBUG:boto:Method: GET 2023-10-12T00:56:54.208 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.208 DEBUG:boto:Data: 2023-10-12T00:56:54.209 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.210 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.210 DEBUG:boto:Params: {} 2023-10-12T00:56:54.210 DEBUG:boto:Token: None 2023-10-12T00:56:54.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.211 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb647d7e77616a7-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.262 DEBUG:boto:b'wafmoa-58obj000000000000000001331000falseobj000000000000000001332023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.263 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.263 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.264 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.264 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.265 DEBUG:boto:Data: 2023-10-12T00:56:54.265 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.266 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.266 DEBUG:boto:Params: {} 2023-10-12T00:56:54.267 DEBUG:boto:Token: None 2023-10-12T00:56:54.267 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xtIjJY9/9DBycC4qNeuWhIAGe+U= 2023-10-12T00:56:54.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xtIjJY9/9DBycC4qNeuWhIAGe+U=', 'Content-Length': '0'} 2023-10-12T00:56:54.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3425f90e819ea0a-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.271 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.271 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.272 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.272 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.272 DEBUG:boto:Data: 2023-10-12T00:56:54.273 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.274 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.274 DEBUG:boto:Params: {} 2023-10-12T00:56:54.274 DEBUG:boto:Token: None 2023-10-12T00:56:54.275 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xtIjJY9/9DBycC4qNeuWhIAGe+U= 2023-10-12T00:56:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xtIjJY9/9DBycC4qNeuWhIAGe+U=', 'Content-Length': '0'} 2023-10-12T00:56:54.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000843a171ff2a4b14-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-12T00:56:54.279 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.279 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.280 DEBUG:boto:Method: GET 2023-10-12T00:56:54.280 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.281 DEBUG:boto:Data: 2023-10-12T00:56:54.281 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.282 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.282 DEBUG:boto:Params: {} 2023-10-12T00:56:54.283 DEBUG:boto:Token: None 2023-10-12T00:56:54.283 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DVjnlIZpIRuKWhXux7y3mMUVF2c= 2023-10-12T00:56:54.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DVjnlIZpIRuKWhXux7y3mMUVF2c=', 'Content-Length': '0'} 2023-10-12T00:56:54.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7de6606f481dcd0-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.287 DEBUG:boto:path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.287 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.288 DEBUG:boto:Method: GET 2023-10-12T00:56:54.288 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.288 DEBUG:boto:Data: 2023-10-12T00:56:54.289 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.290 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.290 DEBUG:boto:Params: {} 2023-10-12T00:56:54.291 DEBUG:boto:Token: None 2023-10-12T00:56:54.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000133 2023-10-12T00:56:54.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DVjnlIZpIRuKWhXux7y3mMUVF2c= 2023-10-12T00:56:54.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DVjnlIZpIRuKWhXux7y3mMUVF2c=', 'Content-Length': '0'} 2023-10-12T00:56:54.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec379e303e0a0e4e-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.295 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.295 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.296 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.296 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.296 DEBUG:boto:Method: GET 2023-10-12T00:56:54.297 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.297 DEBUG:boto:Data: 2023-10-12T00:56:54.298 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.298 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.299 DEBUG:boto:Params: {} 2023-10-12T00:56:54.299 DEBUG:boto:Token: None 2023-10-12T00:56:54.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf5924172d791a7-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.304 DEBUG:boto:b'wafmoa-58obj000000000000000001331000falseobj000000000000000001332023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.305 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.305 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.306 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.306 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.306 DEBUG:boto:Method: GET 2023-10-12T00:56:54.307 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000133 2023-10-12T00:56:54.307 DEBUG:boto:Data: 2023-10-12T00:56:54.308 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.309 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.309 DEBUG:boto:Params: {} 2023-10-12T00:56:54.309 DEBUG:boto:Token: None 2023-10-12T00:56:54.310 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.310 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011562dbe9c5207ff-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.316 DEBUG:boto:b'wafmoa-58obj000000000000000001331000falseobj000000000000000001332023-10-12T00:56:34.984Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-12T00:56:54.317 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.318 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.318 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.319 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.319 DEBUG:boto:Method: GET 2023-10-12T00:56:54.319 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.320 DEBUG:boto:Data: 2023-10-12T00:56:54.320 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.321 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.321 DEBUG:boto:Params: {} 2023-10-12T00:56:54.322 DEBUG:boto:Token: None 2023-10-12T00:56:54.322 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.323 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc= 2023-10-12T00:56:54.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc=', 'Content-Length': '0'} 2023-10-12T00:56:54.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2549a01b7470339-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.366 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.368 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.368 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.368 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.369 DEBUG:boto:Method: GET 2023-10-12T00:56:54.369 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.370 DEBUG:boto:Data: 2023-10-12T00:56:54.370 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.371 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.372 DEBUG:boto:Params: {} 2023-10-12T00:56:54.372 DEBUG:boto:Token: None 2023-10-12T00:56:54.373 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc= 2023-10-12T00:56:54.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc=', 'Content-Length': '0'} 2023-10-12T00:56:54.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5c78774462c771-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.377 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.377 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.378 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.378 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.378 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.379 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.379 DEBUG:boto:Data: 2023-10-12T00:56:54.380 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.380 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.381 DEBUG:boto:Params: {} 2023-10-12T00:56:54.381 DEBUG:boto:Token: None 2023-10-12T00:56:54.382 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cpeZwZckqhZ0dHBwZ+QSyzUg7as= 2023-10-12T00:56:54.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cpeZwZckqhZ0dHBwZ+QSyzUg7as=', 'Content-Length': '0'} 2023-10-12T00:56:54.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661758c2752ce0af-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.386 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.386 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.387 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.387 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.388 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.388 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.389 DEBUG:boto:Data: 2023-10-12T00:56:54.389 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.390 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.391 DEBUG:boto:Params: {} 2023-10-12T00:56:54.391 DEBUG:boto:Token: None 2023-10-12T00:56:54.392 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cpeZwZckqhZ0dHBwZ+QSyzUg7as= 2023-10-12T00:56:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cpeZwZckqhZ0dHBwZ+QSyzUg7as=', 'Content-Length': '0'} 2023-10-12T00:56:54.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c851f86ddc9292-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-12T00:56:54.396 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.397 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.397 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.398 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.398 DEBUG:boto:Method: GET 2023-10-12T00:56:54.399 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.399 DEBUG:boto:Data: 2023-10-12T00:56:54.400 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.401 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.401 DEBUG:boto:Params: {} 2023-10-12T00:56:54.402 DEBUG:boto:Token: None 2023-10-12T00:56:54.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc= 2023-10-12T00:56:54.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc=', 'Content-Length': '0'} 2023-10-12T00:56:54.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000696753b057b2f53b-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.406 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.406 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.407 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.407 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.408 DEBUG:boto:Method: GET 2023-10-12T00:56:54.408 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.409 DEBUG:boto:Data: 2023-10-12T00:56:54.409 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.410 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.410 DEBUG:boto:Params: {} 2023-10-12T00:56:54.411 DEBUG:boto:Token: None 2023-10-12T00:56:54.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162?versionId=null 2023-10-12T00:56:54.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc= 2023-10-12T00:56:54.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bx9ZcxWZMaw+U9z+7426ryKjfxc=', 'Content-Length': '0'} 2023-10-12T00:56:54.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a56f0257a26c13-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.458 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.459 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.459 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.460 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.460 DEBUG:boto:Method: GET 2023-10-12T00:56:54.461 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.461 DEBUG:boto:Data: 2023-10-12T00:56:54.462 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.463 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.463 DEBUG:boto:Params: {} 2023-10-12T00:56:54.464 DEBUG:boto:Token: None 2023-10-12T00:56:54.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077739bfc303f0f42-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.468 DEBUG:boto:b'wafmoa-58obj000000000000000001621000falseobj000000000000000001622023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.469 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.469 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.470 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.470 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.471 DEBUG:boto:Method: GET 2023-10-12T00:56:54.471 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.472 DEBUG:boto:Data: 2023-10-12T00:56:54.472 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.473 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.473 DEBUG:boto:Params: {} 2023-10-12T00:56:54.474 DEBUG:boto:Token: None 2023-10-12T00:56:54.474 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.475 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d66b0401bdff9b01-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.480 DEBUG:boto:b'wafmoa-58obj000000000000000001621000falseobj000000000000000001622023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.481 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.481 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.482 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.482 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.483 DEBUG:boto:Data: 2023-10-12T00:56:54.483 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.484 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.484 DEBUG:boto:Params: {} 2023-10-12T00:56:54.485 DEBUG:boto:Token: None 2023-10-12T00:56:54.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cahFVb8QstVuMfeCKzKTBOioeko= 2023-10-12T00:56:54.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cahFVb8QstVuMfeCKzKTBOioeko=', 'Content-Length': '0'} 2023-10-12T00:56:54.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042754504383849eb-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.489 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.489 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.490 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.490 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.491 DEBUG:boto:Data: 2023-10-12T00:56:54.491 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.492 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.492 DEBUG:boto:Params: {} 2023-10-12T00:56:54.493 DEBUG:boto:Token: None 2023-10-12T00:56:54.493 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cahFVb8QstVuMfeCKzKTBOioeko= 2023-10-12T00:56:54.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cahFVb8QstVuMfeCKzKTBOioeko=', 'Content-Length': '0'} 2023-10-12T00:56:54.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000901e9d15533dc816-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-12T00:56:54.497 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.498 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.498 DEBUG:boto:Method: GET 2023-10-12T00:56:54.499 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.499 DEBUG:boto:Data: 2023-10-12T00:56:54.500 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.501 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.501 DEBUG:boto:Params: {} 2023-10-12T00:56:54.502 DEBUG:boto:Token: None 2023-10-12T00:56:54.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.503 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OJwrPCFywgpfT/M7+clouyASdrA= 2023-10-12T00:56:54.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OJwrPCFywgpfT/M7+clouyASdrA=', 'Content-Length': '0'} 2023-10-12T00:56:54.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e05144c46dd713e-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.506 DEBUG:boto:path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.507 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.507 DEBUG:boto:Method: GET 2023-10-12T00:56:54.508 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.508 DEBUG:boto:Data: 2023-10-12T00:56:54.509 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.509 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.510 DEBUG:boto:Params: {} 2023-10-12T00:56:54.510 DEBUG:boto:Token: None 2023-10-12T00:56:54.511 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000162 2023-10-12T00:56:54.511 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OJwrPCFywgpfT/M7+clouyASdrA= 2023-10-12T00:56:54.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OJwrPCFywgpfT/M7+clouyASdrA=', 'Content-Length': '0'} 2023-10-12T00:56:54.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a640c69f37b98a-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.515 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.515 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.516 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.516 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.517 DEBUG:boto:Method: GET 2023-10-12T00:56:54.517 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.518 DEBUG:boto:Data: 2023-10-12T00:56:54.518 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.519 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.519 DEBUG:boto:Params: {} 2023-10-12T00:56:54.520 DEBUG:boto:Token: None 2023-10-12T00:56:54.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009bd307ca0800520-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.525 DEBUG:boto:b'wafmoa-58obj000000000000000001621000falseobj000000000000000001622023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.525 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.526 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.527 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.527 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.527 DEBUG:boto:Method: GET 2023-10-12T00:56:54.528 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000162 2023-10-12T00:56:54.528 DEBUG:boto:Data: 2023-10-12T00:56:54.529 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.529 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.530 DEBUG:boto:Params: {} 2023-10-12T00:56:54.530 DEBUG:boto:Token: None 2023-10-12T00:56:54.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ec0e8db7afeab1-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.536 DEBUG:boto:b'wafmoa-58obj000000000000000001621000falseobj000000000000000001622023-10-12T00:56:35.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-12T00:56:54.537 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.538 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.538 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.538 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.539 DEBUG:boto:Method: GET 2023-10-12T00:56:54.539 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.540 DEBUG:boto:Data: 2023-10-12T00:56:54.540 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.541 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.541 DEBUG:boto:Params: {} 2023-10-12T00:56:54.542 DEBUG:boto:Token: None 2023-10-12T00:56:54.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0= 2023-10-12T00:56:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0=', 'Content-Length': '0'} 2023-10-12T00:56:54.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f404f18911220f0-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.586 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.587 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.587 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.588 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.588 DEBUG:boto:Method: GET 2023-10-12T00:56:54.589 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.589 DEBUG:boto:Data: 2023-10-12T00:56:54.590 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.591 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.591 DEBUG:boto:Params: {} 2023-10-12T00:56:54.592 DEBUG:boto:Token: None 2023-10-12T00:56:54.592 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0= 2023-10-12T00:56:54.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0=', 'Content-Length': '0'} 2023-10-12T00:56:54.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000018903e70e9a98ff-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.596 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.597 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.597 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.598 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.598 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.598 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.599 DEBUG:boto:Data: 2023-10-12T00:56:54.599 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.600 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.601 DEBUG:boto:Params: {} 2023-10-12T00:56:54.601 DEBUG:boto:Token: None 2023-10-12T00:56:54.601 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.602 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oZrzNgNGWNcoad1hQYnIaJILxQc= 2023-10-12T00:56:54.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oZrzNgNGWNcoad1hQYnIaJILxQc=', 'Content-Length': '0'} 2023-10-12T00:56:54.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c32bffa9f223c1-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.605 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.605 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.606 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.606 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.607 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.607 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.607 DEBUG:boto:Data: 2023-10-12T00:56:54.608 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.609 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.609 DEBUG:boto:Params: {} 2023-10-12T00:56:54.610 DEBUG:boto:Token: None 2023-10-12T00:56:54.610 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oZrzNgNGWNcoad1hQYnIaJILxQc= 2023-10-12T00:56:54.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oZrzNgNGWNcoad1hQYnIaJILxQc=', 'Content-Length': '0'} 2023-10-12T00:56:54.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d451c19335c4014-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-12T00:56:54.614 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.614 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.615 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.615 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.616 DEBUG:boto:Method: GET 2023-10-12T00:56:54.616 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.617 DEBUG:boto:Data: 2023-10-12T00:56:54.617 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.618 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.618 DEBUG:boto:Params: {} 2023-10-12T00:56:54.619 DEBUG:boto:Token: None 2023-10-12T00:56:54.619 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0= 2023-10-12T00:56:54.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0=', 'Content-Length': '0'} 2023-10-12T00:56:54.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea19485e9b5b0913-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.623 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.624 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.624 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.625 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.625 DEBUG:boto:Method: GET 2023-10-12T00:56:54.625 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.626 DEBUG:boto:Data: 2023-10-12T00:56:54.626 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.627 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.628 DEBUG:boto:Params: {} 2023-10-12T00:56:54.628 DEBUG:boto:Token: None 2023-10-12T00:56:54.629 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271?versionId=null 2023-10-12T00:56:54.629 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0= 2023-10-12T00:56:54.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5yZXgYeYlbpB1Hh7ENQPhVZmnv0=', 'Content-Length': '0'} 2023-10-12T00:56:54.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f8a1bff4a44946-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.633 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.633 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.634 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.634 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.635 DEBUG:boto:Method: GET 2023-10-12T00:56:54.635 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.635 DEBUG:boto:Data: 2023-10-12T00:56:54.636 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.637 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.637 DEBUG:boto:Params: {} 2023-10-12T00:56:54.638 DEBUG:boto:Token: None 2023-10-12T00:56:54.638 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3872464ccb39da8-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.643 DEBUG:boto:b'wafmoa-58obj000000000000000002711000falseobj000000000000000002712023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.643 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.644 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.644 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.645 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.645 DEBUG:boto:Method: GET 2023-10-12T00:56:54.646 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.646 DEBUG:boto:Data: 2023-10-12T00:56:54.647 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.648 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.648 DEBUG:boto:Params: {} 2023-10-12T00:56:54.648 DEBUG:boto:Token: None 2023-10-12T00:56:54.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac52b504229244a-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.655 DEBUG:boto:b'wafmoa-58obj000000000000000002711000falseobj000000000000000002712023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.656 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.656 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.657 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.657 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.657 DEBUG:boto:Data: 2023-10-12T00:56:54.658 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.659 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.659 DEBUG:boto:Params: {} 2023-10-12T00:56:54.659 DEBUG:boto:Token: None 2023-10-12T00:56:54.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlbBD3WTy9l0WdoSFNUItnAJxpY= 2023-10-12T00:56:54.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlbBD3WTy9l0WdoSFNUItnAJxpY=', 'Content-Length': '0'} 2023-10-12T00:56:54.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b5abb35161dcf3-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.664 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.664 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.664 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.665 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.665 DEBUG:boto:Data: 2023-10-12T00:56:54.666 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.667 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.667 DEBUG:boto:Params: {} 2023-10-12T00:56:54.667 DEBUG:boto:Token: None 2023-10-12T00:56:54.668 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MlbBD3WTy9l0WdoSFNUItnAJxpY= 2023-10-12T00:56:54.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MlbBD3WTy9l0WdoSFNUItnAJxpY=', 'Content-Length': '0'} 2023-10-12T00:56:54.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aea2f47fde5dcd8-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-12T00:56:54.672 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.672 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.673 DEBUG:boto:Method: GET 2023-10-12T00:56:54.673 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.674 DEBUG:boto:Data: 2023-10-12T00:56:54.674 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.675 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.675 DEBUG:boto:Params: {} 2023-10-12T00:56:54.676 DEBUG:boto:Token: None 2023-10-12T00:56:54.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sWue3iNAtyTvzyzYTRyLYsQN2kk= 2023-10-12T00:56:54.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sWue3iNAtyTvzyzYTRyLYsQN2kk=', 'Content-Length': '0'} 2023-10-12T00:56:54.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6059ad95bdf8bcd-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.722 DEBUG:boto:path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.723 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.723 DEBUG:boto:Method: GET 2023-10-12T00:56:54.724 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.724 DEBUG:boto:Data: 2023-10-12T00:56:54.725 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.726 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.726 DEBUG:boto:Params: {} 2023-10-12T00:56:54.727 DEBUG:boto:Token: None 2023-10-12T00:56:54.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000271 2023-10-12T00:56:54.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sWue3iNAtyTvzyzYTRyLYsQN2kk= 2023-10-12T00:56:54.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sWue3iNAtyTvzyzYTRyLYsQN2kk=', 'Content-Length': '0'} 2023-10-12T00:56:54.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a559eae8d20777-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.731 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.731 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.732 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.732 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.732 DEBUG:boto:Method: GET 2023-10-12T00:56:54.733 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.733 DEBUG:boto:Data: 2023-10-12T00:56:54.734 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.735 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.735 DEBUG:boto:Params: {} 2023-10-12T00:56:54.736 DEBUG:boto:Token: None 2023-10-12T00:56:54.736 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052442c972a8783a0-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.741 DEBUG:boto:b'wafmoa-58obj000000000000000002711000falseobj000000000000000002712023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.741 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.742 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.742 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.743 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.743 DEBUG:boto:Method: GET 2023-10-12T00:56:54.744 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000271 2023-10-12T00:56:54.744 DEBUG:boto:Data: 2023-10-12T00:56:54.745 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.746 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.746 DEBUG:boto:Params: {} 2023-10-12T00:56:54.746 DEBUG:boto:Token: None 2023-10-12T00:56:54.747 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ac876a0b2b0fa8-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.753 DEBUG:boto:b'wafmoa-58obj000000000000000002711000falseobj000000000000000002712023-10-12T00:56:35.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-12T00:56:54.754 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.755 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.755 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.756 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.756 DEBUG:boto:Method: GET 2023-10-12T00:56:54.757 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.757 DEBUG:boto:Data: 2023-10-12T00:56:54.757 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.758 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.759 DEBUG:boto:Params: {} 2023-10-12T00:56:54.759 DEBUG:boto:Token: None 2023-10-12T00:56:54.760 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o= 2023-10-12T00:56:54.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o=', 'Content-Length': '0'} 2023-10-12T00:56:54.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543d2347a2bb262b-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.764 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.765 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.765 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.765 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.766 DEBUG:boto:Method: GET 2023-10-12T00:56:54.766 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.767 DEBUG:boto:Data: 2023-10-12T00:56:54.767 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.768 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.768 DEBUG:boto:Params: {} 2023-10-12T00:56:54.769 DEBUG:boto:Token: None 2023-10-12T00:56:54.769 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o= 2023-10-12T00:56:54.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o=', 'Content-Length': '0'} 2023-10-12T00:56:54.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdf31236cacd4e2-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.773 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.774 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.774 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.774 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.775 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.775 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.776 DEBUG:boto:Data: 2023-10-12T00:56:54.776 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.777 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.778 DEBUG:boto:Params: {} 2023-10-12T00:56:54.778 DEBUG:boto:Token: None 2023-10-12T00:56:54.779 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ouG+79Px9s7YwwS3h8MMppDXpzQ= 2023-10-12T00:56:54.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ouG+79Px9s7YwwS3h8MMppDXpzQ=', 'Content-Length': '0'} 2023-10-12T00:56:54.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019174267a7b4c573-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.782 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.783 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.783 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.784 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.784 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.785 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.785 DEBUG:boto:Data: 2023-10-12T00:56:54.785 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.786 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.787 DEBUG:boto:Params: {} 2023-10-12T00:56:54.787 DEBUG:boto:Token: None 2023-10-12T00:56:54.788 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.788 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ouG+79Px9s7YwwS3h8MMppDXpzQ= 2023-10-12T00:56:54.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ouG+79Px9s7YwwS3h8MMppDXpzQ=', 'Content-Length': '0'} 2023-10-12T00:56:54.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000582eb56264533705-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-12T00:56:54.792 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.792 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.793 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.793 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.794 DEBUG:boto:Method: GET 2023-10-12T00:56:54.794 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.795 DEBUG:boto:Data: 2023-10-12T00:56:54.795 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.796 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.796 DEBUG:boto:Params: {} 2023-10-12T00:56:54.797 DEBUG:boto:Token: None 2023-10-12T00:56:54.797 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o= 2023-10-12T00:56:54.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o=', 'Content-Length': '0'} 2023-10-12T00:56:54.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005121a5eb2ba290be-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.842 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.843 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.843 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.844 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.844 DEBUG:boto:Method: GET 2023-10-12T00:56:54.845 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.845 DEBUG:boto:Data: 2023-10-12T00:56:54.846 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.847 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.847 DEBUG:boto:Params: {} 2023-10-12T00:56:54.848 DEBUG:boto:Token: None 2023-10-12T00:56:54.848 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283?versionId=null 2023-10-12T00:56:54.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o= 2023-10-12T00:56:54.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5UoLPPnE/G3igriSFBkSmsWmT7o=', 'Content-Length': '0'} 2023-10-12T00:56:54.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b9689b80223077-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.852 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.852 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.853 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.853 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.854 DEBUG:boto:Method: GET 2023-10-12T00:56:54.854 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.855 DEBUG:boto:Data: 2023-10-12T00:56:54.855 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.856 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.856 DEBUG:boto:Params: {} 2023-10-12T00:56:54.857 DEBUG:boto:Token: None 2023-10-12T00:56:54.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725c8236e5618672-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.861 DEBUG:boto:b'wafmoa-58obj000000000000000002831000falseobj000000000000000002832023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.862 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.863 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.863 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.863 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.864 DEBUG:boto:Method: GET 2023-10-12T00:56:54.864 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.865 DEBUG:boto:Data: 2023-10-12T00:56:54.865 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.866 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.866 DEBUG:boto:Params: {} 2023-10-12T00:56:54.867 DEBUG:boto:Token: None 2023-10-12T00:56:54.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e84338518cc0a8-0065274456-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.914 DEBUG:boto:b'wafmoa-58obj000000000000000002831000falseobj000000000000000002832023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:54.915 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.916 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.916 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.916 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.917 DEBUG:boto:Data: 2023-10-12T00:56:54.917 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.918 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.919 DEBUG:boto:Params: {} 2023-10-12T00:56:54.919 DEBUG:boto:Token: None 2023-10-12T00:56:54.920 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v22X/7esXs4OB/jXmvqqh8Zk2OE= 2023-10-12T00:56:54.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v22X/7esXs4OB/jXmvqqh8Zk2OE=', 'Content-Length': '0'} 2023-10-12T00:56:54.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e66e6df320ef35-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.923 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.924 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.924 DEBUG:boto:Method: HEAD 2023-10-12T00:56:54.924 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.925 DEBUG:boto:Data: 2023-10-12T00:56:54.925 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.926 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.927 DEBUG:boto:Params: {} 2023-10-12T00:56:54.927 DEBUG:boto:Token: None 2023-10-12T00:56:54.928 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v22X/7esXs4OB/jXmvqqh8Zk2OE= 2023-10-12T00:56:54.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v22X/7esXs4OB/jXmvqqh8Zk2OE=', 'Content-Length': '0'} 2023-10-12T00:56:54.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ed89b3e220deacc-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-12T00:56:54.931 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.932 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.932 DEBUG:boto:Method: GET 2023-10-12T00:56:54.933 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.933 DEBUG:boto:Data: 2023-10-12T00:56:54.934 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.934 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.935 DEBUG:boto:Params: {} 2023-10-12T00:56:54.935 DEBUG:boto:Token: None 2023-10-12T00:56:54.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XzJ9wWKMSBJQalJYAfyBbD4pKJk= 2023-10-12T00:56:54.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XzJ9wWKMSBJQalJYAfyBbD4pKJk=', 'Content-Length': '0'} 2023-10-12T00:56:54.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9710b49139df7c-0065274456-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.940 DEBUG:boto:path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.940 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.941 DEBUG:boto:Method: GET 2023-10-12T00:56:54.941 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.942 DEBUG:boto:Data: 2023-10-12T00:56:54.942 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:54.943 DEBUG:boto:Port: 8000 2023-10-12T00:56:54.943 DEBUG:boto:Params: {} 2023-10-12T00:56:54.944 DEBUG:boto:Token: None 2023-10-12T00:56:54.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/obj00000000000000000283 2023-10-12T00:56:54.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XzJ9wWKMSBJQalJYAfyBbD4pKJk= 2023-10-12T00:56:54.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XzJ9wWKMSBJQalJYAfyBbD4pKJk=', 'Content-Length': '0'} 2023-10-12T00:56:54.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4aa8e9d8065859-0065274456-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:54.948 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:54.948 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:54.949 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.949 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.950 DEBUG:boto:Method: GET 2023-10-12T00:56:54.950 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:54.951 DEBUG:boto:Data: 2023-10-12T00:56:54.951 DEBUG:boto:Headers: {} 2023-10-12T00:56:54.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:54.952 DEBUG:boto:Port: 8001 2023-10-12T00:56:54.952 DEBUG:boto:Params: {} 2023-10-12T00:56:54.953 DEBUG:boto:Token: None 2023-10-12T00:56:54.953 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:54 GMT /wafmoa-58/ 2023-10-12T00:56:54.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY= 2023-10-12T00:56:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NbaIE7YdElDtNyeq5F2Y/bOPehY=', 'Content-Length': '0'} 2023-10-12T00:56:54.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b262c7a81cd410f-0065274456-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.002 DEBUG:boto:b'wafmoa-58obj000000000000000002831000falseobj000000000000000002832023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.003 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.004 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.004 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:55.005 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:55.005 DEBUG:boto:Method: GET 2023-10-12T00:56:55.005 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000283 2023-10-12T00:56:55.006 DEBUG:boto:Data: 2023-10-12T00:56:55.006 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.007 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.008 DEBUG:boto:Params: {} 2023-10-12T00:56:55.008 DEBUG:boto:Token: None 2023-10-12T00:56:55.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422fce319856394b-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.014 DEBUG:boto:b'wafmoa-58obj000000000000000002831000falseobj000000000000000002832023-10-12T00:56:35.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-12T00:56:55.016 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.016 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.016 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.017 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.017 DEBUG:boto:Method: GET 2023-10-12T00:56:55.018 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.018 DEBUG:boto:Data: 2023-10-12T00:56:55.019 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.020 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.020 DEBUG:boto:Params: {} 2023-10-12T00:56:55.021 DEBUG:boto:Token: None 2023-10-12T00:56:55.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg= 2023-10-12T00:56:55.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg=', 'Content-Length': '0'} 2023-10-12T00:56:55.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d0e2f75488a42e-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.025 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.026 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.026 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.027 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.027 DEBUG:boto:Method: GET 2023-10-12T00:56:55.028 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.028 DEBUG:boto:Data: 2023-10-12T00:56:55.029 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.029 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.030 DEBUG:boto:Params: {} 2023-10-12T00:56:55.030 DEBUG:boto:Token: None 2023-10-12T00:56:55.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.031 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg= 2023-10-12T00:56:55.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg=', 'Content-Length': '0'} 2023-10-12T00:56:55.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274e5ea97718c51c-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.035 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.035 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.035 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.036 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.036 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.037 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.037 DEBUG:boto:Data: 2023-10-12T00:56:55.037 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.038 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.039 DEBUG:boto:Params: {} 2023-10-12T00:56:55.039 DEBUG:boto:Token: None 2023-10-12T00:56:55.040 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rmXJqg9xcmEruC5emWaVyqgU/Zk= 2023-10-12T00:56:55.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rmXJqg9xcmEruC5emWaVyqgU/Zk=', 'Content-Length': '0'} 2023-10-12T00:56:55.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1497fe81bb61839-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.043 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.044 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.044 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.044 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.045 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.045 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.046 DEBUG:boto:Data: 2023-10-12T00:56:55.046 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.047 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.047 DEBUG:boto:Params: {} 2023-10-12T00:56:55.048 DEBUG:boto:Token: None 2023-10-12T00:56:55.048 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rmXJqg9xcmEruC5emWaVyqgU/Zk= 2023-10-12T00:56:55.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rmXJqg9xcmEruC5emWaVyqgU/Zk=', 'Content-Length': '0'} 2023-10-12T00:56:55.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e62958bdb1da5b-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-12T00:56:55.052 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.053 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.053 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.054 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.054 DEBUG:boto:Method: GET 2023-10-12T00:56:55.055 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.055 DEBUG:boto:Data: 2023-10-12T00:56:55.055 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.056 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.057 DEBUG:boto:Params: {} 2023-10-12T00:56:55.057 DEBUG:boto:Token: None 2023-10-12T00:56:55.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg= 2023-10-12T00:56:55.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg=', 'Content-Length': '0'} 2023-10-12T00:56:55.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052923889b1340f89-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.062 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.062 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.063 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.063 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.063 DEBUG:boto:Method: GET 2023-10-12T00:56:55.064 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.064 DEBUG:boto:Data: 2023-10-12T00:56:55.065 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.066 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.066 DEBUG:boto:Params: {} 2023-10-12T00:56:55.067 DEBUG:boto:Token: None 2023-10-12T00:56:55.067 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317?versionId=null 2023-10-12T00:56:55.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg= 2023-10-12T00:56:55.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0RUk8aztxlnJLUHfOPTZIP4g/Xg=', 'Content-Length': '0'} 2023-10-12T00:56:55.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844de75e1bec4e6c-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.071 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.071 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.072 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.072 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.073 DEBUG:boto:Method: GET 2023-10-12T00:56:55.073 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.073 DEBUG:boto:Data: 2023-10-12T00:56:55.074 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.075 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.075 DEBUG:boto:Params: {} 2023-10-12T00:56:55.076 DEBUG:boto:Token: None 2023-10-12T00:56:55.076 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.077 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd2b4e0d97792d66-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.080 DEBUG:boto:b'wafmoa-58obj000000000000000003171000falseobj000000000000000003172023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.081 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.081 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.082 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.082 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.082 DEBUG:boto:Method: GET 2023-10-12T00:56:55.083 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.083 DEBUG:boto:Data: 2023-10-12T00:56:55.084 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.085 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.085 DEBUG:boto:Params: {} 2023-10-12T00:56:55.086 DEBUG:boto:Token: None 2023-10-12T00:56:55.086 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007654059833286933-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.134 DEBUG:boto:b'wafmoa-58obj000000000000000003171000falseobj000000000000000003172023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.135 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.136 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.136 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.137 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.137 DEBUG:boto:Data: 2023-10-12T00:56:55.138 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.139 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.139 DEBUG:boto:Params: {} 2023-10-12T00:56:55.139 DEBUG:boto:Token: None 2023-10-12T00:56:55.140 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hFcOgLjOfpBuL32F0qkp9udUueY= 2023-10-12T00:56:55.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hFcOgLjOfpBuL32F0qkp9udUueY=', 'Content-Length': '0'} 2023-10-12T00:56:55.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b3942918bc1e68-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.144 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.144 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.145 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.145 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.146 DEBUG:boto:Data: 2023-10-12T00:56:55.146 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.147 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.147 DEBUG:boto:Params: {} 2023-10-12T00:56:55.148 DEBUG:boto:Token: None 2023-10-12T00:56:55.148 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hFcOgLjOfpBuL32F0qkp9udUueY= 2023-10-12T00:56:55.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hFcOgLjOfpBuL32F0qkp9udUueY=', 'Content-Length': '0'} 2023-10-12T00:56:55.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004495057514dfbfc3-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-12T00:56:55.152 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.153 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.153 DEBUG:boto:Method: GET 2023-10-12T00:56:55.154 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.154 DEBUG:boto:Data: 2023-10-12T00:56:55.155 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.156 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.156 DEBUG:boto:Params: {} 2023-10-12T00:56:55.157 DEBUG:boto:Token: None 2023-10-12T00:56:55.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eF/d4I/A5TYVGLkC2QKWN0Nv1QI= 2023-10-12T00:56:55.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eF/d4I/A5TYVGLkC2QKWN0Nv1QI=', 'Content-Length': '0'} 2023-10-12T00:56:55.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b52dfb7766fb9e13-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.206 DEBUG:boto:path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.207 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.208 DEBUG:boto:Method: GET 2023-10-12T00:56:55.208 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.208 DEBUG:boto:Data: 2023-10-12T00:56:55.209 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.210 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.210 DEBUG:boto:Params: {} 2023-10-12T00:56:55.211 DEBUG:boto:Token: None 2023-10-12T00:56:55.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000317 2023-10-12T00:56:55.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eF/d4I/A5TYVGLkC2QKWN0Nv1QI= 2023-10-12T00:56:55.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eF/d4I/A5TYVGLkC2QKWN0Nv1QI=', 'Content-Length': '0'} 2023-10-12T00:56:55.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897f0b5ec18cfd64-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.216 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.216 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.217 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.217 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.218 DEBUG:boto:Method: GET 2023-10-12T00:56:55.218 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.219 DEBUG:boto:Data: 2023-10-12T00:56:55.219 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.220 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.220 DEBUG:boto:Params: {} 2023-10-12T00:56:55.221 DEBUG:boto:Token: None 2023-10-12T00:56:55.221 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33196002c1a20fe-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.226 DEBUG:boto:b'wafmoa-58obj000000000000000003171000falseobj000000000000000003172023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.226 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.227 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.227 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.228 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.228 DEBUG:boto:Method: GET 2023-10-12T00:56:55.229 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000317 2023-10-12T00:56:55.229 DEBUG:boto:Data: 2023-10-12T00:56:55.230 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.231 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.231 DEBUG:boto:Params: {} 2023-10-12T00:56:55.232 DEBUG:boto:Token: None 2023-10-12T00:56:55.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016647a08e37e3a38-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.237 DEBUG:boto:b'wafmoa-58obj000000000000000003171000falseobj000000000000000003172023-10-12T00:56:34.752Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-12T00:56:55.239 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.239 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.239 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.240 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.240 DEBUG:boto:Method: GET 2023-10-12T00:56:55.241 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.241 DEBUG:boto:Data: 2023-10-12T00:56:55.242 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.242 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.243 DEBUG:boto:Params: {} 2023-10-12T00:56:55.243 DEBUG:boto:Token: None 2023-10-12T00:56:55.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA= 2023-10-12T00:56:55.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA=', 'Content-Length': '0'} 2023-10-12T00:56:55.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a976a266b2cee2-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.248 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.249 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.249 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.249 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.250 DEBUG:boto:Method: GET 2023-10-12T00:56:55.251 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.251 DEBUG:boto:Data: 2023-10-12T00:56:55.251 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.252 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.253 DEBUG:boto:Params: {} 2023-10-12T00:56:55.253 DEBUG:boto:Token: None 2023-10-12T00:56:55.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA= 2023-10-12T00:56:55.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA=', 'Content-Length': '0'} 2023-10-12T00:56:55.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8f0d8d5549a1ef-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.298 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.299 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.300 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.300 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.300 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.301 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.301 DEBUG:boto:Data: 2023-10-12T00:56:55.302 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.303 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.303 DEBUG:boto:Params: {} 2023-10-12T00:56:55.304 DEBUG:boto:Token: None 2023-10-12T00:56:55.304 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CLhnKp+OhMdEnT0YTBkT0FZbhtw= 2023-10-12T00:56:55.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CLhnKp+OhMdEnT0YTBkT0FZbhtw=', 'Content-Length': '0'} 2023-10-12T00:56:55.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7da420dd500b0a-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.309 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.309 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.310 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.310 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.311 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.311 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.312 DEBUG:boto:Data: 2023-10-12T00:56:55.312 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.313 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.313 DEBUG:boto:Params: {} 2023-10-12T00:56:55.314 DEBUG:boto:Token: None 2023-10-12T00:56:55.314 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CLhnKp+OhMdEnT0YTBkT0FZbhtw= 2023-10-12T00:56:55.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CLhnKp+OhMdEnT0YTBkT0FZbhtw=', 'Content-Length': '0'} 2023-10-12T00:56:55.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a10f757df799e12b-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-12T00:56:55.319 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.319 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.320 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.320 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.321 DEBUG:boto:Method: GET 2023-10-12T00:56:55.321 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.321 DEBUG:boto:Data: 2023-10-12T00:56:55.322 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.323 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.324 DEBUG:boto:Params: {} 2023-10-12T00:56:55.325 DEBUG:boto:Token: None 2023-10-12T00:56:55.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA= 2023-10-12T00:56:55.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA=', 'Content-Length': '0'} 2023-10-12T00:56:55.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0112df90f82f6d-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.331 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.331 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.332 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.332 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.332 DEBUG:boto:Method: GET 2023-10-12T00:56:55.333 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.333 DEBUG:boto:Data: 2023-10-12T00:56:55.334 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.335 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.335 DEBUG:boto:Params: {} 2023-10-12T00:56:55.336 DEBUG:boto:Token: None 2023-10-12T00:56:55.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423?versionId=null 2023-10-12T00:56:55.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA= 2023-10-12T00:56:55.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dfLnDXMYkU2Wg1AbLoYIdyZiFWA=', 'Content-Length': '0'} 2023-10-12T00:56:55.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073447bd7dd2d12ea-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.340 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.341 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.341 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.342 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.343 DEBUG:boto:Method: GET 2023-10-12T00:56:55.344 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.345 DEBUG:boto:Data: 2023-10-12T00:56:55.345 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.347 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.347 DEBUG:boto:Params: {} 2023-10-12T00:56:55.348 DEBUG:boto:Token: None 2023-10-12T00:56:55.349 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df8bdee2317fcac3-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.354 DEBUG:boto:b'wafmoa-58obj000000000000000004231000falseobj000000000000000004232023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.355 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.355 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.355 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.356 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.356 DEBUG:boto:Method: GET 2023-10-12T00:56:55.357 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.357 DEBUG:boto:Data: 2023-10-12T00:56:55.358 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.359 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.359 DEBUG:boto:Params: {} 2023-10-12T00:56:55.360 DEBUG:boto:Token: None 2023-10-12T00:56:55.360 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f59a4069ed90aab3-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.366 DEBUG:boto:b'wafmoa-58obj000000000000000004231000falseobj000000000000000004232023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.366 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.367 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.367 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.367 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.368 DEBUG:boto:Data: 2023-10-12T00:56:55.368 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.369 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.370 DEBUG:boto:Params: {} 2023-10-12T00:56:55.370 DEBUG:boto:Token: None 2023-10-12T00:56:55.370 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35D3X+95FpAQg1JpNu84f8l8DOU= 2023-10-12T00:56:55.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35D3X+95FpAQg1JpNu84f8l8DOU=', 'Content-Length': '0'} 2023-10-12T00:56:55.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e418663beb61bbcc-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.374 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.374 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.375 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.375 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.376 DEBUG:boto:Data: 2023-10-12T00:56:55.377 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.378 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.379 DEBUG:boto:Params: {} 2023-10-12T00:56:55.380 DEBUG:boto:Token: None 2023-10-12T00:56:55.381 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.382 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35D3X+95FpAQg1JpNu84f8l8DOU= 2023-10-12T00:56:55.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35D3X+95FpAQg1JpNu84f8l8DOU=', 'Content-Length': '0'} 2023-10-12T00:56:55.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffec72e1f926b3f9-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-12T00:56:55.389 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.390 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.391 DEBUG:boto:Method: GET 2023-10-12T00:56:55.392 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.392 DEBUG:boto:Data: 2023-10-12T00:56:55.393 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.395 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.395 DEBUG:boto:Params: {} 2023-10-12T00:56:55.396 DEBUG:boto:Token: None 2023-10-12T00:56:55.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NXYgoVQ3TeEmEy5K0rGGMyqJm+w= 2023-10-12T00:56:55.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NXYgoVQ3TeEmEy5K0rGGMyqJm+w=', 'Content-Length': '0'} 2023-10-12T00:56:55.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91f851d843e5449-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.402 DEBUG:boto:path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.402 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.403 DEBUG:boto:Method: GET 2023-10-12T00:56:55.403 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.404 DEBUG:boto:Data: 2023-10-12T00:56:55.404 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.405 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.405 DEBUG:boto:Params: {} 2023-10-12T00:56:55.406 DEBUG:boto:Token: None 2023-10-12T00:56:55.406 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000423 2023-10-12T00:56:55.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NXYgoVQ3TeEmEy5K0rGGMyqJm+w= 2023-10-12T00:56:55.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NXYgoVQ3TeEmEy5K0rGGMyqJm+w=', 'Content-Length': '0'} 2023-10-12T00:56:55.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ca0bb198660191-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.410 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.411 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.411 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.412 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.412 DEBUG:boto:Method: GET 2023-10-12T00:56:55.413 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.413 DEBUG:boto:Data: 2023-10-12T00:56:55.414 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.415 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.415 DEBUG:boto:Params: {} 2023-10-12T00:56:55.416 DEBUG:boto:Token: None 2023-10-12T00:56:55.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955c94c98ce8837e-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.420 DEBUG:boto:b'wafmoa-58obj000000000000000004231000falseobj000000000000000004232023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.421 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.422 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.423 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.423 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.424 DEBUG:boto:Method: GET 2023-10-12T00:56:55.424 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000423 2023-10-12T00:56:55.425 DEBUG:boto:Data: 2023-10-12T00:56:55.425 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.426 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.427 DEBUG:boto:Params: {} 2023-10-12T00:56:55.427 DEBUG:boto:Token: None 2023-10-12T00:56:55.427 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db932a1b4719b02a-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.433 DEBUG:boto:b'wafmoa-58obj000000000000000004231000falseobj000000000000000004232023-10-12T00:56:35.068Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-12T00:56:55.434 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.435 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.436 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.437 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.438 DEBUG:boto:Method: GET 2023-10-12T00:56:55.438 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.439 DEBUG:boto:Data: 2023-10-12T00:56:55.440 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.441 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.442 DEBUG:boto:Params: {} 2023-10-12T00:56:55.443 DEBUG:boto:Token: None 2023-10-12T00:56:55.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg= 2023-10-12T00:56:55.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg=', 'Content-Length': '0'} 2023-10-12T00:56:55.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009125cefd4ddbee34-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.450 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.450 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.451 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.452 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.453 DEBUG:boto:Method: GET 2023-10-12T00:56:55.454 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.454 DEBUG:boto:Data: 2023-10-12T00:56:55.455 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.457 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.457 DEBUG:boto:Params: {} 2023-10-12T00:56:55.458 DEBUG:boto:Token: None 2023-10-12T00:56:55.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg= 2023-10-12T00:56:55.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg=', 'Content-Length': '0'} 2023-10-12T00:56:55.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a9b514e8b9d2e0-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.502 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.503 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.503 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.504 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.504 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.505 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.505 DEBUG:boto:Data: 2023-10-12T00:56:55.505 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.506 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.507 DEBUG:boto:Params: {} 2023-10-12T00:56:55.507 DEBUG:boto:Token: None 2023-10-12T00:56:55.508 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9PoE1rGMdmPr7FTTJNK6IR8irVw= 2023-10-12T00:56:55.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9PoE1rGMdmPr7FTTJNK6IR8irVw=', 'Content-Length': '0'} 2023-10-12T00:56:55.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162a0e08ed5701fc-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.511 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.512 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.512 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.512 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.513 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.513 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.514 DEBUG:boto:Data: 2023-10-12T00:56:55.514 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.515 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.515 DEBUG:boto:Params: {} 2023-10-12T00:56:55.516 DEBUG:boto:Token: None 2023-10-12T00:56:55.516 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9PoE1rGMdmPr7FTTJNK6IR8irVw= 2023-10-12T00:56:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9PoE1rGMdmPr7FTTJNK6IR8irVw=', 'Content-Length': '0'} 2023-10-12T00:56:55.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002edcdeba11d24321-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-12T00:56:55.520 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.521 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.521 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.522 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.522 DEBUG:boto:Method: GET 2023-10-12T00:56:55.523 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.523 DEBUG:boto:Data: 2023-10-12T00:56:55.523 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.524 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.525 DEBUG:boto:Params: {} 2023-10-12T00:56:55.525 DEBUG:boto:Token: None 2023-10-12T00:56:55.526 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg= 2023-10-12T00:56:55.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg=', 'Content-Length': '0'} 2023-10-12T00:56:55.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903512de5801af47-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.529 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.530 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.530 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.530 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.531 DEBUG:boto:Method: GET 2023-10-12T00:56:55.531 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.532 DEBUG:boto:Data: 2023-10-12T00:56:55.532 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.533 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.533 DEBUG:boto:Params: {} 2023-10-12T00:56:55.534 DEBUG:boto:Token: None 2023-10-12T00:56:55.534 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467?versionId=null 2023-10-12T00:56:55.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg= 2023-10-12T00:56:55.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GySYlT2T04W9fgEMuw5bGigCwhg=', 'Content-Length': '0'} 2023-10-12T00:56:55.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008adef05839732a4-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.538 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.539 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.539 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.539 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.540 DEBUG:boto:Method: GET 2023-10-12T00:56:55.540 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.541 DEBUG:boto:Data: 2023-10-12T00:56:55.541 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.542 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.542 DEBUG:boto:Params: {} 2023-10-12T00:56:55.543 DEBUG:boto:Token: None 2023-10-12T00:56:55.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f694a934599295bd-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.548 DEBUG:boto:b'wafmoa-58obj000000000000000004671000falseobj000000000000000004672023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.548 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.549 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.549 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.550 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.550 DEBUG:boto:Method: GET 2023-10-12T00:56:55.551 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.551 DEBUG:boto:Data: 2023-10-12T00:56:55.551 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.552 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.553 DEBUG:boto:Params: {} 2023-10-12T00:56:55.553 DEBUG:boto:Token: None 2023-10-12T00:56:55.554 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa78fd7350af30f1-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.560 DEBUG:boto:b'wafmoa-58obj000000000000000004671000falseobj000000000000000004672023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.560 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.561 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.561 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.562 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.562 DEBUG:boto:Data: 2023-10-12T00:56:55.563 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.563 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.564 DEBUG:boto:Params: {} 2023-10-12T00:56:55.564 DEBUG:boto:Token: None 2023-10-12T00:56:55.565 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qRdT2xWcuq+Py/mlA3sfz4kAuBc= 2023-10-12T00:56:55.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qRdT2xWcuq+Py/mlA3sfz4kAuBc=', 'Content-Length': '0'} 2023-10-12T00:56:55.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efaab0f81fe9d7a8-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.568 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.569 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.569 DEBUG:boto:Method: HEAD 2023-10-12T00:56:55.570 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.570 DEBUG:boto:Data: 2023-10-12T00:56:55.571 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.571 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.572 DEBUG:boto:Params: {} 2023-10-12T00:56:55.572 DEBUG:boto:Token: None 2023-10-12T00:56:55.573 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.573 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qRdT2xWcuq+Py/mlA3sfz4kAuBc= 2023-10-12T00:56:55.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qRdT2xWcuq+Py/mlA3sfz4kAuBc=', 'Content-Length': '0'} 2023-10-12T00:56:55.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e988372e35b3fde-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-12T00:56:55.577 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.577 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.577 DEBUG:boto:Method: GET 2023-10-12T00:56:55.578 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.578 DEBUG:boto:Data: 2023-10-12T00:56:55.579 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.580 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.580 DEBUG:boto:Params: {} 2023-10-12T00:56:55.580 DEBUG:boto:Token: None 2023-10-12T00:56:55.581 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:37gnTlip7+hXIUCXYhFjewQ3X4I= 2023-10-12T00:56:55.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:37gnTlip7+hXIUCXYhFjewQ3X4I=', 'Content-Length': '0'} 2023-10-12T00:56:55.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-58:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1ae28215afb8c2-0065274457-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.626 DEBUG:boto:path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.627 DEBUG:boto:auth_path=/wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.628 DEBUG:boto:Method: GET 2023-10-12T00:56:55.628 DEBUG:boto:Path: /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.628 DEBUG:boto:Data: 2023-10-12T00:56:55.629 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.630 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.630 DEBUG:boto:Params: {} 2023-10-12T00:56:55.631 DEBUG:boto:Token: None 2023-10-12T00:56:55.631 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/obj00000000000000000467 2023-10-12T00:56:55.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:37gnTlip7+hXIUCXYhFjewQ3X4I= 2023-10-12T00:56:55.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:37gnTlip7+hXIUCXYhFjewQ3X4I=', 'Content-Length': '0'} 2023-10-12T00:56:55.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:56:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f98de28fbe0ed90-0065274457-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.635 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.635 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.636 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.636 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.637 DEBUG:boto:Method: GET 2023-10-12T00:56:55.637 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.637 DEBUG:boto:Data: 2023-10-12T00:56:55.638 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:56:55.639 DEBUG:boto:Port: 8001 2023-10-12T00:56:55.639 DEBUG:boto:Params: {} 2023-10-12T00:56:55.640 DEBUG:boto:Token: None 2023-10-12T00:56:55.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ea3d8cb734e9777-0065274457-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.644 DEBUG:boto:b'wafmoa-58obj000000000000000004671000falseobj000000000000000004672023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.645 DEBUG:boto:path=/wafmoa-58/ 2023-10-12T00:56:55.645 DEBUG:boto:auth_path=/wafmoa-58/ 2023-10-12T00:56:55.646 DEBUG:boto:path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.646 DEBUG:boto:auth_path=/wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.647 DEBUG:boto:Method: GET 2023-10-12T00:56:55.647 DEBUG:boto:Path: /wafmoa-58/?prefix=obj00000000000000000467 2023-10-12T00:56:55.648 DEBUG:boto:Data: 2023-10-12T00:56:55.648 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.649 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.649 DEBUG:boto:Params: {} 2023-10-12T00:56:55.650 DEBUG:boto:Token: None 2023-10-12T00:56:55.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-58/ 2023-10-12T00:56:55.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0= 2023-10-12T00:56:55.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/gWo0xQHV7L1mBxxcRrNEn8fS0=', 'Content-Length': '0'} 2023-10-12T00:56:55.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e62458232358d3-0065274457-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.656 DEBUG:boto:b'wafmoa-58obj000000000000000004671000falseobj000000000000000004672023-10-12T00:56:35.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:56:55.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-58 zones={a2, a1} 2023-10-12T00:56:55.658 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-10-12T00:56:55.660 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:56:55.661 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:56:55.661 DEBUG:boto:Method: PUT 2023-10-12T00:56:55.661 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:56:55.662 DEBUG:boto:Data: 2023-10-12T00:56:55.662 DEBUG:boto:Headers: {} 2023-10-12T00:56:55.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:56:55.663 DEBUG:boto:Port: 8000 2023-10-12T00:56:55.663 DEBUG:boto:Params: {} 2023-10-12T00:56:55.664 DEBUG:boto:Token: None 2023-10-12T00:56:55.664 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 00:56:55 GMT /wafmoa-59/ 2023-10-12T00:56:55.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2LIpNTJfDeO/t63i41ggEwCT4K0= 2023-10-12T00:56:55.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:56:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2LIpNTJfDeO/t63i41ggEwCT4K0=', 'Content-Length': '0'} 2023-10-12T00:56:55.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb5c7d54dce27d20-0065274457-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:56:55.684 DEBUG:rgw_multi.tests:created bucket=wafmoa-59 2023-10-12T00:56:55.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:55.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:55.686 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:55.772 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T00:56:55.785 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T00:56:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:18:29.315956Z" 2023-10-12T00:56:55.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:55.787 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-10-12T00:56:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072215.682437_610.1", 2023-10-12T00:56:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:56:55.682437Z" 2023-10-12T00:56:55.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T00:56:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.789 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T00:56:55.791 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:55.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T00:56:55.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:55.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:56:55.792 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:56:55.842 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:56:55.843 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:56:55.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:55.870+0000 7fa711a52780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:55.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:56:55.871+0000 7fa711a52780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:56:55.884 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:56:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:56:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:56:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:56:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:56:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:56:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:56:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:56:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:56:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:56:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:56:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:56:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:56:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:56:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:56:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:56:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:56:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:56:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:56:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:56:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:56:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:56:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:56:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:56:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:56:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:56:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:56:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:56:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:56:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:56:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:56:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:56:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:56:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:56:55Z" 2023-10-12T00:56:55.898 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:56:55.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:56:55Z" } 2023-10-12T00:56:55.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:56:55.900 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:55.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:56:55.901 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:00.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:00.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:00.903 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:00.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:00.996+0000 7f6432a50780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:00.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:00.997+0000 7f6432a50780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:01.010 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:01.010 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:00Z" 2023-10-12T00:57:01.026 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:01.027 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:00Z" } 2023-10-12T00:57:01.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:01.028 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:01.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:01.029 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:06.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:06.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:06.031 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:06.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:06.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:06.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:06.120+0000 7f02d6215780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:06.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:06.121+0000 7f02d6215780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:06.134 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:06.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:06.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:06.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:06.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:06.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:06.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:06.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:06.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:06.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:06.139 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:06.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:06.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:06.140 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:06.140 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:06.140 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:06.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:06.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:06.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:06.142 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:06.142 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:06.142 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:06.143 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:06.143 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:06.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:06.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:06.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:06.144 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:06.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:06.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:06.145 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:06.145 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:06.145 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:06.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:06.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:06.146 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:06.146 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:06.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:06.146 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:06.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:06.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:06.147 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:06.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.147 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:06.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:06.148 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:06.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:06.148 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:06Z" 2023-10-12T00:57:06.148 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:06.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:06Z" } 2023-10-12T00:57:06.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:06.151 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:06.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:06.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:11.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:11.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:11.154 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:11.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:11.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:11.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:11.245+0000 7fabe6903780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:11.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:11.246+0000 7fabe6903780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:11.259 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:11.260 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:11.260 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:11.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:11.260 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:11.260 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:11.261 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:11.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.261 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:11.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:11.262 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:11.262 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:11.262 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:11.262 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:11.262 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:11.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:11.263 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:11.263 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:11.263 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:11.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:11.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:11.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:11.264 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:11.264 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:11.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:11.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:11.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:11.265 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:11.266 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:11.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:11.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:11.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:11.267 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:11.267 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:11.267 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:11.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:11.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:11.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:11.268 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:11.268 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:11.268 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:11.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:11.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:11.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:11.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:11.270 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:11.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:11.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:11.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:11.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:11.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:11.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:11.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:11.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:11.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:11.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.272 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:11.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:11.272 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:11.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:11.273 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:11Z" 2023-10-12T00:57:11.273 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:11.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:11Z" } 2023-10-12T00:57:11.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:11.275 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:11.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:11.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:16.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:16.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:16.278 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:16.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:16.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:16.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:16.367+0000 7f271071d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:16.368+0000 7f271071d780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:16.382 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:16.383 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:16.383 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:16.383 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:16.384 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:16.384 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:16.385 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:16.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.386 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:16.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:16.386 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:16.387 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:16.387 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:16.388 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:16.388 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:16.388 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:16Z" 2023-10-12T00:57:16.403 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:16.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:16Z" } 2023-10-12T00:57:16.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:16.405 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:16.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:16.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:21.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:21.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:21.408 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:21.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:21.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:21.497+0000 7fe705382780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:21.498+0000 7fe705382780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:21.512 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:21.513 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:21.513 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:21.513 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:21.513 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:21.514 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:21.514 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:21.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.514 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:21.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:21.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:21.515 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:21.515 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:21.516 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:21.516 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:21.516 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:21.517 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:21.517 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:21.517 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:21.517 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:21.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:21.518 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:21.518 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:21.519 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:21.519 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:21.519 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:21.519 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:21.520 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:21.520 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:21.520 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:21.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:21.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:21.521 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:21.521 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:21.521 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:21.522 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:21.522 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:21.522 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:21.522 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:21.523 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:21.523 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:21.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:21.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:21.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:21.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:21.524 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:21.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:21.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:21.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:21.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:21.526 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:21.526 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:21.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:21.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:21.527 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:21.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.527 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:21.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:21.527 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:21.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:21.528 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:21Z" 2023-10-12T00:57:21.528 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:21.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:21Z" } 2023-10-12T00:57:21.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:21.530 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:21.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:21.531 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:26.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:26.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:26.534 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:26.594 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:26.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:26.626+0000 7f3cf1ba8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:26.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:26.627+0000 7f3cf1ba8780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:26.641 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:26.642 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:26.642 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:26.642 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:26.642 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:26.643 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:26.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:26.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.643 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:26.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:26.644 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:26.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:26.644 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:26.645 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:26.645 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:26.645 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:26.645 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:26.645 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:26.646 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:26.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:26.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:26.647 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:26.647 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:26.647 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:26.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:26.648 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:26.648 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:26.648 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:26.648 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:26.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:26.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:26.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:26.649 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:26.650 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:26.650 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:26.650 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:26.650 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:26.651 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:26.651 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:26.651 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:26.651 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:26.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:26.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:26.652 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:26.652 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:26.653 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:26.653 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:26.653 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:26.653 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:26.653 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:26.654 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:26.654 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:26.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:26.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:26.655 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:26.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.655 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:26.655 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:26Z" 2023-10-12T00:57:26.656 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:26.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:26Z" } 2023-10-12T00:57:26.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:26.659 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:26.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:26.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:31.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:31.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:31.662 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:31.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:31.748+0000 7fd67b464780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:31.749+0000 7fd67b464780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:31.763 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:31Z" 2023-10-12T00:57:31.777 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:31.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:31Z" } 2023-10-12T00:57:31.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:31.779 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:31.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:31.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:36.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:36.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:36.782 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:36.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:36.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:36.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:36.870+0000 7f993f645780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:36.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:36.871+0000 7f993f645780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:36.885 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:36.885 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:36.885 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:36.885 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:36.886 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:36.886 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:36.886 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:36.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.887 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:36.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:36.887 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:36.887 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:36.887 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:36.888 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:36.889 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:36.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:36.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:36.890 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:36.890 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:36.890 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:36.890 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:36.890 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:36.891 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:36.891 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:36.891 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:36.891 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:36.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:36.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:36.892 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:36.893 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:36.893 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:36.893 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072160.758530_576.1", 2023-10-12T00:57:36.893 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:36.893 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:36.894 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:36.894 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:00.758530Z", 2023-10-12T00:57:36.894 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:36.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:36.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:36.895 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:36.895 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:36.896 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:36.896 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:36.896 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:36.896 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:36.896 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:36.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:36.897 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:36.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:36.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:36.897 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:36.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.898 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:36.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:36.898 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:36.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:36.899 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:36Z" 2023-10-12T00:57:36.899 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:36.900 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072160.758530_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:00.758530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:36Z" } 2023-10-12T00:57:36.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072160.758530_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:00.758530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:36.901 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:36.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072160.758530_576.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:36.902 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072160.758530_576.1 2023-10-12T00:57:41.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:41.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:41.904 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:41.955 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:41.955 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:41.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:41.983+0000 7f5381964780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:41.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:41.984+0000 7f5381964780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:41.996 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:41.996 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:41.997 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:41.997 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:41.997 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:41.997 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:41.998 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:41.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:41.998 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:41.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:41.999 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:41.999 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:41.999 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:41.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:41.999 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:42.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:42.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:42.000 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:42.000 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:42.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:42.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:42.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:42.001 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:42.001 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072190.277606_583.1", 2023-10-12T00:57:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:30.277606Z", 2023-10-12T00:57:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:42.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:42.010 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:41Z" 2023-10-12T00:57:42.010 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:42.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072190.277606_583.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:30.277606Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:41Z" } 2023-10-12T00:57:42.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072190.277606_583.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:30.277606Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:42.012 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:42.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072190.277606_583.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:42.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697072215.682437_610.1 target=1_1697072190.277606_583.1 2023-10-12T00:57:47.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:47.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T00:57:47.015 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T00:57:47.072 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:47.073 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:47.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:47.105+0000 7fa595173780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:47.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:47.106+0000 7fa595173780 20 RGW-SYNC:meta: read sync status 2023-10-12T00:57:47.120 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T00:57:47.120 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T00:57:47.120 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T00:57:47.121 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T00:57:47.121 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T00:57:47.121 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T00:57:47.121 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:47.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.122 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T00:57:47.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:47.122 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T00:57:47.123 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:47.123 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:47.123 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T00:57:47.123 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:47.123 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T00:57:47.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:47.124 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T00:57:47.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:47.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:47.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:47.125 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T00:57:47.125 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:47.125 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:47.126 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T00:57:47.126 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:47.126 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:47.126 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:47.127 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T00:57:47.127 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:47.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:47.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:47.128 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T00:57:47.128 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:47.128 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:47.128 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072215.682437_610.1", 2023-10-12T00:57:47.128 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:47.129 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:47.129 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:47.129 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:56:55.682437Z", 2023-10-12T00:57:47.129 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:47.129 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:47.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T00:57:47.130 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T00:57:47.130 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T00:57:47.131 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T00:57:47.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T00:57:47.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T00:57:47.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T00:57:47.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T00:57:47.132 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T00:57:47.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T00:57:47.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:47.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T00:57:47.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T00:57:47.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.133 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T00:57:47.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T00:57:47.133 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T00:57:47.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T00:57:47.134 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T00:57:47Z" 2023-10-12T00:57:47.134 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T00:57:47.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072215.682437_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:56:55.682437Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T00:57:47Z" } 2023-10-12T00:57:47.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072215.682437_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:56:55.682437Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T00:57:47.136 DEBUG:rgw_multi.tests:log_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:47.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072215.682437_610.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T00:57:47.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T00:57:47.137 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:57:47.138 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:57:47.138 DEBUG:boto:Method: HEAD 2023-10-12T00:57:47.139 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:57:47.139 DEBUG:boto:Data: 2023-10-12T00:57:47.140 DEBUG:boto:Headers: {} 2023-10-12T00:57:47.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.141 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.141 DEBUG:boto:Params: {} 2023-10-12T00:57:47.141 DEBUG:boto:Token: None 2023-10-12T00:57:47.142 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/ 2023-10-12T00:57:47.143 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI= 2023-10-12T00:57:47.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI=', 'Content-Length': '0'} 2023-10-12T00:57:47.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030d9971ef73b415d-006527448b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.147 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:57:47.147 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:57:47.148 DEBUG:boto:Method: PUT 2023-10-12T00:57:47.148 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T00:57:47.149 DEBUG:boto:Data: 2023-10-12T00:57:47.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:57:47.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.150 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.151 DEBUG:boto:Params: {} 2023-10-12T00:57:47.151 DEBUG:boto:Token: None 2023-10-12T00:57:47.152 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/obj11 2023-10-12T00:57:47.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RZUtwt0ozUfihHsrqXPfM5zpRcM= 2023-10-12T00:57:47.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RZUtwt0ozUfihHsrqXPfM5zpRcM='} 2023-10-12T00:57:47.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023f920ebe1dcaec4-006527448b-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.161 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:57:47.162 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:57:47.162 DEBUG:boto:Method: HEAD 2023-10-12T00:57:47.163 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:57:47.163 DEBUG:boto:Data: 2023-10-12T00:57:47.163 DEBUG:boto:Headers: {} 2023-10-12T00:57:47.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.164 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.165 DEBUG:boto:Params: {} 2023-10-12T00:57:47.165 DEBUG:boto:Token: None 2023-10-12T00:57:47.166 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/ 2023-10-12T00:57:47.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI= 2023-10-12T00:57:47.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI=', 'Content-Length': '0'} 2023-10-12T00:57:47.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003173353c3f8d574-006527448b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.169 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:57:47.170 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:57:47.170 DEBUG:boto:Method: PUT 2023-10-12T00:57:47.171 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T00:57:47.171 DEBUG:boto:Data: 2023-10-12T00:57:47.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:57:47.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.173 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.173 DEBUG:boto:Params: {} 2023-10-12T00:57:47.173 DEBUG:boto:Token: None 2023-10-12T00:57:47.174 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/obj22 2023-10-12T00:57:47.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xK/L+PWKn3p2TEdSkQY6jiaTRtc= 2023-10-12T00:57:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xK/L+PWKn3p2TEdSkQY6jiaTRtc='} 2023-10-12T00:57:47.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000323972756a2ded47-006527448b-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.223 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:57:47.224 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:57:47.224 DEBUG:boto:Method: HEAD 2023-10-12T00:57:47.224 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:57:47.225 DEBUG:boto:Data: 2023-10-12T00:57:47.225 DEBUG:boto:Headers: {} 2023-10-12T00:57:47.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.227 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.227 DEBUG:boto:Params: {} 2023-10-12T00:57:47.228 DEBUG:boto:Token: None 2023-10-12T00:57:47.228 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/ 2023-10-12T00:57:47.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI= 2023-10-12T00:57:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI=', 'Content-Length': '0'} 2023-10-12T00:57:47.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000964b7a53916c7c57-006527448b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.232 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:57:47.233 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:57:47.233 DEBUG:boto:Method: PUT 2023-10-12T00:57:47.234 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T00:57:47.234 DEBUG:boto:Data: 2023-10-12T00:57:47.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:57:47.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.236 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.236 DEBUG:boto:Params: {} 2023-10-12T00:57:47.237 DEBUG:boto:Token: None 2023-10-12T00:57:47.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/obj33 2023-10-12T00:57:47.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jLR+mKxXxDTPwQ/A5pbhG6U8vPc= 2023-10-12T00:57:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jLR+mKxXxDTPwQ/A5pbhG6U8vPc='} 2023-10-12T00:57:47.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000140a86c22e009048-006527448b-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.244 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:57:47.244 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:57:47.244 DEBUG:boto:Method: HEAD 2023-10-12T00:57:47.245 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:57:47.245 DEBUG:boto:Data: 2023-10-12T00:57:47.246 DEBUG:boto:Headers: {} 2023-10-12T00:57:47.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.247 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.249 DEBUG:boto:Params: {} 2023-10-12T00:57:47.249 DEBUG:boto:Token: None 2023-10-12T00:57:47.250 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/ 2023-10-12T00:57:47.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI= 2023-10-12T00:57:47.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI=', 'Content-Length': '0'} 2023-10-12T00:57:47.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000250814be56f3e233-006527448b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.254 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:57:47.255 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:57:47.255 DEBUG:boto:Method: PUT 2023-10-12T00:57:47.255 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T00:57:47.256 DEBUG:boto:Data: 2023-10-12T00:57:47.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:57:47.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.257 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.257 DEBUG:boto:Params: {} 2023-10-12T00:57:47.258 DEBUG:boto:Token: None 2023-10-12T00:57:47.259 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/obj44 2023-10-12T00:57:47.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4oz2e/8Fr+JidJCYXD+1qS8wNvs= 2023-10-12T00:57:47.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4oz2e/8Fr+JidJCYXD+1qS8wNvs='} 2023-10-12T00:57:47.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e34fa78ff7c8daec-006527448b-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.265 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:57:47.266 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:57:47.266 DEBUG:boto:Method: HEAD 2023-10-12T00:57:47.266 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:57:47.267 DEBUG:boto:Data: 2023-10-12T00:57:47.267 DEBUG:boto:Headers: {} 2023-10-12T00:57:47.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.268 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.268 DEBUG:boto:Params: {} 2023-10-12T00:57:47.269 DEBUG:boto:Token: None 2023-10-12T00:57:47.269 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/ 2023-10-12T00:57:47.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI= 2023-10-12T00:57:47.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErCnpIsCPjkHaSKqM8dGkEHZsbI=', 'Content-Length': '0'} 2023-10-12T00:57:47.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba53fcb8d69c84ca-006527448b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.273 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:57:47.274 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:57:47.274 DEBUG:boto:Method: PUT 2023-10-12T00:57:47.275 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T00:57:47.275 DEBUG:boto:Data: 2023-10-12T00:57:47.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:57:47.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:57:47.276 DEBUG:boto:Port: 8000 2023-10-12T00:57:47.277 DEBUG:boto:Params: {} 2023-10-12T00:57:47.277 DEBUG:boto:Token: None 2023-10-12T00:57:47.278 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:57:47 GMT /wafmoa-59/obj55 2023-10-12T00:57:47.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pxKS32GdmtVQ5jd+1oaP9Pf7qSo= 2023-10-12T00:57:47.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:57:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pxKS32GdmtVQ5jd+1oaP9Pf7qSo='} 2023-10-12T00:57:47.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000382140e9bdd53e83-006527448b-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:57:47.323 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T00:57:47.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:57:47.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:57:47.324 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:57:47.382 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:57:47.383 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:57:47.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:47.418+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:57:52.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:52.418+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:57:57.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:57:57.418+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:02.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:02.419+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:07.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:07.420+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:12.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:12.420+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:17.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:17.422+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:22.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:22.423+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:27.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:27.423+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:32.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:32.423+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:37.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:37.424+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:42.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:42.424+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:47.425+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:52.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:52.426+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:58:57.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:58:57.428+0000 7f1a1c651780 1 waiting to reach incremental sync.. 2023-10-12T00:59:02.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:02.429+0000 7f1a1c651780 1 bucket sync caught up with source: 2023-10-12T00:59:02.429 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.68.6, 00000000002.82.6, 00000000002.69.6] 2023-10-12T00:59:02.430 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.68.6, 00000000002.82.6, 00000000002.69.6] 2023-10-12T00:59:02.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:02.429+0000 7f1a1c651780 0 bucket checkpoint complete 2023-10-12T00:59:02.448 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T00:59:02.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:59:02.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:59:02.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:59:02.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:59:02.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:59:02.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:02.555+0000 7f72ae7a7780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:02.555 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.271.6, 00000000001.226.6, 00000000002.165.6] 2023-10-12T00:59:02.555 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:07.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:07.556+0000 7f72ae7a7780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:07.556 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.271.6, 00000000001.226.6, 00000000002.165.6] 2023-10-12T00:59:07.556 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:12.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:12.556+0000 7f72ae7a7780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:12.557 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.271.6, 00000000001.226.6, 00000000002.165.6] 2023-10-12T00:59:12.558 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:17.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:17.558+0000 7f72ae7a7780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:17.559 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.271.6, 00000000001.226.6, 00000000002.165.6] 2023-10-12T00:59:17.559 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:20.214 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T00:59:20.213+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T00:59:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:22.559+0000 7f72ae7a7780 1 bucket sync caught up with source: 2023-10-12T00:59:22.560 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:22.560 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.271.6, 00000000002.228.6, 00000000002.165.6] 2023-10-12T00:59:22.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:22.559+0000 7f72ae7a7780 0 bucket checkpoint complete 2023-10-12T00:59:22.577 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T00:59:22.578 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.578 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.579 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.579 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:22.580 DEBUG:boto:Data: 2023-10-12T00:59:22.580 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.581 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.581 DEBUG:boto:Params: {} 2023-10-12T00:59:22.582 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T00:59:22.583 DEBUG:boto:Token: None 2023-10-12T00:59:22.583 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RsiH/3XevPspxDBErB6tsIN9ZhQ= 2023-10-12T00:59:22.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RsiH/3XevPspxDBErB6tsIN9ZhQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2636926cf8ed2fa-00652744ea-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.589 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.589 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.590 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.590 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:22.591 DEBUG:boto:Data: 2023-10-12T00:59:22.591 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.592 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.592 DEBUG:boto:Params: {} 2023-10-12T00:59:22.593 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T00:59:22.593 DEBUG:boto:Token: None 2023-10-12T00:59:22.594 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RsiH/3XevPspxDBErB6tsIN9ZhQ= 2023-10-12T00:59:22.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RsiH/3XevPspxDBErB6tsIN9ZhQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcd12362199270d7-00652744ea-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T00:59:22.599 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.600 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.600 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.601 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.601 DEBUG:boto:Method: GET 2023-10-12T00:59:22.601 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.602 DEBUG:boto:Data: 2023-10-12T00:59:22.602 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.603 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.604 DEBUG:boto:Params: {} 2023-10-12T00:59:22.604 DEBUG:boto:Token: None 2023-10-12T00:59:22.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/?versions 2023-10-12T00:59:22.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ= 2023-10-12T00:59:22.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a33b638a131d1883-00652744ea-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.609 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T00:59:22.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T00:59:22.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T00:59:22.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T00:59:22.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T00:59:22.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T00:59:22.612 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.613 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.613 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.614 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.614 DEBUG:boto:Method: GET 2023-10-12T00:59:22.615 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.615 DEBUG:boto:Data: 2023-10-12T00:59:22.615 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.616 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.617 DEBUG:boto:Params: {} 2023-10-12T00:59:22.617 DEBUG:boto:Token: None 2023-10-12T00:59:22.618 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/?versions 2023-10-12T00:59:22.618 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ= 2023-10-12T00:59:22.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13f324172fe7c5f-00652744ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.622 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T00:59:22.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T00:59:22.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T00:59:22.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T00:59:22.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T00:59:22.625 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.625 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.626 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.626 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.627 DEBUG:boto:Method: GET 2023-10-12T00:59:22.627 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.628 DEBUG:boto:Data: 2023-10-12T00:59:22.630 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.631 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.632 DEBUG:boto:Params: {} 2023-10-12T00:59:22.632 DEBUG:boto:Token: None 2023-10-12T00:59:22.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/?versions 2023-10-12T00:59:22.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ= 2023-10-12T00:59:22.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1a1bfbbbde1b5cd-00652744ea-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.638 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.639 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.639 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.640 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.640 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.640 DEBUG:boto:Method: GET 2023-10-12T00:59:22.641 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T00:59:22.641 DEBUG:boto:Data: 2023-10-12T00:59:22.642 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.643 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.643 DEBUG:boto:Params: {} 2023-10-12T00:59:22.643 DEBUG:boto:Token: None 2023-10-12T00:59:22.644 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/?versions 2023-10-12T00:59:22.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ= 2023-10-12T00:59:22.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IBWzIpcOf7vfwcM0lfl401JttUQ=', 'Content-Length': '0'} 2023-10-12T00:59:22.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9ca3f2a3302eff-00652744ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.648 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T00:59:22.650 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.650 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.651 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.651 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.652 DEBUG:boto:Method: GET 2023-10-12T00:59:22.652 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.652 DEBUG:boto:Data: 2023-10-12T00:59:22.653 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.654 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.654 DEBUG:boto:Params: {} 2023-10-12T00:59:22.655 DEBUG:boto:Token: None 2023-10-12T00:59:22.655 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk= 2023-10-12T00:59:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk=', 'Content-Length': '0'} 2023-10-12T00:59:22.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196b68e6cfbae37f-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.659 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.659 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.660 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.660 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.661 DEBUG:boto:Method: GET 2023-10-12T00:59:22.661 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.662 DEBUG:boto:Data: 2023-10-12T00:59:22.662 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.663 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.663 DEBUG:boto:Params: {} 2023-10-12T00:59:22.664 DEBUG:boto:Token: None 2023-10-12T00:59:22.665 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk= 2023-10-12T00:59:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk=', 'Content-Length': '0'} 2023-10-12T00:59:22.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e548ff6ea1d31ec3-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.668 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.669 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.669 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.669 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.670 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.670 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.671 DEBUG:boto:Data: 2023-10-12T00:59:22.671 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.672 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.672 DEBUG:boto:Params: {} 2023-10-12T00:59:22.673 DEBUG:boto:Token: None 2023-10-12T00:59:22.673 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3DX1x++yHWdrqDBvt9EIakkSfRE= 2023-10-12T00:59:22.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3DX1x++yHWdrqDBvt9EIakkSfRE=', 'Content-Length': '0'} 2023-10-12T00:59:22.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002b316226c1ae26-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.677 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.677 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.678 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.678 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.678 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.679 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.679 DEBUG:boto:Data: 2023-10-12T00:59:22.680 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.680 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.681 DEBUG:boto:Params: {} 2023-10-12T00:59:22.681 DEBUG:boto:Token: None 2023-10-12T00:59:22.682 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3DX1x++yHWdrqDBvt9EIakkSfRE= 2023-10-12T00:59:22.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3DX1x++yHWdrqDBvt9EIakkSfRE=', 'Content-Length': '0'} 2023-10-12T00:59:22.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a9b6821373ed67-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T00:59:22.686 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.686 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.687 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.687 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.688 DEBUG:boto:Method: GET 2023-10-12T00:59:22.688 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.689 DEBUG:boto:Data: 2023-10-12T00:59:22.689 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.690 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.690 DEBUG:boto:Params: {} 2023-10-12T00:59:22.691 DEBUG:boto:Token: None 2023-10-12T00:59:22.691 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk= 2023-10-12T00:59:22.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk=', 'Content-Length': '0'} 2023-10-12T00:59:22.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e4b14c90d8d079-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.695 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.695 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.696 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.696 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.697 DEBUG:boto:Method: GET 2023-10-12T00:59:22.697 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.697 DEBUG:boto:Data: 2023-10-12T00:59:22.698 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.699 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.699 DEBUG:boto:Params: {} 2023-10-12T00:59:22.700 DEBUG:boto:Token: None 2023-10-12T00:59:22.700 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T00:59:22.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk= 2023-10-12T00:59:22.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dACWVmy6dX6Chs8M+P2Vq3XAYwk=', 'Content-Length': '0'} 2023-10-12T00:59:22.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7b3612884ae0c6-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.704 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.704 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.705 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.705 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.706 DEBUG:boto:Method: GET 2023-10-12T00:59:22.706 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.706 DEBUG:boto:Data: 2023-10-12T00:59:22.707 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.708 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.708 DEBUG:boto:Params: {} 2023-10-12T00:59:22.709 DEBUG:boto:Token: None 2023-10-12T00:59:22.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ee0abc144f9e22-00652744ea-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.713 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.714 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.715 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.715 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.716 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.716 DEBUG:boto:Method: GET 2023-10-12T00:59:22.716 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.717 DEBUG:boto:Data: 2023-10-12T00:59:22.717 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.718 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.719 DEBUG:boto:Params: {} 2023-10-12T00:59:22.719 DEBUG:boto:Token: None 2023-10-12T00:59:22.720 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fdafb5943ebd6b2-00652744ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.766 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.767 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.768 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.768 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.769 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T00:59:22.769 DEBUG:boto:Data: 2023-10-12T00:59:22.769 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.770 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.771 DEBUG:boto:Params: {} 2023-10-12T00:59:22.771 DEBUG:boto:Token: None 2023-10-12T00:59:22.772 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11 2023-10-12T00:59:22.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GcHN4zxjWQXYfJGPt2FVrpYjjGY= 2023-10-12T00:59:22.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GcHN4zxjWQXYfJGPt2FVrpYjjGY=', 'Content-Length': '0'} 2023-10-12T00:59:22.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240a9255d3f0b57a-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.775 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.776 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.776 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.777 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T00:59:22.777 DEBUG:boto:Data: 2023-10-12T00:59:22.778 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.779 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.779 DEBUG:boto:Params: {} 2023-10-12T00:59:22.780 DEBUG:boto:Token: None 2023-10-12T00:59:22.780 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11 2023-10-12T00:59:22.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GcHN4zxjWQXYfJGPt2FVrpYjjGY= 2023-10-12T00:59:22.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GcHN4zxjWQXYfJGPt2FVrpYjjGY=', 'Content-Length': '0'} 2023-10-12T00:59:22.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf457d14792aaf1-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T00:59:22.783 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.784 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.784 DEBUG:boto:Method: GET 2023-10-12T00:59:22.785 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T00:59:22.785 DEBUG:boto:Data: 2023-10-12T00:59:22.786 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.787 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.787 DEBUG:boto:Params: {} 2023-10-12T00:59:22.788 DEBUG:boto:Token: None 2023-10-12T00:59:22.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11 2023-10-12T00:59:22.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uP80r+G66XPo2+jGnYQhzl3Iygg= 2023-10-12T00:59:22.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uP80r+G66XPo2+jGnYQhzl3Iygg=', 'Content-Length': '0'} 2023-10-12T00:59:22.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ededcad97594606-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.834 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T00:59:22.835 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T00:59:22.835 DEBUG:boto:Method: GET 2023-10-12T00:59:22.836 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T00:59:22.836 DEBUG:boto:Data: 2023-10-12T00:59:22.837 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.838 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.838 DEBUG:boto:Params: {} 2023-10-12T00:59:22.839 DEBUG:boto:Token: None 2023-10-12T00:59:22.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj11 2023-10-12T00:59:22.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uP80r+G66XPo2+jGnYQhzl3Iygg= 2023-10-12T00:59:22.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uP80r+G66XPo2+jGnYQhzl3Iygg=', 'Content-Length': '0'} 2023-10-12T00:59:22.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d898b6eec1c49cbf-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.843 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.846 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.847 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.847 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.848 DEBUG:boto:Method: GET 2023-10-12T00:59:22.848 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.849 DEBUG:boto:Data: 2023-10-12T00:59:22.849 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.850 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.850 DEBUG:boto:Params: {} 2023-10-12T00:59:22.851 DEBUG:boto:Token: None 2023-10-12T00:59:22.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017387e0a9ed42800-00652744ea-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.856 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.857 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.857 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.858 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.858 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.859 DEBUG:boto:Method: GET 2023-10-12T00:59:22.859 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T00:59:22.859 DEBUG:boto:Data: 2023-10-12T00:59:22.860 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.861 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.861 DEBUG:boto:Params: {} 2023-10-12T00:59:22.862 DEBUG:boto:Token: None 2023-10-12T00:59:22.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2101fe7f9005905-00652744ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.866 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T00:59:22.868 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.868 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.869 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.869 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.870 DEBUG:boto:Method: GET 2023-10-12T00:59:22.870 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.871 DEBUG:boto:Data: 2023-10-12T00:59:22.871 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.872 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.872 DEBUG:boto:Params: {} 2023-10-12T00:59:22.873 DEBUG:boto:Token: None 2023-10-12T00:59:22.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI= 2023-10-12T00:59:22.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI=', 'Content-Length': '0'} 2023-10-12T00:59:22.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005305787ecc3f2d77-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.877 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.878 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.878 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.879 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.879 DEBUG:boto:Method: GET 2023-10-12T00:59:22.880 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.880 DEBUG:boto:Data: 2023-10-12T00:59:22.881 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.882 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.882 DEBUG:boto:Params: {} 2023-10-12T00:59:22.883 DEBUG:boto:Token: None 2023-10-12T00:59:22.883 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI= 2023-10-12T00:59:22.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI=', 'Content-Length': '0'} 2023-10-12T00:59:22.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad25941f47ab6cc0-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.887 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.887 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.888 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.888 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.889 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.889 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.889 DEBUG:boto:Data: 2023-10-12T00:59:22.890 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.891 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.891 DEBUG:boto:Params: {} 2023-10-12T00:59:22.892 DEBUG:boto:Token: None 2023-10-12T00:59:22.892 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.892 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e7Pk+Qbwnafieu6H8an1pWWshJY= 2023-10-12T00:59:22.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e7Pk+Qbwnafieu6H8an1pWWshJY=', 'Content-Length': '0'} 2023-10-12T00:59:22.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dafa8fad869e93-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.896 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.896 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.896 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.897 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.897 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.898 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.898 DEBUG:boto:Data: 2023-10-12T00:59:22.899 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.899 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.900 DEBUG:boto:Params: {} 2023-10-12T00:59:22.900 DEBUG:boto:Token: None 2023-10-12T00:59:22.901 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e7Pk+Qbwnafieu6H8an1pWWshJY= 2023-10-12T00:59:22.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e7Pk+Qbwnafieu6H8an1pWWshJY=', 'Content-Length': '0'} 2023-10-12T00:59:22.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77891e5bb86ae76-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T00:59:22.905 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.906 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.906 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.907 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.908 DEBUG:boto:Method: GET 2023-10-12T00:59:22.908 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.909 DEBUG:boto:Data: 2023-10-12T00:59:22.910 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.911 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.911 DEBUG:boto:Params: {} 2023-10-12T00:59:22.912 DEBUG:boto:Token: None 2023-10-12T00:59:22.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI= 2023-10-12T00:59:22.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI=', 'Content-Length': '0'} 2023-10-12T00:59:22.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b59acbc6b5113c0-00652744ea-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.962 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.963 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.963 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.964 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.964 DEBUG:boto:Method: GET 2023-10-12T00:59:22.965 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.965 DEBUG:boto:Data: 2023-10-12T00:59:22.966 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.967 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.970 DEBUG:boto:Params: {} 2023-10-12T00:59:22.971 DEBUG:boto:Token: None 2023-10-12T00:59:22.971 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T00:59:22.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI= 2023-10-12T00:59:22.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7+y5FR7rRUWboCc+W7ovWV5CtEI=', 'Content-Length': '0'} 2023-10-12T00:59:22.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3dbb7cd2c94720e-00652744ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.975 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.976 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.976 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.977 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.977 DEBUG:boto:Method: GET 2023-10-12T00:59:22.977 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.978 DEBUG:boto:Data: 2023-10-12T00:59:22.978 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.979 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.980 DEBUG:boto:Params: {} 2023-10-12T00:59:22.980 DEBUG:boto:Token: None 2023-10-12T00:59:22.981 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab87ea16221feb6-00652744ea-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.984 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.985 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:22.986 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:22.986 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.987 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.987 DEBUG:boto:Method: GET 2023-10-12T00:59:22.988 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T00:59:22.988 DEBUG:boto:Data: 2023-10-12T00:59:22.988 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:22.989 DEBUG:boto:Port: 8000 2023-10-12T00:59:22.990 DEBUG:boto:Params: {} 2023-10-12T00:59:22.990 DEBUG:boto:Token: None 2023-10-12T00:59:22.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:22 GMT /wafmoa-59/ 2023-10-12T00:59:22.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA= 2023-10-12T00:59:22.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0SzjU20kWmFwVHQ6RdtL1CiMGyA=', 'Content-Length': '0'} 2023-10-12T00:59:22.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7be7a06958469a2-00652744ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:22.995 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:22.996 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:22.996 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:22.997 DEBUG:boto:Method: HEAD 2023-10-12T00:59:22.997 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T00:59:22.997 DEBUG:boto:Data: 2023-10-12T00:59:22.998 DEBUG:boto:Headers: {} 2023-10-12T00:59:22.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:22.999 DEBUG:boto:Port: 8001 2023-10-12T00:59:22.999 DEBUG:boto:Params: {} 2023-10-12T00:59:23.000 DEBUG:boto:Token: None 2023-10-12T00:59:23.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj22 2023-10-12T00:59:23.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fc1SqMJc/nMF5jcmOlTrayq7eB0= 2023-10-12T00:59:23.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fc1SqMJc/nMF5jcmOlTrayq7eB0=', 'Content-Length': '0'} 2023-10-12T00:59:23.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff053d34e5d9a80a-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.003 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:23.004 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:23.004 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.005 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T00:59:23.005 DEBUG:boto:Data: 2023-10-12T00:59:23.006 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.006 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.007 DEBUG:boto:Params: {} 2023-10-12T00:59:23.007 DEBUG:boto:Token: None 2023-10-12T00:59:23.008 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj22 2023-10-12T00:59:23.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fc1SqMJc/nMF5jcmOlTrayq7eB0= 2023-10-12T00:59:23.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fc1SqMJc/nMF5jcmOlTrayq7eB0=', 'Content-Length': '0'} 2023-10-12T00:59:23.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d5189cef34e242-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T00:59:23.012 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:23.012 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:23.012 DEBUG:boto:Method: GET 2023-10-12T00:59:23.013 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T00:59:23.013 DEBUG:boto:Data: 2023-10-12T00:59:23.014 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.015 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.015 DEBUG:boto:Params: {} 2023-10-12T00:59:23.016 DEBUG:boto:Token: None 2023-10-12T00:59:23.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj22 2023-10-12T00:59:23.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGK7lKzn+ku5Fzo87gmo+kXn6gY= 2023-10-12T00:59:23.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGK7lKzn+ku5Fzo87gmo+kXn6gY=', 'Content-Length': '0'} 2023-10-12T00:59:23.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91a433b5202614a-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.020 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T00:59:23.020 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T00:59:23.020 DEBUG:boto:Method: GET 2023-10-12T00:59:23.021 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T00:59:23.021 DEBUG:boto:Data: 2023-10-12T00:59:23.022 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.023 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.023 DEBUG:boto:Params: {} 2023-10-12T00:59:23.024 DEBUG:boto:Token: None 2023-10-12T00:59:23.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj22 2023-10-12T00:59:23.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGK7lKzn+ku5Fzo87gmo+kXn6gY= 2023-10-12T00:59:23.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGK7lKzn+ku5Fzo87gmo+kXn6gY=', 'Content-Length': '0'} 2023-10-12T00:59:23.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087dfa29eb6e1880-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.070 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.071 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.072 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.072 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.072 DEBUG:boto:Method: GET 2023-10-12T00:59:23.073 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.073 DEBUG:boto:Data: 2023-10-12T00:59:23.074 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.075 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.075 DEBUG:boto:Params: {} 2023-10-12T00:59:23.076 DEBUG:boto:Token: None 2023-10-12T00:59:23.076 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.077 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8283bc0723824f1-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.080 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.081 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.081 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.082 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.082 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.083 DEBUG:boto:Method: GET 2023-10-12T00:59:23.083 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T00:59:23.084 DEBUG:boto:Data: 2023-10-12T00:59:23.084 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.085 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.085 DEBUG:boto:Params: {} 2023-10-12T00:59:23.086 DEBUG:boto:Token: None 2023-10-12T00:59:23.086 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001231bb8b3558ab7d-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.090 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T00:59:23.092 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.092 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.092 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.093 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.093 DEBUG:boto:Method: GET 2023-10-12T00:59:23.094 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.094 DEBUG:boto:Data: 2023-10-12T00:59:23.095 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.096 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.096 DEBUG:boto:Params: {} 2023-10-12T00:59:23.096 DEBUG:boto:Token: None 2023-10-12T00:59:23.097 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs= 2023-10-12T00:59:23.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e5e1c4aafe904d-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.142 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.143 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.143 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.144 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.144 DEBUG:boto:Method: GET 2023-10-12T00:59:23.145 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.145 DEBUG:boto:Data: 2023-10-12T00:59:23.146 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.147 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.147 DEBUG:boto:Params: {} 2023-10-12T00:59:23.148 DEBUG:boto:Token: None 2023-10-12T00:59:23.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs= 2023-10-12T00:59:23.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7e60817dea775f-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.152 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.152 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.153 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.153 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.153 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.154 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.154 DEBUG:boto:Data: 2023-10-12T00:59:23.155 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.156 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.156 DEBUG:boto:Params: {} 2023-10-12T00:59:23.156 DEBUG:boto:Token: None 2023-10-12T00:59:23.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lIfsv3cgSj18Ex6JN0ghqFHHmHM= 2023-10-12T00:59:23.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lIfsv3cgSj18Ex6JN0ghqFHHmHM=', 'Content-Length': '0'} 2023-10-12T00:59:23.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f167105f85425dd-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.161 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.161 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.161 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.162 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.162 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.163 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.163 DEBUG:boto:Data: 2023-10-12T00:59:23.163 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.164 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.165 DEBUG:boto:Params: {} 2023-10-12T00:59:23.165 DEBUG:boto:Token: None 2023-10-12T00:59:23.166 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lIfsv3cgSj18Ex6JN0ghqFHHmHM= 2023-10-12T00:59:23.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lIfsv3cgSj18Ex6JN0ghqFHHmHM=', 'Content-Length': '0'} 2023-10-12T00:59:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7f2ec1a5b39aab-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T00:59:23.170 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.170 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.171 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.171 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.171 DEBUG:boto:Method: GET 2023-10-12T00:59:23.172 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.172 DEBUG:boto:Data: 2023-10-12T00:59:23.173 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.174 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.174 DEBUG:boto:Params: {} 2023-10-12T00:59:23.175 DEBUG:boto:Token: None 2023-10-12T00:59:23.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs= 2023-10-12T00:59:23.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440cce2bb9352778-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.179 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.179 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.180 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.180 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.181 DEBUG:boto:Method: GET 2023-10-12T00:59:23.181 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.181 DEBUG:boto:Data: 2023-10-12T00:59:23.182 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.183 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.183 DEBUG:boto:Params: {} 2023-10-12T00:59:23.184 DEBUG:boto:Token: None 2023-10-12T00:59:23.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T00:59:23.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs= 2023-10-12T00:59:23.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I8EO3LsuHosz2dqCPbkEdq2BnXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae71b4a71c526d65-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.188 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.188 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.189 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.189 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.189 DEBUG:boto:Method: GET 2023-10-12T00:59:23.190 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.190 DEBUG:boto:Data: 2023-10-12T00:59:23.191 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.192 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.192 DEBUG:boto:Params: {} 2023-10-12T00:59:23.193 DEBUG:boto:Token: None 2023-10-12T00:59:23.193 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.194 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac65f54a8200c50f-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.197 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.198 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.198 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.199 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.199 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.200 DEBUG:boto:Method: GET 2023-10-12T00:59:23.200 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.201 DEBUG:boto:Data: 2023-10-12T00:59:23.201 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.202 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.202 DEBUG:boto:Params: {} 2023-10-12T00:59:23.203 DEBUG:boto:Token: None 2023-10-12T00:59:23.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a2a8b96b5bb2c6-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.250 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.251 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.252 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.252 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.253 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T00:59:23.253 DEBUG:boto:Data: 2023-10-12T00:59:23.253 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.254 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.255 DEBUG:boto:Params: {} 2023-10-12T00:59:23.255 DEBUG:boto:Token: None 2023-10-12T00:59:23.257 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33 2023-10-12T00:59:23.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NWF8euHGY2womOu5u+rfkL7lUtE= 2023-10-12T00:59:23.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NWF8euHGY2womOu5u+rfkL7lUtE=', 'Content-Length': '0'} 2023-10-12T00:59:23.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a3c7eae8a66544-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.261 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.262 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.262 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.263 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T00:59:23.263 DEBUG:boto:Data: 2023-10-12T00:59:23.263 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.264 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.265 DEBUG:boto:Params: {} 2023-10-12T00:59:23.265 DEBUG:boto:Token: None 2023-10-12T00:59:23.266 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33 2023-10-12T00:59:23.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NWF8euHGY2womOu5u+rfkL7lUtE= 2023-10-12T00:59:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NWF8euHGY2womOu5u+rfkL7lUtE=', 'Content-Length': '0'} 2023-10-12T00:59:23.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d811da4417dd51c6-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T00:59:23.270 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.270 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.271 DEBUG:boto:Method: GET 2023-10-12T00:59:23.271 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T00:59:23.271 DEBUG:boto:Data: 2023-10-12T00:59:23.272 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.273 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.273 DEBUG:boto:Params: {} 2023-10-12T00:59:23.274 DEBUG:boto:Token: None 2023-10-12T00:59:23.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33 2023-10-12T00:59:23.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Fk43PLTUM70hzPjAf9CTSE0XuE= 2023-10-12T00:59:23.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Fk43PLTUM70hzPjAf9CTSE0XuE=', 'Content-Length': '0'} 2023-10-12T00:59:23.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca7d6d06b749f10-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.278 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T00:59:23.278 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T00:59:23.279 DEBUG:boto:Method: GET 2023-10-12T00:59:23.279 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T00:59:23.280 DEBUG:boto:Data: 2023-10-12T00:59:23.280 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.281 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.281 DEBUG:boto:Params: {} 2023-10-12T00:59:23.282 DEBUG:boto:Token: None 2023-10-12T00:59:23.282 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj33 2023-10-12T00:59:23.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9Fk43PLTUM70hzPjAf9CTSE0XuE= 2023-10-12T00:59:23.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9Fk43PLTUM70hzPjAf9CTSE0XuE=', 'Content-Length': '0'} 2023-10-12T00:59:23.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba5ed58c66e4675-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.286 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.286 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.287 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.287 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.288 DEBUG:boto:Method: GET 2023-10-12T00:59:23.288 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.288 DEBUG:boto:Data: 2023-10-12T00:59:23.289 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.290 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.290 DEBUG:boto:Params: {} 2023-10-12T00:59:23.291 DEBUG:boto:Token: None 2023-10-12T00:59:23.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7bc20bde1257a93-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.295 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.296 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.297 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.297 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.297 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.298 DEBUG:boto:Method: GET 2023-10-12T00:59:23.298 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T00:59:23.299 DEBUG:boto:Data: 2023-10-12T00:59:23.299 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.300 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.300 DEBUG:boto:Params: {} 2023-10-12T00:59:23.301 DEBUG:boto:Token: None 2023-10-12T00:59:23.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85f00fc08fa9adc-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.306 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T00:59:23.307 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.307 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.308 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.308 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.309 DEBUG:boto:Method: GET 2023-10-12T00:59:23.309 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.309 DEBUG:boto:Data: 2023-10-12T00:59:23.310 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.311 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.311 DEBUG:boto:Params: {} 2023-10-12T00:59:23.312 DEBUG:boto:Token: None 2023-10-12T00:59:23.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE= 2023-10-12T00:59:23.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE=', 'Content-Length': '0'} 2023-10-12T00:59:23.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22375c0c384e2c4-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.358 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.359 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.359 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.360 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.360 DEBUG:boto:Method: GET 2023-10-12T00:59:23.361 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.361 DEBUG:boto:Data: 2023-10-12T00:59:23.361 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.362 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.363 DEBUG:boto:Params: {} 2023-10-12T00:59:23.363 DEBUG:boto:Token: None 2023-10-12T00:59:23.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE= 2023-10-12T00:59:23.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE=', 'Content-Length': '0'} 2023-10-12T00:59:23.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1a103109d5771e-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.367 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.368 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.368 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.369 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.369 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.370 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.370 DEBUG:boto:Data: 2023-10-12T00:59:23.370 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.371 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.372 DEBUG:boto:Params: {} 2023-10-12T00:59:23.372 DEBUG:boto:Token: None 2023-10-12T00:59:23.372 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5Eeeydlhe5F0C0wmI3mvqn0HEXs= 2023-10-12T00:59:23.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5Eeeydlhe5F0C0wmI3mvqn0HEXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004b986ee4dad114a-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.376 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.377 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.377 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.377 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.378 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.378 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.379 DEBUG:boto:Data: 2023-10-12T00:59:23.379 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.380 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.380 DEBUG:boto:Params: {} 2023-10-12T00:59:23.381 DEBUG:boto:Token: None 2023-10-12T00:59:23.381 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5Eeeydlhe5F0C0wmI3mvqn0HEXs= 2023-10-12T00:59:23.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5Eeeydlhe5F0C0wmI3mvqn0HEXs=', 'Content-Length': '0'} 2023-10-12T00:59:23.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943d4e35b8c0a135-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T00:59:23.385 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.385 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.386 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.386 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.387 DEBUG:boto:Method: GET 2023-10-12T00:59:23.387 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.388 DEBUG:boto:Data: 2023-10-12T00:59:23.388 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.389 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.389 DEBUG:boto:Params: {} 2023-10-12T00:59:23.390 DEBUG:boto:Token: None 2023-10-12T00:59:23.390 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE= 2023-10-12T00:59:23.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE=', 'Content-Length': '0'} 2023-10-12T00:59:23.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff4b1f1f3cdbc29-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.394 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.394 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.395 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.395 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.396 DEBUG:boto:Method: GET 2023-10-12T00:59:23.396 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.397 DEBUG:boto:Data: 2023-10-12T00:59:23.397 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.398 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.398 DEBUG:boto:Params: {} 2023-10-12T00:59:23.399 DEBUG:boto:Token: None 2023-10-12T00:59:23.399 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T00:59:23.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE= 2023-10-12T00:59:23.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r/x9aWy/hW3iV+DdHrBYGgyd6lE=', 'Content-Length': '0'} 2023-10-12T00:59:23.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43524d5e9393ff5-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.403 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.403 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.404 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.404 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.405 DEBUG:boto:Method: GET 2023-10-12T00:59:23.405 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.405 DEBUG:boto:Data: 2023-10-12T00:59:23.406 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.407 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.407 DEBUG:boto:Params: {} 2023-10-12T00:59:23.408 DEBUG:boto:Token: None 2023-10-12T00:59:23.408 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.409 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec16b8f6df3ba0b3-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.412 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.413 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.413 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.414 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.414 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.415 DEBUG:boto:Method: GET 2023-10-12T00:59:23.415 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.415 DEBUG:boto:Data: 2023-10-12T00:59:23.416 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.417 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.417 DEBUG:boto:Params: {} 2023-10-12T00:59:23.418 DEBUG:boto:Token: None 2023-10-12T00:59:23.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e160dec038005cd0-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.462 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.463 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.464 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.464 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.465 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T00:59:23.465 DEBUG:boto:Data: 2023-10-12T00:59:23.465 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.466 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.467 DEBUG:boto:Params: {} 2023-10-12T00:59:23.467 DEBUG:boto:Token: None 2023-10-12T00:59:23.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44 2023-10-12T00:59:23.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BNI1TAs7+7sVf5ZyQmT2eTZhAUo= 2023-10-12T00:59:23.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BNI1TAs7+7sVf5ZyQmT2eTZhAUo=', 'Content-Length': '0'} 2023-10-12T00:59:23.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c0e143c906ed82-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.471 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.472 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.472 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.473 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T00:59:23.473 DEBUG:boto:Data: 2023-10-12T00:59:23.473 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.474 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.475 DEBUG:boto:Params: {} 2023-10-12T00:59:23.475 DEBUG:boto:Token: None 2023-10-12T00:59:23.476 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44 2023-10-12T00:59:23.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BNI1TAs7+7sVf5ZyQmT2eTZhAUo= 2023-10-12T00:59:23.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BNI1TAs7+7sVf5ZyQmT2eTZhAUo=', 'Content-Length': '0'} 2023-10-12T00:59:23.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ce77e6df0d5b02-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T00:59:23.480 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.480 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.481 DEBUG:boto:Method: GET 2023-10-12T00:59:23.481 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T00:59:23.482 DEBUG:boto:Data: 2023-10-12T00:59:23.482 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.483 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.484 DEBUG:boto:Params: {} 2023-10-12T00:59:23.484 DEBUG:boto:Token: None 2023-10-12T00:59:23.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44 2023-10-12T00:59:23.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4MqCqPy5L5D80dGLVYeJrp8V4ok= 2023-10-12T00:59:23.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4MqCqPy5L5D80dGLVYeJrp8V4ok=', 'Content-Length': '0'} 2023-10-12T00:59:23.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001116029333ae4aaa-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.488 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T00:59:23.489 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T00:59:23.489 DEBUG:boto:Method: GET 2023-10-12T00:59:23.489 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T00:59:23.490 DEBUG:boto:Data: 2023-10-12T00:59:23.490 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.491 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.492 DEBUG:boto:Params: {} 2023-10-12T00:59:23.492 DEBUG:boto:Token: None 2023-10-12T00:59:23.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj44 2023-10-12T00:59:23.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4MqCqPy5L5D80dGLVYeJrp8V4ok= 2023-10-12T00:59:23.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4MqCqPy5L5D80dGLVYeJrp8V4ok=', 'Content-Length': '0'} 2023-10-12T00:59:23.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ae88c0806e5efb-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.499 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.499 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.499 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.500 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.500 DEBUG:boto:Method: GET 2023-10-12T00:59:23.501 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.501 DEBUG:boto:Data: 2023-10-12T00:59:23.502 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.503 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.503 DEBUG:boto:Params: {} 2023-10-12T00:59:23.504 DEBUG:boto:Token: None 2023-10-12T00:59:23.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae08b59bc375d126-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.509 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.510 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.511 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.511 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.511 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.512 DEBUG:boto:Method: GET 2023-10-12T00:59:23.513 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T00:59:23.513 DEBUG:boto:Data: 2023-10-12T00:59:23.514 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.515 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.515 DEBUG:boto:Params: {} 2023-10-12T00:59:23.516 DEBUG:boto:Token: None 2023-10-12T00:59:23.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043f9d61b7ec942c5-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.521 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T00:59:23.522 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.522 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.523 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.523 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.524 DEBUG:boto:Method: GET 2023-10-12T00:59:23.524 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.525 DEBUG:boto:Data: 2023-10-12T00:59:23.525 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.526 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.526 DEBUG:boto:Params: {} 2023-10-12T00:59:23.527 DEBUG:boto:Token: None 2023-10-12T00:59:23.527 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY= 2023-10-12T00:59:23.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY=', 'Content-Length': '0'} 2023-10-12T00:59:23.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656e0892a0332221-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.532 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.532 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.533 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.533 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.533 DEBUG:boto:Method: GET 2023-10-12T00:59:23.534 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.534 DEBUG:boto:Data: 2023-10-12T00:59:23.535 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.536 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.536 DEBUG:boto:Params: {} 2023-10-12T00:59:23.537 DEBUG:boto:Token: None 2023-10-12T00:59:23.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY= 2023-10-12T00:59:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY=', 'Content-Length': '0'} 2023-10-12T00:59:23.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647c9b36cf737091-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.582 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.583 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.583 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.584 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.584 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.585 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.585 DEBUG:boto:Data: 2023-10-12T00:59:23.585 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.586 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.587 DEBUG:boto:Params: {} 2023-10-12T00:59:23.588 DEBUG:boto:Token: None 2023-10-12T00:59:23.589 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SqOpc5+23NkoMiLoQ8/qLle51s4= 2023-10-12T00:59:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SqOpc5+23NkoMiLoQ8/qLle51s4=', 'Content-Length': '0'} 2023-10-12T00:59:23.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d7f01c0049d27ff-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.593 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.593 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.594 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.594 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.595 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.595 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.596 DEBUG:boto:Data: 2023-10-12T00:59:23.596 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.597 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.597 DEBUG:boto:Params: {} 2023-10-12T00:59:23.598 DEBUG:boto:Token: None 2023-10-12T00:59:23.598 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.599 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SqOpc5+23NkoMiLoQ8/qLle51s4= 2023-10-12T00:59:23.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SqOpc5+23NkoMiLoQ8/qLle51s4=', 'Content-Length': '0'} 2023-10-12T00:59:23.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076679cc6bf2ee838-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T00:59:23.603 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.603 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.604 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.604 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.604 DEBUG:boto:Method: GET 2023-10-12T00:59:23.605 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.605 DEBUG:boto:Data: 2023-10-12T00:59:23.606 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.607 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.607 DEBUG:boto:Params: {} 2023-10-12T00:59:23.607 DEBUG:boto:Token: None 2023-10-12T00:59:23.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY= 2023-10-12T00:59:23.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY=', 'Content-Length': '0'} 2023-10-12T00:59:23.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9bed03c55baf94c-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.612 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.612 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.613 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.613 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.614 DEBUG:boto:Method: GET 2023-10-12T00:59:23.614 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.615 DEBUG:boto:Data: 2023-10-12T00:59:23.615 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.616 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.616 DEBUG:boto:Params: {} 2023-10-12T00:59:23.617 DEBUG:boto:Token: None 2023-10-12T00:59:23.617 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T00:59:23.618 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY= 2023-10-12T00:59:23.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fM9hyfLiCySm8gsicsc7cFmQNrY=', 'Content-Length': '0'} 2023-10-12T00:59:23.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008981f87b41bb1d24-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.621 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.622 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.622 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.623 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.623 DEBUG:boto:Method: GET 2023-10-12T00:59:23.624 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.624 DEBUG:boto:Data: 2023-10-12T00:59:23.625 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.626 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.626 DEBUG:boto:Params: {} 2023-10-12T00:59:23.627 DEBUG:boto:Token: None 2023-10-12T00:59:23.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017201eb31efed21c-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.674 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.677 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.677 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.677 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.678 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.678 DEBUG:boto:Method: GET 2023-10-12T00:59:23.679 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.679 DEBUG:boto:Data: 2023-10-12T00:59:23.680 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.681 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.681 DEBUG:boto:Params: {} 2023-10-12T00:59:23.681 DEBUG:boto:Token: None 2023-10-12T00:59:23.682 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b89ff72139dd9af-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.687 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.688 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.688 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.688 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.689 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T00:59:23.689 DEBUG:boto:Data: 2023-10-12T00:59:23.690 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.691 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.691 DEBUG:boto:Params: {} 2023-10-12T00:59:23.692 DEBUG:boto:Token: None 2023-10-12T00:59:23.692 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55 2023-10-12T00:59:23.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVd90B3b/hQWZlzTBd+OB5ZMFLM= 2023-10-12T00:59:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVd90B3b/hQWZlzTBd+OB5ZMFLM=', 'Content-Length': '0'} 2023-10-12T00:59:23.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5559f1bbb43e56-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.696 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.696 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.697 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.697 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T00:59:23.698 DEBUG:boto:Data: 2023-10-12T00:59:23.698 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.699 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.700 DEBUG:boto:Params: {} 2023-10-12T00:59:23.700 DEBUG:boto:Token: None 2023-10-12T00:59:23.701 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55 2023-10-12T00:59:23.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVd90B3b/hQWZlzTBd+OB5ZMFLM= 2023-10-12T00:59:23.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVd90B3b/hQWZlzTBd+OB5ZMFLM=', 'Content-Length': '0'} 2023-10-12T00:59:23.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3a92e9c01cf9cb-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T00:59:23.705 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.706 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.706 DEBUG:boto:Method: GET 2023-10-12T00:59:23.707 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T00:59:23.707 DEBUG:boto:Data: 2023-10-12T00:59:23.708 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.708 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.709 DEBUG:boto:Params: {} 2023-10-12T00:59:23.709 DEBUG:boto:Token: None 2023-10-12T00:59:23.710 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55 2023-10-12T00:59:23.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:00Gzc7LiV4a4ySZGWzhwnPCfWaU= 2023-10-12T00:59:23.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:00Gzc7LiV4a4ySZGWzhwnPCfWaU=', 'Content-Length': '0'} 2023-10-12T00:59:23.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093bd5d3f9b76d5f0-00652744eb-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.714 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T00:59:23.714 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T00:59:23.715 DEBUG:boto:Method: GET 2023-10-12T00:59:23.715 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T00:59:23.715 DEBUG:boto:Data: 2023-10-12T00:59:23.716 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.717 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.717 DEBUG:boto:Params: {} 2023-10-12T00:59:23.718 DEBUG:boto:Token: None 2023-10-12T00:59:23.718 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj55 2023-10-12T00:59:23.719 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:00Gzc7LiV4a4ySZGWzhwnPCfWaU= 2023-10-12T00:59:23.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:00Gzc7LiV4a4ySZGWzhwnPCfWaU=', 'Content-Length': '0'} 2023-10-12T00:59:23.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d9102b9006ae25-00652744eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.722 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.722 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.723 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.723 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.724 DEBUG:boto:Method: GET 2023-10-12T00:59:23.724 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.725 DEBUG:boto:Data: 2023-10-12T00:59:23.725 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T00:59:23.726 DEBUG:boto:Port: 8001 2023-10-12T00:59:23.727 DEBUG:boto:Params: {} 2023-10-12T00:59:23.727 DEBUG:boto:Token: None 2023-10-12T00:59:23.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e02ff4995a5a52-00652744eb-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.732 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.733 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.733 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.734 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.734 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.735 DEBUG:boto:Method: GET 2023-10-12T00:59:23.735 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T00:59:23.736 DEBUG:boto:Data: 2023-10-12T00:59:23.736 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.738 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.738 DEBUG:boto:Params: {} 2023-10-12T00:59:23.738 DEBUG:boto:Token: None 2023-10-12T00:59:23.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4= 2023-10-12T00:59:23.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qNOdyIGWknFRTHDI171tzYJl6X4=', 'Content-Length': '0'} 2023-10-12T00:59:23.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2653071990d7d9b-00652744eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.743 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T00:59:23.744 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T00:59:23.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:59:23.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T00:59:23.745 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T00:59:23.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:59:23.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:59:23.858 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T00:59:23.859 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T00:59:23.862 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-10-12T00:59:23.899 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.900 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.900 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.901 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:23.901 DEBUG:boto:Data: 2023-10-12T00:59:23.902 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.902 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.903 DEBUG:boto:Params: {} 2023-10-12T00:59:23.903 DEBUG:boto:Token: None 2023-10-12T00:59:23.904 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk= 2023-10-12T00:59:23.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk=', 'Content-Length': '0'} 2023-10-12T00:59:23.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2885dfe8387bf18-00652744eb-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.908 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T00:59:23.909 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T00:59:23.909 DEBUG:boto:Method: PUT 2023-10-12T00:59:23.910 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T00:59:23.911 DEBUG:boto:Data: 2023-10-12T00:59:23.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:59:23.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.912 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.913 DEBUG:boto:Params: {} 2023-10-12T00:59:23.913 DEBUG:boto:Token: None 2023-10-12T00:59:23.914 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj17 2023-10-12T00:59:23.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JSIewZoOKmceEIE6KhPT5fljcz8= 2023-10-12T00:59:23.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JSIewZoOKmceEIE6KhPT5fljcz8='} 2023-10-12T00:59:23.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c447232e23940f1d-00652744eb-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.959 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.960 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.960 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.960 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:23.961 DEBUG:boto:Data: 2023-10-12T00:59:23.961 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.962 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.963 DEBUG:boto:Params: {} 2023-10-12T00:59:23.963 DEBUG:boto:Token: None 2023-10-12T00:59:23.964 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk= 2023-10-12T00:59:23.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk=', 'Content-Length': '0'} 2023-10-12T00:59:23.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a782adc44b292f-00652744eb-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.968 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T00:59:23.969 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T00:59:23.969 DEBUG:boto:Method: PUT 2023-10-12T00:59:23.969 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T00:59:23.970 DEBUG:boto:Data: 2023-10-12T00:59:23.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:59:23.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.971 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.972 DEBUG:boto:Params: {} 2023-10-12T00:59:23.972 DEBUG:boto:Token: None 2023-10-12T00:59:23.973 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj34 2023-10-12T00:59:23.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:myLUL1LczFAFWgA3MBV0fGn171E= 2023-10-12T00:59:23.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:myLUL1LczFAFWgA3MBV0fGn171E='} 2023-10-12T00:59:23.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024b6668573309c0b-00652744eb-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.979 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:23.980 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:23.980 DEBUG:boto:Method: HEAD 2023-10-12T00:59:23.981 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:23.981 DEBUG:boto:Data: 2023-10-12T00:59:23.982 DEBUG:boto:Headers: {} 2023-10-12T00:59:23.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.982 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.983 DEBUG:boto:Params: {} 2023-10-12T00:59:23.983 DEBUG:boto:Token: None 2023-10-12T00:59:23.984 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/ 2023-10-12T00:59:23.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk= 2023-10-12T00:59:23.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYVbfFIxjQjKFWI6lbkh1lhrsmk=', 'Content-Length': '0'} 2023-10-12T00:59:23.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d6844c13b52f230-00652744eb-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:23.988 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T00:59:23.988 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T00:59:23.989 DEBUG:boto:Method: PUT 2023-10-12T00:59:23.989 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T00:59:23.990 DEBUG:boto:Data: 2023-10-12T00:59:23.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:59:23.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:23.991 DEBUG:boto:Port: 8000 2023-10-12T00:59:23.991 DEBUG:boto:Params: {} 2023-10-12T00:59:23.992 DEBUG:boto:Token: None 2023-10-12T00:59:23.992 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:59:23 GMT /wafmoa-59/obj51 2023-10-12T00:59:23.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FapElzllEnR0HVP8yj81mEaIoiA= 2023-10-12T00:59:23.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:59:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FapElzllEnR0HVP8yj81mEaIoiA='} 2023-10-12T00:59:24.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f4661f74e7abc3e-00652744eb-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:24.039 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:24.040 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:24.040 DEBUG:boto:Method: HEAD 2023-10-12T00:59:24.041 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:24.041 DEBUG:boto:Data: 2023-10-12T00:59:24.042 DEBUG:boto:Headers: {} 2023-10-12T00:59:24.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:24.043 DEBUG:boto:Port: 8000 2023-10-12T00:59:24.043 DEBUG:boto:Params: {} 2023-10-12T00:59:24.044 DEBUG:boto:Token: None 2023-10-12T00:59:24.044 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:24 GMT /wafmoa-59/ 2023-10-12T00:59:24.045 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eNT4Idhyb+v2eQcvG1n5IjR4x2Q= 2023-10-12T00:59:24.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eNT4Idhyb+v2eQcvG1n5IjR4x2Q=', 'Content-Length': '0'} 2023-10-12T00:59:24.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbcc709666b9636b-00652744ec-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:24.049 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T00:59:24.050 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T00:59:24.050 DEBUG:boto:Method: PUT 2023-10-12T00:59:24.051 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T00:59:24.051 DEBUG:boto:Data: 2023-10-12T00:59:24.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:59:24.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:24.053 DEBUG:boto:Port: 8000 2023-10-12T00:59:24.053 DEBUG:boto:Params: {} 2023-10-12T00:59:24.054 DEBUG:boto:Token: None 2023-10-12T00:59:24.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:59:24 GMT /wafmoa-59/obj68 2023-10-12T00:59:24.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:31swAdqtdSMWxZljqfXEop8UIg8= 2023-10-12T00:59:24.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:59:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:31swAdqtdSMWxZljqfXEop8UIg8='} 2023-10-12T00:59:24.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1c7366874704950-00652744ec-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:24.061 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T00:59:24.062 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T00:59:24.063 DEBUG:boto:Method: HEAD 2023-10-12T00:59:24.063 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T00:59:24.063 DEBUG:boto:Data: 2023-10-12T00:59:24.064 DEBUG:boto:Headers: {} 2023-10-12T00:59:24.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:24.065 DEBUG:boto:Port: 8000 2023-10-12T00:59:24.065 DEBUG:boto:Params: {} 2023-10-12T00:59:24.066 DEBUG:boto:Token: None 2023-10-12T00:59:24.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 00:59:24 GMT /wafmoa-59/ 2023-10-12T00:59:24.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eNT4Idhyb+v2eQcvG1n5IjR4x2Q= 2023-10-12T00:59:24.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 00:59:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eNT4Idhyb+v2eQcvG1n5IjR4x2Q=', 'Content-Length': '0'} 2023-10-12T00:59:24.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f990e3f8488de175-00652744ec-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:24.070 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T00:59:24.070 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T00:59:24.071 DEBUG:boto:Method: PUT 2023-10-12T00:59:24.071 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T00:59:24.072 DEBUG:boto:Data: 2023-10-12T00:59:24.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T00:59:24.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T00:59:24.073 DEBUG:boto:Port: 8000 2023-10-12T00:59:24.073 DEBUG:boto:Params: {} 2023-10-12T00:59:24.074 DEBUG:boto:Token: None 2023-10-12T00:59:24.074 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 00:59:24 GMT /wafmoa-59/obj85 2023-10-12T00:59:24.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MbSBHuZlJHjYhqK55RvEfEUj/jo= 2023-10-12T00:59:24.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 00:59:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MbSBHuZlJHjYhqK55RvEfEUj/jo='} 2023-10-12T00:59:24.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e4c35642159eb38-00652744ec-168d-a1'), ('Date', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T00:59:24.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T00:59:24.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:59:24.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T00:59:24.085 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T00:59:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T00:59:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T00:59:24.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:24.173+0000 7fbd218b5780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:59:29.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:29.174+0000 7fbd218b5780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:59:34.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:34.175+0000 7fbd218b5780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:59:39.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:39.176+0000 7fbd218b5780 1 waiting to reach latest gen 1, on 0.. 2023-10-12T00:59:44.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:44.178+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:44.178 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-12T00:59:44.178 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T00:59:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:49.178+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:49.179 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-12T00:59:49.180 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T00:59:54.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:54.180+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:54.181 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-12T00:59:54.181 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T00:59:59.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T00:59:59.182+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T00:59:59.184 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-12T00:59:59.184 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:04.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:04.184+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:04.185 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.70.6, , , , , , , , , , ] 2023-10-12T01:00:04.185 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:09.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:09.185+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:09.186 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.70.6, , , , , , , , , , ] 2023-10-12T01:00:09.186 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:14.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:14.187+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:14.188 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.70.6, , , , , , , , , , ] 2023-10-12T01:00:14.188 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:19.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:19.189+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:19.189 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.70.6, , , , , , , , , , ] 2023-10-12T01:00:19.189 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:24.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:24.190+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:24.191 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, , , , ] 2023-10-12T01:00:24.191 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:29.191+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:29.193 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, , , , ] 2023-10-12T01:00:29.193 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:34.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:34.193+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:34.194 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, , , , ] 2023-10-12T01:00:34.194 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:39.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:39.194+0000 7fbd218b5780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:39.196 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, , , , ] 2023-10-12T01:00:39.196 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:44.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:44.196+0000 7fbd218b5780 1 bucket sync caught up with source: 2023-10-12T01:00:44.197 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:44.197 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.88.6, , , , , , 00000000001.70.6, , 00000000001.54.6, , , , 00000000001.56.6, 00000000001.102.6, , , ] 2023-10-12T01:00:44.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:44.196+0000 7fbd218b5780 0 bucket checkpoint complete 2023-10-12T01:00:44.214 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:00:44.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:00:44.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:00:44.216 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:00:44.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:00:44.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:00:44.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:44.318+0000 7f7e92ae6780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:44.319 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.324.6, , , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:44.320 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:49.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:49.321+0000 7f7e92ae6780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:49.321 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.324.6, , , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:49.321 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:54.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:54.322+0000 7f7e92ae6780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:54.322 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.324.6, , , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:54.322 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:59.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:00:59.323+0000 7f7e92ae6780 1 waiting for incremental sync to catch up: 2023-10-12T01:00:59.323 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.324.6, , , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:00:59.323 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:01:04.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:04.324+0000 7f7e92ae6780 1 bucket sync caught up with source: 2023-10-12T01:01:04.325 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:01:04.325 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.324.6, 00000000001.281.6, , , 00000000001.181.6, , , 00000000001.271.6, 00000000003.234.6, 00000000003.179.6] 2023-10-12T01:01:04.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:04.324+0000 7f7e92ae6780 0 bucket checkpoint complete 2023-10-12T01:01:04.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:01:04.343 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.344 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.345 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.346 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:04.347 DEBUG:boto:Data: 2023-10-12T01:01:04.347 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.349 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.350 DEBUG:boto:Params: {} 2023-10-12T01:01:04.351 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:01:04.352 DEBUG:boto:Token: None 2023-10-12T01:01:04.353 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q+qF5LBvsZh+LAdweb7UyXSzqQY= 2023-10-12T01:01:04.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q+qF5LBvsZh+LAdweb7UyXSzqQY=', 'Content-Length': '0'} 2023-10-12T01:01:04.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac148819bf972191-0065274550-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.363 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.364 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.364 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.365 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:04.365 DEBUG:boto:Data: 2023-10-12T01:01:04.366 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.367 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.367 DEBUG:boto:Params: {} 2023-10-12T01:01:04.368 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:01:04.368 DEBUG:boto:Token: None 2023-10-12T01:01:04.369 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q+qF5LBvsZh+LAdweb7UyXSzqQY= 2023-10-12T01:01:04.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q+qF5LBvsZh+LAdweb7UyXSzqQY=', 'Content-Length': '0'} 2023-10-12T01:01:04.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000048a5b651cc36080-0065274550-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:01:04.375 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.375 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.376 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.376 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.377 DEBUG:boto:Method: GET 2023-10-12T01:01:04.377 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.378 DEBUG:boto:Data: 2023-10-12T01:01:04.378 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.379 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.379 DEBUG:boto:Params: {} 2023-10-12T01:01:04.380 DEBUG:boto:Token: None 2023-10-12T01:01:04.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/?versions 2023-10-12T01:01:04.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A= 2023-10-12T01:01:04.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A=', 'Content-Length': '0'} 2023-10-12T01:01:04.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6caf1ce5069abf1-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.385 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:01:04.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:01:04.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:01:04.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:01:04.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:01:04.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:01:04.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:01:04.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:01:04.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:01:04.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:01:04.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:01:04.391 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.391 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.391 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.392 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.392 DEBUG:boto:Method: GET 2023-10-12T01:01:04.393 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.393 DEBUG:boto:Data: 2023-10-12T01:01:04.394 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.394 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.395 DEBUG:boto:Params: {} 2023-10-12T01:01:04.395 DEBUG:boto:Token: None 2023-10-12T01:01:04.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/?versions 2023-10-12T01:01:04.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A= 2023-10-12T01:01:04.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A=', 'Content-Length': '0'} 2023-10-12T01:01:04.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a0578425243912-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.401 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:01:04.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:01:04.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:01:04.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:01:04.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:01:04.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:01:04.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:01:04.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:01:04.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:01:04.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:01:04.405 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.406 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.406 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.407 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.407 DEBUG:boto:Method: GET 2023-10-12T01:01:04.408 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.408 DEBUG:boto:Data: 2023-10-12T01:01:04.408 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.409 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.410 DEBUG:boto:Params: {} 2023-10-12T01:01:04.410 DEBUG:boto:Token: None 2023-10-12T01:01:04.411 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/?versions 2023-10-12T01:01:04.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A= 2023-10-12T01:01:04.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A=', 'Content-Length': '0'} 2023-10-12T01:01:04.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bd0222c2be002e-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.415 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.416 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.416 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.417 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.417 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.418 DEBUG:boto:Method: GET 2023-10-12T01:01:04.418 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:01:04.418 DEBUG:boto:Data: 2023-10-12T01:01:04.419 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.420 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.420 DEBUG:boto:Params: {} 2023-10-12T01:01:04.420 DEBUG:boto:Token: None 2023-10-12T01:01:04.421 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/?versions 2023-10-12T01:01:04.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A= 2023-10-12T01:01:04.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gply3F1HQppRKkkvJCxiihyQ/+A=', 'Content-Length': '0'} 2023-10-12T01:01:04.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1d86b3d27a2b37-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.425 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:01:04.427 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.427 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.428 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.428 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.428 DEBUG:boto:Method: GET 2023-10-12T01:01:04.429 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.429 DEBUG:boto:Data: 2023-10-12T01:01:04.430 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.431 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.431 DEBUG:boto:Params: {} 2023-10-12T01:01:04.432 DEBUG:boto:Token: None 2023-10-12T01:01:04.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w= 2023-10-12T01:01:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w=', 'Content-Length': '0'} 2023-10-12T01:01:04.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3324df9ea74f1cc-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.436 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.436 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.437 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.437 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.437 DEBUG:boto:Method: GET 2023-10-12T01:01:04.438 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.438 DEBUG:boto:Data: 2023-10-12T01:01:04.439 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.440 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.440 DEBUG:boto:Params: {} 2023-10-12T01:01:04.441 DEBUG:boto:Token: None 2023-10-12T01:01:04.441 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w= 2023-10-12T01:01:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w=', 'Content-Length': '0'} 2023-10-12T01:01:04.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0adf10b74dae78-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.445 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.446 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.446 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.447 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.447 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.447 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.448 DEBUG:boto:Data: 2023-10-12T01:01:04.448 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.449 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.449 DEBUG:boto:Params: {} 2023-10-12T01:01:04.450 DEBUG:boto:Token: None 2023-10-12T01:01:04.450 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErF7pQTavnw1N6rzVt2n6pk2qUA= 2023-10-12T01:01:04.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErF7pQTavnw1N6rzVt2n6pk2qUA=', 'Content-Length': '0'} 2023-10-12T01:01:04.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000849effbbb2d5b59-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.454 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.455 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.455 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.455 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.456 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.456 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.457 DEBUG:boto:Data: 2023-10-12T01:01:04.457 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.458 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.458 DEBUG:boto:Params: {} 2023-10-12T01:01:04.459 DEBUG:boto:Token: None 2023-10-12T01:01:04.459 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ErF7pQTavnw1N6rzVt2n6pk2qUA= 2023-10-12T01:01:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ErF7pQTavnw1N6rzVt2n6pk2qUA=', 'Content-Length': '0'} 2023-10-12T01:01:04.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a41af45e3ad972b0-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:01:04.464 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.464 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.465 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.465 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.466 DEBUG:boto:Method: GET 2023-10-12T01:01:04.466 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.467 DEBUG:boto:Data: 2023-10-12T01:01:04.467 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.468 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.469 DEBUG:boto:Params: {} 2023-10-12T01:01:04.469 DEBUG:boto:Token: None 2023-10-12T01:01:04.470 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w= 2023-10-12T01:01:04.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w=', 'Content-Length': '0'} 2023-10-12T01:01:04.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6d2eee969750ad-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.515 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.516 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.516 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.517 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.517 DEBUG:boto:Method: GET 2023-10-12T01:01:04.518 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.518 DEBUG:boto:Data: 2023-10-12T01:01:04.519 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.520 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.520 DEBUG:boto:Params: {} 2023-10-12T01:01:04.520 DEBUG:boto:Token: None 2023-10-12T01:01:04.521 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:01:04.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w= 2023-10-12T01:01:04.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:12wJQoYJnDyQ4pnOoYp0qdii20w=', 'Content-Length': '0'} 2023-10-12T01:01:04.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fef759d64684c7e-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.525 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.525 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.526 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.526 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.526 DEBUG:boto:Method: GET 2023-10-12T01:01:04.527 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.527 DEBUG:boto:Data: 2023-10-12T01:01:04.528 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.529 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.529 DEBUG:boto:Params: {} 2023-10-12T01:01:04.530 DEBUG:boto:Token: None 2023-10-12T01:01:04.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab989fe2b4082b9-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.534 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.535 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.535 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.536 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.536 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.537 DEBUG:boto:Method: GET 2023-10-12T01:01:04.537 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.538 DEBUG:boto:Data: 2023-10-12T01:01:04.538 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.539 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.539 DEBUG:boto:Params: {} 2023-10-12T01:01:04.540 DEBUG:boto:Token: None 2023-10-12T01:01:04.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae90d7fe146829ab-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.545 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.545 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.546 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.546 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.547 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:01:04.547 DEBUG:boto:Data: 2023-10-12T01:01:04.548 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.548 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.549 DEBUG:boto:Params: {} 2023-10-12T01:01:04.549 DEBUG:boto:Token: None 2023-10-12T01:01:04.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11 2023-10-12T01:01:04.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5fZD5sa0G3hsdjxPnSNGa9El40Y= 2023-10-12T01:01:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5fZD5sa0G3hsdjxPnSNGa9El40Y=', 'Content-Length': '0'} 2023-10-12T01:01:04.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d8af3f8051e2d9-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.553 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.554 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.554 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.555 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:01:04.556 DEBUG:boto:Data: 2023-10-12T01:01:04.557 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.558 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.561 DEBUG:boto:Params: {} 2023-10-12T01:01:04.562 DEBUG:boto:Token: None 2023-10-12T01:01:04.563 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11 2023-10-12T01:01:04.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5fZD5sa0G3hsdjxPnSNGa9El40Y= 2023-10-12T01:01:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5fZD5sa0G3hsdjxPnSNGa9El40Y=', 'Content-Length': '0'} 2023-10-12T01:01:04.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1fe4e137578fdd7-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:01:04.569 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.570 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.570 DEBUG:boto:Method: GET 2023-10-12T01:01:04.571 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:01:04.572 DEBUG:boto:Data: 2023-10-12T01:01:04.573 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.574 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.575 DEBUG:boto:Params: {} 2023-10-12T01:01:04.576 DEBUG:boto:Token: None 2023-10-12T01:01:04.576 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11 2023-10-12T01:01:04.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AJqI6M4LdVv0gWdSTkLQ1pUuXVE= 2023-10-12T01:01:04.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AJqI6M4LdVv0gWdSTkLQ1pUuXVE=', 'Content-Length': '0'} 2023-10-12T01:01:04.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6fe690bb54e9c6d-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.580 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:01:04.581 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:01:04.581 DEBUG:boto:Method: GET 2023-10-12T01:01:04.582 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:01:04.582 DEBUG:boto:Data: 2023-10-12T01:01:04.583 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.584 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.584 DEBUG:boto:Params: {} 2023-10-12T01:01:04.584 DEBUG:boto:Token: None 2023-10-12T01:01:04.585 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj11 2023-10-12T01:01:04.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AJqI6M4LdVv0gWdSTkLQ1pUuXVE= 2023-10-12T01:01:04.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AJqI6M4LdVv0gWdSTkLQ1pUuXVE=', 'Content-Length': '0'} 2023-10-12T01:01:04.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8092769e1d4733-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.630 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.631 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.632 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.632 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.633 DEBUG:boto:Method: GET 2023-10-12T01:01:04.634 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.635 DEBUG:boto:Data: 2023-10-12T01:01:04.636 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.637 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.638 DEBUG:boto:Params: {} 2023-10-12T01:01:04.639 DEBUG:boto:Token: None 2023-10-12T01:01:04.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c1da9648e422582-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.645 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.646 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.647 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.647 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.648 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.648 DEBUG:boto:Method: GET 2023-10-12T01:01:04.649 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:01:04.649 DEBUG:boto:Data: 2023-10-12T01:01:04.649 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.650 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.651 DEBUG:boto:Params: {} 2023-10-12T01:01:04.651 DEBUG:boto:Token: None 2023-10-12T01:01:04.652 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba79aaf4a9a99649-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.657 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:01:04.659 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.660 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.661 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.662 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.663 DEBUG:boto:Method: GET 2023-10-12T01:01:04.663 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.664 DEBUG:boto:Data: 2023-10-12T01:01:04.664 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.665 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.665 DEBUG:boto:Params: {} 2023-10-12T01:01:04.666 DEBUG:boto:Token: None 2023-10-12T01:01:04.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0= 2023-10-12T01:01:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0=', 'Content-Length': '0'} 2023-10-12T01:01:04.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331365563077048b-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.672 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.673 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.674 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.674 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.675 DEBUG:boto:Method: GET 2023-10-12T01:01:04.676 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.676 DEBUG:boto:Data: 2023-10-12T01:01:04.677 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.679 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.679 DEBUG:boto:Params: {} 2023-10-12T01:01:04.680 DEBUG:boto:Token: None 2023-10-12T01:01:04.680 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.681 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0= 2023-10-12T01:01:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0=', 'Content-Length': '0'} 2023-10-12T01:01:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b88a27a00de4b6-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.685 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.686 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.686 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.687 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.687 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.688 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.688 DEBUG:boto:Data: 2023-10-12T01:01:04.688 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.689 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.690 DEBUG:boto:Params: {} 2023-10-12T01:01:04.690 DEBUG:boto:Token: None 2023-10-12T01:01:04.691 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.691 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BZ5BIVu8bdf7a66p5YXC21eQFk4= 2023-10-12T01:01:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BZ5BIVu8bdf7a66p5YXC21eQFk4=', 'Content-Length': '0'} 2023-10-12T01:01:04.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71292658c4b9e86-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.694 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.695 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.695 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.695 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.696 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.696 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.697 DEBUG:boto:Data: 2023-10-12T01:01:04.697 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.698 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.699 DEBUG:boto:Params: {} 2023-10-12T01:01:04.700 DEBUG:boto:Token: None 2023-10-12T01:01:04.700 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BZ5BIVu8bdf7a66p5YXC21eQFk4= 2023-10-12T01:01:04.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BZ5BIVu8bdf7a66p5YXC21eQFk4=', 'Content-Length': '0'} 2023-10-12T01:01:04.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c618c851f42cbb-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:01:04.706 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.707 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.708 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.708 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.709 DEBUG:boto:Method: GET 2023-10-12T01:01:04.710 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.710 DEBUG:boto:Data: 2023-10-12T01:01:04.711 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.712 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.713 DEBUG:boto:Params: {} 2023-10-12T01:01:04.713 DEBUG:boto:Token: None 2023-10-12T01:01:04.714 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0= 2023-10-12T01:01:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0=', 'Content-Length': '0'} 2023-10-12T01:01:04.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df7b4ae4b0fd5eb-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.719 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.720 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.721 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.721 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.722 DEBUG:boto:Method: GET 2023-10-12T01:01:04.722 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.723 DEBUG:boto:Data: 2023-10-12T01:01:04.723 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.724 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.724 DEBUG:boto:Params: {} 2023-10-12T01:01:04.725 DEBUG:boto:Token: None 2023-10-12T01:01:04.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:01:04.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0= 2023-10-12T01:01:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+3bT9bO6OtEIURTR35ggcfmqVJ0=', 'Content-Length': '0'} 2023-10-12T01:01:04.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe23db2d4f897a78-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.730 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.730 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.731 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.731 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.732 DEBUG:boto:Method: GET 2023-10-12T01:01:04.732 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.732 DEBUG:boto:Data: 2023-10-12T01:01:04.733 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.734 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.734 DEBUG:boto:Params: {} 2023-10-12T01:01:04.735 DEBUG:boto:Token: None 2023-10-12T01:01:04.735 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095ea203a242eac5-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.786 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.787 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.788 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.789 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.789 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.789 DEBUG:boto:Method: GET 2023-10-12T01:01:04.790 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.790 DEBUG:boto:Data: 2023-10-12T01:01:04.791 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.792 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.792 DEBUG:boto:Params: {} 2023-10-12T01:01:04.793 DEBUG:boto:Token: None 2023-10-12T01:01:04.793 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab8ef3afbe3145c-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.798 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.799 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.799 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.800 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.800 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:01:04.801 DEBUG:boto:Data: 2023-10-12T01:01:04.801 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.802 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.803 DEBUG:boto:Params: {} 2023-10-12T01:01:04.803 DEBUG:boto:Token: None 2023-10-12T01:01:04.804 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17 2023-10-12T01:01:04.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TVSY9Ow/wtgI6VX/poqNPg/cMGs= 2023-10-12T01:01:04.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TVSY9Ow/wtgI6VX/poqNPg/cMGs=', 'Content-Length': '0'} 2023-10-12T01:01:04.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b787fadba2b6089-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.808 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.808 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.808 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.809 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:01:04.809 DEBUG:boto:Data: 2023-10-12T01:01:04.810 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.811 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.812 DEBUG:boto:Params: {} 2023-10-12T01:01:04.812 DEBUG:boto:Token: None 2023-10-12T01:01:04.813 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17 2023-10-12T01:01:04.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TVSY9Ow/wtgI6VX/poqNPg/cMGs= 2023-10-12T01:01:04.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TVSY9Ow/wtgI6VX/poqNPg/cMGs=', 'Content-Length': '0'} 2023-10-12T01:01:04.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9febc8a5e5d964-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:01:04.817 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.818 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.818 DEBUG:boto:Method: GET 2023-10-12T01:01:04.819 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:01:04.819 DEBUG:boto:Data: 2023-10-12T01:01:04.820 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.821 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.821 DEBUG:boto:Params: {} 2023-10-12T01:01:04.822 DEBUG:boto:Token: None 2023-10-12T01:01:04.823 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17 2023-10-12T01:01:04.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qApUmlimikhxtrkuEvaPnvAbeRg= 2023-10-12T01:01:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qApUmlimikhxtrkuEvaPnvAbeRg=', 'Content-Length': '0'} 2023-10-12T01:01:04.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa152cbf2008c083-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.828 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:01:04.829 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:01:04.830 DEBUG:boto:Method: GET 2023-10-12T01:01:04.831 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:01:04.831 DEBUG:boto:Data: 2023-10-12T01:01:04.832 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.834 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.834 DEBUG:boto:Params: {} 2023-10-12T01:01:04.835 DEBUG:boto:Token: None 2023-10-12T01:01:04.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj17 2023-10-12T01:01:04.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qApUmlimikhxtrkuEvaPnvAbeRg= 2023-10-12T01:01:04.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qApUmlimikhxtrkuEvaPnvAbeRg=', 'Content-Length': '0'} 2023-10-12T01:01:04.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096bfcc35b8f30fc8-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.842 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.843 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.844 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.844 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.845 DEBUG:boto:Method: GET 2023-10-12T01:01:04.846 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.847 DEBUG:boto:Data: 2023-10-12T01:01:04.848 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.849 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.850 DEBUG:boto:Params: {} 2023-10-12T01:01:04.851 DEBUG:boto:Token: None 2023-10-12T01:01:04.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412683b80a8a849c-0065274550-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.902 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.904 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.905 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.905 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.906 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.907 DEBUG:boto:Method: GET 2023-10-12T01:01:04.907 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:01:04.908 DEBUG:boto:Data: 2023-10-12T01:01:04.909 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.910 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.911 DEBUG:boto:Params: {} 2023-10-12T01:01:04.911 DEBUG:boto:Token: None 2023-10-12T01:01:04.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:04.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a42fcfafefb899-0065274550-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.918 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:04.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:01:04.919 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.920 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.921 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.921 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.922 DEBUG:boto:Method: GET 2023-10-12T01:01:04.923 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.923 DEBUG:boto:Data: 2023-10-12T01:01:04.924 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.925 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.926 DEBUG:boto:Params: {} 2023-10-12T01:01:04.927 DEBUG:boto:Token: None 2023-10-12T01:01:04.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY= 2023-10-12T01:01:04.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY=', 'Content-Length': '0'} 2023-10-12T01:01:04.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008332f6d6810d6f9f-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.932 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.933 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.934 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.934 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.935 DEBUG:boto:Method: GET 2023-10-12T01:01:04.936 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.936 DEBUG:boto:Data: 2023-10-12T01:01:04.937 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.938 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.939 DEBUG:boto:Params: {} 2023-10-12T01:01:04.940 DEBUG:boto:Token: None 2023-10-12T01:01:04.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY= 2023-10-12T01:01:04.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY=', 'Content-Length': '0'} 2023-10-12T01:01:04.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6e03338372b362-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.946 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.947 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.947 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.948 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.948 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.949 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.950 DEBUG:boto:Data: 2023-10-12T01:01:04.950 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.952 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.952 DEBUG:boto:Params: {} 2023-10-12T01:01:04.953 DEBUG:boto:Token: None 2023-10-12T01:01:04.954 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OIBWcY5q4PoaEtK+dbz4yVioL5k= 2023-10-12T01:01:04.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OIBWcY5q4PoaEtK+dbz4yVioL5k=', 'Content-Length': '0'} 2023-10-12T01:01:04.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119c2110fc0d95a0-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.959 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.959 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.960 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.960 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.961 DEBUG:boto:Method: HEAD 2023-10-12T01:01:04.962 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.962 DEBUG:boto:Data: 2023-10-12T01:01:04.963 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.964 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.965 DEBUG:boto:Params: {} 2023-10-12T01:01:04.966 DEBUG:boto:Token: None 2023-10-12T01:01:04.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OIBWcY5q4PoaEtK+dbz4yVioL5k= 2023-10-12T01:01:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OIBWcY5q4PoaEtK+dbz4yVioL5k=', 'Content-Length': '0'} 2023-10-12T01:01:04.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f12cb5ee98096bf5-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:01:04.974 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.975 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.975 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.975 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.976 DEBUG:boto:Method: GET 2023-10-12T01:01:04.976 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.977 DEBUG:boto:Data: 2023-10-12T01:01:04.977 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.978 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.979 DEBUG:boto:Params: {} 2023-10-12T01:01:04.979 DEBUG:boto:Token: None 2023-10-12T01:01:04.980 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY= 2023-10-12T01:01:04.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY=', 'Content-Length': '0'} 2023-10-12T01:01:04.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9aee422ab6b1db-0065274550-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.984 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:04.984 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:04.985 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.985 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.985 DEBUG:boto:Method: GET 2023-10-12T01:01:04.986 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.986 DEBUG:boto:Data: 2023-10-12T01:01:04.987 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:04.988 DEBUG:boto:Port: 8000 2023-10-12T01:01:04.988 DEBUG:boto:Params: {} 2023-10-12T01:01:04.989 DEBUG:boto:Token: None 2023-10-12T01:01:04.989 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:01:04.990 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY= 2023-10-12T01:01:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NL402369Kz4qmCy+uWhQX6SH7dY=', 'Content-Length': '0'} 2023-10-12T01:01:04.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c88f089d2ac84cf-0065274550-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:04.993 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:04.994 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:04.994 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:04.994 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:04.995 DEBUG:boto:Method: GET 2023-10-12T01:01:04.995 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:01:04.996 DEBUG:boto:Data: 2023-10-12T01:01:04.996 DEBUG:boto:Headers: {} 2023-10-12T01:01:04.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:04.997 DEBUG:boto:Port: 8001 2023-10-12T01:01:04.997 DEBUG:boto:Params: {} 2023-10-12T01:01:04.998 DEBUG:boto:Token: None 2023-10-12T01:01:04.998 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:04 GMT /wafmoa-59/ 2023-10-12T01:01:04.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY= 2023-10-12T01:01:04.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jIptlSu+uDC6wkFMpnoM/0XVJVY=', 'Content-Length': '0'} 2023-10-12T01:01:05.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e232f4a9f5f12fa6-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.046 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.047 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.048 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.048 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.049 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.049 DEBUG:boto:Method: GET 2023-10-12T01:01:05.049 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.050 DEBUG:boto:Data: 2023-10-12T01:01:05.050 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.051 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.052 DEBUG:boto:Params: {} 2023-10-12T01:01:05.052 DEBUG:boto:Token: None 2023-10-12T01:01:05.053 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.053 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e2ab441329f4bc-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.057 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.058 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:05.059 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:05.059 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.060 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:01:05.060 DEBUG:boto:Data: 2023-10-12T01:01:05.060 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.061 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.062 DEBUG:boto:Params: {} 2023-10-12T01:01:05.062 DEBUG:boto:Token: None 2023-10-12T01:01:05.063 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj22 2023-10-12T01:01:05.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gws7zMGiy+BCHpP/zpYHkHl7qL4= 2023-10-12T01:01:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gws7zMGiy+BCHpP/zpYHkHl7qL4=', 'Content-Length': '0'} 2023-10-12T01:01:05.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6d8c80fa90d81e7-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.066 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:05.066 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:05.067 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.067 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:01:05.068 DEBUG:boto:Data: 2023-10-12T01:01:05.068 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.069 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.069 DEBUG:boto:Params: {} 2023-10-12T01:01:05.070 DEBUG:boto:Token: None 2023-10-12T01:01:05.070 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj22 2023-10-12T01:01:05.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gws7zMGiy+BCHpP/zpYHkHl7qL4= 2023-10-12T01:01:05.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gws7zMGiy+BCHpP/zpYHkHl7qL4=', 'Content-Length': '0'} 2023-10-12T01:01:05.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac83ae0c4023319c-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:01:05.074 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:05.075 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:05.075 DEBUG:boto:Method: GET 2023-10-12T01:01:05.076 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:01:05.076 DEBUG:boto:Data: 2023-10-12T01:01:05.077 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.078 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.078 DEBUG:boto:Params: {} 2023-10-12T01:01:05.079 DEBUG:boto:Token: None 2023-10-12T01:01:05.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj22 2023-10-12T01:01:05.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2nCPTkzp3jfV9wNO++kptusT0JY= 2023-10-12T01:01:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2nCPTkzp3jfV9wNO++kptusT0JY=', 'Content-Length': '0'} 2023-10-12T01:01:05.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8f0faf4063c463-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.083 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:01:05.083 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:01:05.084 DEBUG:boto:Method: GET 2023-10-12T01:01:05.084 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:01:05.085 DEBUG:boto:Data: 2023-10-12T01:01:05.085 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.086 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.086 DEBUG:boto:Params: {} 2023-10-12T01:01:05.087 DEBUG:boto:Token: None 2023-10-12T01:01:05.087 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj22 2023-10-12T01:01:05.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2nCPTkzp3jfV9wNO++kptusT0JY= 2023-10-12T01:01:05.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2nCPTkzp3jfV9wNO++kptusT0JY=', 'Content-Length': '0'} 2023-10-12T01:01:05.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009205b64ba1f1b5a5-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.091 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.091 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.092 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.092 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.093 DEBUG:boto:Method: GET 2023-10-12T01:01:05.093 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.094 DEBUG:boto:Data: 2023-10-12T01:01:05.094 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.095 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.095 DEBUG:boto:Params: {} 2023-10-12T01:01:05.096 DEBUG:boto:Token: None 2023-10-12T01:01:05.096 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d07318f9271630d-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.101 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.101 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.102 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.102 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.103 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.103 DEBUG:boto:Method: GET 2023-10-12T01:01:05.103 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:01:05.104 DEBUG:boto:Data: 2023-10-12T01:01:05.104 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.105 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.106 DEBUG:boto:Params: {} 2023-10-12T01:01:05.106 DEBUG:boto:Token: None 2023-10-12T01:01:05.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a98070f7b916318-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.154 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:01:05.156 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.156 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.157 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.157 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.158 DEBUG:boto:Method: GET 2023-10-12T01:01:05.158 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.159 DEBUG:boto:Data: 2023-10-12T01:01:05.159 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.160 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.160 DEBUG:boto:Params: {} 2023-10-12T01:01:05.161 DEBUG:boto:Token: None 2023-10-12T01:01:05.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig= 2023-10-12T01:01:05.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig=', 'Content-Length': '0'} 2023-10-12T01:01:05.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec68c0aa70e2b56a-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.165 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.166 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.166 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.167 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.167 DEBUG:boto:Method: GET 2023-10-12T01:01:05.168 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.168 DEBUG:boto:Data: 2023-10-12T01:01:05.169 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.169 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.170 DEBUG:boto:Params: {} 2023-10-12T01:01:05.171 DEBUG:boto:Token: None 2023-10-12T01:01:05.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig= 2023-10-12T01:01:05.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig=', 'Content-Length': '0'} 2023-10-12T01:01:05.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7203158489352e-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.178 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.178 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.179 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.179 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.179 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.180 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.180 DEBUG:boto:Data: 2023-10-12T01:01:05.181 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.181 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.182 DEBUG:boto:Params: {} 2023-10-12T01:01:05.182 DEBUG:boto:Token: None 2023-10-12T01:01:05.183 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qlk9QK9Ka4A+hOCPs/8JoCBmIdo= 2023-10-12T01:01:05.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qlk9QK9Ka4A+hOCPs/8JoCBmIdo=', 'Content-Length': '0'} 2023-10-12T01:01:05.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1eb8b26f650827-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.187 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.187 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.188 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.188 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.188 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.189 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.189 DEBUG:boto:Data: 2023-10-12T01:01:05.190 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.191 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.191 DEBUG:boto:Params: {} 2023-10-12T01:01:05.192 DEBUG:boto:Token: None 2023-10-12T01:01:05.192 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qlk9QK9Ka4A+hOCPs/8JoCBmIdo= 2023-10-12T01:01:05.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qlk9QK9Ka4A+hOCPs/8JoCBmIdo=', 'Content-Length': '0'} 2023-10-12T01:01:05.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035033852f8c17aaa-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:01:05.196 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.197 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.197 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.198 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.198 DEBUG:boto:Method: GET 2023-10-12T01:01:05.199 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.199 DEBUG:boto:Data: 2023-10-12T01:01:05.200 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.201 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.201 DEBUG:boto:Params: {} 2023-10-12T01:01:05.202 DEBUG:boto:Token: None 2023-10-12T01:01:05.202 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.203 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig= 2023-10-12T01:01:05.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig=', 'Content-Length': '0'} 2023-10-12T01:01:05.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c5c7f5e27f667c-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.206 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.206 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.207 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.207 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.208 DEBUG:boto:Method: GET 2023-10-12T01:01:05.208 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.209 DEBUG:boto:Data: 2023-10-12T01:01:05.209 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.210 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.210 DEBUG:boto:Params: {} 2023-10-12T01:01:05.211 DEBUG:boto:Token: None 2023-10-12T01:01:05.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:01:05.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig= 2023-10-12T01:01:05.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pjXOPjXMqTRnin1bfdj7d3xSDig=', 'Content-Length': '0'} 2023-10-12T01:01:05.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008093ddf75555ddd3-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.258 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.259 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.259 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.260 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.260 DEBUG:boto:Method: GET 2023-10-12T01:01:05.261 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.261 DEBUG:boto:Data: 2023-10-12T01:01:05.262 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.262 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.263 DEBUG:boto:Params: {} 2023-10-12T01:01:05.263 DEBUG:boto:Token: None 2023-10-12T01:01:05.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54d1020178a7487-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.268 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.269 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.269 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.270 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.270 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.271 DEBUG:boto:Method: GET 2023-10-12T01:01:05.271 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.272 DEBUG:boto:Data: 2023-10-12T01:01:05.272 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.273 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.273 DEBUG:boto:Params: {} 2023-10-12T01:01:05.274 DEBUG:boto:Token: None 2023-10-12T01:01:05.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d2eb5ee969bacbc-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.279 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.280 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.280 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.281 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.281 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:01:05.282 DEBUG:boto:Data: 2023-10-12T01:01:05.282 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.283 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.283 DEBUG:boto:Params: {} 2023-10-12T01:01:05.284 DEBUG:boto:Token: None 2023-10-12T01:01:05.284 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33 2023-10-12T01:01:05.285 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TcflHC0zhY01Aw21oYhGnZO7kq0= 2023-10-12T01:01:05.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TcflHC0zhY01Aw21oYhGnZO7kq0=', 'Content-Length': '0'} 2023-10-12T01:01:05.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a44db14d40adf7-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.288 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.288 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.289 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.289 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:01:05.289 DEBUG:boto:Data: 2023-10-12T01:01:05.290 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.291 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.291 DEBUG:boto:Params: {} 2023-10-12T01:01:05.291 DEBUG:boto:Token: None 2023-10-12T01:01:05.292 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33 2023-10-12T01:01:05.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TcflHC0zhY01Aw21oYhGnZO7kq0= 2023-10-12T01:01:05.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TcflHC0zhY01Aw21oYhGnZO7kq0=', 'Content-Length': '0'} 2023-10-12T01:01:05.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca7714cf763937c-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:01:05.296 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.296 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.297 DEBUG:boto:Method: GET 2023-10-12T01:01:05.297 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:01:05.298 DEBUG:boto:Data: 2023-10-12T01:01:05.298 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.299 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.299 DEBUG:boto:Params: {} 2023-10-12T01:01:05.300 DEBUG:boto:Token: None 2023-10-12T01:01:05.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33 2023-10-12T01:01:05.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z02TT8UgHnRKb7Xp4jqEoJVDG1A= 2023-10-12T01:01:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z02TT8UgHnRKb7Xp4jqEoJVDG1A=', 'Content-Length': '0'} 2023-10-12T01:01:05.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae810c607ad9c43-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.346 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:01:05.347 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:01:05.347 DEBUG:boto:Method: GET 2023-10-12T01:01:05.348 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:01:05.348 DEBUG:boto:Data: 2023-10-12T01:01:05.349 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.350 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.350 DEBUG:boto:Params: {} 2023-10-12T01:01:05.350 DEBUG:boto:Token: None 2023-10-12T01:01:05.351 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj33 2023-10-12T01:01:05.351 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z02TT8UgHnRKb7Xp4jqEoJVDG1A= 2023-10-12T01:01:05.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z02TT8UgHnRKb7Xp4jqEoJVDG1A=', 'Content-Length': '0'} 2023-10-12T01:01:05.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae60ec1600d6406a-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.355 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.356 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.356 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.356 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.357 DEBUG:boto:Method: GET 2023-10-12T01:01:05.357 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.358 DEBUG:boto:Data: 2023-10-12T01:01:05.358 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.359 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.360 DEBUG:boto:Params: {} 2023-10-12T01:01:05.360 DEBUG:boto:Token: None 2023-10-12T01:01:05.361 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7c50b72f3eb064-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.365 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.366 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.366 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.367 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.367 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.367 DEBUG:boto:Method: GET 2023-10-12T01:01:05.368 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:01:05.368 DEBUG:boto:Data: 2023-10-12T01:01:05.369 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.370 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.370 DEBUG:boto:Params: {} 2023-10-12T01:01:05.370 DEBUG:boto:Token: None 2023-10-12T01:01:05.371 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c909d92426f98f1b-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.419 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:01:05.420 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.421 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.421 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.422 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.422 DEBUG:boto:Method: GET 2023-10-12T01:01:05.423 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.423 DEBUG:boto:Data: 2023-10-12T01:01:05.424 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.425 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.425 DEBUG:boto:Params: {} 2023-10-12T01:01:05.426 DEBUG:boto:Token: None 2023-10-12T01:01:05.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.427 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4= 2023-10-12T01:01:05.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4=', 'Content-Length': '0'} 2023-10-12T01:01:05.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e594ed9592b6e55-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.430 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.431 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.431 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.432 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.432 DEBUG:boto:Method: GET 2023-10-12T01:01:05.433 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.433 DEBUG:boto:Data: 2023-10-12T01:01:05.434 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.435 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.435 DEBUG:boto:Params: {} 2023-10-12T01:01:05.436 DEBUG:boto:Token: None 2023-10-12T01:01:05.436 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4= 2023-10-12T01:01:05.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4=', 'Content-Length': '0'} 2023-10-12T01:01:05.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24c38500adbfb33-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.440 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.441 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.441 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.441 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.442 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.442 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.443 DEBUG:boto:Data: 2023-10-12T01:01:05.443 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.444 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.445 DEBUG:boto:Params: {} 2023-10-12T01:01:05.445 DEBUG:boto:Token: None 2023-10-12T01:01:05.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LE2WDRIo6at+76gJmDpD3AoF7A8= 2023-10-12T01:01:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LE2WDRIo6at+76gJmDpD3AoF7A8=', 'Content-Length': '0'} 2023-10-12T01:01:05.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023ff9a17b3bd3983-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.449 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.450 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.450 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.450 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.451 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.451 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.452 DEBUG:boto:Data: 2023-10-12T01:01:05.452 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.453 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.453 DEBUG:boto:Params: {} 2023-10-12T01:01:05.454 DEBUG:boto:Token: None 2023-10-12T01:01:05.455 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LE2WDRIo6at+76gJmDpD3AoF7A8= 2023-10-12T01:01:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LE2WDRIo6at+76gJmDpD3AoF7A8=', 'Content-Length': '0'} 2023-10-12T01:01:05.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097665d1ca53f2839-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:01:05.459 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.459 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.460 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.460 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.461 DEBUG:boto:Method: GET 2023-10-12T01:01:05.461 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.461 DEBUG:boto:Data: 2023-10-12T01:01:05.462 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.463 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.463 DEBUG:boto:Params: {} 2023-10-12T01:01:05.464 DEBUG:boto:Token: None 2023-10-12T01:01:05.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.465 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4= 2023-10-12T01:01:05.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4=', 'Content-Length': '0'} 2023-10-12T01:01:05.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afbe56a6b35f0115-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.468 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.469 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.469 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.470 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.470 DEBUG:boto:Method: GET 2023-10-12T01:01:05.472 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.473 DEBUG:boto:Data: 2023-10-12T01:01:05.473 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.474 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.474 DEBUG:boto:Params: {} 2023-10-12T01:01:05.475 DEBUG:boto:Token: None 2023-10-12T01:01:05.475 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:01:05.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4= 2023-10-12T01:01:05.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SEEb0bUaWbYL74gbFn0z/RRRy4=', 'Content-Length': '0'} 2023-10-12T01:01:05.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000876080f583e4ba9-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.479 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.480 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.481 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.481 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.481 DEBUG:boto:Method: GET 2023-10-12T01:01:05.482 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.483 DEBUG:boto:Data: 2023-10-12T01:01:05.483 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.484 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.484 DEBUG:boto:Params: {} 2023-10-12T01:01:05.485 DEBUG:boto:Token: None 2023-10-12T01:01:05.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704cc8f5410972e6-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.490 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.491 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.491 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.491 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.492 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.492 DEBUG:boto:Method: GET 2023-10-12T01:01:05.493 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.493 DEBUG:boto:Data: 2023-10-12T01:01:05.494 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.495 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.495 DEBUG:boto:Params: {} 2023-10-12T01:01:05.495 DEBUG:boto:Token: None 2023-10-12T01:01:05.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db6fc521a05429b-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.501 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.501 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.502 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.502 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.503 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:01:05.503 DEBUG:boto:Data: 2023-10-12T01:01:05.503 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.504 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.505 DEBUG:boto:Params: {} 2023-10-12T01:01:05.505 DEBUG:boto:Token: None 2023-10-12T01:01:05.506 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34 2023-10-12T01:01:05.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnftb2AC43+tcuGfpjqNYGFUVWs= 2023-10-12T01:01:05.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnftb2AC43+tcuGfpjqNYGFUVWs=', 'Content-Length': '0'} 2023-10-12T01:01:05.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002942a3c287108b57-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.509 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.510 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.510 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.511 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:01:05.511 DEBUG:boto:Data: 2023-10-12T01:01:05.511 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.512 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.513 DEBUG:boto:Params: {} 2023-10-12T01:01:05.513 DEBUG:boto:Token: None 2023-10-12T01:01:05.514 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34 2023-10-12T01:01:05.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xnftb2AC43+tcuGfpjqNYGFUVWs= 2023-10-12T01:01:05.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xnftb2AC43+tcuGfpjqNYGFUVWs=', 'Content-Length': '0'} 2023-10-12T01:01:05.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c82f4754130b4f-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:01:05.517 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.518 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.518 DEBUG:boto:Method: GET 2023-10-12T01:01:05.519 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:01:05.519 DEBUG:boto:Data: 2023-10-12T01:01:05.520 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.521 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.521 DEBUG:boto:Params: {} 2023-10-12T01:01:05.521 DEBUG:boto:Token: None 2023-10-12T01:01:05.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34 2023-10-12T01:01:05.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/rM4rRxxLtF8syGlAn0L1PCukrw= 2023-10-12T01:01:05.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/rM4rRxxLtF8syGlAn0L1PCukrw=', 'Content-Length': '0'} 2023-10-12T01:01:05.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000608758f029a430a1-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.526 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:01:05.526 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:01:05.527 DEBUG:boto:Method: GET 2023-10-12T01:01:05.527 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:01:05.527 DEBUG:boto:Data: 2023-10-12T01:01:05.528 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.529 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.529 DEBUG:boto:Params: {} 2023-10-12T01:01:05.530 DEBUG:boto:Token: None 2023-10-12T01:01:05.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj34 2023-10-12T01:01:05.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/rM4rRxxLtF8syGlAn0L1PCukrw= 2023-10-12T01:01:05.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/rM4rRxxLtF8syGlAn0L1PCukrw=', 'Content-Length': '0'} 2023-10-12T01:01:05.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca853746b5f2968-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.574 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.575 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.575 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.576 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.576 DEBUG:boto:Method: GET 2023-10-12T01:01:05.577 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.577 DEBUG:boto:Data: 2023-10-12T01:01:05.578 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.579 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.579 DEBUG:boto:Params: {} 2023-10-12T01:01:05.580 DEBUG:boto:Token: None 2023-10-12T01:01:05.580 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636718957d6349eb-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.585 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.585 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.586 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.586 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.589 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.590 DEBUG:boto:Method: GET 2023-10-12T01:01:05.590 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:01:05.591 DEBUG:boto:Data: 2023-10-12T01:01:05.591 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.592 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.592 DEBUG:boto:Params: {} 2023-10-12T01:01:05.593 DEBUG:boto:Token: None 2023-10-12T01:01:05.593 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a8d6187c5e2f90-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.598 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:01:05.599 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.600 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.600 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.600 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.601 DEBUG:boto:Method: GET 2023-10-12T01:01:05.601 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.602 DEBUG:boto:Data: 2023-10-12T01:01:05.602 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.603 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.604 DEBUG:boto:Params: {} 2023-10-12T01:01:05.604 DEBUG:boto:Token: None 2023-10-12T01:01:05.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk= 2023-10-12T01:01:05.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk=', 'Content-Length': '0'} 2023-10-12T01:01:05.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311652d09cd5576a-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.608 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.609 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.609 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.610 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.610 DEBUG:boto:Method: GET 2023-10-12T01:01:05.611 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.611 DEBUG:boto:Data: 2023-10-12T01:01:05.612 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.612 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.613 DEBUG:boto:Params: {} 2023-10-12T01:01:05.613 DEBUG:boto:Token: None 2023-10-12T01:01:05.614 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.614 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk= 2023-10-12T01:01:05.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk=', 'Content-Length': '0'} 2023-10-12T01:01:05.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b82e324ee3abba-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.617 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.618 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.618 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.619 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.619 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.620 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.620 DEBUG:boto:Data: 2023-10-12T01:01:05.620 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.621 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.622 DEBUG:boto:Params: {} 2023-10-12T01:01:05.622 DEBUG:boto:Token: None 2023-10-12T01:01:05.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m5xv/EKtXZf+rqm1mnPpYwE8aJ8= 2023-10-12T01:01:05.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m5xv/EKtXZf+rqm1mnPpYwE8aJ8=', 'Content-Length': '0'} 2023-10-12T01:01:05.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15a7c3591ba0b12-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.626 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.627 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.627 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.628 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.628 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.629 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.629 DEBUG:boto:Data: 2023-10-12T01:01:05.629 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.630 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.630 DEBUG:boto:Params: {} 2023-10-12T01:01:05.631 DEBUG:boto:Token: None 2023-10-12T01:01:05.631 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m5xv/EKtXZf+rqm1mnPpYwE8aJ8= 2023-10-12T01:01:05.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m5xv/EKtXZf+rqm1mnPpYwE8aJ8=', 'Content-Length': '0'} 2023-10-12T01:01:05.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5b498efd08d7b7-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:01:05.635 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.636 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.636 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.637 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.637 DEBUG:boto:Method: GET 2023-10-12T01:01:05.638 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.638 DEBUG:boto:Data: 2023-10-12T01:01:05.638 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.639 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.640 DEBUG:boto:Params: {} 2023-10-12T01:01:05.640 DEBUG:boto:Token: None 2023-10-12T01:01:05.641 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk= 2023-10-12T01:01:05.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk=', 'Content-Length': '0'} 2023-10-12T01:01:05.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a63fd240b715798-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.644 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.645 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.645 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.646 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.646 DEBUG:boto:Method: GET 2023-10-12T01:01:05.647 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.647 DEBUG:boto:Data: 2023-10-12T01:01:05.648 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.648 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.649 DEBUG:boto:Params: {} 2023-10-12T01:01:05.649 DEBUG:boto:Token: None 2023-10-12T01:01:05.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:01:05.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk= 2023-10-12T01:01:05.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hInqT71JB4dNQwlQtfPyCJTmWVk=', 'Content-Length': '0'} 2023-10-12T01:01:05.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000937901bf0ce1a105-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.653 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.654 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.654 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.655 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.655 DEBUG:boto:Method: GET 2023-10-12T01:01:05.656 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.656 DEBUG:boto:Data: 2023-10-12T01:01:05.657 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.657 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.658 DEBUG:boto:Params: {} 2023-10-12T01:01:05.658 DEBUG:boto:Token: None 2023-10-12T01:01:05.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb6542efdc89ecf7-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.663 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.664 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.664 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.665 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.665 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.665 DEBUG:boto:Method: GET 2023-10-12T01:01:05.666 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.666 DEBUG:boto:Data: 2023-10-12T01:01:05.667 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.668 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.668 DEBUG:boto:Params: {} 2023-10-12T01:01:05.668 DEBUG:boto:Token: None 2023-10-12T01:01:05.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012feb255d2e1e3fe-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.673 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.674 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.674 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.675 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.675 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:01:05.676 DEBUG:boto:Data: 2023-10-12T01:01:05.676 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.677 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.677 DEBUG:boto:Params: {} 2023-10-12T01:01:05.678 DEBUG:boto:Token: None 2023-10-12T01:01:05.678 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44 2023-10-12T01:01:05.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w4pPayMKLh+c90A+12cj5jX2T8o= 2023-10-12T01:01:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w4pPayMKLh+c90A+12cj5jX2T8o=', 'Content-Length': '0'} 2023-10-12T01:01:05.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4a2fbfafe17a87-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.682 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.682 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.683 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.683 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:01:05.684 DEBUG:boto:Data: 2023-10-12T01:01:05.684 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.685 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.685 DEBUG:boto:Params: {} 2023-10-12T01:01:05.686 DEBUG:boto:Token: None 2023-10-12T01:01:05.686 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44 2023-10-12T01:01:05.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w4pPayMKLh+c90A+12cj5jX2T8o= 2023-10-12T01:01:05.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w4pPayMKLh+c90A+12cj5jX2T8o=', 'Content-Length': '0'} 2023-10-12T01:01:05.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf0349d285ce1e5-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:01:05.690 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.690 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.691 DEBUG:boto:Method: GET 2023-10-12T01:01:05.691 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:01:05.692 DEBUG:boto:Data: 2023-10-12T01:01:05.692 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.693 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.693 DEBUG:boto:Params: {} 2023-10-12T01:01:05.694 DEBUG:boto:Token: None 2023-10-12T01:01:05.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44 2023-10-12T01:01:05.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PpBUALtOumOcdHhLBZR+FWCULi4= 2023-10-12T01:01:05.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PpBUALtOumOcdHhLBZR+FWCULi4=', 'Content-Length': '0'} 2023-10-12T01:01:05.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2f495cd83837a1-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.738 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:01:05.739 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:01:05.739 DEBUG:boto:Method: GET 2023-10-12T01:01:05.740 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:01:05.740 DEBUG:boto:Data: 2023-10-12T01:01:05.741 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.742 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.742 DEBUG:boto:Params: {} 2023-10-12T01:01:05.743 DEBUG:boto:Token: None 2023-10-12T01:01:05.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj44 2023-10-12T01:01:05.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PpBUALtOumOcdHhLBZR+FWCULi4= 2023-10-12T01:01:05.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PpBUALtOumOcdHhLBZR+FWCULi4=', 'Content-Length': '0'} 2023-10-12T01:01:05.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5afb95970a727ec-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.747 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.747 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.748 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.748 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.749 DEBUG:boto:Method: GET 2023-10-12T01:01:05.749 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.749 DEBUG:boto:Data: 2023-10-12T01:01:05.750 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.751 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.751 DEBUG:boto:Params: {} 2023-10-12T01:01:05.752 DEBUG:boto:Token: None 2023-10-12T01:01:05.752 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c14fad13a6ea0c-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.756 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.757 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.757 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.758 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.758 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.759 DEBUG:boto:Method: GET 2023-10-12T01:01:05.759 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:01:05.760 DEBUG:boto:Data: 2023-10-12T01:01:05.761 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.762 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.763 DEBUG:boto:Params: {} 2023-10-12T01:01:05.763 DEBUG:boto:Token: None 2023-10-12T01:01:05.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f26e75f51a4e42a0-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.810 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:01:05.811 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.812 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.812 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.813 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.813 DEBUG:boto:Method: GET 2023-10-12T01:01:05.813 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.814 DEBUG:boto:Data: 2023-10-12T01:01:05.814 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.815 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.816 DEBUG:boto:Params: {} 2023-10-12T01:01:05.816 DEBUG:boto:Token: None 2023-10-12T01:01:05.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk= 2023-10-12T01:01:05.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk=', 'Content-Length': '0'} 2023-10-12T01:01:05.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f144740e4530c14e-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.820 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.821 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.821 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.822 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.822 DEBUG:boto:Method: GET 2023-10-12T01:01:05.823 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.823 DEBUG:boto:Data: 2023-10-12T01:01:05.823 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.824 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.825 DEBUG:boto:Params: {} 2023-10-12T01:01:05.825 DEBUG:boto:Token: None 2023-10-12T01:01:05.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk= 2023-10-12T01:01:05.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk=', 'Content-Length': '0'} 2023-10-12T01:01:05.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f8042065ce2800-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.830 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.830 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.830 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.831 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.831 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.832 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.832 DEBUG:boto:Data: 2023-10-12T01:01:05.832 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.833 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.834 DEBUG:boto:Params: {} 2023-10-12T01:01:05.834 DEBUG:boto:Token: None 2023-10-12T01:01:05.834 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mn2hZj3GhfWyWkudBuvk+1ADM7w= 2023-10-12T01:01:05.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mn2hZj3GhfWyWkudBuvk+1ADM7w=', 'Content-Length': '0'} 2023-10-12T01:01:05.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f831f1f24d42d851-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.838 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.839 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.839 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.839 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.840 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.840 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.841 DEBUG:boto:Data: 2023-10-12T01:01:05.841 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.842 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.842 DEBUG:boto:Params: {} 2023-10-12T01:01:05.843 DEBUG:boto:Token: None 2023-10-12T01:01:05.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.843 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mn2hZj3GhfWyWkudBuvk+1ADM7w= 2023-10-12T01:01:05.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mn2hZj3GhfWyWkudBuvk+1ADM7w=', 'Content-Length': '0'} 2023-10-12T01:01:05.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023ef9c7400533db0-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:01:05.847 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.848 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.848 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.848 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.849 DEBUG:boto:Method: GET 2023-10-12T01:01:05.849 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.850 DEBUG:boto:Data: 2023-10-12T01:01:05.850 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.851 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.851 DEBUG:boto:Params: {} 2023-10-12T01:01:05.852 DEBUG:boto:Token: None 2023-10-12T01:01:05.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk= 2023-10-12T01:01:05.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk=', 'Content-Length': '0'} 2023-10-12T01:01:05.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016cd8b0bc591ae82-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.856 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.856 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.857 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.857 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.858 DEBUG:boto:Method: GET 2023-10-12T01:01:05.858 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.858 DEBUG:boto:Data: 2023-10-12T01:01:05.859 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.859 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.860 DEBUG:boto:Params: {} 2023-10-12T01:01:05.860 DEBUG:boto:Token: None 2023-10-12T01:01:05.861 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:01:05.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk= 2023-10-12T01:01:05.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7sefO73Yuv6dNeBDg79NqZQlgk=', 'Content-Length': '0'} 2023-10-12T01:01:05.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c94757cbb0a52b1-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.864 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.865 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.865 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.866 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.866 DEBUG:boto:Method: GET 2023-10-12T01:01:05.867 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.867 DEBUG:boto:Data: 2023-10-12T01:01:05.868 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.868 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.869 DEBUG:boto:Params: {} 2023-10-12T01:01:05.869 DEBUG:boto:Token: None 2023-10-12T01:01:05.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d78ac11a4eefcebd-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.874 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.875 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.875 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.875 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.876 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.876 DEBUG:boto:Method: GET 2023-10-12T01:01:05.877 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.877 DEBUG:boto:Data: 2023-10-12T01:01:05.878 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.879 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.879 DEBUG:boto:Params: {} 2023-10-12T01:01:05.880 DEBUG:boto:Token: None 2023-10-12T01:01:05.880 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057650aa4f7a8e2ef-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.884 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.885 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.885 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.886 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.886 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:01:05.886 DEBUG:boto:Data: 2023-10-12T01:01:05.887 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.888 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.888 DEBUG:boto:Params: {} 2023-10-12T01:01:05.888 DEBUG:boto:Token: None 2023-10-12T01:01:05.889 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51 2023-10-12T01:01:05.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sguvgOwESMpJzT63rFw91PdA2Bo= 2023-10-12T01:01:05.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sguvgOwESMpJzT63rFw91PdA2Bo=', 'Content-Length': '0'} 2023-10-12T01:01:05.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cffb0af71530f638-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.893 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.893 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.893 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.894 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:01:05.894 DEBUG:boto:Data: 2023-10-12T01:01:05.895 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.896 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.896 DEBUG:boto:Params: {} 2023-10-12T01:01:05.896 DEBUG:boto:Token: None 2023-10-12T01:01:05.897 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51 2023-10-12T01:01:05.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sguvgOwESMpJzT63rFw91PdA2Bo= 2023-10-12T01:01:05.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sguvgOwESMpJzT63rFw91PdA2Bo=', 'Content-Length': '0'} 2023-10-12T01:01:05.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab377c14e28b59e2-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:01:05.901 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.901 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.902 DEBUG:boto:Method: GET 2023-10-12T01:01:05.902 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:01:05.902 DEBUG:boto:Data: 2023-10-12T01:01:05.903 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.904 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.904 DEBUG:boto:Params: {} 2023-10-12T01:01:05.905 DEBUG:boto:Token: None 2023-10-12T01:01:05.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51 2023-10-12T01:01:05.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4Gk2JabopnPt2Zr5KqparALqN0= 2023-10-12T01:01:05.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4Gk2JabopnPt2Zr5KqparALqN0=', 'Content-Length': '0'} 2023-10-12T01:01:05.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9051d4879ad59a-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.909 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:01:05.909 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:01:05.910 DEBUG:boto:Method: GET 2023-10-12T01:01:05.910 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:01:05.910 DEBUG:boto:Data: 2023-10-12T01:01:05.911 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.912 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.912 DEBUG:boto:Params: {} 2023-10-12T01:01:05.912 DEBUG:boto:Token: None 2023-10-12T01:01:05.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj51 2023-10-12T01:01:05.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V4Gk2JabopnPt2Zr5KqparALqN0= 2023-10-12T01:01:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V4Gk2JabopnPt2Zr5KqparALqN0=', 'Content-Length': '0'} 2023-10-12T01:01:05.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ca7c684a34896e-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.917 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.917 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.918 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.918 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.918 DEBUG:boto:Method: GET 2023-10-12T01:01:05.919 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.919 DEBUG:boto:Data: 2023-10-12T01:01:05.920 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.920 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.921 DEBUG:boto:Params: {} 2023-10-12T01:01:05.921 DEBUG:boto:Token: None 2023-10-12T01:01:05.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d14546d8f26a23-0065274551-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.926 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.927 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:05.927 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:05.927 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.928 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.928 DEBUG:boto:Method: GET 2023-10-12T01:01:05.928 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:01:05.929 DEBUG:boto:Data: 2023-10-12T01:01:05.929 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.930 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.930 DEBUG:boto:Params: {} 2023-10-12T01:01:05.931 DEBUG:boto:Token: None 2023-10-12T01:01:05.931 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/ 2023-10-12T01:01:05.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s= 2023-10-12T01:01:05.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ0ED0hAX5glpqc49w8l4dD+N4s=', 'Content-Length': '0'} 2023-10-12T01:01:05.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e0918b37646b77f-0065274551-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.936 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:05.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:01:05.937 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:05.938 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:05.938 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.938 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.939 DEBUG:boto:Method: GET 2023-10-12T01:01:05.939 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.940 DEBUG:boto:Data: 2023-10-12T01:01:05.940 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.941 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.941 DEBUG:boto:Params: {} 2023-10-12T01:01:05.942 DEBUG:boto:Token: None 2023-10-12T01:01:05.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.943 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iCikAjuS/DPGBDL5FTOYb5MPUl8= 2023-10-12T01:01:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iCikAjuS/DPGBDL5FTOYb5MPUl8=', 'Content-Length': '0'} 2023-10-12T01:01:05.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005464880fd6b0e1c9-0065274551-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.946 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:05.946 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:05.947 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.947 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.948 DEBUG:boto:Method: GET 2023-10-12T01:01:05.948 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.948 DEBUG:boto:Data: 2023-10-12T01:01:05.949 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:05.950 DEBUG:boto:Port: 8000 2023-10-12T01:01:05.950 DEBUG:boto:Params: {} 2023-10-12T01:01:05.951 DEBUG:boto:Token: None 2023-10-12T01:01:05.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:05 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iCikAjuS/DPGBDL5FTOYb5MPUl8= 2023-10-12T01:01:05.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iCikAjuS/DPGBDL5FTOYb5MPUl8=', 'Content-Length': '0'} 2023-10-12T01:01:05.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008956c2e098cf8ecf-0065274551-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:05.994 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:05.995 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:05.996 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.996 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.997 DEBUG:boto:Method: HEAD 2023-10-12T01:01:05.997 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:05.998 DEBUG:boto:Data: 2023-10-12T01:01:05.998 DEBUG:boto:Headers: {} 2023-10-12T01:01:05.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:05.999 DEBUG:boto:Port: 8001 2023-10-12T01:01:05.999 DEBUG:boto:Params: {} 2023-10-12T01:01:06.000 DEBUG:boto:Token: None 2023-10-12T01:01:06.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:COMDtcbjJU9YL9CrXZZPbqRkDgc= 2023-10-12T01:01:06.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:COMDtcbjJU9YL9CrXZZPbqRkDgc=', 'Content-Length': '0'} 2023-10-12T01:01:06.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddbeae25ac3f3ced-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.004 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.004 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.005 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.005 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.006 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.006 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.006 DEBUG:boto:Data: 2023-10-12T01:01:06.007 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.008 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.008 DEBUG:boto:Params: {} 2023-10-12T01:01:06.009 DEBUG:boto:Token: None 2023-10-12T01:01:06.009 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.010 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:COMDtcbjJU9YL9CrXZZPbqRkDgc= 2023-10-12T01:01:06.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:COMDtcbjJU9YL9CrXZZPbqRkDgc=', 'Content-Length': '0'} 2023-10-12T01:01:06.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004780a71500b15b02-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:01:06.013 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.014 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.014 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.015 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.015 DEBUG:boto:Method: GET 2023-10-12T01:01:06.016 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.016 DEBUG:boto:Data: 2023-10-12T01:01:06.017 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.017 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.018 DEBUG:boto:Params: {} 2023-10-12T01:01:06.018 DEBUG:boto:Token: None 2023-10-12T01:01:06.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUMnxaru9fxTSMZFhvhkSdB0LcA= 2023-10-12T01:01:06.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUMnxaru9fxTSMZFhvhkSdB0LcA=', 'Content-Length': '0'} 2023-10-12T01:01:06.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20768f680dde2ad-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.062 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.063 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.063 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.064 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.065 DEBUG:boto:Method: GET 2023-10-12T01:01:06.065 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.065 DEBUG:boto:Data: 2023-10-12T01:01:06.066 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.067 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.067 DEBUG:boto:Params: {} 2023-10-12T01:01:06.068 DEBUG:boto:Token: None 2023-10-12T01:01:06.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:01:06.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUMnxaru9fxTSMZFhvhkSdB0LcA= 2023-10-12T01:01:06.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUMnxaru9fxTSMZFhvhkSdB0LcA=', 'Content-Length': '0'} 2023-10-12T01:01:06.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45e98e9ab9b09e8-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.072 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.072 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.073 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.073 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.074 DEBUG:boto:Method: GET 2023-10-12T01:01:06.074 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.075 DEBUG:boto:Data: 2023-10-12T01:01:06.075 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.076 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.076 DEBUG:boto:Params: {} 2023-10-12T01:01:06.077 DEBUG:boto:Token: None 2023-10-12T01:01:06.077 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.078 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005891b8c02bab4a97-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.081 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.082 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.083 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.083 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.083 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.084 DEBUG:boto:Method: GET 2023-10-12T01:01:06.084 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.085 DEBUG:boto:Data: 2023-10-12T01:01:06.085 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.086 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.087 DEBUG:boto:Params: {} 2023-10-12T01:01:06.087 DEBUG:boto:Token: None 2023-10-12T01:01:06.087 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000762fd7e4a5501705-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.092 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.093 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.093 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.093 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.094 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:01:06.094 DEBUG:boto:Data: 2023-10-12T01:01:06.095 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.095 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.096 DEBUG:boto:Params: {} 2023-10-12T01:01:06.096 DEBUG:boto:Token: None 2023-10-12T01:01:06.097 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55 2023-10-12T01:01:06.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zj1MABNMaTZUsqfIdtsnZWg2u3U= 2023-10-12T01:01:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zj1MABNMaTZUsqfIdtsnZWg2u3U=', 'Content-Length': '0'} 2023-10-12T01:01:06.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2a4092d74557dd-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.100 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.100 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.101 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.101 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:01:06.102 DEBUG:boto:Data: 2023-10-12T01:01:06.102 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.103 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.103 DEBUG:boto:Params: {} 2023-10-12T01:01:06.104 DEBUG:boto:Token: None 2023-10-12T01:01:06.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55 2023-10-12T01:01:06.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zj1MABNMaTZUsqfIdtsnZWg2u3U= 2023-10-12T01:01:06.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zj1MABNMaTZUsqfIdtsnZWg2u3U=', 'Content-Length': '0'} 2023-10-12T01:01:06.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf5816e9633435b-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:01:06.108 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.109 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.109 DEBUG:boto:Method: GET 2023-10-12T01:01:06.110 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:01:06.110 DEBUG:boto:Data: 2023-10-12T01:01:06.111 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.112 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.112 DEBUG:boto:Params: {} 2023-10-12T01:01:06.113 DEBUG:boto:Token: None 2023-10-12T01:01:06.113 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55 2023-10-12T01:01:06.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l27hLbdY07T0raSF4EeSJoHwkC8= 2023-10-12T01:01:06.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l27hLbdY07T0raSF4EeSJoHwkC8=', 'Content-Length': '0'} 2023-10-12T01:01:06.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f46a3dc18453ab46-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.117 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:01:06.117 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:01:06.118 DEBUG:boto:Method: GET 2023-10-12T01:01:06.118 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:01:06.119 DEBUG:boto:Data: 2023-10-12T01:01:06.119 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.120 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.120 DEBUG:boto:Params: {} 2023-10-12T01:01:06.121 DEBUG:boto:Token: None 2023-10-12T01:01:06.121 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj55 2023-10-12T01:01:06.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l27hLbdY07T0raSF4EeSJoHwkC8= 2023-10-12T01:01:06.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l27hLbdY07T0raSF4EeSJoHwkC8=', 'Content-Length': '0'} 2023-10-12T01:01:06.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000627dd262db0a6f2-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.170 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.171 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.172 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.172 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.173 DEBUG:boto:Method: GET 2023-10-12T01:01:06.173 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.173 DEBUG:boto:Data: 2023-10-12T01:01:06.174 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.175 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.175 DEBUG:boto:Params: {} 2023-10-12T01:01:06.176 DEBUG:boto:Token: None 2023-10-12T01:01:06.176 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909c476545b7f3b7-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.181 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.182 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.182 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.183 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.183 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.184 DEBUG:boto:Method: GET 2023-10-12T01:01:06.184 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:01:06.184 DEBUG:boto:Data: 2023-10-12T01:01:06.185 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.186 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.186 DEBUG:boto:Params: {} 2023-10-12T01:01:06.187 DEBUG:boto:Token: None 2023-10-12T01:01:06.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198488d5b2556baf-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.192 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:01:06.193 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.193 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.194 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.194 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.195 DEBUG:boto:Method: GET 2023-10-12T01:01:06.195 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.196 DEBUG:boto:Data: 2023-10-12T01:01:06.196 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.197 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.197 DEBUG:boto:Params: {} 2023-10-12T01:01:06.198 DEBUG:boto:Token: None 2023-10-12T01:01:06.198 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.201 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc= 2023-10-12T01:01:06.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc=', 'Content-Length': '0'} 2023-10-12T01:01:06.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa10bbb4d922f0b-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.206 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.206 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.207 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.207 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.208 DEBUG:boto:Method: GET 2023-10-12T01:01:06.208 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.209 DEBUG:boto:Data: 2023-10-12T01:01:06.209 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.210 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.210 DEBUG:boto:Params: {} 2023-10-12T01:01:06.211 DEBUG:boto:Token: None 2023-10-12T01:01:06.212 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc= 2023-10-12T01:01:06.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc=', 'Content-Length': '0'} 2023-10-12T01:01:06.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa5859ed93c72da-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.215 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.216 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.216 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.216 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.217 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.217 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.218 DEBUG:boto:Data: 2023-10-12T01:01:06.218 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.219 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.219 DEBUG:boto:Params: {} 2023-10-12T01:01:06.220 DEBUG:boto:Token: None 2023-10-12T01:01:06.220 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EGLyKNwgVCKl393xmiVCblMtSZA= 2023-10-12T01:01:06.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EGLyKNwgVCKl393xmiVCblMtSZA=', 'Content-Length': '0'} 2023-10-12T01:01:06.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c9c694f6f357bf-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.224 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.224 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.225 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.225 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.225 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.226 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.226 DEBUG:boto:Data: 2023-10-12T01:01:06.227 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.228 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.228 DEBUG:boto:Params: {} 2023-10-12T01:01:06.228 DEBUG:boto:Token: None 2023-10-12T01:01:06.229 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EGLyKNwgVCKl393xmiVCblMtSZA= 2023-10-12T01:01:06.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EGLyKNwgVCKl393xmiVCblMtSZA=', 'Content-Length': '0'} 2023-10-12T01:01:06.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c3372c5bab7759-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:01:06.233 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.233 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.234 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.234 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.234 DEBUG:boto:Method: GET 2023-10-12T01:01:06.235 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.235 DEBUG:boto:Data: 2023-10-12T01:01:06.236 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.237 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.237 DEBUG:boto:Params: {} 2023-10-12T01:01:06.238 DEBUG:boto:Token: None 2023-10-12T01:01:06.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc= 2023-10-12T01:01:06.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc=', 'Content-Length': '0'} 2023-10-12T01:01:06.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160ddaf5770d43bc-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.242 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.242 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.243 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.243 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.244 DEBUG:boto:Method: GET 2023-10-12T01:01:06.244 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.244 DEBUG:boto:Data: 2023-10-12T01:01:06.245 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.246 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.246 DEBUG:boto:Params: {} 2023-10-12T01:01:06.247 DEBUG:boto:Token: None 2023-10-12T01:01:06.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:01:06.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc= 2023-10-12T01:01:06.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fMJDAwe5+BnymDnuY6i3O/o16jc=', 'Content-Length': '0'} 2023-10-12T01:01:06.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ab898502f8890f-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.294 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.295 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.295 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.296 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.296 DEBUG:boto:Method: GET 2023-10-12T01:01:06.297 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.297 DEBUG:boto:Data: 2023-10-12T01:01:06.297 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.298 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.299 DEBUG:boto:Params: {} 2023-10-12T01:01:06.299 DEBUG:boto:Token: None 2023-10-12T01:01:06.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eb24827fa77b179-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.304 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.304 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.305 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.305 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.306 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.306 DEBUG:boto:Method: GET 2023-10-12T01:01:06.307 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.307 DEBUG:boto:Data: 2023-10-12T01:01:06.307 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.308 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.309 DEBUG:boto:Params: {} 2023-10-12T01:01:06.309 DEBUG:boto:Token: None 2023-10-12T01:01:06.309 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.310 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a254c622cde7a12b-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.314 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.315 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.315 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.316 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.316 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:01:06.317 DEBUG:boto:Data: 2023-10-12T01:01:06.317 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.318 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.318 DEBUG:boto:Params: {} 2023-10-12T01:01:06.318 DEBUG:boto:Token: None 2023-10-12T01:01:06.319 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68 2023-10-12T01:01:06.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eRb8MYnyNNdjlCmGTCN2exiTvjc= 2023-10-12T01:01:06.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eRb8MYnyNNdjlCmGTCN2exiTvjc=', 'Content-Length': '0'} 2023-10-12T01:01:06.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003291ce83bad3f553-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.322 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.323 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.323 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.323 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:01:06.324 DEBUG:boto:Data: 2023-10-12T01:01:06.324 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.325 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.325 DEBUG:boto:Params: {} 2023-10-12T01:01:06.326 DEBUG:boto:Token: None 2023-10-12T01:01:06.326 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68 2023-10-12T01:01:06.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eRb8MYnyNNdjlCmGTCN2exiTvjc= 2023-10-12T01:01:06.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eRb8MYnyNNdjlCmGTCN2exiTvjc=', 'Content-Length': '0'} 2023-10-12T01:01:06.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a7895c93b850a9-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:01:06.330 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.331 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.331 DEBUG:boto:Method: GET 2023-10-12T01:01:06.332 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:01:06.332 DEBUG:boto:Data: 2023-10-12T01:01:06.333 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.333 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.334 DEBUG:boto:Params: {} 2023-10-12T01:01:06.334 DEBUG:boto:Token: None 2023-10-12T01:01:06.335 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68 2023-10-12T01:01:06.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SwP23rBOwkwUceUkggkZ/J9yh2M= 2023-10-12T01:01:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SwP23rBOwkwUceUkggkZ/J9yh2M=', 'Content-Length': '0'} 2023-10-12T01:01:06.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9aca25e43199a7-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.378 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:01:06.379 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:01:06.379 DEBUG:boto:Method: GET 2023-10-12T01:01:06.380 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:01:06.380 DEBUG:boto:Data: 2023-10-12T01:01:06.381 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.382 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.382 DEBUG:boto:Params: {} 2023-10-12T01:01:06.383 DEBUG:boto:Token: None 2023-10-12T01:01:06.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj68 2023-10-12T01:01:06.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SwP23rBOwkwUceUkggkZ/J9yh2M= 2023-10-12T01:01:06.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SwP23rBOwkwUceUkggkZ/J9yh2M=', 'Content-Length': '0'} 2023-10-12T01:01:06.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d10fe2618f78a18-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.387 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.387 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.388 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.388 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.389 DEBUG:boto:Method: GET 2023-10-12T01:01:06.389 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.390 DEBUG:boto:Data: 2023-10-12T01:01:06.390 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.391 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.391 DEBUG:boto:Params: {} 2023-10-12T01:01:06.392 DEBUG:boto:Token: None 2023-10-12T01:01:06.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0a7926ad096b71-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.396 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.397 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.398 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.398 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.398 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.399 DEBUG:boto:Method: GET 2023-10-12T01:01:06.399 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:01:06.400 DEBUG:boto:Data: 2023-10-12T01:01:06.400 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.401 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.401 DEBUG:boto:Params: {} 2023-10-12T01:01:06.402 DEBUG:boto:Token: None 2023-10-12T01:01:06.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d7d128f66ddba9-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.410 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:01:06.411 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.411 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.412 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.412 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.413 DEBUG:boto:Method: GET 2023-10-12T01:01:06.413 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.414 DEBUG:boto:Data: 2023-10-12T01:01:06.414 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.415 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.415 DEBUG:boto:Params: {} 2023-10-12T01:01:06.416 DEBUG:boto:Token: None 2023-10-12T01:01:06.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc= 2023-10-12T01:01:06.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc=', 'Content-Length': '0'} 2023-10-12T01:01:06.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92e7469ad036450-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.420 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.420 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.421 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.421 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.421 DEBUG:boto:Method: GET 2023-10-12T01:01:06.422 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.422 DEBUG:boto:Data: 2023-10-12T01:01:06.423 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.424 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.424 DEBUG:boto:Params: {} 2023-10-12T01:01:06.425 DEBUG:boto:Token: None 2023-10-12T01:01:06.425 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc= 2023-10-12T01:01:06.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc=', 'Content-Length': '0'} 2023-10-12T01:01:06.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cdb5c2f17aa190-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.429 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.429 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.430 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.430 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.430 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.431 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.431 DEBUG:boto:Data: 2023-10-12T01:01:06.432 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.432 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.433 DEBUG:boto:Params: {} 2023-10-12T01:01:06.433 DEBUG:boto:Token: None 2023-10-12T01:01:06.434 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6LsQUo9hyZEqOIXrfbFsNQwiG2I= 2023-10-12T01:01:06.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6LsQUo9hyZEqOIXrfbFsNQwiG2I=', 'Content-Length': '0'} 2023-10-12T01:01:06.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c11955b3c847465-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.437 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.437 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.438 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.438 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.439 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.439 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.439 DEBUG:boto:Data: 2023-10-12T01:01:06.440 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.441 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.441 DEBUG:boto:Params: {} 2023-10-12T01:01:06.441 DEBUG:boto:Token: None 2023-10-12T01:01:06.442 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6LsQUo9hyZEqOIXrfbFsNQwiG2I= 2023-10-12T01:01:06.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6LsQUo9hyZEqOIXrfbFsNQwiG2I=', 'Content-Length': '0'} 2023-10-12T01:01:06.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad62a13d9707d350-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:01:06.446 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.446 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.447 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.447 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.448 DEBUG:boto:Method: GET 2023-10-12T01:01:06.448 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.449 DEBUG:boto:Data: 2023-10-12T01:01:06.449 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.450 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.450 DEBUG:boto:Params: {} 2023-10-12T01:01:06.451 DEBUG:boto:Token: None 2023-10-12T01:01:06.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc= 2023-10-12T01:01:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc=', 'Content-Length': '0'} 2023-10-12T01:01:06.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9fd1ce6c8f86255-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.502 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.503 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.503 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.504 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.504 DEBUG:boto:Method: GET 2023-10-12T01:01:06.505 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.505 DEBUG:boto:Data: 2023-10-12T01:01:06.506 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.507 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.507 DEBUG:boto:Params: {} 2023-10-12T01:01:06.508 DEBUG:boto:Token: None 2023-10-12T01:01:06.508 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:01:06.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc= 2023-10-12T01:01:06.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZTxceyEhTer0KrZ1d0EzISAtbBc=', 'Content-Length': '0'} 2023-10-12T01:01:06.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d88f3fd5e8999d-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.512 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.512 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.513 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.513 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.514 DEBUG:boto:Method: GET 2023-10-12T01:01:06.514 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.514 DEBUG:boto:Data: 2023-10-12T01:01:06.515 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.516 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.516 DEBUG:boto:Params: {} 2023-10-12T01:01:06.517 DEBUG:boto:Token: None 2023-10-12T01:01:06.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee8696cbd246395-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.521 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.522 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.523 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.523 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.524 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.524 DEBUG:boto:Method: GET 2023-10-12T01:01:06.524 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.525 DEBUG:boto:Data: 2023-10-12T01:01:06.525 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.526 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.526 DEBUG:boto:Params: {} 2023-10-12T01:01:06.527 DEBUG:boto:Token: None 2023-10-12T01:01:06.527 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60a76fe79ee6f93-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.578 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.579 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.580 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.580 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.581 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:01:06.581 DEBUG:boto:Data: 2023-10-12T01:01:06.581 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.582 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.583 DEBUG:boto:Params: {} 2023-10-12T01:01:06.583 DEBUG:boto:Token: None 2023-10-12T01:01:06.584 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85 2023-10-12T01:01:06.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+TxmAcwGqeowArNuE2xv8FjLToo= 2023-10-12T01:01:06.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+TxmAcwGqeowArNuE2xv8FjLToo=', 'Content-Length': '0'} 2023-10-12T01:01:06.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213e717d10206a87-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.587 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.588 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.588 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.589 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:01:06.589 DEBUG:boto:Data: 2023-10-12T01:01:06.589 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.590 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.591 DEBUG:boto:Params: {} 2023-10-12T01:01:06.591 DEBUG:boto:Token: None 2023-10-12T01:01:06.592 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85 2023-10-12T01:01:06.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+TxmAcwGqeowArNuE2xv8FjLToo= 2023-10-12T01:01:06.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+TxmAcwGqeowArNuE2xv8FjLToo=', 'Content-Length': '0'} 2023-10-12T01:01:06.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a95099cf30181e4-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:01:06.595 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.596 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.596 DEBUG:boto:Method: GET 2023-10-12T01:01:06.597 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:01:06.597 DEBUG:boto:Data: 2023-10-12T01:01:06.597 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.598 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.599 DEBUG:boto:Params: {} 2023-10-12T01:01:06.599 DEBUG:boto:Token: None 2023-10-12T01:01:06.600 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85 2023-10-12T01:01:06.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+DH5fVXcBA5fGyC67DjN0sGvPbk= 2023-10-12T01:01:06.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+DH5fVXcBA5fGyC67DjN0sGvPbk=', 'Content-Length': '0'} 2023-10-12T01:01:06.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe484ae07f5c36f-0065274552-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.604 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:01:06.604 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:01:06.604 DEBUG:boto:Method: GET 2023-10-12T01:01:06.605 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:01:06.605 DEBUG:boto:Data: 2023-10-12T01:01:06.606 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.607 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.607 DEBUG:boto:Params: {} 2023-10-12T01:01:06.607 DEBUG:boto:Token: None 2023-10-12T01:01:06.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj85 2023-10-12T01:01:06.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+DH5fVXcBA5fGyC67DjN0sGvPbk= 2023-10-12T01:01:06.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+DH5fVXcBA5fGyC67DjN0sGvPbk=', 'Content-Length': '0'} 2023-10-12T01:01:06.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa57290cf76c6ce-0065274552-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.614 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.615 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.615 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.616 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.616 DEBUG:boto:Method: GET 2023-10-12T01:01:06.616 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.617 DEBUG:boto:Data: 2023-10-12T01:01:06.617 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:01:06.618 DEBUG:boto:Port: 8001 2023-10-12T01:01:06.618 DEBUG:boto:Params: {} 2023-10-12T01:01:06.619 DEBUG:boto:Token: None 2023-10-12T01:01:06.619 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c2b069688cf3922-0065274552-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.670 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.671 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.672 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.672 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.673 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.673 DEBUG:boto:Method: GET 2023-10-12T01:01:06.673 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:01:06.674 DEBUG:boto:Data: 2023-10-12T01:01:06.674 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.675 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.676 DEBUG:boto:Params: {} 2023-10-12T01:01:06.676 DEBUG:boto:Token: None 2023-10-12T01:01:06.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4= 2023-10-12T01:01:06.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cHT5+JCDUQvM+L2YvCm/gSl3OB4=', 'Content-Length': '0'} 2023-10-12T01:01:06.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdfb3bbe7a3762c-0065274552-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.682 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:01:06.683 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:01:06.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:01:06.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:01:06.685 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:01:06.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:01:06.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:01:06.790 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:01:06.791 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T01:01:06.796 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-10-12T01:01:06.839 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.839 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.840 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.840 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:06.841 DEBUG:boto:Data: 2023-10-12T01:01:06.841 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.842 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.842 DEBUG:boto:Params: {} 2023-10-12T01:01:06.843 DEBUG:boto:Token: None 2023-10-12T01:01:06.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8= 2023-10-12T01:01:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8=', 'Content-Length': '0'} 2023-10-12T01:01:06.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fad8dbcdfb2dc6fc-0065274552-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.848 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:01:06.848 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:01:06.848 DEBUG:boto:Method: PUT 2023-10-12T01:01:06.849 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:01:06.849 DEBUG:boto:Data: 2023-10-12T01:01:06.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:01:06.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.851 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.851 DEBUG:boto:Params: {} 2023-10-12T01:01:06.851 DEBUG:boto:Token: None 2023-10-12T01:01:06.852 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj19 2023-10-12T01:01:06.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CP0oM2pbpy5+injJT1jFn9SgpbA= 2023-10-12T01:01:06.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CP0oM2pbpy5+injJT1jFn9SgpbA='} 2023-10-12T01:01:06.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000316ada7d76b216c3-0065274552-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.895 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.896 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.896 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.896 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:06.897 DEBUG:boto:Data: 2023-10-12T01:01:06.897 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.898 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.898 DEBUG:boto:Params: {} 2023-10-12T01:01:06.899 DEBUG:boto:Token: None 2023-10-12T01:01:06.899 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8= 2023-10-12T01:01:06.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8=', 'Content-Length': '0'} 2023-10-12T01:01:06.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ddd4be01fcd99be-0065274552-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.903 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:01:06.904 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:01:06.904 DEBUG:boto:Method: PUT 2023-10-12T01:01:06.904 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:01:06.905 DEBUG:boto:Data: 2023-10-12T01:01:06.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:01:06.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.906 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.906 DEBUG:boto:Params: {} 2023-10-12T01:01:06.907 DEBUG:boto:Token: None 2023-10-12T01:01:06.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj38 2023-10-12T01:01:06.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hQzpYEfz6ClhAnPf47l/ALIveiI= 2023-10-12T01:01:06.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hQzpYEfz6ClhAnPf47l/ALIveiI='} 2023-10-12T01:01:06.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008db221de9cd502b3-0065274552-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.913 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.913 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.914 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.914 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:06.914 DEBUG:boto:Data: 2023-10-12T01:01:06.915 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.916 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.916 DEBUG:boto:Params: {} 2023-10-12T01:01:06.916 DEBUG:boto:Token: None 2023-10-12T01:01:06.917 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8= 2023-10-12T01:01:06.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8=', 'Content-Length': '0'} 2023-10-12T01:01:06.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d77363a45365ec1-0065274552-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.921 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:01:06.922 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:01:06.922 DEBUG:boto:Method: PUT 2023-10-12T01:01:06.923 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:01:06.923 DEBUG:boto:Data: 2023-10-12T01:01:06.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:01:06.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.924 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.925 DEBUG:boto:Params: {} 2023-10-12T01:01:06.925 DEBUG:boto:Token: None 2023-10-12T01:01:06.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj57 2023-10-12T01:01:06.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+tm8C/dVoYzfucPSqAXXKJUQZ/o= 2023-10-12T01:01:06.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+tm8C/dVoYzfucPSqAXXKJUQZ/o='} 2023-10-12T01:01:06.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a9c3e0ea4ed00e0-0065274552-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.975 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.976 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.976 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.976 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:06.977 DEBUG:boto:Data: 2023-10-12T01:01:06.978 DEBUG:boto:Headers: {} 2023-10-12T01:01:06.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.978 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.979 DEBUG:boto:Params: {} 2023-10-12T01:01:06.979 DEBUG:boto:Token: None 2023-10-12T01:01:06.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/ 2023-10-12T01:01:06.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8= 2023-10-12T01:01:06.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRBtRs7cbhPD9YnrlJObr+kDqU8=', 'Content-Length': '0'} 2023-10-12T01:01:06.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080cdc7e108c639b3-0065274552-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.984 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:01:06.985 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:01:06.985 DEBUG:boto:Method: PUT 2023-10-12T01:01:06.985 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:01:06.986 DEBUG:boto:Data: 2023-10-12T01:01:06.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:01:06.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:06.987 DEBUG:boto:Port: 8000 2023-10-12T01:01:06.988 DEBUG:boto:Params: {} 2023-10-12T01:01:06.988 DEBUG:boto:Token: None 2023-10-12T01:01:06.989 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:01:06 GMT /wafmoa-59/obj76 2023-10-12T01:01:06.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qR9PffO9bwWXWi8l4aZQ+HupPno= 2023-10-12T01:01:06.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:01:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qR9PffO9bwWXWi8l4aZQ+HupPno='} 2023-10-12T01:01:06.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f58025d28a69744d-0065274552-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:06.996 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:01:06.997 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:01:06.997 DEBUG:boto:Method: HEAD 2023-10-12T01:01:06.998 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:01:06.999 DEBUG:boto:Data: 2023-10-12T01:01:06.999 DEBUG:boto:Headers: {} 2023-10-12T01:01:07.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:07.001 DEBUG:boto:Port: 8000 2023-10-12T01:01:07.002 DEBUG:boto:Params: {} 2023-10-12T01:01:07.002 DEBUG:boto:Token: None 2023-10-12T01:01:07.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:01:07 GMT /wafmoa-59/ 2023-10-12T01:01:07.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eJujyP5XJGSx7Q0vSaOix2a37wQ= 2023-10-12T01:01:07.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:01:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eJujyP5XJGSx7Q0vSaOix2a37wQ=', 'Content-Length': '0'} 2023-10-12T01:01:07.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f97ca316e598184-0065274553-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:07.007 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:01:07.008 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:01:07.008 DEBUG:boto:Method: PUT 2023-10-12T01:01:07.009 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:01:07.009 DEBUG:boto:Data: 2023-10-12T01:01:07.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:01:07.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:01:07.011 DEBUG:boto:Port: 8000 2023-10-12T01:01:07.011 DEBUG:boto:Params: {} 2023-10-12T01:01:07.012 DEBUG:boto:Token: None 2023-10-12T01:01:07.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:01:07 GMT /wafmoa-59/obj95 2023-10-12T01:01:07.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l3G1BxB0p9eJB1AI25sAaY/iTsk= 2023-10-12T01:01:07.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:01:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l3G1BxB0p9eJB1AI25sAaY/iTsk='} 2023-10-12T01:01:07.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072b60703ce36d110-0065274553-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:01:07.059 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:01:07.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:01:07.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:01:07.061 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:01:07.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:01:07.118 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:01:07.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:07.150+0000 7fa0248fa780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T01:01:12.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:12.151+0000 7fa0248fa780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T01:01:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:17.152+0000 7fa0248fa780 1 waiting to reach latest gen 2, on 1.. 2023-10-12T01:01:22.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:22.154+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:22.155 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-12T01:01:22.155 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:27.156+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:27.156 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-12T01:01:27.157 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:32.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:32.158+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:32.158 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-12T01:01:32.158 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:37.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:37.160+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:37.160 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-12T01:01:37.160 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:42.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:42.160+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:42.162 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:01:42.162 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:47.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:47.162+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:47.163 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:01:47.163 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:52.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:52.163+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:52.165 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:01:52.165 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:01:57.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:01:57.165+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:01:57.166 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:01:57.167 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:02.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:02.167+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:02.168 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:02.168 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:07.170+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:07.170 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:07.170 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:12.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:12.171+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:12.172 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:12.172 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:17.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:17.173+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:17.173 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:17.173 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:22.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:22.174+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:22.175 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:22.175 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:27.175+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:27.176 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:27.176 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:32.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:32.177+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:32.178 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:32.178 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:37.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:37.178+0000 7fa0248fa780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:37.179 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000002.97.6, , , , , , , , , , , ] 2023-10-12T01:02:37.180 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:42.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:42.181+0000 7fa0248fa780 1 bucket sync caught up with source: 2023-10-12T01:02:42.181 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:42.181 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.34.6, 00000000004.101.6, , , , , , , , , , , ] 2023-10-12T01:02:42.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:42.181+0000 7fa0248fa780 0 bucket checkpoint complete 2023-10-12T01:02:42.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:02:42.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:02:42.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:02:42.200 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:02:42.261 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:02:42.262 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:02:42.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:42.299+0000 7f2249945780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:42.299 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000001.281.6, , 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:42.299 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:47.300+0000 7f2249945780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:47.300 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000001.281.6, , 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:47.300 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:52.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:52.301+0000 7f2249945780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:52.301 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000001.281.6, , 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:52.302 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:57.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:02:57.301+0000 7f2249945780 1 waiting for incremental sync to catch up: 2023-10-12T01:02:57.303 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000001.281.6, , 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:02:57.303 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:03:02.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:02.303+0000 7f2249945780 1 bucket sync caught up with source: 2023-10-12T01:03:02.304 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:03:02.304 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000001.281.6, 00000000001.181.6, 00000000001.189.6, 00000000001.181.6, , , 00000000002.287.6, 00000000004.238.6, 00000000003.179.6] 2023-10-12T01:03:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:02.303+0000 7f2249945780 0 bucket checkpoint complete 2023-10-12T01:03:02.320 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:03:02.320 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.321 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.322 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.322 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:02.322 DEBUG:boto:Data: 2023-10-12T01:03:02.323 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.324 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.324 DEBUG:boto:Params: {} 2023-10-12T01:03:02.325 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:03:02.325 DEBUG:boto:Token: None 2023-10-12T01:03:02.326 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7wNuD76BZtnrRJv6g/5fJxqAEiA= 2023-10-12T01:03:02.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7wNuD76BZtnrRJv6g/5fJxqAEiA=', 'Content-Length': '0'} 2023-10-12T01:03:02.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c6f3a93c322209a-00652745c6-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.332 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.333 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.333 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.334 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:02.334 DEBUG:boto:Data: 2023-10-12T01:03:02.334 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.335 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.336 DEBUG:boto:Params: {} 2023-10-12T01:03:02.336 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:03:02.337 DEBUG:boto:Token: None 2023-10-12T01:03:02.337 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7wNuD76BZtnrRJv6g/5fJxqAEiA= 2023-10-12T01:03:02.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7wNuD76BZtnrRJv6g/5fJxqAEiA=', 'Content-Length': '0'} 2023-10-12T01:03:02.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9966910dd418197-00652745c6-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:03:02.343 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.344 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.344 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.344 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.345 DEBUG:boto:Method: GET 2023-10-12T01:03:02.345 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.346 DEBUG:boto:Data: 2023-10-12T01:03:02.346 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.347 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.347 DEBUG:boto:Params: {} 2023-10-12T01:03:02.348 DEBUG:boto:Token: None 2023-10-12T01:03:02.348 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/?versions 2023-10-12T01:03:02.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g= 2023-10-12T01:03:02.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g=', 'Content-Length': '0'} 2023-10-12T01:03:02.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d3e000aedf36ce3-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.352 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:03:02.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:03:02.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:03:02.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:03:02.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:03:02.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:03:02.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:03:02.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:03:02.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:03:02.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:03:02.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:03:02.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:03:02.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:03:02.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:03:02.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:03:02.360 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:03:02.360 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.360 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.361 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.361 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.362 DEBUG:boto:Method: GET 2023-10-12T01:03:02.362 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.363 DEBUG:boto:Data: 2023-10-12T01:03:02.363 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.364 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.365 DEBUG:boto:Params: {} 2023-10-12T01:03:02.365 DEBUG:boto:Token: None 2023-10-12T01:03:02.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/?versions 2023-10-12T01:03:02.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g= 2023-10-12T01:03:02.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g=', 'Content-Length': '0'} 2023-10-12T01:03:02.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000070411991c31ff8-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.370 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:03:02.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:03:02.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:03:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:03:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:03:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:03:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:03:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:03:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:03:02.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:03:02.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:03:02.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:03:02.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:03:02.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:03:02.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:03:02.377 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.378 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.378 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.379 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.379 DEBUG:boto:Method: GET 2023-10-12T01:03:02.380 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.380 DEBUG:boto:Data: 2023-10-12T01:03:02.380 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.381 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.382 DEBUG:boto:Params: {} 2023-10-12T01:03:02.382 DEBUG:boto:Token: None 2023-10-12T01:03:02.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/?versions 2023-10-12T01:03:02.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g= 2023-10-12T01:03:02.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g=', 'Content-Length': '0'} 2023-10-12T01:03:02.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d30888405e3c437-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.387 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.388 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.389 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.389 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.390 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.390 DEBUG:boto:Method: GET 2023-10-12T01:03:02.391 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:03:02.391 DEBUG:boto:Data: 2023-10-12T01:03:02.391 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.392 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.393 DEBUG:boto:Params: {} 2023-10-12T01:03:02.393 DEBUG:boto:Token: None 2023-10-12T01:03:02.394 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/?versions 2023-10-12T01:03:02.394 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g= 2023-10-12T01:03:02.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fxlk52PL30dLObKf/DezFXMs/+g=', 'Content-Length': '0'} 2023-10-12T01:03:02.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e0ab50049b94cc3-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.398 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:03:02.400 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.400 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.401 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.401 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.402 DEBUG:boto:Method: GET 2023-10-12T01:03:02.402 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.403 DEBUG:boto:Data: 2023-10-12T01:03:02.403 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.404 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.404 DEBUG:boto:Params: {} 2023-10-12T01:03:02.405 DEBUG:boto:Token: None 2023-10-12T01:03:02.405 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4= 2023-10-12T01:03:02.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4=', 'Content-Length': '0'} 2023-10-12T01:03:02.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc425292fb8603f-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.409 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.410 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.410 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.410 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.411 DEBUG:boto:Method: GET 2023-10-12T01:03:02.411 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.412 DEBUG:boto:Data: 2023-10-12T01:03:02.412 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.413 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.414 DEBUG:boto:Params: {} 2023-10-12T01:03:02.414 DEBUG:boto:Token: None 2023-10-12T01:03:02.415 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4= 2023-10-12T01:03:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4=', 'Content-Length': '0'} 2023-10-12T01:03:02.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119c1bb2794caccf-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.419 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.419 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.419 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.420 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.420 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.421 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.421 DEBUG:boto:Data: 2023-10-12T01:03:02.422 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.422 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.423 DEBUG:boto:Params: {} 2023-10-12T01:03:02.423 DEBUG:boto:Token: None 2023-10-12T01:03:02.424 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TCOa0/83EuEMxczdjkbcp3ypYBw= 2023-10-12T01:03:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TCOa0/83EuEMxczdjkbcp3ypYBw=', 'Content-Length': '0'} 2023-10-12T01:03:02.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b64d2b762fa2379-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.427 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.428 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.428 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.428 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.429 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.429 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.430 DEBUG:boto:Data: 2023-10-12T01:03:02.430 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.431 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.431 DEBUG:boto:Params: {} 2023-10-12T01:03:02.432 DEBUG:boto:Token: None 2023-10-12T01:03:02.432 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TCOa0/83EuEMxczdjkbcp3ypYBw= 2023-10-12T01:03:02.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TCOa0/83EuEMxczdjkbcp3ypYBw=', 'Content-Length': '0'} 2023-10-12T01:03:02.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a55cf2f05d34d3f-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:03:02.436 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.437 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.437 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.437 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.438 DEBUG:boto:Method: GET 2023-10-12T01:03:02.438 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.439 DEBUG:boto:Data: 2023-10-12T01:03:02.439 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.440 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.440 DEBUG:boto:Params: {} 2023-10-12T01:03:02.441 DEBUG:boto:Token: None 2023-10-12T01:03:02.441 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4= 2023-10-12T01:03:02.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4=', 'Content-Length': '0'} 2023-10-12T01:03:02.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000508760930d8e7cb0-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.490 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.491 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.492 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.492 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.493 DEBUG:boto:Method: GET 2023-10-12T01:03:02.493 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.494 DEBUG:boto:Data: 2023-10-12T01:03:02.494 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.495 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.496 DEBUG:boto:Params: {} 2023-10-12T01:03:02.496 DEBUG:boto:Token: None 2023-10-12T01:03:02.497 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:03:02.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4= 2023-10-12T01:03:02.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:047D/WMrORYoyDAVAwJUY+7JN+4=', 'Content-Length': '0'} 2023-10-12T01:03:02.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f3a194b01c6b02-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.501 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.501 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.501 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.502 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.502 DEBUG:boto:Method: GET 2023-10-12T01:03:02.503 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.503 DEBUG:boto:Data: 2023-10-12T01:03:02.504 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.504 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.505 DEBUG:boto:Params: {} 2023-10-12T01:03:02.505 DEBUG:boto:Token: None 2023-10-12T01:03:02.506 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98535e72b1a8b70-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.510 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.512 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.512 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.512 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.513 DEBUG:boto:Method: GET 2023-10-12T01:03:02.513 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.514 DEBUG:boto:Data: 2023-10-12T01:03:02.514 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.515 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.515 DEBUG:boto:Params: {} 2023-10-12T01:03:02.516 DEBUG:boto:Token: None 2023-10-12T01:03:02.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000939e3e2c2590dfad-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.521 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.522 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.522 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.523 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.523 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:03:02.524 DEBUG:boto:Data: 2023-10-12T01:03:02.524 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.525 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.525 DEBUG:boto:Params: {} 2023-10-12T01:03:02.526 DEBUG:boto:Token: None 2023-10-12T01:03:02.526 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11 2023-10-12T01:03:02.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G8Te1r3lUusRTIGL5K8HuXfDy2o= 2023-10-12T01:03:02.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G8Te1r3lUusRTIGL5K8HuXfDy2o=', 'Content-Length': '0'} 2023-10-12T01:03:02.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fd7b6d9d716b8d-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.530 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.530 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.530 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.531 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:03:02.531 DEBUG:boto:Data: 2023-10-12T01:03:02.532 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.532 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.533 DEBUG:boto:Params: {} 2023-10-12T01:03:02.533 DEBUG:boto:Token: None 2023-10-12T01:03:02.534 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11 2023-10-12T01:03:02.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G8Te1r3lUusRTIGL5K8HuXfDy2o= 2023-10-12T01:03:02.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G8Te1r3lUusRTIGL5K8HuXfDy2o=', 'Content-Length': '0'} 2023-10-12T01:03:02.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f43bd7813b4514-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:03:02.538 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.538 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.539 DEBUG:boto:Method: GET 2023-10-12T01:03:02.539 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:03:02.539 DEBUG:boto:Data: 2023-10-12T01:03:02.540 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.541 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.541 DEBUG:boto:Params: {} 2023-10-12T01:03:02.542 DEBUG:boto:Token: None 2023-10-12T01:03:02.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11 2023-10-12T01:03:02.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4fDEwP8lrCQz7eEea55XnAaMYG0= 2023-10-12T01:03:02.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4fDEwP8lrCQz7eEea55XnAaMYG0=', 'Content-Length': '0'} 2023-10-12T01:03:02.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064de74707de4e9ba-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.546 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:03:02.546 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:03:02.547 DEBUG:boto:Method: GET 2023-10-12T01:03:02.547 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:03:02.548 DEBUG:boto:Data: 2023-10-12T01:03:02.548 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.549 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.549 DEBUG:boto:Params: {} 2023-10-12T01:03:02.550 DEBUG:boto:Token: None 2023-10-12T01:03:02.550 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj11 2023-10-12T01:03:02.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4fDEwP8lrCQz7eEea55XnAaMYG0= 2023-10-12T01:03:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4fDEwP8lrCQz7eEea55XnAaMYG0=', 'Content-Length': '0'} 2023-10-12T01:03:02.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c2975cb5b570b2-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.554 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.554 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.555 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.555 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.555 DEBUG:boto:Method: GET 2023-10-12T01:03:02.556 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.556 DEBUG:boto:Data: 2023-10-12T01:03:02.557 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.558 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.558 DEBUG:boto:Params: {} 2023-10-12T01:03:02.559 DEBUG:boto:Token: None 2023-10-12T01:03:02.559 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed453977026e46e9-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.563 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.564 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.565 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.565 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.565 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.566 DEBUG:boto:Method: GET 2023-10-12T01:03:02.566 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:03:02.567 DEBUG:boto:Data: 2023-10-12T01:03:02.567 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.568 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.569 DEBUG:boto:Params: {} 2023-10-12T01:03:02.569 DEBUG:boto:Token: None 2023-10-12T01:03:02.570 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b0cb522d25da283-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.574 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:03:02.575 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.576 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.576 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.577 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.577 DEBUG:boto:Method: GET 2023-10-12T01:03:02.577 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.578 DEBUG:boto:Data: 2023-10-12T01:03:02.578 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.579 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.580 DEBUG:boto:Params: {} 2023-10-12T01:03:02.580 DEBUG:boto:Token: None 2023-10-12T01:03:02.581 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE= 2023-10-12T01:03:02.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE=', 'Content-Length': '0'} 2023-10-12T01:03:02.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04f40ce6514d72b-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.584 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.585 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.585 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.586 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.586 DEBUG:boto:Method: GET 2023-10-12T01:03:02.587 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.590 DEBUG:boto:Data: 2023-10-12T01:03:02.590 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.591 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.591 DEBUG:boto:Params: {} 2023-10-12T01:03:02.592 DEBUG:boto:Token: None 2023-10-12T01:03:02.592 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE= 2023-10-12T01:03:02.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE=', 'Content-Length': '0'} 2023-10-12T01:03:02.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051c1c455c97c1d5-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.596 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.596 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.597 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.597 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.597 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.598 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.598 DEBUG:boto:Data: 2023-10-12T01:03:02.598 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.599 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.600 DEBUG:boto:Params: {} 2023-10-12T01:03:02.600 DEBUG:boto:Token: None 2023-10-12T01:03:02.600 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bkV+Lbler4DYRtZN0gvrXuj15lI= 2023-10-12T01:03:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bkV+Lbler4DYRtZN0gvrXuj15lI=', 'Content-Length': '0'} 2023-10-12T01:03:02.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b070bd85b194db42-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.604 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.604 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.605 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.605 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.605 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.606 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.606 DEBUG:boto:Data: 2023-10-12T01:03:02.607 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.607 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.608 DEBUG:boto:Params: {} 2023-10-12T01:03:02.608 DEBUG:boto:Token: None 2023-10-12T01:03:02.608 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bkV+Lbler4DYRtZN0gvrXuj15lI= 2023-10-12T01:03:02.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bkV+Lbler4DYRtZN0gvrXuj15lI=', 'Content-Length': '0'} 2023-10-12T01:03:02.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7805a6c396dbac-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:03:02.612 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.613 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.613 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.614 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.614 DEBUG:boto:Method: GET 2023-10-12T01:03:02.614 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.615 DEBUG:boto:Data: 2023-10-12T01:03:02.615 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.616 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.617 DEBUG:boto:Params: {} 2023-10-12T01:03:02.617 DEBUG:boto:Token: None 2023-10-12T01:03:02.618 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.618 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE= 2023-10-12T01:03:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE=', 'Content-Length': '0'} 2023-10-12T01:03:02.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d072b72ef4eb741-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.662 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.663 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.664 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.664 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.665 DEBUG:boto:Method: GET 2023-10-12T01:03:02.665 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.665 DEBUG:boto:Data: 2023-10-12T01:03:02.666 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.667 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.667 DEBUG:boto:Params: {} 2023-10-12T01:03:02.668 DEBUG:boto:Token: None 2023-10-12T01:03:02.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:03:02.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE= 2023-10-12T01:03:02.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oluuvvkyBezbz9XfWnw5dT6wVyE=', 'Content-Length': '0'} 2023-10-12T01:03:02.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a6ad5bcae54010-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.672 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.673 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.673 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.674 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.674 DEBUG:boto:Method: GET 2023-10-12T01:03:02.675 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.675 DEBUG:boto:Data: 2023-10-12T01:03:02.676 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.676 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.677 DEBUG:boto:Params: {} 2023-10-12T01:03:02.677 DEBUG:boto:Token: None 2023-10-12T01:03:02.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6b5061352a9551-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.682 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.683 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.683 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.684 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.684 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.684 DEBUG:boto:Method: GET 2023-10-12T01:03:02.685 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.685 DEBUG:boto:Data: 2023-10-12T01:03:02.686 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.687 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.687 DEBUG:boto:Params: {} 2023-10-12T01:03:02.688 DEBUG:boto:Token: None 2023-10-12T01:03:02.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.690 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8a43e660b6e2a4-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.694 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.695 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.696 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.696 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.696 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:03:02.697 DEBUG:boto:Data: 2023-10-12T01:03:02.697 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.698 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.699 DEBUG:boto:Params: {} 2023-10-12T01:03:02.699 DEBUG:boto:Token: None 2023-10-12T01:03:02.700 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17 2023-10-12T01:03:02.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wMzw3ohv4cg+9AC9Lj4A3yszm+0= 2023-10-12T01:03:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wMzw3ohv4cg+9AC9Lj4A3yszm+0=', 'Content-Length': '0'} 2023-10-12T01:03:02.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097fca7ba38e0597-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.703 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.703 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.704 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.704 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:03:02.705 DEBUG:boto:Data: 2023-10-12T01:03:02.705 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.706 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.707 DEBUG:boto:Params: {} 2023-10-12T01:03:02.707 DEBUG:boto:Token: None 2023-10-12T01:03:02.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17 2023-10-12T01:03:02.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wMzw3ohv4cg+9AC9Lj4A3yszm+0= 2023-10-12T01:03:02.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wMzw3ohv4cg+9AC9Lj4A3yszm+0=', 'Content-Length': '0'} 2023-10-12T01:03:02.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbfe259364b987c-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:03:02.711 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.712 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.712 DEBUG:boto:Method: GET 2023-10-12T01:03:02.713 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:03:02.713 DEBUG:boto:Data: 2023-10-12T01:03:02.714 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.715 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.715 DEBUG:boto:Params: {} 2023-10-12T01:03:02.716 DEBUG:boto:Token: None 2023-10-12T01:03:02.716 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17 2023-10-12T01:03:02.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i7UoSBjbAxChiveVnPdfvMxyl/c= 2023-10-12T01:03:02.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i7UoSBjbAxChiveVnPdfvMxyl/c=', 'Content-Length': '0'} 2023-10-12T01:03:02.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edc9198fa4399805-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.720 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:03:02.720 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:03:02.721 DEBUG:boto:Method: GET 2023-10-12T01:03:02.721 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:03:02.722 DEBUG:boto:Data: 2023-10-12T01:03:02.722 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.723 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.724 DEBUG:boto:Params: {} 2023-10-12T01:03:02.724 DEBUG:boto:Token: None 2023-10-12T01:03:02.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj17 2023-10-12T01:03:02.725 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i7UoSBjbAxChiveVnPdfvMxyl/c= 2023-10-12T01:03:02.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i7UoSBjbAxChiveVnPdfvMxyl/c=', 'Content-Length': '0'} 2023-10-12T01:03:02.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd2c57cf9ff8127-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.770 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.771 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.771 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.772 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.772 DEBUG:boto:Method: GET 2023-10-12T01:03:02.773 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.773 DEBUG:boto:Data: 2023-10-12T01:03:02.774 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.775 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.775 DEBUG:boto:Params: {} 2023-10-12T01:03:02.776 DEBUG:boto:Token: None 2023-10-12T01:03:02.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f241684f80c076-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.781 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.781 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.782 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.782 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.783 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.783 DEBUG:boto:Method: GET 2023-10-12T01:03:02.784 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:03:02.784 DEBUG:boto:Data: 2023-10-12T01:03:02.784 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.785 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.786 DEBUG:boto:Params: {} 2023-10-12T01:03:02.786 DEBUG:boto:Token: None 2023-10-12T01:03:02.787 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a2441a47c12731a-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.792 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:03:02.793 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.793 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.793 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.794 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.794 DEBUG:boto:Method: GET 2023-10-12T01:03:02.795 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.795 DEBUG:boto:Data: 2023-10-12T01:03:02.796 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.797 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.797 DEBUG:boto:Params: {} 2023-10-12T01:03:02.798 DEBUG:boto:Token: None 2023-10-12T01:03:02.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0= 2023-10-12T01:03:02.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0=', 'Content-Length': '0'} 2023-10-12T01:03:02.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb66ed83d74e019-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.846 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.847 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.848 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.849 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.849 DEBUG:boto:Method: GET 2023-10-12T01:03:02.850 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.850 DEBUG:boto:Data: 2023-10-12T01:03:02.851 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.851 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.852 DEBUG:boto:Params: {} 2023-10-12T01:03:02.852 DEBUG:boto:Token: None 2023-10-12T01:03:02.853 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0= 2023-10-12T01:03:02.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0=', 'Content-Length': '0'} 2023-10-12T01:03:02.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a0061ceb655690-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.857 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.857 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.858 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.858 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.859 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.859 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.860 DEBUG:boto:Data: 2023-10-12T01:03:02.860 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.861 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.861 DEBUG:boto:Params: {} 2023-10-12T01:03:02.862 DEBUG:boto:Token: None 2023-10-12T01:03:02.862 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LRhZGPItZMapMOjcdLRUM6gt0b4= 2023-10-12T01:03:02.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LRhZGPItZMapMOjcdLRUM6gt0b4=', 'Content-Length': '0'} 2023-10-12T01:03:02.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed61b7cb9f7ba12-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.866 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.866 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.867 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.867 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.868 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.868 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.869 DEBUG:boto:Data: 2023-10-12T01:03:02.869 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.870 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.871 DEBUG:boto:Params: {} 2023-10-12T01:03:02.872 DEBUG:boto:Token: None 2023-10-12T01:03:02.872 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LRhZGPItZMapMOjcdLRUM6gt0b4= 2023-10-12T01:03:02.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LRhZGPItZMapMOjcdLRUM6gt0b4=', 'Content-Length': '0'} 2023-10-12T01:03:02.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dcfcd49941faca0-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:03:02.876 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.876 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.877 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.877 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.878 DEBUG:boto:Method: GET 2023-10-12T01:03:02.878 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.879 DEBUG:boto:Data: 2023-10-12T01:03:02.879 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.880 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.881 DEBUG:boto:Params: {} 2023-10-12T01:03:02.881 DEBUG:boto:Token: None 2023-10-12T01:03:02.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0= 2023-10-12T01:03:02.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0=', 'Content-Length': '0'} 2023-10-12T01:03:02.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8712e167b7a947-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.886 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.886 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.887 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.887 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.888 DEBUG:boto:Method: GET 2023-10-12T01:03:02.888 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.889 DEBUG:boto:Data: 2023-10-12T01:03:02.889 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.890 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.891 DEBUG:boto:Params: {} 2023-10-12T01:03:02.891 DEBUG:boto:Token: None 2023-10-12T01:03:02.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:03:02.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0= 2023-10-12T01:03:02.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bu0Oim7d940YdmZXH1hLimUzmg0=', 'Content-Length': '0'} 2023-10-12T01:03:02.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074895eab7b28dfda-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.898 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.898 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.899 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.899 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.900 DEBUG:boto:Method: GET 2023-10-12T01:03:02.901 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.901 DEBUG:boto:Data: 2023-10-12T01:03:02.902 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.903 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.903 DEBUG:boto:Params: {} 2023-10-12T01:03:02.904 DEBUG:boto:Token: None 2023-10-12T01:03:02.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd717af474ec82f-00652745c6-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.909 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.910 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.911 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.911 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.911 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.912 DEBUG:boto:Method: GET 2023-10-12T01:03:02.912 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.913 DEBUG:boto:Data: 2023-10-12T01:03:02.913 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.914 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.915 DEBUG:boto:Params: {} 2023-10-12T01:03:02.915 DEBUG:boto:Token: None 2023-10-12T01:03:02.915 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/ 2023-10-12T01:03:02.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI= 2023-10-12T01:03:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aPPYGdOuBrwc+Eazhp6L8rTTIOI=', 'Content-Length': '0'} 2023-10-12T01:03:02.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e7c9ddff4f5459-00652745c6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.920 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:02.920 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.921 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.921 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.922 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:03:02.922 DEBUG:boto:Data: 2023-10-12T01:03:02.923 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.923 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.924 DEBUG:boto:Params: {} 2023-10-12T01:03:02.924 DEBUG:boto:Token: None 2023-10-12T01:03:02.925 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19 2023-10-12T01:03:02.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/x5HFqdRGT8unH8FeptskgaXo4= 2023-10-12T01:03:02.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/x5HFqdRGT8unH8FeptskgaXo4=', 'Content-Length': '0'} 2023-10-12T01:03:02.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fc4d66a08c2921-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.928 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.929 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.929 DEBUG:boto:Method: HEAD 2023-10-12T01:03:02.930 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:03:02.930 DEBUG:boto:Data: 2023-10-12T01:03:02.930 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.931 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.932 DEBUG:boto:Params: {} 2023-10-12T01:03:02.932 DEBUG:boto:Token: None 2023-10-12T01:03:02.933 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19 2023-10-12T01:03:02.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/x5HFqdRGT8unH8FeptskgaXo4= 2023-10-12T01:03:02.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/x5HFqdRGT8unH8FeptskgaXo4=', 'Content-Length': '0'} 2023-10-12T01:03:02.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e7c55564fc91f7-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:03:02.937 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.937 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.938 DEBUG:boto:Method: GET 2023-10-12T01:03:02.938 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:03:02.939 DEBUG:boto:Data: 2023-10-12T01:03:02.939 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.940 DEBUG:boto:Port: 8001 2023-10-12T01:03:02.941 DEBUG:boto:Params: {} 2023-10-12T01:03:02.941 DEBUG:boto:Token: None 2023-10-12T01:03:02.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19 2023-10-12T01:03:02.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cEDkV82nQpMGrqdHfC1cgbRRfLk= 2023-10-12T01:03:02.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cEDkV82nQpMGrqdHfC1cgbRRfLk=', 'Content-Length': '0'} 2023-10-12T01:03:02.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9695a29019adcd1-00652745c6-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.945 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:03:02.946 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:03:02.946 DEBUG:boto:Method: GET 2023-10-12T01:03:02.947 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:03:02.947 DEBUG:boto:Data: 2023-10-12T01:03:02.948 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:02.949 DEBUG:boto:Port: 8000 2023-10-12T01:03:02.949 DEBUG:boto:Params: {} 2023-10-12T01:03:02.950 DEBUG:boto:Token: None 2023-10-12T01:03:02.950 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:02 GMT /wafmoa-59/obj19 2023-10-12T01:03:02.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cEDkV82nQpMGrqdHfC1cgbRRfLk= 2023-10-12T01:03:02.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cEDkV82nQpMGrqdHfC1cgbRRfLk=', 'Content-Length': '0'} 2023-10-12T01:03:02.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208515e100ab1ab1-00652745c6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:02.994 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:02.995 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:02.996 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.996 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.997 DEBUG:boto:Method: GET 2023-10-12T01:03:02.997 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:03:02.998 DEBUG:boto:Data: 2023-10-12T01:03:02.998 DEBUG:boto:Headers: {} 2023-10-12T01:03:02.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:02.999 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.000 DEBUG:boto:Params: {} 2023-10-12T01:03:03.001 DEBUG:boto:Token: None 2023-10-12T01:03:03.001 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e00a7c4820c79e9-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.006 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.007 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.007 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.008 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:03.008 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:03:03.009 DEBUG:boto:Method: GET 2023-10-12T01:03:03.009 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:03:03.010 DEBUG:boto:Data: 2023-10-12T01:03:03.010 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.011 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.011 DEBUG:boto:Params: {} 2023-10-12T01:03:03.012 DEBUG:boto:Token: None 2023-10-12T01:03:03.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061ea7edaccb42756-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.017 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:03:03.018 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.019 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.019 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.020 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.020 DEBUG:boto:Method: GET 2023-10-12T01:03:03.021 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.021 DEBUG:boto:Data: 2023-10-12T01:03:03.022 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.022 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.023 DEBUG:boto:Params: {} 2023-10-12T01:03:03.023 DEBUG:boto:Token: None 2023-10-12T01:03:03.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI= 2023-10-12T01:03:03.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI=', 'Content-Length': '0'} 2023-10-12T01:03:03.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd0dcbe7558196e-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.028 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.028 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.029 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.029 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.030 DEBUG:boto:Method: GET 2023-10-12T01:03:03.030 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.030 DEBUG:boto:Data: 2023-10-12T01:03:03.031 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.032 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.032 DEBUG:boto:Params: {} 2023-10-12T01:03:03.033 DEBUG:boto:Token: None 2023-10-12T01:03:03.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.034 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI= 2023-10-12T01:03:03.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI=', 'Content-Length': '0'} 2023-10-12T01:03:03.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027cc074e68b7f44f-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.037 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.037 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.038 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.038 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.039 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.039 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.040 DEBUG:boto:Data: 2023-10-12T01:03:03.040 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.041 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.041 DEBUG:boto:Params: {} 2023-10-12T01:03:03.043 DEBUG:boto:Token: None 2023-10-12T01:03:03.043 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.044 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rZyCzthnv1HJRtIWcK6LKLk6I3I= 2023-10-12T01:03:03.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rZyCzthnv1HJRtIWcK6LKLk6I3I=', 'Content-Length': '0'} 2023-10-12T01:03:03.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000725381a1c7aa0a-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.047 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.048 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.048 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.049 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.050 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.050 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.051 DEBUG:boto:Data: 2023-10-12T01:03:03.052 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.053 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.053 DEBUG:boto:Params: {} 2023-10-12T01:03:03.054 DEBUG:boto:Token: None 2023-10-12T01:03:03.055 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rZyCzthnv1HJRtIWcK6LKLk6I3I= 2023-10-12T01:03:03.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rZyCzthnv1HJRtIWcK6LKLk6I3I=', 'Content-Length': '0'} 2023-10-12T01:03:03.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099fb6fd12ce8f08c-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:03:03.059 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.060 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.060 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.060 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.061 DEBUG:boto:Method: GET 2023-10-12T01:03:03.061 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.062 DEBUG:boto:Data: 2023-10-12T01:03:03.062 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.063 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.063 DEBUG:boto:Params: {} 2023-10-12T01:03:03.064 DEBUG:boto:Token: None 2023-10-12T01:03:03.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI= 2023-10-12T01:03:03.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI=', 'Content-Length': '0'} 2023-10-12T01:03:03.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c5812f7432a80c4-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.114 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.115 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.116 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.117 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.118 DEBUG:boto:Method: GET 2023-10-12T01:03:03.119 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.120 DEBUG:boto:Data: 2023-10-12T01:03:03.120 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.122 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.123 DEBUG:boto:Params: {} 2023-10-12T01:03:03.124 DEBUG:boto:Token: None 2023-10-12T01:03:03.125 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:03:03.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI= 2023-10-12T01:03:03.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l/KVa0NUjsZCiWWvIS/ma9AQVuI=', 'Content-Length': '0'} 2023-10-12T01:03:03.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f48a848311dd23d6-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.130 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.131 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.132 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.133 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.134 DEBUG:boto:Method: GET 2023-10-12T01:03:03.135 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.135 DEBUG:boto:Data: 2023-10-12T01:03:03.136 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.138 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.139 DEBUG:boto:Params: {} 2023-10-12T01:03:03.140 DEBUG:boto:Token: None 2023-10-12T01:03:03.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586ae0733fc15d2d-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.147 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.148 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.149 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.149 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.150 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.151 DEBUG:boto:Method: GET 2023-10-12T01:03:03.152 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.153 DEBUG:boto:Data: 2023-10-12T01:03:03.153 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.155 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.156 DEBUG:boto:Params: {} 2023-10-12T01:03:03.157 DEBUG:boto:Token: None 2023-10-12T01:03:03.158 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001593a03647a88b09-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.163 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.164 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.164 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.165 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.165 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:03:03.165 DEBUG:boto:Data: 2023-10-12T01:03:03.166 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.167 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.167 DEBUG:boto:Params: {} 2023-10-12T01:03:03.168 DEBUG:boto:Token: None 2023-10-12T01:03:03.168 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22 2023-10-12T01:03:03.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DRbxahnFem8hFaGI35iMOceFpCs= 2023-10-12T01:03:03.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DRbxahnFem8hFaGI35iMOceFpCs=', 'Content-Length': '0'} 2023-10-12T01:03:03.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db00ebecc701eaa3-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.171 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.172 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.172 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.173 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:03:03.173 DEBUG:boto:Data: 2023-10-12T01:03:03.173 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.174 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.175 DEBUG:boto:Params: {} 2023-10-12T01:03:03.175 DEBUG:boto:Token: None 2023-10-12T01:03:03.176 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22 2023-10-12T01:03:03.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DRbxahnFem8hFaGI35iMOceFpCs= 2023-10-12T01:03:03.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DRbxahnFem8hFaGI35iMOceFpCs=', 'Content-Length': '0'} 2023-10-12T01:03:03.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc7df6043150188-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:03:03.180 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.180 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.181 DEBUG:boto:Method: GET 2023-10-12T01:03:03.181 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:03:03.181 DEBUG:boto:Data: 2023-10-12T01:03:03.182 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.183 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.183 DEBUG:boto:Params: {} 2023-10-12T01:03:03.184 DEBUG:boto:Token: None 2023-10-12T01:03:03.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22 2023-10-12T01:03:03.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y6Cw8QCM1X7SP1rWN51rBNDWyfs= 2023-10-12T01:03:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y6Cw8QCM1X7SP1rWN51rBNDWyfs=', 'Content-Length': '0'} 2023-10-12T01:03:03.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf5758fea1cd5a3b-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.188 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:03:03.188 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:03:03.189 DEBUG:boto:Method: GET 2023-10-12T01:03:03.189 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:03:03.189 DEBUG:boto:Data: 2023-10-12T01:03:03.190 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.191 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.191 DEBUG:boto:Params: {} 2023-10-12T01:03:03.192 DEBUG:boto:Token: None 2023-10-12T01:03:03.192 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj22 2023-10-12T01:03:03.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y6Cw8QCM1X7SP1rWN51rBNDWyfs= 2023-10-12T01:03:03.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y6Cw8QCM1X7SP1rWN51rBNDWyfs=', 'Content-Length': '0'} 2023-10-12T01:03:03.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd87d341ea66dea-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.238 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.239 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.239 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.240 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.240 DEBUG:boto:Method: GET 2023-10-12T01:03:03.241 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.241 DEBUG:boto:Data: 2023-10-12T01:03:03.242 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.243 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.244 DEBUG:boto:Params: {} 2023-10-12T01:03:03.245 DEBUG:boto:Token: None 2023-10-12T01:03:03.245 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000948c7b26d3a5c45-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.251 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.252 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.252 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.252 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.253 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.253 DEBUG:boto:Method: GET 2023-10-12T01:03:03.254 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:03:03.254 DEBUG:boto:Data: 2023-10-12T01:03:03.255 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.256 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.256 DEBUG:boto:Params: {} 2023-10-12T01:03:03.258 DEBUG:boto:Token: None 2023-10-12T01:03:03.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6cee3571625f470-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.264 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:03:03.265 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.265 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.266 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.266 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.267 DEBUG:boto:Method: GET 2023-10-12T01:03:03.267 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.268 DEBUG:boto:Data: 2023-10-12T01:03:03.268 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.269 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.269 DEBUG:boto:Params: {} 2023-10-12T01:03:03.270 DEBUG:boto:Token: None 2023-10-12T01:03:03.270 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.271 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA= 2023-10-12T01:03:03.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA=', 'Content-Length': '0'} 2023-10-12T01:03:03.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374a3fe1fa27d956-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.314 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.315 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.316 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.316 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.316 DEBUG:boto:Method: GET 2023-10-12T01:03:03.317 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.317 DEBUG:boto:Data: 2023-10-12T01:03:03.318 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.319 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.319 DEBUG:boto:Params: {} 2023-10-12T01:03:03.320 DEBUG:boto:Token: None 2023-10-12T01:03:03.320 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA= 2023-10-12T01:03:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA=', 'Content-Length': '0'} 2023-10-12T01:03:03.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6ab6b5a65bc9b6-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.324 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.324 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.325 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.325 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.326 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.326 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.327 DEBUG:boto:Data: 2023-10-12T01:03:03.327 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.328 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.328 DEBUG:boto:Params: {} 2023-10-12T01:03:03.329 DEBUG:boto:Token: None 2023-10-12T01:03:03.329 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHZdY2tjfEp0dE4c4zJDbB4LH90= 2023-10-12T01:03:03.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHZdY2tjfEp0dE4c4zJDbB4LH90=', 'Content-Length': '0'} 2023-10-12T01:03:03.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fe737ebeaad6fc-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.333 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.333 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.334 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.334 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.334 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.335 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.335 DEBUG:boto:Data: 2023-10-12T01:03:03.336 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.337 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.337 DEBUG:boto:Params: {} 2023-10-12T01:03:03.338 DEBUG:boto:Token: None 2023-10-12T01:03:03.338 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JHZdY2tjfEp0dE4c4zJDbB4LH90= 2023-10-12T01:03:03.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JHZdY2tjfEp0dE4c4zJDbB4LH90=', 'Content-Length': '0'} 2023-10-12T01:03:03.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8a0b9c63dbe371-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:03:03.342 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.342 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.343 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.343 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.344 DEBUG:boto:Method: GET 2023-10-12T01:03:03.344 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.344 DEBUG:boto:Data: 2023-10-12T01:03:03.345 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.346 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.346 DEBUG:boto:Params: {} 2023-10-12T01:03:03.347 DEBUG:boto:Token: None 2023-10-12T01:03:03.347 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA= 2023-10-12T01:03:03.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA=', 'Content-Length': '0'} 2023-10-12T01:03:03.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d1a486cbcc019f-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.351 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.352 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.352 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.352 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.353 DEBUG:boto:Method: GET 2023-10-12T01:03:03.353 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.354 DEBUG:boto:Data: 2023-10-12T01:03:03.354 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.355 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.355 DEBUG:boto:Params: {} 2023-10-12T01:03:03.356 DEBUG:boto:Token: None 2023-10-12T01:03:03.356 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:03:03.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA= 2023-10-12T01:03:03.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+lELuf+Z8iG+/EVCZ7MIVpyyYA=', 'Content-Length': '0'} 2023-10-12T01:03:03.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266e09816aacaea6-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.361 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.361 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.361 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.362 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.362 DEBUG:boto:Method: GET 2023-10-12T01:03:03.363 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.363 DEBUG:boto:Data: 2023-10-12T01:03:03.364 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.364 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.365 DEBUG:boto:Params: {} 2023-10-12T01:03:03.365 DEBUG:boto:Token: None 2023-10-12T01:03:03.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60f339267ccae60-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.370 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.371 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.371 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.372 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.372 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.373 DEBUG:boto:Method: GET 2023-10-12T01:03:03.373 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.374 DEBUG:boto:Data: 2023-10-12T01:03:03.374 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.375 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.375 DEBUG:boto:Params: {} 2023-10-12T01:03:03.376 DEBUG:boto:Token: None 2023-10-12T01:03:03.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52bfe5141deb39e-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.381 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.382 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.382 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.382 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.383 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:03:03.383 DEBUG:boto:Data: 2023-10-12T01:03:03.384 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.384 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.385 DEBUG:boto:Params: {} 2023-10-12T01:03:03.385 DEBUG:boto:Token: None 2023-10-12T01:03:03.386 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33 2023-10-12T01:03:03.386 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iy6adBiE/5aTpcXQU1edzp37THM= 2023-10-12T01:03:03.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iy6adBiE/5aTpcXQU1edzp37THM=', 'Content-Length': '0'} 2023-10-12T01:03:03.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c338edfa103f1100-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.389 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.390 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.390 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.391 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:03:03.391 DEBUG:boto:Data: 2023-10-12T01:03:03.391 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.392 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.393 DEBUG:boto:Params: {} 2023-10-12T01:03:03.393 DEBUG:boto:Token: None 2023-10-12T01:03:03.394 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33 2023-10-12T01:03:03.394 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iy6adBiE/5aTpcXQU1edzp37THM= 2023-10-12T01:03:03.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iy6adBiE/5aTpcXQU1edzp37THM=', 'Content-Length': '0'} 2023-10-12T01:03:03.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f576bf14381e8758-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:03:03.398 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.398 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.399 DEBUG:boto:Method: GET 2023-10-12T01:03:03.399 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:03:03.400 DEBUG:boto:Data: 2023-10-12T01:03:03.400 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.401 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.401 DEBUG:boto:Params: {} 2023-10-12T01:03:03.402 DEBUG:boto:Token: None 2023-10-12T01:03:03.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33 2023-10-12T01:03:03.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gw8sK85xuJq45rcYvY1VUoZtAvA= 2023-10-12T01:03:03.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gw8sK85xuJq45rcYvY1VUoZtAvA=', 'Content-Length': '0'} 2023-10-12T01:03:03.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca46c43caee24a0-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.406 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:03:03.406 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:03:03.407 DEBUG:boto:Method: GET 2023-10-12T01:03:03.407 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:03:03.408 DEBUG:boto:Data: 2023-10-12T01:03:03.408 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.409 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.409 DEBUG:boto:Params: {} 2023-10-12T01:03:03.410 DEBUG:boto:Token: None 2023-10-12T01:03:03.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj33 2023-10-12T01:03:03.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gw8sK85xuJq45rcYvY1VUoZtAvA= 2023-10-12T01:03:03.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gw8sK85xuJq45rcYvY1VUoZtAvA=', 'Content-Length': '0'} 2023-10-12T01:03:03.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf596fe6143980c-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.454 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.455 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.455 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.456 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.456 DEBUG:boto:Method: GET 2023-10-12T01:03:03.457 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.457 DEBUG:boto:Data: 2023-10-12T01:03:03.458 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.459 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.459 DEBUG:boto:Params: {} 2023-10-12T01:03:03.459 DEBUG:boto:Token: None 2023-10-12T01:03:03.460 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b57f8646b8ba56f-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.464 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.465 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.466 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.466 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.466 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.467 DEBUG:boto:Method: GET 2023-10-12T01:03:03.467 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:03:03.468 DEBUG:boto:Data: 2023-10-12T01:03:03.468 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.469 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.469 DEBUG:boto:Params: {} 2023-10-12T01:03:03.470 DEBUG:boto:Token: None 2023-10-12T01:03:03.470 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629113543ad53fa0-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.475 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:03:03.476 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.476 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.477 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.477 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.478 DEBUG:boto:Method: GET 2023-10-12T01:03:03.478 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.479 DEBUG:boto:Data: 2023-10-12T01:03:03.479 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.480 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.480 DEBUG:boto:Params: {} 2023-10-12T01:03:03.481 DEBUG:boto:Token: None 2023-10-12T01:03:03.481 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI= 2023-10-12T01:03:03.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI=', 'Content-Length': '0'} 2023-10-12T01:03:03.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e670e91e6581c8-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.526 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.527 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.527 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.528 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.528 DEBUG:boto:Method: GET 2023-10-12T01:03:03.529 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.529 DEBUG:boto:Data: 2023-10-12T01:03:03.530 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.530 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.531 DEBUG:boto:Params: {} 2023-10-12T01:03:03.531 DEBUG:boto:Token: None 2023-10-12T01:03:03.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI= 2023-10-12T01:03:03.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI=', 'Content-Length': '0'} 2023-10-12T01:03:03.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ed52b6fbf04756-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.536 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.536 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.537 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.537 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.538 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.538 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.538 DEBUG:boto:Data: 2023-10-12T01:03:03.539 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.540 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.540 DEBUG:boto:Params: {} 2023-10-12T01:03:03.541 DEBUG:boto:Token: None 2023-10-12T01:03:03.541 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0+Ly6jGsq0DO4O+e6OQvhutvpTA= 2023-10-12T01:03:03.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0+Ly6jGsq0DO4O+e6OQvhutvpTA=', 'Content-Length': '0'} 2023-10-12T01:03:03.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078abfd7ee47107bb-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.545 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.545 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.545 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.546 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.546 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.547 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.547 DEBUG:boto:Data: 2023-10-12T01:03:03.547 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.548 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.549 DEBUG:boto:Params: {} 2023-10-12T01:03:03.549 DEBUG:boto:Token: None 2023-10-12T01:03:03.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0+Ly6jGsq0DO4O+e6OQvhutvpTA= 2023-10-12T01:03:03.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0+Ly6jGsq0DO4O+e6OQvhutvpTA=', 'Content-Length': '0'} 2023-10-12T01:03:03.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d529c6b0fec36d4d-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:03:03.553 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.554 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.554 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.555 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.555 DEBUG:boto:Method: GET 2023-10-12T01:03:03.556 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.556 DEBUG:boto:Data: 2023-10-12T01:03:03.556 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.557 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.558 DEBUG:boto:Params: {} 2023-10-12T01:03:03.558 DEBUG:boto:Token: None 2023-10-12T01:03:03.559 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.559 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI= 2023-10-12T01:03:03.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI=', 'Content-Length': '0'} 2023-10-12T01:03:03.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d7c68a66fa029a-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.563 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.563 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.563 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.564 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.564 DEBUG:boto:Method: GET 2023-10-12T01:03:03.565 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.565 DEBUG:boto:Data: 2023-10-12T01:03:03.566 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.566 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.567 DEBUG:boto:Params: {} 2023-10-12T01:03:03.567 DEBUG:boto:Token: None 2023-10-12T01:03:03.568 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:03:03.568 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI= 2023-10-12T01:03:03.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5e/e5GkKvejtCzCHIBaFJSdqPMI=', 'Content-Length': '0'} 2023-10-12T01:03:03.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86ea330cd7fb779-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.572 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.572 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.573 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.573 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.573 DEBUG:boto:Method: GET 2023-10-12T01:03:03.574 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.574 DEBUG:boto:Data: 2023-10-12T01:03:03.575 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.575 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.576 DEBUG:boto:Params: {} 2023-10-12T01:03:03.576 DEBUG:boto:Token: None 2023-10-12T01:03:03.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032849b1a75b41c46-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.581 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.582 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.582 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.583 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.583 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.583 DEBUG:boto:Method: GET 2023-10-12T01:03:03.584 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.584 DEBUG:boto:Data: 2023-10-12T01:03:03.585 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.586 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.586 DEBUG:boto:Params: {} 2023-10-12T01:03:03.586 DEBUG:boto:Token: None 2023-10-12T01:03:03.587 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b46e1f2c81e1de0-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.634 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.635 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.636 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.636 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.637 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:03:03.637 DEBUG:boto:Data: 2023-10-12T01:03:03.637 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.638 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.639 DEBUG:boto:Params: {} 2023-10-12T01:03:03.639 DEBUG:boto:Token: None 2023-10-12T01:03:03.640 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34 2023-10-12T01:03:03.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bp7GfYpTt9ZnPmZ0AO/EYKa56Po= 2023-10-12T01:03:03.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bp7GfYpTt9ZnPmZ0AO/EYKa56Po=', 'Content-Length': '0'} 2023-10-12T01:03:03.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007153f451384bcee6-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.643 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.643 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.644 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.644 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:03:03.645 DEBUG:boto:Data: 2023-10-12T01:03:03.645 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.646 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.646 DEBUG:boto:Params: {} 2023-10-12T01:03:03.647 DEBUG:boto:Token: None 2023-10-12T01:03:03.647 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34 2023-10-12T01:03:03.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bp7GfYpTt9ZnPmZ0AO/EYKa56Po= 2023-10-12T01:03:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bp7GfYpTt9ZnPmZ0AO/EYKa56Po=', 'Content-Length': '0'} 2023-10-12T01:03:03.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f9cf059d25de6de-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:03:03.651 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.652 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.652 DEBUG:boto:Method: GET 2023-10-12T01:03:03.652 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:03:03.653 DEBUG:boto:Data: 2023-10-12T01:03:03.653 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.654 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.655 DEBUG:boto:Params: {} 2023-10-12T01:03:03.655 DEBUG:boto:Token: None 2023-10-12T01:03:03.656 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34 2023-10-12T01:03:03.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Q4xSkZCHliRwzyufs3tRm4snvA= 2023-10-12T01:03:03.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Q4xSkZCHliRwzyufs3tRm4snvA=', 'Content-Length': '0'} 2023-10-12T01:03:03.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000917f79f24a42b862-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.659 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:03:03.659 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:03:03.660 DEBUG:boto:Method: GET 2023-10-12T01:03:03.660 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:03:03.661 DEBUG:boto:Data: 2023-10-12T01:03:03.661 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.662 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.663 DEBUG:boto:Params: {} 2023-10-12T01:03:03.663 DEBUG:boto:Token: None 2023-10-12T01:03:03.664 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj34 2023-10-12T01:03:03.664 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Q4xSkZCHliRwzyufs3tRm4snvA= 2023-10-12T01:03:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Q4xSkZCHliRwzyufs3tRm4snvA=', 'Content-Length': '0'} 2023-10-12T01:03:03.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad8631fed462afe4-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.667 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.667 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.668 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.668 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.669 DEBUG:boto:Method: GET 2023-10-12T01:03:03.669 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.670 DEBUG:boto:Data: 2023-10-12T01:03:03.670 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.671 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.671 DEBUG:boto:Params: {} 2023-10-12T01:03:03.672 DEBUG:boto:Token: None 2023-10-12T01:03:03.672 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.673 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3791d4c1b03c55f-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.722 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.723 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.724 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.724 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.725 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.725 DEBUG:boto:Method: GET 2023-10-12T01:03:03.726 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:03:03.726 DEBUG:boto:Data: 2023-10-12T01:03:03.726 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.727 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.728 DEBUG:boto:Params: {} 2023-10-12T01:03:03.728 DEBUG:boto:Token: None 2023-10-12T01:03:03.729 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ade8d5c26ac960b-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.734 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:03:03.735 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.735 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.736 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.736 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.737 DEBUG:boto:Method: GET 2023-10-12T01:03:03.737 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.737 DEBUG:boto:Data: 2023-10-12T01:03:03.738 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.740 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.740 DEBUG:boto:Params: {} 2023-10-12T01:03:03.741 DEBUG:boto:Token: None 2023-10-12T01:03:03.741 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ= 2023-10-12T01:03:03.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ=', 'Content-Length': '0'} 2023-10-12T01:03:03.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3266f89b4549b4-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.746 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.746 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.747 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.747 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.748 DEBUG:boto:Method: GET 2023-10-12T01:03:03.748 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.749 DEBUG:boto:Data: 2023-10-12T01:03:03.749 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.750 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.751 DEBUG:boto:Params: {} 2023-10-12T01:03:03.751 DEBUG:boto:Token: None 2023-10-12T01:03:03.751 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ= 2023-10-12T01:03:03.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ=', 'Content-Length': '0'} 2023-10-12T01:03:03.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d96bf8eb1d08b7-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.755 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.755 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.756 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.756 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.757 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.757 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.758 DEBUG:boto:Data: 2023-10-12T01:03:03.758 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.759 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.759 DEBUG:boto:Params: {} 2023-10-12T01:03:03.760 DEBUG:boto:Token: None 2023-10-12T01:03:03.760 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zw6i4empwmYTGCtbPYdyuJTOCzg= 2023-10-12T01:03:03.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zw6i4empwmYTGCtbPYdyuJTOCzg=', 'Content-Length': '0'} 2023-10-12T01:03:03.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f870669955aedc-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.763 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.764 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.764 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.765 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.765 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.765 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.766 DEBUG:boto:Data: 2023-10-12T01:03:03.766 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.767 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.768 DEBUG:boto:Params: {} 2023-10-12T01:03:03.768 DEBUG:boto:Token: None 2023-10-12T01:03:03.769 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zw6i4empwmYTGCtbPYdyuJTOCzg= 2023-10-12T01:03:03.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zw6i4empwmYTGCtbPYdyuJTOCzg=', 'Content-Length': '0'} 2023-10-12T01:03:03.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825eaf8c2335b3bc-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:03:03.773 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.773 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.773 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.774 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.774 DEBUG:boto:Method: GET 2023-10-12T01:03:03.775 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.775 DEBUG:boto:Data: 2023-10-12T01:03:03.776 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.777 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.777 DEBUG:boto:Params: {} 2023-10-12T01:03:03.778 DEBUG:boto:Token: None 2023-10-12T01:03:03.778 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ= 2023-10-12T01:03:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ=', 'Content-Length': '0'} 2023-10-12T01:03:03.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088aa15a70a410eff-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.782 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.782 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.783 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.783 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.783 DEBUG:boto:Method: GET 2023-10-12T01:03:03.784 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.784 DEBUG:boto:Data: 2023-10-12T01:03:03.785 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.786 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.786 DEBUG:boto:Params: {} 2023-10-12T01:03:03.787 DEBUG:boto:Token: None 2023-10-12T01:03:03.787 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:03:03.788 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ= 2023-10-12T01:03:03.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OnHsQYKGcyXGkG+Xq4iQHYFz6JQ=', 'Content-Length': '0'} 2023-10-12T01:03:03.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a55bb53dba46e88-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.834 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.835 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.835 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.836 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.836 DEBUG:boto:Method: GET 2023-10-12T01:03:03.837 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.837 DEBUG:boto:Data: 2023-10-12T01:03:03.837 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.838 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.839 DEBUG:boto:Params: {} 2023-10-12T01:03:03.839 DEBUG:boto:Token: None 2023-10-12T01:03:03.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074384248e230438e-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.844 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.845 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.845 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.846 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.846 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.847 DEBUG:boto:Method: GET 2023-10-12T01:03:03.847 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.848 DEBUG:boto:Data: 2023-10-12T01:03:03.848 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.849 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.850 DEBUG:boto:Params: {} 2023-10-12T01:03:03.850 DEBUG:boto:Token: None 2023-10-12T01:03:03.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000134d42be9d7d1c03-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.855 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.856 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.856 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.857 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.857 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:03:03.858 DEBUG:boto:Data: 2023-10-12T01:03:03.858 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.859 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.859 DEBUG:boto:Params: {} 2023-10-12T01:03:03.860 DEBUG:boto:Token: None 2023-10-12T01:03:03.860 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38 2023-10-12T01:03:03.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qctNWTFXxe70MO0hzVzNfNCpvLM= 2023-10-12T01:03:03.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qctNWTFXxe70MO0hzVzNfNCpvLM=', 'Content-Length': '0'} 2023-10-12T01:03:03.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d460a79ad9b82e23-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.864 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.864 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.865 DEBUG:boto:Method: HEAD 2023-10-12T01:03:03.865 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:03:03.866 DEBUG:boto:Data: 2023-10-12T01:03:03.866 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.867 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.868 DEBUG:boto:Params: {} 2023-10-12T01:03:03.868 DEBUG:boto:Token: None 2023-10-12T01:03:03.868 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38 2023-10-12T01:03:03.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qctNWTFXxe70MO0hzVzNfNCpvLM= 2023-10-12T01:03:03.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qctNWTFXxe70MO0hzVzNfNCpvLM=', 'Content-Length': '0'} 2023-10-12T01:03:03.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da6598d11e94d191-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:03:03.872 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.873 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.873 DEBUG:boto:Method: GET 2023-10-12T01:03:03.874 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:03:03.874 DEBUG:boto:Data: 2023-10-12T01:03:03.875 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.875 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.876 DEBUG:boto:Params: {} 2023-10-12T01:03:03.876 DEBUG:boto:Token: None 2023-10-12T01:03:03.877 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38 2023-10-12T01:03:03.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wmKy5VnMbU+o/e437Id5oDa1yQc= 2023-10-12T01:03:03.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wmKy5VnMbU+o/e437Id5oDa1yQc=', 'Content-Length': '0'} 2023-10-12T01:03:03.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b94cbd47d50f674-00652745c7-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.922 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:03:03.923 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:03:03.923 DEBUG:boto:Method: GET 2023-10-12T01:03:03.924 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:03:03.924 DEBUG:boto:Data: 2023-10-12T01:03:03.925 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.926 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.926 DEBUG:boto:Params: {} 2023-10-12T01:03:03.926 DEBUG:boto:Token: None 2023-10-12T01:03:03.927 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/obj38 2023-10-12T01:03:03.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wmKy5VnMbU+o/e437Id5oDa1yQc= 2023-10-12T01:03:03.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wmKy5VnMbU+o/e437Id5oDa1yQc=', 'Content-Length': '0'} 2023-10-12T01:03:03.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002df89f7b837cca3-00652745c7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.931 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.931 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.932 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.932 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.933 DEBUG:boto:Method: GET 2023-10-12T01:03:03.933 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.933 DEBUG:boto:Data: 2023-10-12T01:03:03.934 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:03.935 DEBUG:boto:Port: 8001 2023-10-12T01:03:03.935 DEBUG:boto:Params: {} 2023-10-12T01:03:03.936 DEBUG:boto:Token: None 2023-10-12T01:03:03.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba2565a4f201350-00652745c7-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.941 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.942 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:03.942 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:03.942 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.943 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.943 DEBUG:boto:Method: GET 2023-10-12T01:03:03.944 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:03:03.944 DEBUG:boto:Data: 2023-10-12T01:03:03.945 DEBUG:boto:Headers: {} 2023-10-12T01:03:03.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:03.946 DEBUG:boto:Port: 8000 2023-10-12T01:03:03.946 DEBUG:boto:Params: {} 2023-10-12T01:03:03.947 DEBUG:boto:Token: None 2023-10-12T01:03:03.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:03 GMT /wafmoa-59/ 2023-10-12T01:03:03.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU= 2023-10-12T01:03:03.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIaC99gKV1gG4jylbQh8QaSrHYU=', 'Content-Length': '0'} 2023-10-12T01:03:03.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070c5209378a8a10d-00652745c7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:03.994 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:03.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:03:03.997 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:03.997 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:03.998 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:03.998 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:03.999 DEBUG:boto:Method: GET 2023-10-12T01:03:03.999 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.001 DEBUG:boto:Data: 2023-10-12T01:03:04.001 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.002 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.003 DEBUG:boto:Params: {} 2023-10-12T01:03:04.003 DEBUG:boto:Token: None 2023-10-12T01:03:04.004 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ= 2023-10-12T01:03:04.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efdf6a00a9d6add3-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.008 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.008 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.009 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.010 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.010 DEBUG:boto:Method: GET 2023-10-12T01:03:04.010 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.011 DEBUG:boto:Data: 2023-10-12T01:03:04.012 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.013 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.013 DEBUG:boto:Params: {} 2023-10-12T01:03:04.014 DEBUG:boto:Token: None 2023-10-12T01:03:04.014 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ= 2023-10-12T01:03:04.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072c5558f27d4a64d-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.019 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.019 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.019 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.020 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.020 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.021 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.021 DEBUG:boto:Data: 2023-10-12T01:03:04.022 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.023 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.024 DEBUG:boto:Params: {} 2023-10-12T01:03:04.024 DEBUG:boto:Token: None 2023-10-12T01:03:04.025 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XUdDB6sPdAOc9BxoGjFTQSbd6VA= 2023-10-12T01:03:04.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XUdDB6sPdAOc9BxoGjFTQSbd6VA=', 'Content-Length': '0'} 2023-10-12T01:03:04.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502cc6171650607b-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.029 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.030 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.030 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.031 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.032 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.033 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.033 DEBUG:boto:Data: 2023-10-12T01:03:04.034 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.034 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.035 DEBUG:boto:Params: {} 2023-10-12T01:03:04.035 DEBUG:boto:Token: None 2023-10-12T01:03:04.036 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XUdDB6sPdAOc9BxoGjFTQSbd6VA= 2023-10-12T01:03:04.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XUdDB6sPdAOc9BxoGjFTQSbd6VA=', 'Content-Length': '0'} 2023-10-12T01:03:04.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000345acbe9989af92-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:03:04.040 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.040 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.041 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.041 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.043 DEBUG:boto:Method: GET 2023-10-12T01:03:04.043 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.044 DEBUG:boto:Data: 2023-10-12T01:03:04.044 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.045 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.045 DEBUG:boto:Params: {} 2023-10-12T01:03:04.046 DEBUG:boto:Token: None 2023-10-12T01:03:04.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ= 2023-10-12T01:03:04.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d954505ef7865ba-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.050 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.051 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.051 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.052 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.052 DEBUG:boto:Method: GET 2023-10-12T01:03:04.053 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.053 DEBUG:boto:Data: 2023-10-12T01:03:04.054 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.055 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.055 DEBUG:boto:Params: {} 2023-10-12T01:03:04.056 DEBUG:boto:Token: None 2023-10-12T01:03:04.056 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:03:04.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ= 2023-10-12T01:03:04.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ndalYnh+OH3ZR0Dnp4J+qdAxJAQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d22889a77fd036-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.060 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.061 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.062 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.062 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.063 DEBUG:boto:Method: GET 2023-10-12T01:03:04.063 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.064 DEBUG:boto:Data: 2023-10-12T01:03:04.064 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.065 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.065 DEBUG:boto:Params: {} 2023-10-12T01:03:04.066 DEBUG:boto:Token: None 2023-10-12T01:03:04.066 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1338bebf982a71c-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.071 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.072 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.072 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.073 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.073 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.074 DEBUG:boto:Method: GET 2023-10-12T01:03:04.074 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.075 DEBUG:boto:Data: 2023-10-12T01:03:04.075 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.076 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.077 DEBUG:boto:Params: {} 2023-10-12T01:03:04.077 DEBUG:boto:Token: None 2023-10-12T01:03:04.078 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d714e17ddb830f8-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.084 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.085 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.085 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.085 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.086 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:03:04.086 DEBUG:boto:Data: 2023-10-12T01:03:04.087 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.088 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.089 DEBUG:boto:Params: {} 2023-10-12T01:03:04.090 DEBUG:boto:Token: None 2023-10-12T01:03:04.091 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44 2023-10-12T01:03:04.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sJE9Bk1QoNVwY8DNXMkbUCHVtPk= 2023-10-12T01:03:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sJE9Bk1QoNVwY8DNXMkbUCHVtPk=', 'Content-Length': '0'} 2023-10-12T01:03:04.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f0799df3bf54b2-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.094 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.095 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.095 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.096 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:03:04.096 DEBUG:boto:Data: 2023-10-12T01:03:04.097 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.097 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.098 DEBUG:boto:Params: {} 2023-10-12T01:03:04.098 DEBUG:boto:Token: None 2023-10-12T01:03:04.099 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44 2023-10-12T01:03:04.099 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sJE9Bk1QoNVwY8DNXMkbUCHVtPk= 2023-10-12T01:03:04.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sJE9Bk1QoNVwY8DNXMkbUCHVtPk=', 'Content-Length': '0'} 2023-10-12T01:03:04.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a56393abc704ee-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:03:04.103 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.104 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.104 DEBUG:boto:Method: GET 2023-10-12T01:03:04.104 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:03:04.105 DEBUG:boto:Data: 2023-10-12T01:03:04.105 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.106 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.107 DEBUG:boto:Params: {} 2023-10-12T01:03:04.107 DEBUG:boto:Token: None 2023-10-12T01:03:04.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44 2023-10-12T01:03:04.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d0ikrNuTFbnc7EkEMXc02nLFOdQ= 2023-10-12T01:03:04.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d0ikrNuTFbnc7EkEMXc02nLFOdQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d920c3ec07bdd56c-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.154 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:03:04.155 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:03:04.155 DEBUG:boto:Method: GET 2023-10-12T01:03:04.156 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:03:04.156 DEBUG:boto:Data: 2023-10-12T01:03:04.157 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.158 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.158 DEBUG:boto:Params: {} 2023-10-12T01:03:04.159 DEBUG:boto:Token: None 2023-10-12T01:03:04.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj44 2023-10-12T01:03:04.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d0ikrNuTFbnc7EkEMXc02nLFOdQ= 2023-10-12T01:03:04.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d0ikrNuTFbnc7EkEMXc02nLFOdQ=', 'Content-Length': '0'} 2023-10-12T01:03:04.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e83017b9ba891c-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.163 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.163 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.164 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.164 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.165 DEBUG:boto:Method: GET 2023-10-12T01:03:04.165 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.166 DEBUG:boto:Data: 2023-10-12T01:03:04.166 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.167 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.167 DEBUG:boto:Params: {} 2023-10-12T01:03:04.168 DEBUG:boto:Token: None 2023-10-12T01:03:04.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ea20ac1866c614-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.173 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.174 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.174 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.175 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.175 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.175 DEBUG:boto:Method: GET 2023-10-12T01:03:04.176 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:03:04.176 DEBUG:boto:Data: 2023-10-12T01:03:04.177 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.178 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.178 DEBUG:boto:Params: {} 2023-10-12T01:03:04.179 DEBUG:boto:Token: None 2023-10-12T01:03:04.179 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4c716fcc267fa5-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.183 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:03:04.185 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.185 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.185 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.186 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.186 DEBUG:boto:Method: GET 2023-10-12T01:03:04.187 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.187 DEBUG:boto:Data: 2023-10-12T01:03:04.188 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.188 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.189 DEBUG:boto:Params: {} 2023-10-12T01:03:04.189 DEBUG:boto:Token: None 2023-10-12T01:03:04.190 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.190 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU= 2023-10-12T01:03:04.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU=', 'Content-Length': '0'} 2023-10-12T01:03:04.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3b2ea127f9835c-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.194 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.194 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.195 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.195 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.195 DEBUG:boto:Method: GET 2023-10-12T01:03:04.196 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.196 DEBUG:boto:Data: 2023-10-12T01:03:04.197 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.198 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.198 DEBUG:boto:Params: {} 2023-10-12T01:03:04.199 DEBUG:boto:Token: None 2023-10-12T01:03:04.199 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.199 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU= 2023-10-12T01:03:04.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU=', 'Content-Length': '0'} 2023-10-12T01:03:04.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000533a86f201fa33eb-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.203 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.203 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.204 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.204 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.204 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.205 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.205 DEBUG:boto:Data: 2023-10-12T01:03:04.206 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.206 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.207 DEBUG:boto:Params: {} 2023-10-12T01:03:04.208 DEBUG:boto:Token: None 2023-10-12T01:03:04.209 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rFfjMu+AI2AQr5sdk5T9e8lSx6I= 2023-10-12T01:03:04.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rFfjMu+AI2AQr5sdk5T9e8lSx6I=', 'Content-Length': '0'} 2023-10-12T01:03:04.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8d6d5cf3d9fdef-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.213 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.213 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.213 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.214 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.214 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.215 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.215 DEBUG:boto:Data: 2023-10-12T01:03:04.215 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.216 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.217 DEBUG:boto:Params: {} 2023-10-12T01:03:04.217 DEBUG:boto:Token: None 2023-10-12T01:03:04.218 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rFfjMu+AI2AQr5sdk5T9e8lSx6I= 2023-10-12T01:03:04.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rFfjMu+AI2AQr5sdk5T9e8lSx6I=', 'Content-Length': '0'} 2023-10-12T01:03:04.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe92a20e9f8764a-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:03:04.222 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.222 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.223 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.223 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.224 DEBUG:boto:Method: GET 2023-10-12T01:03:04.224 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.225 DEBUG:boto:Data: 2023-10-12T01:03:04.225 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.226 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.226 DEBUG:boto:Params: {} 2023-10-12T01:03:04.227 DEBUG:boto:Token: None 2023-10-12T01:03:04.227 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU= 2023-10-12T01:03:04.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU=', 'Content-Length': '0'} 2023-10-12T01:03:04.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122eb3f4a7967679-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.231 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.231 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.232 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.232 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.232 DEBUG:boto:Method: GET 2023-10-12T01:03:04.233 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.233 DEBUG:boto:Data: 2023-10-12T01:03:04.234 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.235 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.235 DEBUG:boto:Params: {} 2023-10-12T01:03:04.235 DEBUG:boto:Token: None 2023-10-12T01:03:04.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:03:04.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU= 2023-10-12T01:03:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sez3KoQ2lMigFO4jPZl+mx1l3VU=', 'Content-Length': '0'} 2023-10-12T01:03:04.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000782cca18a85ef9bd-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.240 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.240 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.240 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.241 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.241 DEBUG:boto:Method: GET 2023-10-12T01:03:04.242 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.242 DEBUG:boto:Data: 2023-10-12T01:03:04.243 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.244 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.244 DEBUG:boto:Params: {} 2023-10-12T01:03:04.245 DEBUG:boto:Token: None 2023-10-12T01:03:04.245 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c52efd850b53b231-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.249 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.250 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.251 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.251 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.252 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.252 DEBUG:boto:Method: GET 2023-10-12T01:03:04.253 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.253 DEBUG:boto:Data: 2023-10-12T01:03:04.254 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.255 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.255 DEBUG:boto:Params: {} 2023-10-12T01:03:04.256 DEBUG:boto:Token: None 2023-10-12T01:03:04.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73e74fc4ded7d94-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.302 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.303 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.304 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.304 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.305 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:03:04.305 DEBUG:boto:Data: 2023-10-12T01:03:04.306 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.307 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.307 DEBUG:boto:Params: {} 2023-10-12T01:03:04.308 DEBUG:boto:Token: None 2023-10-12T01:03:04.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51 2023-10-12T01:03:04.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PQuhXL5S/L83A96wlbIgiuxz+8Q= 2023-10-12T01:03:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PQuhXL5S/L83A96wlbIgiuxz+8Q=', 'Content-Length': '0'} 2023-10-12T01:03:04.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ba8d46fe39ec86-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.311 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.312 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.312 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.313 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:03:04.313 DEBUG:boto:Data: 2023-10-12T01:03:04.313 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.314 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.315 DEBUG:boto:Params: {} 2023-10-12T01:03:04.315 DEBUG:boto:Token: None 2023-10-12T01:03:04.316 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51 2023-10-12T01:03:04.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PQuhXL5S/L83A96wlbIgiuxz+8Q= 2023-10-12T01:03:04.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PQuhXL5S/L83A96wlbIgiuxz+8Q=', 'Content-Length': '0'} 2023-10-12T01:03:04.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000499aa063e6c6db00-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:03:04.320 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.320 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.321 DEBUG:boto:Method: GET 2023-10-12T01:03:04.321 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:03:04.322 DEBUG:boto:Data: 2023-10-12T01:03:04.322 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.323 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.324 DEBUG:boto:Params: {} 2023-10-12T01:03:04.324 DEBUG:boto:Token: None 2023-10-12T01:03:04.325 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51 2023-10-12T01:03:04.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jJjMpNSRFCIEj3paGn5FYwTe6FY= 2023-10-12T01:03:04.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jJjMpNSRFCIEj3paGn5FYwTe6FY=', 'Content-Length': '0'} 2023-10-12T01:03:04.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066593793bd3c3c5c-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.374 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:03:04.375 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:03:04.375 DEBUG:boto:Method: GET 2023-10-12T01:03:04.376 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:03:04.376 DEBUG:boto:Data: 2023-10-12T01:03:04.377 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.378 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.378 DEBUG:boto:Params: {} 2023-10-12T01:03:04.379 DEBUG:boto:Token: None 2023-10-12T01:03:04.379 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj51 2023-10-12T01:03:04.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jJjMpNSRFCIEj3paGn5FYwTe6FY= 2023-10-12T01:03:04.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jJjMpNSRFCIEj3paGn5FYwTe6FY=', 'Content-Length': '0'} 2023-10-12T01:03:04.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e28b532ac7ef155-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.383 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.383 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.384 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.384 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.385 DEBUG:boto:Method: GET 2023-10-12T01:03:04.385 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.386 DEBUG:boto:Data: 2023-10-12T01:03:04.386 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.387 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.387 DEBUG:boto:Params: {} 2023-10-12T01:03:04.388 DEBUG:boto:Token: None 2023-10-12T01:03:04.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfd3346bc719bb7-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.392 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.393 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.394 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.394 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.394 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.395 DEBUG:boto:Method: GET 2023-10-12T01:03:04.395 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:03:04.396 DEBUG:boto:Data: 2023-10-12T01:03:04.396 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.397 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.398 DEBUG:boto:Params: {} 2023-10-12T01:03:04.398 DEBUG:boto:Token: None 2023-10-12T01:03:04.399 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.399 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d72ff39fbd926ae-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.403 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:03:04.405 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.405 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.405 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.406 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.406 DEBUG:boto:Method: GET 2023-10-12T01:03:04.407 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.407 DEBUG:boto:Data: 2023-10-12T01:03:04.407 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.408 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.409 DEBUG:boto:Params: {} 2023-10-12T01:03:04.409 DEBUG:boto:Token: None 2023-10-12T01:03:04.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo= 2023-10-12T01:03:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo=', 'Content-Length': '0'} 2023-10-12T01:03:04.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd20cc833425dbd2-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.413 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.414 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.414 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.415 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.415 DEBUG:boto:Method: GET 2023-10-12T01:03:04.416 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.416 DEBUG:boto:Data: 2023-10-12T01:03:04.417 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.417 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.418 DEBUG:boto:Params: {} 2023-10-12T01:03:04.418 DEBUG:boto:Token: None 2023-10-12T01:03:04.419 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo= 2023-10-12T01:03:04.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo=', 'Content-Length': '0'} 2023-10-12T01:03:04.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9669a47189db8e2-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.462 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.463 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.463 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.464 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.464 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.464 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.465 DEBUG:boto:Data: 2023-10-12T01:03:04.465 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.466 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.467 DEBUG:boto:Params: {} 2023-10-12T01:03:04.467 DEBUG:boto:Token: None 2023-10-12T01:03:04.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BBv2mDkUqB3tCwMJULhEQyp3jEk= 2023-10-12T01:03:04.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BBv2mDkUqB3tCwMJULhEQyp3jEk=', 'Content-Length': '0'} 2023-10-12T01:03:04.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc342e8b9ae9e6b5-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.471 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.472 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.472 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.473 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.473 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.473 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.474 DEBUG:boto:Data: 2023-10-12T01:03:04.474 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.475 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.476 DEBUG:boto:Params: {} 2023-10-12T01:03:04.476 DEBUG:boto:Token: None 2023-10-12T01:03:04.477 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BBv2mDkUqB3tCwMJULhEQyp3jEk= 2023-10-12T01:03:04.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BBv2mDkUqB3tCwMJULhEQyp3jEk=', 'Content-Length': '0'} 2023-10-12T01:03:04.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4871458ae9a7f9-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:03:04.481 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.481 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.481 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.482 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.482 DEBUG:boto:Method: GET 2023-10-12T01:03:04.483 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.483 DEBUG:boto:Data: 2023-10-12T01:03:04.484 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.485 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.485 DEBUG:boto:Params: {} 2023-10-12T01:03:04.486 DEBUG:boto:Token: None 2023-10-12T01:03:04.486 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.487 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo= 2023-10-12T01:03:04.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo=', 'Content-Length': '0'} 2023-10-12T01:03:04.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5d0c859a4f9719-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.534 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.535 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.535 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.536 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.536 DEBUG:boto:Method: GET 2023-10-12T01:03:04.537 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.537 DEBUG:boto:Data: 2023-10-12T01:03:04.538 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.539 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.539 DEBUG:boto:Params: {} 2023-10-12T01:03:04.540 DEBUG:boto:Token: None 2023-10-12T01:03:04.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:03:04.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo= 2023-10-12T01:03:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4r1psshizBr7/vBXuv9Sqva7CYo=', 'Content-Length': '0'} 2023-10-12T01:03:04.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f7ec7377a996baf-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.544 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.545 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.545 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.545 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.546 DEBUG:boto:Method: GET 2023-10-12T01:03:04.546 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.547 DEBUG:boto:Data: 2023-10-12T01:03:04.547 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.548 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.548 DEBUG:boto:Params: {} 2023-10-12T01:03:04.549 DEBUG:boto:Token: None 2023-10-12T01:03:04.549 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af212a28984e489-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.554 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.555 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.555 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.555 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.556 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.556 DEBUG:boto:Method: GET 2023-10-12T01:03:04.557 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.557 DEBUG:boto:Data: 2023-10-12T01:03:04.558 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.559 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.559 DEBUG:boto:Params: {} 2023-10-12T01:03:04.560 DEBUG:boto:Token: None 2023-10-12T01:03:04.560 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1fcb9c0461df6e7-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.564 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.565 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.566 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.566 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.566 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:03:04.567 DEBUG:boto:Data: 2023-10-12T01:03:04.567 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.568 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.569 DEBUG:boto:Params: {} 2023-10-12T01:03:04.569 DEBUG:boto:Token: None 2023-10-12T01:03:04.569 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55 2023-10-12T01:03:04.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZVC0RlQ+upD0RH9LsoQX8WA89n8= 2023-10-12T01:03:04.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZVC0RlQ+upD0RH9LsoQX8WA89n8=', 'Content-Length': '0'} 2023-10-12T01:03:04.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039bbf41a5e3eddad-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.573 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.573 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.574 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.574 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:03:04.575 DEBUG:boto:Data: 2023-10-12T01:03:04.575 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.576 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.577 DEBUG:boto:Params: {} 2023-10-12T01:03:04.577 DEBUG:boto:Token: None 2023-10-12T01:03:04.577 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55 2023-10-12T01:03:04.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZVC0RlQ+upD0RH9LsoQX8WA89n8= 2023-10-12T01:03:04.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZVC0RlQ+upD0RH9LsoQX8WA89n8=', 'Content-Length': '0'} 2023-10-12T01:03:04.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be18ad6093dd9138-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:03:04.581 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.582 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.582 DEBUG:boto:Method: GET 2023-10-12T01:03:04.583 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:03:04.583 DEBUG:boto:Data: 2023-10-12T01:03:04.584 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.585 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.585 DEBUG:boto:Params: {} 2023-10-12T01:03:04.586 DEBUG:boto:Token: None 2023-10-12T01:03:04.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55 2023-10-12T01:03:04.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FiE72gdw5b5OOS+2JeSUDXGSDVI= 2023-10-12T01:03:04.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FiE72gdw5b5OOS+2JeSUDXGSDVI=', 'Content-Length': '0'} 2023-10-12T01:03:04.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15013b67b3f7d71-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.590 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:03:04.590 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:03:04.591 DEBUG:boto:Method: GET 2023-10-12T01:03:04.591 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:03:04.591 DEBUG:boto:Data: 2023-10-12T01:03:04.592 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.593 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.593 DEBUG:boto:Params: {} 2023-10-12T01:03:04.594 DEBUG:boto:Token: None 2023-10-12T01:03:04.594 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj55 2023-10-12T01:03:04.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FiE72gdw5b5OOS+2JeSUDXGSDVI= 2023-10-12T01:03:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FiE72gdw5b5OOS+2JeSUDXGSDVI=', 'Content-Length': '0'} 2023-10-12T01:03:04.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d0595e59f531a7-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.638 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.639 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.639 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.640 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.640 DEBUG:boto:Method: GET 2023-10-12T01:03:04.641 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.641 DEBUG:boto:Data: 2023-10-12T01:03:04.642 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.643 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.643 DEBUG:boto:Params: {} 2023-10-12T01:03:04.644 DEBUG:boto:Token: None 2023-10-12T01:03:04.644 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb295824c21136dd-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.649 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.649 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.650 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.650 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.651 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.652 DEBUG:boto:Method: GET 2023-10-12T01:03:04.652 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:03:04.653 DEBUG:boto:Data: 2023-10-12T01:03:04.653 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.654 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.655 DEBUG:boto:Params: {} 2023-10-12T01:03:04.655 DEBUG:boto:Token: None 2023-10-12T01:03:04.656 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c707b75cfb12f486-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.660 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:03:04.662 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.662 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.663 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.663 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.663 DEBUG:boto:Method: GET 2023-10-12T01:03:04.664 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.664 DEBUG:boto:Data: 2023-10-12T01:03:04.665 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.666 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.666 DEBUG:boto:Params: {} 2023-10-12T01:03:04.667 DEBUG:boto:Token: None 2023-10-12T01:03:04.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc= 2023-10-12T01:03:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc=', 'Content-Length': '0'} 2023-10-12T01:03:04.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000404b49dc9c9077c8-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.671 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.672 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.672 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.673 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.673 DEBUG:boto:Method: GET 2023-10-12T01:03:04.673 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.674 DEBUG:boto:Data: 2023-10-12T01:03:04.674 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.675 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.676 DEBUG:boto:Params: {} 2023-10-12T01:03:04.676 DEBUG:boto:Token: None 2023-10-12T01:03:04.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc= 2023-10-12T01:03:04.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc=', 'Content-Length': '0'} 2023-10-12T01:03:04.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000385dfe9be4762a35-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.680 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.681 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.681 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.682 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.682 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.683 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.683 DEBUG:boto:Data: 2023-10-12T01:03:04.684 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.684 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.685 DEBUG:boto:Params: {} 2023-10-12T01:03:04.685 DEBUG:boto:Token: None 2023-10-12T01:03:04.686 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HPbQqrl8Wct4LtcmRMJZKh3G8qE= 2023-10-12T01:03:04.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HPbQqrl8Wct4LtcmRMJZKh3G8qE=', 'Content-Length': '0'} 2023-10-12T01:03:04.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4aabf3df93bbe9c-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.689 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.689 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.690 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.690 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.691 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.691 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.691 DEBUG:boto:Data: 2023-10-12T01:03:04.692 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.693 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.693 DEBUG:boto:Params: {} 2023-10-12T01:03:04.694 DEBUG:boto:Token: None 2023-10-12T01:03:04.694 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HPbQqrl8Wct4LtcmRMJZKh3G8qE= 2023-10-12T01:03:04.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HPbQqrl8Wct4LtcmRMJZKh3G8qE=', 'Content-Length': '0'} 2023-10-12T01:03:04.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff2e1e705d13307-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:03:04.698 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.698 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.699 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.699 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.700 DEBUG:boto:Method: GET 2023-10-12T01:03:04.700 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.701 DEBUG:boto:Data: 2023-10-12T01:03:04.701 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.702 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.702 DEBUG:boto:Params: {} 2023-10-12T01:03:04.703 DEBUG:boto:Token: None 2023-10-12T01:03:04.703 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc= 2023-10-12T01:03:04.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc=', 'Content-Length': '0'} 2023-10-12T01:03:04.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9fcdc10e7b7c000-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.755 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.755 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.756 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.756 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.757 DEBUG:boto:Method: GET 2023-10-12T01:03:04.757 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.758 DEBUG:boto:Data: 2023-10-12T01:03:04.758 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.759 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.760 DEBUG:boto:Params: {} 2023-10-12T01:03:04.761 DEBUG:boto:Token: None 2023-10-12T01:03:04.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:03:04.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc= 2023-10-12T01:03:04.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:spAplZ4GmSq4/URNxyfmK9m0IUc=', 'Content-Length': '0'} 2023-10-12T01:03:04.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f87be364b9303b36-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.767 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.767 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.767 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.768 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.768 DEBUG:boto:Method: GET 2023-10-12T01:03:04.769 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.770 DEBUG:boto:Data: 2023-10-12T01:03:04.770 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.772 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.772 DEBUG:boto:Params: {} 2023-10-12T01:03:04.773 DEBUG:boto:Token: None 2023-10-12T01:03:04.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f806ce392fb88e-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.778 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.779 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.779 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.779 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.780 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.780 DEBUG:boto:Method: GET 2023-10-12T01:03:04.781 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.782 DEBUG:boto:Data: 2023-10-12T01:03:04.783 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.784 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.784 DEBUG:boto:Params: {} 2023-10-12T01:03:04.785 DEBUG:boto:Token: None 2023-10-12T01:03:04.785 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525a28c370ed2e14-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.790 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.791 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.791 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.792 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.792 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:03:04.792 DEBUG:boto:Data: 2023-10-12T01:03:04.793 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.794 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.795 DEBUG:boto:Params: {} 2023-10-12T01:03:04.795 DEBUG:boto:Token: None 2023-10-12T01:03:04.796 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57 2023-10-12T01:03:04.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:78W5RkuLYNkSZxJVI0b5JZmDZHM= 2023-10-12T01:03:04.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:78W5RkuLYNkSZxJVI0b5JZmDZHM=', 'Content-Length': '0'} 2023-10-12T01:03:04.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2cdcef6606b7d1-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.801 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.801 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.802 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.802 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:03:04.803 DEBUG:boto:Data: 2023-10-12T01:03:04.803 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.804 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.804 DEBUG:boto:Params: {} 2023-10-12T01:03:04.805 DEBUG:boto:Token: None 2023-10-12T01:03:04.805 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57 2023-10-12T01:03:04.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:78W5RkuLYNkSZxJVI0b5JZmDZHM= 2023-10-12T01:03:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:78W5RkuLYNkSZxJVI0b5JZmDZHM=', 'Content-Length': '0'} 2023-10-12T01:03:04.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da41209e6a87f9ad-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:03:04.810 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.811 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.811 DEBUG:boto:Method: GET 2023-10-12T01:03:04.812 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:03:04.812 DEBUG:boto:Data: 2023-10-12T01:03:04.813 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.813 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.814 DEBUG:boto:Params: {} 2023-10-12T01:03:04.816 DEBUG:boto:Token: None 2023-10-12T01:03:04.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57 2023-10-12T01:03:04.818 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VGEV2FeJ4lmENrJkF/h596paL3s= 2023-10-12T01:03:04.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VGEV2FeJ4lmENrJkF/h596paL3s=', 'Content-Length': '0'} 2023-10-12T01:03:04.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75115791b707ce9-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.822 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:03:04.823 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:03:04.824 DEBUG:boto:Method: GET 2023-10-12T01:03:04.825 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:03:04.825 DEBUG:boto:Data: 2023-10-12T01:03:04.826 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.828 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.828 DEBUG:boto:Params: {} 2023-10-12T01:03:04.829 DEBUG:boto:Token: None 2023-10-12T01:03:04.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj57 2023-10-12T01:03:04.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VGEV2FeJ4lmENrJkF/h596paL3s= 2023-10-12T01:03:04.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VGEV2FeJ4lmENrJkF/h596paL3s=', 'Content-Length': '0'} 2023-10-12T01:03:04.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd22d16ffd59900-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.834 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.835 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.835 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.835 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.836 DEBUG:boto:Method: GET 2023-10-12T01:03:04.836 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.837 DEBUG:boto:Data: 2023-10-12T01:03:04.837 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.838 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.838 DEBUG:boto:Params: {} 2023-10-12T01:03:04.839 DEBUG:boto:Token: None 2023-10-12T01:03:04.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681e547d9e0257d1-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.844 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.844 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.845 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.845 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.846 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.846 DEBUG:boto:Method: GET 2023-10-12T01:03:04.847 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:03:04.847 DEBUG:boto:Data: 2023-10-12T01:03:04.848 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.848 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.849 DEBUG:boto:Params: {} 2023-10-12T01:03:04.849 DEBUG:boto:Token: None 2023-10-12T01:03:04.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002825e7b40086c282-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.854 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:03:04.856 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.856 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.856 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.857 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.857 DEBUG:boto:Method: GET 2023-10-12T01:03:04.858 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.858 DEBUG:boto:Data: 2023-10-12T01:03:04.859 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.860 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.860 DEBUG:boto:Params: {} 2023-10-12T01:03:04.861 DEBUG:boto:Token: None 2023-10-12T01:03:04.861 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg= 2023-10-12T01:03:04.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg=', 'Content-Length': '0'} 2023-10-12T01:03:04.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee9aeef24119180-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.865 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.866 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.866 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.867 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.867 DEBUG:boto:Method: GET 2023-10-12T01:03:04.868 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.868 DEBUG:boto:Data: 2023-10-12T01:03:04.868 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.869 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.870 DEBUG:boto:Params: {} 2023-10-12T01:03:04.870 DEBUG:boto:Token: None 2023-10-12T01:03:04.871 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg= 2023-10-12T01:03:04.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg=', 'Content-Length': '0'} 2023-10-12T01:03:04.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd31f31fffdd1aa0-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.875 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.875 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.875 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.876 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.876 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.877 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.877 DEBUG:boto:Data: 2023-10-12T01:03:04.877 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.878 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.879 DEBUG:boto:Params: {} 2023-10-12T01:03:04.879 DEBUG:boto:Token: None 2023-10-12T01:03:04.880 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GjUARMf6VxZBQ+BqPZHurPbGHZo= 2023-10-12T01:03:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GjUARMf6VxZBQ+BqPZHurPbGHZo=', 'Content-Length': '0'} 2023-10-12T01:03:04.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000104aae69ecd6fb00-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.883 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.884 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.884 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.884 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.885 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.885 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.886 DEBUG:boto:Data: 2023-10-12T01:03:04.886 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.887 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.887 DEBUG:boto:Params: {} 2023-10-12T01:03:04.888 DEBUG:boto:Token: None 2023-10-12T01:03:04.888 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GjUARMf6VxZBQ+BqPZHurPbGHZo= 2023-10-12T01:03:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GjUARMf6VxZBQ+BqPZHurPbGHZo=', 'Content-Length': '0'} 2023-10-12T01:03:04.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e28bdfd5116f7e-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:03:04.892 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.893 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.893 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.894 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.894 DEBUG:boto:Method: GET 2023-10-12T01:03:04.895 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.895 DEBUG:boto:Data: 2023-10-12T01:03:04.895 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.896 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.897 DEBUG:boto:Params: {} 2023-10-12T01:03:04.897 DEBUG:boto:Token: None 2023-10-12T01:03:04.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg= 2023-10-12T01:03:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg=', 'Content-Length': '0'} 2023-10-12T01:03:04.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f294cb27c2fbfc86-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.946 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.947 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.947 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.948 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.948 DEBUG:boto:Method: GET 2023-10-12T01:03:04.949 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.949 DEBUG:boto:Data: 2023-10-12T01:03:04.950 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.951 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.951 DEBUG:boto:Params: {} 2023-10-12T01:03:04.952 DEBUG:boto:Token: None 2023-10-12T01:03:04.952 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:03:04.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg= 2023-10-12T01:03:04.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5FH1WH4/kbbyBrhkM/3exfgMZg=', 'Content-Length': '0'} 2023-10-12T01:03:04.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049651f5c7648b81e-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.956 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.956 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.957 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.957 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.958 DEBUG:boto:Method: GET 2023-10-12T01:03:04.958 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.959 DEBUG:boto:Data: 2023-10-12T01:03:04.959 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.960 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.960 DEBUG:boto:Params: {} 2023-10-12T01:03:04.961 DEBUG:boto:Token: None 2023-10-12T01:03:04.961 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087570d3faaf3a938-00652745c8-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.966 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.966 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:04.967 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:04.967 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.968 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.968 DEBUG:boto:Method: GET 2023-10-12T01:03:04.969 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:03:04.969 DEBUG:boto:Data: 2023-10-12T01:03:04.969 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.970 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.971 DEBUG:boto:Params: {} 2023-10-12T01:03:04.971 DEBUG:boto:Token: None 2023-10-12T01:03:04.972 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/ 2023-10-12T01:03:04.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg= 2023-10-12T01:03:04.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Ym67EvqjwSKqTZgaaTvigEzrWg=', 'Content-Length': '0'} 2023-10-12T01:03:04.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081633be22e564596-00652745c8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.977 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:04.977 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.978 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.978 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.979 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:03:04.979 DEBUG:boto:Data: 2023-10-12T01:03:04.980 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.980 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.981 DEBUG:boto:Params: {} 2023-10-12T01:03:04.981 DEBUG:boto:Token: None 2023-10-12T01:03:04.982 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68 2023-10-12T01:03:04.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nX9ImZRaQ5vO91S/H/7z68qMuhE= 2023-10-12T01:03:04.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nX9ImZRaQ5vO91S/H/7z68qMuhE=', 'Content-Length': '0'} 2023-10-12T01:03:04.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adf58908ee1e87b-00652745c8-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.985 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.986 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.986 DEBUG:boto:Method: HEAD 2023-10-12T01:03:04.986 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:03:04.987 DEBUG:boto:Data: 2023-10-12T01:03:04.987 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:04.988 DEBUG:boto:Port: 8000 2023-10-12T01:03:04.989 DEBUG:boto:Params: {} 2023-10-12T01:03:04.989 DEBUG:boto:Token: None 2023-10-12T01:03:04.990 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68 2023-10-12T01:03:04.990 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nX9ImZRaQ5vO91S/H/7z68qMuhE= 2023-10-12T01:03:04.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nX9ImZRaQ5vO91S/H/7z68qMuhE=', 'Content-Length': '0'} 2023-10-12T01:03:04.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009feecbd1ed55d04b-00652745c8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:04.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:03:04.994 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:04.994 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:04.995 DEBUG:boto:Method: GET 2023-10-12T01:03:04.995 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:03:04.996 DEBUG:boto:Data: 2023-10-12T01:03:04.996 DEBUG:boto:Headers: {} 2023-10-12T01:03:04.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:04.997 DEBUG:boto:Port: 8001 2023-10-12T01:03:04.997 DEBUG:boto:Params: {} 2023-10-12T01:03:04.998 DEBUG:boto:Token: None 2023-10-12T01:03:04.998 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:04 GMT /wafmoa-59/obj68 2023-10-12T01:03:04.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c32L/Vp30NqsL6cLjsM/54D6WbI= 2023-10-12T01:03:04.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c32L/Vp30NqsL6cLjsM/54D6WbI=', 'Content-Length': '0'} 2023-10-12T01:03:05.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a88f136df10116-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.002 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:03:05.002 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:03:05.003 DEBUG:boto:Method: GET 2023-10-12T01:03:05.003 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:03:05.004 DEBUG:boto:Data: 2023-10-12T01:03:05.004 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.005 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.005 DEBUG:boto:Params: {} 2023-10-12T01:03:05.006 DEBUG:boto:Token: None 2023-10-12T01:03:05.006 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj68 2023-10-12T01:03:05.007 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n3Ii2JtrwpNX/CNtDynNWwy44yw= 2023-10-12T01:03:05.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n3Ii2JtrwpNX/CNtDynNWwy44yw=', 'Content-Length': '0'} 2023-10-12T01:03:05.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe5545ff279e2e4-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.054 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.055 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.056 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.056 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.056 DEBUG:boto:Method: GET 2023-10-12T01:03:05.057 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.057 DEBUG:boto:Data: 2023-10-12T01:03:05.058 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.059 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.059 DEBUG:boto:Params: {} 2023-10-12T01:03:05.059 DEBUG:boto:Token: None 2023-10-12T01:03:05.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f239563b5bb72a4b-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.064 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.065 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.066 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.066 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.066 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.067 DEBUG:boto:Method: GET 2023-10-12T01:03:05.067 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:03:05.068 DEBUG:boto:Data: 2023-10-12T01:03:05.068 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.069 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.070 DEBUG:boto:Params: {} 2023-10-12T01:03:05.070 DEBUG:boto:Token: None 2023-10-12T01:03:05.071 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0202935be64fb66-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.076 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:03:05.077 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.077 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.078 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.078 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.078 DEBUG:boto:Method: GET 2023-10-12T01:03:05.079 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.079 DEBUG:boto:Data: 2023-10-12T01:03:05.080 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.081 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.081 DEBUG:boto:Params: {} 2023-10-12T01:03:05.082 DEBUG:boto:Token: None 2023-10-12T01:03:05.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I= 2023-10-12T01:03:05.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I=', 'Content-Length': '0'} 2023-10-12T01:03:05.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11d533b341402ea-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.087 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.087 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.087 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.088 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.088 DEBUG:boto:Method: GET 2023-10-12T01:03:05.089 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.089 DEBUG:boto:Data: 2023-10-12T01:03:05.090 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.091 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.091 DEBUG:boto:Params: {} 2023-10-12T01:03:05.091 DEBUG:boto:Token: None 2023-10-12T01:03:05.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I= 2023-10-12T01:03:05.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I=', 'Content-Length': '0'} 2023-10-12T01:03:05.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006625d13ce1022201-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.096 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.096 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.097 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.097 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.097 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.098 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.098 DEBUG:boto:Data: 2023-10-12T01:03:05.099 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.100 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.100 DEBUG:boto:Params: {} 2023-10-12T01:03:05.101 DEBUG:boto:Token: None 2023-10-12T01:03:05.101 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W5P/uzuJleiSGYqSXfF/qtv9Wp8= 2023-10-12T01:03:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W5P/uzuJleiSGYqSXfF/qtv9Wp8=', 'Content-Length': '0'} 2023-10-12T01:03:05.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b645e185f29d8ae3-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.104 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.105 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.105 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.106 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.106 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.107 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.107 DEBUG:boto:Data: 2023-10-12T01:03:05.108 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.108 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.109 DEBUG:boto:Params: {} 2023-10-12T01:03:05.109 DEBUG:boto:Token: None 2023-10-12T01:03:05.110 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W5P/uzuJleiSGYqSXfF/qtv9Wp8= 2023-10-12T01:03:05.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W5P/uzuJleiSGYqSXfF/qtv9Wp8=', 'Content-Length': '0'} 2023-10-12T01:03:05.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24630a6d332fddc-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:03:05.114 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.114 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.115 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.115 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.116 DEBUG:boto:Method: GET 2023-10-12T01:03:05.116 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.117 DEBUG:boto:Data: 2023-10-12T01:03:05.117 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.118 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.118 DEBUG:boto:Params: {} 2023-10-12T01:03:05.119 DEBUG:boto:Token: None 2023-10-12T01:03:05.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I= 2023-10-12T01:03:05.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I=', 'Content-Length': '0'} 2023-10-12T01:03:05.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007557007cf067dbbc-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.123 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.124 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.124 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.125 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.125 DEBUG:boto:Method: GET 2023-10-12T01:03:05.126 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.126 DEBUG:boto:Data: 2023-10-12T01:03:05.126 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.127 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.128 DEBUG:boto:Params: {} 2023-10-12T01:03:05.128 DEBUG:boto:Token: None 2023-10-12T01:03:05.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:03:05.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I= 2023-10-12T01:03:05.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+xr7iq7A9dcI3w53lshM/55l6I=', 'Content-Length': '0'} 2023-10-12T01:03:05.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4c6ced5dd77311-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.178 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.179 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.179 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.180 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.180 DEBUG:boto:Method: GET 2023-10-12T01:03:05.181 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.181 DEBUG:boto:Data: 2023-10-12T01:03:05.182 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.182 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.183 DEBUG:boto:Params: {} 2023-10-12T01:03:05.183 DEBUG:boto:Token: None 2023-10-12T01:03:05.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc6cc4075fecb95-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.189 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.190 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.190 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.191 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.191 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.192 DEBUG:boto:Method: GET 2023-10-12T01:03:05.192 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.192 DEBUG:boto:Data: 2023-10-12T01:03:05.193 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.194 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.194 DEBUG:boto:Params: {} 2023-10-12T01:03:05.195 DEBUG:boto:Token: None 2023-10-12T01:03:05.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.198 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a261829ad60e706-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.203 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.204 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.205 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.205 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.205 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:03:05.206 DEBUG:boto:Data: 2023-10-12T01:03:05.206 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.207 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.208 DEBUG:boto:Params: {} 2023-10-12T01:03:05.208 DEBUG:boto:Token: None 2023-10-12T01:03:05.209 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76 2023-10-12T01:03:05.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FQ7HGaVEWkXWXeQ+p27uPsJmWXE= 2023-10-12T01:03:05.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FQ7HGaVEWkXWXeQ+p27uPsJmWXE=', 'Content-Length': '0'} 2023-10-12T01:03:05.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575fec52d7aa2936-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.212 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.213 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.213 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.214 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:03:05.214 DEBUG:boto:Data: 2023-10-12T01:03:05.215 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.216 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.216 DEBUG:boto:Params: {} 2023-10-12T01:03:05.216 DEBUG:boto:Token: None 2023-10-12T01:03:05.217 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76 2023-10-12T01:03:05.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FQ7HGaVEWkXWXeQ+p27uPsJmWXE= 2023-10-12T01:03:05.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FQ7HGaVEWkXWXeQ+p27uPsJmWXE=', 'Content-Length': '0'} 2023-10-12T01:03:05.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335dd5f75d9e3807-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:03:05.221 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.222 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.222 DEBUG:boto:Method: GET 2023-10-12T01:03:05.223 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:03:05.223 DEBUG:boto:Data: 2023-10-12T01:03:05.224 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.224 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.225 DEBUG:boto:Params: {} 2023-10-12T01:03:05.225 DEBUG:boto:Token: None 2023-10-12T01:03:05.226 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76 2023-10-12T01:03:05.227 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0f+FhuTzrBWuuYCYdkSr1omr4C4= 2023-10-12T01:03:05.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0f+FhuTzrBWuuYCYdkSr1omr4C4=', 'Content-Length': '0'} 2023-10-12T01:03:05.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f0111050205e27d-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.230 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:03:05.230 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:03:05.231 DEBUG:boto:Method: GET 2023-10-12T01:03:05.231 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:03:05.232 DEBUG:boto:Data: 2023-10-12T01:03:05.232 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.233 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.234 DEBUG:boto:Params: {} 2023-10-12T01:03:05.234 DEBUG:boto:Token: None 2023-10-12T01:03:05.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj76 2023-10-12T01:03:05.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0f+FhuTzrBWuuYCYdkSr1omr4C4= 2023-10-12T01:03:05.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0f+FhuTzrBWuuYCYdkSr1omr4C4=', 'Content-Length': '0'} 2023-10-12T01:03:05.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b3688cf65e1da7-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.239 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.239 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.239 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.240 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.240 DEBUG:boto:Method: GET 2023-10-12T01:03:05.241 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.241 DEBUG:boto:Data: 2023-10-12T01:03:05.242 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.243 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.243 DEBUG:boto:Params: {} 2023-10-12T01:03:05.244 DEBUG:boto:Token: None 2023-10-12T01:03:05.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e942bbc89365a2db-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.249 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.250 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.250 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.251 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.251 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.252 DEBUG:boto:Method: GET 2023-10-12T01:03:05.252 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:03:05.252 DEBUG:boto:Data: 2023-10-12T01:03:05.253 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.254 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.254 DEBUG:boto:Params: {} 2023-10-12T01:03:05.255 DEBUG:boto:Token: None 2023-10-12T01:03:05.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb49922c6914675f-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.260 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:03:05.261 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.261 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.262 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.262 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.263 DEBUG:boto:Method: GET 2023-10-12T01:03:05.263 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.264 DEBUG:boto:Data: 2023-10-12T01:03:05.264 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.265 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.266 DEBUG:boto:Params: {} 2023-10-12T01:03:05.266 DEBUG:boto:Token: None 2023-10-12T01:03:05.267 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.267 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s= 2023-10-12T01:03:05.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s=', 'Content-Length': '0'} 2023-10-12T01:03:05.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009036a5a3a690eb3a-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.271 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.271 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.272 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.272 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.273 DEBUG:boto:Method: GET 2023-10-12T01:03:05.273 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.274 DEBUG:boto:Data: 2023-10-12T01:03:05.274 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.275 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.276 DEBUG:boto:Params: {} 2023-10-12T01:03:05.276 DEBUG:boto:Token: None 2023-10-12T01:03:05.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s= 2023-10-12T01:03:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s=', 'Content-Length': '0'} 2023-10-12T01:03:05.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9738523388978c5-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.322 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.323 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.323 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.324 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.324 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.325 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.325 DEBUG:boto:Data: 2023-10-12T01:03:05.326 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.326 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.327 DEBUG:boto:Params: {} 2023-10-12T01:03:05.327 DEBUG:boto:Token: None 2023-10-12T01:03:05.328 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DUjBuK6D0627CV3pRV3usV/Jbbs= 2023-10-12T01:03:05.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DUjBuK6D0627CV3pRV3usV/Jbbs=', 'Content-Length': '0'} 2023-10-12T01:03:05.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2403148dfc95e21-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.332 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.332 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.333 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.333 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.334 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.334 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.334 DEBUG:boto:Data: 2023-10-12T01:03:05.335 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.336 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.336 DEBUG:boto:Params: {} 2023-10-12T01:03:05.337 DEBUG:boto:Token: None 2023-10-12T01:03:05.337 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DUjBuK6D0627CV3pRV3usV/Jbbs= 2023-10-12T01:03:05.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DUjBuK6D0627CV3pRV3usV/Jbbs=', 'Content-Length': '0'} 2023-10-12T01:03:05.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9517d28e4c4826-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:03:05.341 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.342 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.342 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.343 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.343 DEBUG:boto:Method: GET 2023-10-12T01:03:05.344 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.344 DEBUG:boto:Data: 2023-10-12T01:03:05.344 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.345 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.346 DEBUG:boto:Params: {} 2023-10-12T01:03:05.346 DEBUG:boto:Token: None 2023-10-12T01:03:05.347 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.347 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s= 2023-10-12T01:03:05.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s=', 'Content-Length': '0'} 2023-10-12T01:03:05.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091a9470bba4c501-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.394 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.395 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.395 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.396 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.396 DEBUG:boto:Method: GET 2023-10-12T01:03:05.397 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.397 DEBUG:boto:Data: 2023-10-12T01:03:05.398 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.399 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.399 DEBUG:boto:Params: {} 2023-10-12T01:03:05.399 DEBUG:boto:Token: None 2023-10-12T01:03:05.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:03:05.400 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s= 2023-10-12T01:03:05.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:do8vO2bPLA3jifrMcW1aj2KyI5s=', 'Content-Length': '0'} 2023-10-12T01:03:05.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f62a3fc8c9682c-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.404 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.405 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.405 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.406 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.406 DEBUG:boto:Method: GET 2023-10-12T01:03:05.406 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.407 DEBUG:boto:Data: 2023-10-12T01:03:05.407 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.408 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.409 DEBUG:boto:Params: {} 2023-10-12T01:03:05.409 DEBUG:boto:Token: None 2023-10-12T01:03:05.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000410e52922265dd87-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.414 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.415 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.415 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.416 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.416 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.417 DEBUG:boto:Method: GET 2023-10-12T01:03:05.417 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.417 DEBUG:boto:Data: 2023-10-12T01:03:05.418 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.419 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.419 DEBUG:boto:Params: {} 2023-10-12T01:03:05.420 DEBUG:boto:Token: None 2023-10-12T01:03:05.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ee0333c470de19-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.425 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.426 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.427 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.427 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.427 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:03:05.428 DEBUG:boto:Data: 2023-10-12T01:03:05.428 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.429 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.429 DEBUG:boto:Params: {} 2023-10-12T01:03:05.430 DEBUG:boto:Token: None 2023-10-12T01:03:05.430 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85 2023-10-12T01:03:05.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uFvaxmYaXt/DHgTxHHbcrD+Ngcg= 2023-10-12T01:03:05.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uFvaxmYaXt/DHgTxHHbcrD+Ngcg=', 'Content-Length': '0'} 2023-10-12T01:03:05.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187dbe99ece24149-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.434 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.435 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.435 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.435 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:03:05.436 DEBUG:boto:Data: 2023-10-12T01:03:05.436 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.437 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.437 DEBUG:boto:Params: {} 2023-10-12T01:03:05.438 DEBUG:boto:Token: None 2023-10-12T01:03:05.438 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85 2023-10-12T01:03:05.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uFvaxmYaXt/DHgTxHHbcrD+Ngcg= 2023-10-12T01:03:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uFvaxmYaXt/DHgTxHHbcrD+Ngcg=', 'Content-Length': '0'} 2023-10-12T01:03:05.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000007a07ef1555da39-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:03:05.442 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.443 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.443 DEBUG:boto:Method: GET 2023-10-12T01:03:05.443 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:03:05.444 DEBUG:boto:Data: 2023-10-12T01:03:05.444 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.445 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.446 DEBUG:boto:Params: {} 2023-10-12T01:03:05.446 DEBUG:boto:Token: None 2023-10-12T01:03:05.446 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85 2023-10-12T01:03:05.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E0E3551qpPtKr447S6BsJkHMVAw= 2023-10-12T01:03:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E0E3551qpPtKr447S6BsJkHMVAw=', 'Content-Length': '0'} 2023-10-12T01:03:05.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036661ffd7be4b2df-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.450 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:03:05.451 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:03:05.451 DEBUG:boto:Method: GET 2023-10-12T01:03:05.452 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:03:05.452 DEBUG:boto:Data: 2023-10-12T01:03:05.452 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.453 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.454 DEBUG:boto:Params: {} 2023-10-12T01:03:05.454 DEBUG:boto:Token: None 2023-10-12T01:03:05.455 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj85 2023-10-12T01:03:05.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E0E3551qpPtKr447S6BsJkHMVAw= 2023-10-12T01:03:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E0E3551qpPtKr447S6BsJkHMVAw=', 'Content-Length': '0'} 2023-10-12T01:03:05.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008042e2dc4d6e76b2-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.498 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.499 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.499 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.500 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.500 DEBUG:boto:Method: GET 2023-10-12T01:03:05.501 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.501 DEBUG:boto:Data: 2023-10-12T01:03:05.502 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.502 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.503 DEBUG:boto:Params: {} 2023-10-12T01:03:05.503 DEBUG:boto:Token: None 2023-10-12T01:03:05.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f685c309b593209-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.508 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.509 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.509 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.510 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.510 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.511 DEBUG:boto:Method: GET 2023-10-12T01:03:05.511 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:03:05.512 DEBUG:boto:Data: 2023-10-12T01:03:05.512 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.513 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.513 DEBUG:boto:Params: {} 2023-10-12T01:03:05.514 DEBUG:boto:Token: None 2023-10-12T01:03:05.514 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038597e761595ad34-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.519 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:03:05.520 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.520 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.521 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.521 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.522 DEBUG:boto:Method: GET 2023-10-12T01:03:05.522 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.523 DEBUG:boto:Data: 2023-10-12T01:03:05.523 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.524 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.524 DEBUG:boto:Params: {} 2023-10-12T01:03:05.525 DEBUG:boto:Token: None 2023-10-12T01:03:05.525 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw= 2023-10-12T01:03:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw=', 'Content-Length': '0'} 2023-10-12T01:03:05.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1c937efdc004a4-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.529 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.530 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.530 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.531 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.531 DEBUG:boto:Method: GET 2023-10-12T01:03:05.532 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.532 DEBUG:boto:Data: 2023-10-12T01:03:05.533 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.533 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.534 DEBUG:boto:Params: {} 2023-10-12T01:03:05.534 DEBUG:boto:Token: None 2023-10-12T01:03:05.535 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw= 2023-10-12T01:03:05.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw=', 'Content-Length': '0'} 2023-10-12T01:03:05.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004839e639818d49cd-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.539 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.539 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.540 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.540 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.540 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.541 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.541 DEBUG:boto:Data: 2023-10-12T01:03:05.542 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.543 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.543 DEBUG:boto:Params: {} 2023-10-12T01:03:05.543 DEBUG:boto:Token: None 2023-10-12T01:03:05.544 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NdWv/y6S/01MpQjP1JvEqp+vXR8= 2023-10-12T01:03:05.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NdWv/y6S/01MpQjP1JvEqp+vXR8=', 'Content-Length': '0'} 2023-10-12T01:03:05.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc890bcba0a99b8c-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.547 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.548 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.548 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.549 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.549 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.549 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.550 DEBUG:boto:Data: 2023-10-12T01:03:05.550 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.551 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.552 DEBUG:boto:Params: {} 2023-10-12T01:03:05.552 DEBUG:boto:Token: None 2023-10-12T01:03:05.552 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NdWv/y6S/01MpQjP1JvEqp+vXR8= 2023-10-12T01:03:05.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NdWv/y6S/01MpQjP1JvEqp+vXR8=', 'Content-Length': '0'} 2023-10-12T01:03:05.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052aa4121848c5392-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:03:05.557 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.557 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.557 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.558 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.558 DEBUG:boto:Method: GET 2023-10-12T01:03:05.559 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.561 DEBUG:boto:Data: 2023-10-12T01:03:05.562 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.563 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.563 DEBUG:boto:Params: {} 2023-10-12T01:03:05.564 DEBUG:boto:Token: None 2023-10-12T01:03:05.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw= 2023-10-12T01:03:05.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw=', 'Content-Length': '0'} 2023-10-12T01:03:05.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2319e20c678542-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.614 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.615 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.615 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.616 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.616 DEBUG:boto:Method: GET 2023-10-12T01:03:05.617 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.617 DEBUG:boto:Data: 2023-10-12T01:03:05.618 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.618 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.619 DEBUG:boto:Params: {} 2023-10-12T01:03:05.619 DEBUG:boto:Token: None 2023-10-12T01:03:05.620 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:03:05.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw= 2023-10-12T01:03:05.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:51+FZO5K+VcuTeTUc6wXUEPKvOw=', 'Content-Length': '0'} 2023-10-12T01:03:05.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055084466cf65d897-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.624 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.624 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.625 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.625 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.626 DEBUG:boto:Method: GET 2023-10-12T01:03:05.626 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.626 DEBUG:boto:Data: 2023-10-12T01:03:05.627 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.628 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.628 DEBUG:boto:Params: {} 2023-10-12T01:03:05.628 DEBUG:boto:Token: None 2023-10-12T01:03:05.629 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.629 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b47acf2303883a4-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.633 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.634 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.634 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.635 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.635 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.635 DEBUG:boto:Method: GET 2023-10-12T01:03:05.636 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.636 DEBUG:boto:Data: 2023-10-12T01:03:05.637 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.637 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.638 DEBUG:boto:Params: {} 2023-10-12T01:03:05.638 DEBUG:boto:Token: None 2023-10-12T01:03:05.639 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cdbb4a804f5f8e0-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.644 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.644 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.645 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.645 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.646 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:03:05.646 DEBUG:boto:Data: 2023-10-12T01:03:05.646 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.647 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.648 DEBUG:boto:Params: {} 2023-10-12T01:03:05.648 DEBUG:boto:Token: None 2023-10-12T01:03:05.649 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95 2023-10-12T01:03:05.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yiMfWqOkTp8Ay27cHs9tx8yYZlI= 2023-10-12T01:03:05.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yiMfWqOkTp8Ay27cHs9tx8yYZlI=', 'Content-Length': '0'} 2023-10-12T01:03:05.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93b116944fed02f-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.652 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.653 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.653 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.654 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:03:05.654 DEBUG:boto:Data: 2023-10-12T01:03:05.654 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.655 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.656 DEBUG:boto:Params: {} 2023-10-12T01:03:05.656 DEBUG:boto:Token: None 2023-10-12T01:03:05.657 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95 2023-10-12T01:03:05.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yiMfWqOkTp8Ay27cHs9tx8yYZlI= 2023-10-12T01:03:05.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yiMfWqOkTp8Ay27cHs9tx8yYZlI=', 'Content-Length': '0'} 2023-10-12T01:03:05.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539dd3bf6efb9038-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:03:05.661 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.661 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.661 DEBUG:boto:Method: GET 2023-10-12T01:03:05.662 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:03:05.662 DEBUG:boto:Data: 2023-10-12T01:03:05.663 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.664 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.664 DEBUG:boto:Params: {} 2023-10-12T01:03:05.665 DEBUG:boto:Token: None 2023-10-12T01:03:05.665 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95 2023-10-12T01:03:05.666 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1lVMubggEvoDZG4M3BeRxhqKDwY= 2023-10-12T01:03:05.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1lVMubggEvoDZG4M3BeRxhqKDwY=', 'Content-Length': '0'} 2023-10-12T01:03:05.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000974fbc218a5576d-00652745c9-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.669 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:03:05.669 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:03:05.670 DEBUG:boto:Method: GET 2023-10-12T01:03:05.670 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:03:05.671 DEBUG:boto:Data: 2023-10-12T01:03:05.671 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.672 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.672 DEBUG:boto:Params: {} 2023-10-12T01:03:05.673 DEBUG:boto:Token: None 2023-10-12T01:03:05.673 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj95 2023-10-12T01:03:05.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1lVMubggEvoDZG4M3BeRxhqKDwY= 2023-10-12T01:03:05.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1lVMubggEvoDZG4M3BeRxhqKDwY=', 'Content-Length': '0'} 2023-10-12T01:03:05.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbcccf8a1770914-00652745c9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.677 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.677 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.678 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.678 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.679 DEBUG:boto:Method: GET 2023-10-12T01:03:05.679 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.680 DEBUG:boto:Data: 2023-10-12T01:03:05.680 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:03:05.681 DEBUG:boto:Port: 8001 2023-10-12T01:03:05.682 DEBUG:boto:Params: {} 2023-10-12T01:03:05.682 DEBUG:boto:Token: None 2023-10-12T01:03:05.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5a7d9d487001028-00652745c9-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.689 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.689 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.690 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.690 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.691 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.691 DEBUG:boto:Method: GET 2023-10-12T01:03:05.692 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:03:05.692 DEBUG:boto:Data: 2023-10-12T01:03:05.692 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.693 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.694 DEBUG:boto:Params: {} 2023-10-12T01:03:05.694 DEBUG:boto:Token: None 2023-10-12T01:03:05.695 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.695 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI= 2023-10-12T01:03:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rXmjiY8kN8gQcmE77M3tH0a2FeI=', 'Content-Length': '0'} 2023-10-12T01:03:05.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060a84b530f32ea61-00652745c9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.700 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:03:05.700 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:03:05.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:03:05.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:03:05.702 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:03:05.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:03:05.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:03:05.810 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:03:05.811 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T01:03:05.817 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-10-12T01:03:05.865 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.866 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.866 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.867 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:05.867 DEBUG:boto:Data: 2023-10-12T01:03:05.868 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.868 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.869 DEBUG:boto:Params: {} 2023-10-12T01:03:05.869 DEBUG:boto:Token: None 2023-10-12T01:03:05.870 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84= 2023-10-12T01:03:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84=', 'Content-Length': '0'} 2023-10-12T01:03:05.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d25b196b6c700ca-00652745c9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.875 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:03:05.875 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:03:05.875 DEBUG:boto:Method: PUT 2023-10-12T01:03:05.876 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:03:05.876 DEBUG:boto:Data: 2023-10-12T01:03:05.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:03:05.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.878 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.878 DEBUG:boto:Params: {} 2023-10-12T01:03:05.878 DEBUG:boto:Token: None 2023-10-12T01:03:05.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj23 2023-10-12T01:03:05.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DNhKoi0AmzOKYvKrrPkxn3lA34s= 2023-10-12T01:03:05.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DNhKoi0AmzOKYvKrrPkxn3lA34s='} 2023-10-12T01:03:05.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6a51927a79b340a-00652745c9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.886 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.886 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.887 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.887 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:05.887 DEBUG:boto:Data: 2023-10-12T01:03:05.888 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.889 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.889 DEBUG:boto:Params: {} 2023-10-12T01:03:05.890 DEBUG:boto:Token: None 2023-10-12T01:03:05.890 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84= 2023-10-12T01:03:05.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84=', 'Content-Length': '0'} 2023-10-12T01:03:05.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d07fec49baee3c9-00652745c9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.894 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:03:05.895 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:03:05.895 DEBUG:boto:Method: PUT 2023-10-12T01:03:05.895 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:03:05.896 DEBUG:boto:Data: 2023-10-12T01:03:05.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:03:05.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.897 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.898 DEBUG:boto:Params: {} 2023-10-12T01:03:05.898 DEBUG:boto:Token: None 2023-10-12T01:03:05.898 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj46 2023-10-12T01:03:05.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tvlbL7yj6J+MLnTcpvZBbcnUcn0= 2023-10-12T01:03:05.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tvlbL7yj6J+MLnTcpvZBbcnUcn0='} 2023-10-12T01:03:05.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bce0d9d0a877b858-00652745c9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.943 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.943 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.944 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.944 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:05.944 DEBUG:boto:Data: 2023-10-12T01:03:05.945 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.945 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.946 DEBUG:boto:Params: {} 2023-10-12T01:03:05.946 DEBUG:boto:Token: None 2023-10-12T01:03:05.947 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84= 2023-10-12T01:03:05.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84=', 'Content-Length': '0'} 2023-10-12T01:03:05.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2038d6e4669313a-00652745c9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.951 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:03:05.952 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:03:05.952 DEBUG:boto:Method: PUT 2023-10-12T01:03:05.953 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:03:05.953 DEBUG:boto:Data: 2023-10-12T01:03:05.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:03:05.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.954 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.954 DEBUG:boto:Params: {} 2023-10-12T01:03:05.955 DEBUG:boto:Token: None 2023-10-12T01:03:05.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj69 2023-10-12T01:03:05.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2Q3tEgL2PYgu7Qjok93rsXBpb4w= 2023-10-12T01:03:05.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2Q3tEgL2PYgu7Qjok93rsXBpb4w='} 2023-10-12T01:03:05.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033342f6101550cdb-00652745c9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.962 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:05.962 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:05.962 DEBUG:boto:Method: HEAD 2023-10-12T01:03:05.963 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:05.963 DEBUG:boto:Data: 2023-10-12T01:03:05.964 DEBUG:boto:Headers: {} 2023-10-12T01:03:05.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.964 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.965 DEBUG:boto:Params: {} 2023-10-12T01:03:05.965 DEBUG:boto:Token: None 2023-10-12T01:03:05.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/ 2023-10-12T01:03:05.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84= 2023-10-12T01:03:05.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XJRqmfAm72oFYp2tB1nBsC8TT84=', 'Content-Length': '0'} 2023-10-12T01:03:05.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0c9fb1675e2e120-00652745c9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:05.970 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:03:05.970 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:03:05.971 DEBUG:boto:Method: PUT 2023-10-12T01:03:05.971 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:03:05.971 DEBUG:boto:Data: 2023-10-12T01:03:05.974 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:03:05.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:05.974 DEBUG:boto:Port: 8000 2023-10-12T01:03:05.975 DEBUG:boto:Params: {} 2023-10-12T01:03:05.975 DEBUG:boto:Token: None 2023-10-12T01:03:05.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:03:05 GMT /wafmoa-59/obj92 2023-10-12T01:03:05.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pzsJJOhTasEsp5wUNSzBJXUGsyI= 2023-10-12T01:03:05.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:03:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pzsJJOhTasEsp5wUNSzBJXUGsyI='} 2023-10-12T01:03:06.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092b6005c8acdd244-00652745c9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:06.023 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:03:06.024 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:03:06.024 DEBUG:boto:Method: HEAD 2023-10-12T01:03:06.024 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:03:06.025 DEBUG:boto:Data: 2023-10-12T01:03:06.025 DEBUG:boto:Headers: {} 2023-10-12T01:03:06.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:06.026 DEBUG:boto:Port: 8000 2023-10-12T01:03:06.026 DEBUG:boto:Params: {} 2023-10-12T01:03:06.027 DEBUG:boto:Token: None 2023-10-12T01:03:06.027 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:03:06 GMT /wafmoa-59/ 2023-10-12T01:03:06.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pf8QQheXzpiJ0QW7bygau+keiwk= 2023-10-12T01:03:06.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:03:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pf8QQheXzpiJ0QW7bygau+keiwk=', 'Content-Length': '0'} 2023-10-12T01:03:06.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee09140473e1dd19-00652745ca-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:06.032 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:03:06.032 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:03:06.033 DEBUG:boto:Method: PUT 2023-10-12T01:03:06.033 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:03:06.034 DEBUG:boto:Data: 2023-10-12T01:03:06.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:03:06.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:03:06.035 DEBUG:boto:Port: 8000 2023-10-12T01:03:06.035 DEBUG:boto:Params: {} 2023-10-12T01:03:06.036 DEBUG:boto:Token: None 2023-10-12T01:03:06.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:03:06 GMT /wafmoa-59/obj115 2023-10-12T01:03:06.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tvsgfB05uXlQrJA2IIsXeSIBoM8= 2023-10-12T01:03:06.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:03:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tvsgfB05uXlQrJA2IIsXeSIBoM8='} 2023-10-12T01:03:06.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c15d932e47606c20-00652745ca-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:03:06.042 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:03:06.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:03:06.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:03:06.044 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:03:06.100 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:03:06.100 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:03:06.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:06.134+0000 7f346c5aa780 1 waiting to reach latest gen 3, on 2.. 2023-10-12T01:03:11.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:11.134+0000 7f346c5aa780 1 waiting to reach latest gen 3, on 2.. 2023-10-12T01:03:16.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:16.135+0000 7f346c5aa780 1 waiting to reach latest gen 3, on 2.. 2023-10-12T01:03:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:21.136+0000 7f346c5aa780 1 waiting to reach latest gen 3, on 2.. 2023-10-12T01:03:26.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:26.138+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:26.139 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:26.139 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:31.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:31.140+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:31.140 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:31.141 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:36.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:36.141+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:36.142 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:36.142 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:41.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:41.143+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:41.144 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:41.144 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:46.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:46.145+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:46.146 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:46.146 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:51.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:51.146+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:51.148 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:51.148 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:56.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:03:56.148+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:03:56.150 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:03:56.150 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:04:01.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:01.151+0000 7f346c5aa780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:01.152 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:04:01.152 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:04:06.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:06.153+0000 7f346c5aa780 1 bucket sync caught up with source: 2023-10-12T01:04:06.153 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:04:06.154 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.41.6, , 00000000002.141.6, , , , , , 00000000001.40.6, , 00000000001.75.6, , , , ] 2023-10-12T01:04:06.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:06.153+0000 7f346c5aa780 0 bucket checkpoint complete 2023-10-12T01:04:06.171 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:04:06.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:04:06.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:04:06.173 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:04:06.236 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:04:06.236 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:04:06.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:06.273+0000 7ff45b2e3780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:06.273 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000004.238.6, 00000000004.210.6] 2023-10-12T01:04:06.274 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:04:11.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:11.275+0000 7ff45b2e3780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:11.275 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000004.238.6, 00000000004.210.6] 2023-10-12T01:04:11.275 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:04:16.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:16.275+0000 7ff45b2e3780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:16.276 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000004.238.6, 00000000004.210.6] 2023-10-12T01:04:16.276 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:04:20.261 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:04:20.260+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:04:21.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:21.276+0000 7ff45b2e3780 1 bucket sync caught up with source: 2023-10-12T01:04:21.277 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:04:21.277 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000002.206.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:04:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:21.276+0000 7ff45b2e3780 0 bucket checkpoint complete 2023-10-12T01:04:21.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:04:21.295 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.295 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.296 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.296 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:21.297 DEBUG:boto:Data: 2023-10-12T01:04:21.297 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.298 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.298 DEBUG:boto:Params: {} 2023-10-12T01:04:21.299 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:04:21.300 DEBUG:boto:Token: None 2023-10-12T01:04:21.300 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ML9jgkfTWIDrKh28aRXLaJCkY0= 2023-10-12T01:04:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ML9jgkfTWIDrKh28aRXLaJCkY0=', 'Content-Length': '0'} 2023-10-12T01:04:21.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000580b5ed944d41847-0065274615-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.307 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.308 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.308 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.309 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:21.309 DEBUG:boto:Data: 2023-10-12T01:04:21.309 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.310 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.311 DEBUG:boto:Params: {} 2023-10-12T01:04:21.311 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:04:21.312 DEBUG:boto:Token: None 2023-10-12T01:04:21.312 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ML9jgkfTWIDrKh28aRXLaJCkY0= 2023-10-12T01:04:21.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ML9jgkfTWIDrKh28aRXLaJCkY0=', 'Content-Length': '0'} 2023-10-12T01:04:21.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e86a7e7bad8a206-0065274615-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.318 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:04:21.319 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.319 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.320 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.320 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.321 DEBUG:boto:Method: GET 2023-10-12T01:04:21.321 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.321 DEBUG:boto:Data: 2023-10-12T01:04:21.322 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.323 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.323 DEBUG:boto:Params: {} 2023-10-12T01:04:21.324 DEBUG:boto:Token: None 2023-10-12T01:04:21.324 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/?versions 2023-10-12T01:04:21.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk= 2023-10-12T01:04:21.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk=', 'Content-Length': '0'} 2023-10-12T01:04:21.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a583b9a2c014876f-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.329 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:04:21.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:04:21.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:04:21.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:04:21.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:04:21.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:04:21.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:04:21.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:04:21.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:04:21.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:04:21.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:04:21.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:04:21.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:04:21.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:04:21.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:04:21.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:04:21.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:04:21.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:04:21.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:04:21.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:04:21.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:04:21.338 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.339 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.339 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.340 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.340 DEBUG:boto:Method: GET 2023-10-12T01:04:21.340 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.341 DEBUG:boto:Data: 2023-10-12T01:04:21.341 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.342 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.343 DEBUG:boto:Params: {} 2023-10-12T01:04:21.343 DEBUG:boto:Token: None 2023-10-12T01:04:21.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/?versions 2023-10-12T01:04:21.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk= 2023-10-12T01:04:21.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk=', 'Content-Length': '0'} 2023-10-12T01:04:21.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007638245e66d8a052-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.349 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:04:21.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:04:21.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:04:21.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:04:21.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:04:21.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:04:21.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:04:21.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:04:21.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:04:21.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:04:21.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:04:21.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:04:21.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:04:21.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:04:21.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:04:21.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:04:21.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:04:21.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:04:21.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:04:21.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:04:21.358 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.358 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.358 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.359 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.359 DEBUG:boto:Method: GET 2023-10-12T01:04:21.360 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.360 DEBUG:boto:Data: 2023-10-12T01:04:21.361 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.361 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.362 DEBUG:boto:Params: {} 2023-10-12T01:04:21.362 DEBUG:boto:Token: None 2023-10-12T01:04:21.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/?versions 2023-10-12T01:04:21.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk= 2023-10-12T01:04:21.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk=', 'Content-Length': '0'} 2023-10-12T01:04:21.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c42c6a217064d9-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.368 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.369 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.369 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.370 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.370 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.371 DEBUG:boto:Method: GET 2023-10-12T01:04:21.371 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:04:21.372 DEBUG:boto:Data: 2023-10-12T01:04:21.372 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.373 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.373 DEBUG:boto:Params: {} 2023-10-12T01:04:21.374 DEBUG:boto:Token: None 2023-10-12T01:04:21.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/?versions 2023-10-12T01:04:21.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk= 2023-10-12T01:04:21.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jWRIeTbXWSdpIqT6HLRZYJCr3tk=', 'Content-Length': '0'} 2023-10-12T01:04:21.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f1a1bf559dc1d9c-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.379 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:04:21.381 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.382 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.382 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.383 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.383 DEBUG:boto:Method: GET 2023-10-12T01:04:21.383 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.384 DEBUG:boto:Data: 2023-10-12T01:04:21.384 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.385 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.386 DEBUG:boto:Params: {} 2023-10-12T01:04:21.386 DEBUG:boto:Token: None 2023-10-12T01:04:21.387 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk= 2023-10-12T01:04:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk=', 'Content-Length': '0'} 2023-10-12T01:04:21.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b256623dc164cad-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.430 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.431 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.431 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.432 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.432 DEBUG:boto:Method: GET 2023-10-12T01:04:21.433 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.433 DEBUG:boto:Data: 2023-10-12T01:04:21.434 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.435 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.435 DEBUG:boto:Params: {} 2023-10-12T01:04:21.435 DEBUG:boto:Token: None 2023-10-12T01:04:21.436 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk= 2023-10-12T01:04:21.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk=', 'Content-Length': '0'} 2023-10-12T01:04:21.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95a97884fbe26ce-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.440 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.440 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.441 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.441 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.442 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.442 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.443 DEBUG:boto:Data: 2023-10-12T01:04:21.443 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.444 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.444 DEBUG:boto:Params: {} 2023-10-12T01:04:21.445 DEBUG:boto:Token: None 2023-10-12T01:04:21.445 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Atg5an58nGHEQAUcR+L+4GADerk= 2023-10-12T01:04:21.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Atg5an58nGHEQAUcR+L+4GADerk=', 'Content-Length': '0'} 2023-10-12T01:04:21.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e0177ce0cc789d-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.449 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.449 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.450 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.450 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.451 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.451 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.452 DEBUG:boto:Data: 2023-10-12T01:04:21.452 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.453 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.453 DEBUG:boto:Params: {} 2023-10-12T01:04:21.454 DEBUG:boto:Token: None 2023-10-12T01:04:21.454 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Atg5an58nGHEQAUcR+L+4GADerk= 2023-10-12T01:04:21.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Atg5an58nGHEQAUcR+L+4GADerk=', 'Content-Length': '0'} 2023-10-12T01:04:21.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c72982abcfe0e4d9-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:04:21.458 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.458 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.459 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.459 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.460 DEBUG:boto:Method: GET 2023-10-12T01:04:21.460 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.461 DEBUG:boto:Data: 2023-10-12T01:04:21.461 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.462 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.462 DEBUG:boto:Params: {} 2023-10-12T01:04:21.463 DEBUG:boto:Token: None 2023-10-12T01:04:21.463 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk= 2023-10-12T01:04:21.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk=', 'Content-Length': '0'} 2023-10-12T01:04:21.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd76d93f15891a63-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.467 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.467 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.468 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.468 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.469 DEBUG:boto:Method: GET 2023-10-12T01:04:21.469 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.469 DEBUG:boto:Data: 2023-10-12T01:04:21.470 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.471 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.471 DEBUG:boto:Params: {} 2023-10-12T01:04:21.472 DEBUG:boto:Token: None 2023-10-12T01:04:21.472 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:04:21.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk= 2023-10-12T01:04:21.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iekJuGiO+1Z0xYD2bPItSvDuIJk=', 'Content-Length': '0'} 2023-10-12T01:04:21.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008711dcebaf38b4a9-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.476 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.476 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.477 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.477 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.478 DEBUG:boto:Method: GET 2023-10-12T01:04:21.478 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.478 DEBUG:boto:Data: 2023-10-12T01:04:21.479 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.480 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.480 DEBUG:boto:Params: {} 2023-10-12T01:04:21.481 DEBUG:boto:Token: None 2023-10-12T01:04:21.481 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068171ac869640a78-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.485 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.486 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.486 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.487 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.487 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.488 DEBUG:boto:Method: GET 2023-10-12T01:04:21.488 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.489 DEBUG:boto:Data: 2023-10-12T01:04:21.489 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.490 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.490 DEBUG:boto:Params: {} 2023-10-12T01:04:21.491 DEBUG:boto:Token: None 2023-10-12T01:04:21.491 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356d06b1bbf1bf8d-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.542 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.543 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.544 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.544 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.545 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:04:21.545 DEBUG:boto:Data: 2023-10-12T01:04:21.545 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.546 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.547 DEBUG:boto:Params: {} 2023-10-12T01:04:21.547 DEBUG:boto:Token: None 2023-10-12T01:04:21.548 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11 2023-10-12T01:04:21.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S1egPXHNoHzTzROR6Z/o87XEhb4= 2023-10-12T01:04:21.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S1egPXHNoHzTzROR6Z/o87XEhb4=', 'Content-Length': '0'} 2023-10-12T01:04:21.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d0499aa1e0a771-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.551 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.552 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.552 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.552 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:04:21.553 DEBUG:boto:Data: 2023-10-12T01:04:21.553 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.554 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.555 DEBUG:boto:Params: {} 2023-10-12T01:04:21.555 DEBUG:boto:Token: None 2023-10-12T01:04:21.555 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11 2023-10-12T01:04:21.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S1egPXHNoHzTzROR6Z/o87XEhb4= 2023-10-12T01:04:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S1egPXHNoHzTzROR6Z/o87XEhb4=', 'Content-Length': '0'} 2023-10-12T01:04:21.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cefe628501f5ee41-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:04:21.560 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.560 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.561 DEBUG:boto:Method: GET 2023-10-12T01:04:21.561 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:04:21.562 DEBUG:boto:Data: 2023-10-12T01:04:21.562 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.563 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.563 DEBUG:boto:Params: {} 2023-10-12T01:04:21.564 DEBUG:boto:Token: None 2023-10-12T01:04:21.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11 2023-10-12T01:04:21.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cfnZFrXLEJlIAyq8jJc7LSrwweo= 2023-10-12T01:04:21.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cfnZFrXLEJlIAyq8jJc7LSrwweo=', 'Content-Length': '0'} 2023-10-12T01:04:21.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b9e8f5ebf33be8-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.614 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:04:21.615 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:04:21.615 DEBUG:boto:Method: GET 2023-10-12T01:04:21.616 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:04:21.616 DEBUG:boto:Data: 2023-10-12T01:04:21.617 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.618 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.618 DEBUG:boto:Params: {} 2023-10-12T01:04:21.619 DEBUG:boto:Token: None 2023-10-12T01:04:21.619 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj11 2023-10-12T01:04:21.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cfnZFrXLEJlIAyq8jJc7LSrwweo= 2023-10-12T01:04:21.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cfnZFrXLEJlIAyq8jJc7LSrwweo=', 'Content-Length': '0'} 2023-10-12T01:04:21.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080412f326e00898c-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.623 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.623 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.624 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.624 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.625 DEBUG:boto:Method: GET 2023-10-12T01:04:21.625 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.626 DEBUG:boto:Data: 2023-10-12T01:04:21.626 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.627 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.627 DEBUG:boto:Params: {} 2023-10-12T01:04:21.628 DEBUG:boto:Token: None 2023-10-12T01:04:21.628 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.629 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e0c2675676944a-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.633 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.633 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.634 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.634 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.635 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.635 DEBUG:boto:Method: GET 2023-10-12T01:04:21.636 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:04:21.636 DEBUG:boto:Data: 2023-10-12T01:04:21.637 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.637 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.638 DEBUG:boto:Params: {} 2023-10-12T01:04:21.638 DEBUG:boto:Token: None 2023-10-12T01:04:21.639 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c69ea9e09ad5c86-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.645 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:04:21.647 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.647 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.647 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.648 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.648 DEBUG:boto:Method: GET 2023-10-12T01:04:21.649 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.649 DEBUG:boto:Data: 2023-10-12T01:04:21.650 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.651 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.651 DEBUG:boto:Params: {} 2023-10-12T01:04:21.651 DEBUG:boto:Token: None 2023-10-12T01:04:21.652 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY= 2023-10-12T01:04:21.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY=', 'Content-Length': '0'} 2023-10-12T01:04:21.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012319658a2451364-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.657 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.657 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.657 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.658 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.658 DEBUG:boto:Method: GET 2023-10-12T01:04:21.659 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.659 DEBUG:boto:Data: 2023-10-12T01:04:21.660 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.660 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.661 DEBUG:boto:Params: {} 2023-10-12T01:04:21.661 DEBUG:boto:Token: None 2023-10-12T01:04:21.662 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.662 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY= 2023-10-12T01:04:21.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY=', 'Content-Length': '0'} 2023-10-12T01:04:21.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe3268d14660d5c-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.710 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.711 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.711 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.712 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.712 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.713 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.713 DEBUG:boto:Data: 2023-10-12T01:04:21.714 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.715 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.715 DEBUG:boto:Params: {} 2023-10-12T01:04:21.716 DEBUG:boto:Token: None 2023-10-12T01:04:21.716 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i84mVX+v7rzm4lxuaYsU/yjiKm0= 2023-10-12T01:04:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i84mVX+v7rzm4lxuaYsU/yjiKm0=', 'Content-Length': '0'} 2023-10-12T01:04:21.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bcc9e4afe49c38-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.720 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.720 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.721 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.721 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.722 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.722 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.722 DEBUG:boto:Data: 2023-10-12T01:04:21.723 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.724 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.724 DEBUG:boto:Params: {} 2023-10-12T01:04:21.725 DEBUG:boto:Token: None 2023-10-12T01:04:21.725 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i84mVX+v7rzm4lxuaYsU/yjiKm0= 2023-10-12T01:04:21.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i84mVX+v7rzm4lxuaYsU/yjiKm0=', 'Content-Length': '0'} 2023-10-12T01:04:21.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef77bf8b7fa58897-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:04:21.730 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.730 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.730 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.731 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.731 DEBUG:boto:Method: GET 2023-10-12T01:04:21.732 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.732 DEBUG:boto:Data: 2023-10-12T01:04:21.733 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.734 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.734 DEBUG:boto:Params: {} 2023-10-12T01:04:21.735 DEBUG:boto:Token: None 2023-10-12T01:04:21.736 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY= 2023-10-12T01:04:21.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY=', 'Content-Length': '0'} 2023-10-12T01:04:21.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e83ae60b44c6bfd-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.739 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.740 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.740 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.741 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.741 DEBUG:boto:Method: GET 2023-10-12T01:04:21.742 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.743 DEBUG:boto:Data: 2023-10-12T01:04:21.744 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.745 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.746 DEBUG:boto:Params: {} 2023-10-12T01:04:21.746 DEBUG:boto:Token: None 2023-10-12T01:04:21.747 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:04:21.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY= 2023-10-12T01:04:21.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HmIITXnuFl/EsZ2uz1em9yfNEY=', 'Content-Length': '0'} 2023-10-12T01:04:21.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b892d0467d953f-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.751 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.751 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.752 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.752 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.753 DEBUG:boto:Method: GET 2023-10-12T01:04:21.753 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.754 DEBUG:boto:Data: 2023-10-12T01:04:21.754 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.755 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.756 DEBUG:boto:Params: {} 2023-10-12T01:04:21.756 DEBUG:boto:Token: None 2023-10-12T01:04:21.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165359f91b22c586-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.802 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.804 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.804 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.805 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.805 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.806 DEBUG:boto:Method: GET 2023-10-12T01:04:21.806 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.807 DEBUG:boto:Data: 2023-10-12T01:04:21.807 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.808 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.808 DEBUG:boto:Params: {} 2023-10-12T01:04:21.809 DEBUG:boto:Token: None 2023-10-12T01:04:21.809 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.810 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2cb7cfcafcb1141-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.814 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.815 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.816 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.816 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.816 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:04:21.817 DEBUG:boto:Data: 2023-10-12T01:04:21.817 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.818 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.819 DEBUG:boto:Params: {} 2023-10-12T01:04:21.819 DEBUG:boto:Token: None 2023-10-12T01:04:21.820 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115 2023-10-12T01:04:21.820 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OXFgBPYArxp/GQZFJFcyEasXCLo= 2023-10-12T01:04:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXFgBPYArxp/GQZFJFcyEasXCLo=', 'Content-Length': '0'} 2023-10-12T01:04:21.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353cfe6e99d4d158-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.823 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.824 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.824 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.825 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:04:21.825 DEBUG:boto:Data: 2023-10-12T01:04:21.826 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.827 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.827 DEBUG:boto:Params: {} 2023-10-12T01:04:21.828 DEBUG:boto:Token: None 2023-10-12T01:04:21.828 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115 2023-10-12T01:04:21.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OXFgBPYArxp/GQZFJFcyEasXCLo= 2023-10-12T01:04:21.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXFgBPYArxp/GQZFJFcyEasXCLo=', 'Content-Length': '0'} 2023-10-12T01:04:21.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4d524649749e0e-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:04:21.833 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.833 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.834 DEBUG:boto:Method: GET 2023-10-12T01:04:21.834 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:04:21.835 DEBUG:boto:Data: 2023-10-12T01:04:21.836 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.836 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.837 DEBUG:boto:Params: {} 2023-10-12T01:04:21.837 DEBUG:boto:Token: None 2023-10-12T01:04:21.838 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115 2023-10-12T01:04:21.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ZCeW3t8RriXedD8bsaZenUf8Hg= 2023-10-12T01:04:21.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ZCeW3t8RriXedD8bsaZenUf8Hg=', 'Content-Length': '0'} 2023-10-12T01:04:21.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001993359838fae377-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.841 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:04:21.842 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:04:21.843 DEBUG:boto:Method: GET 2023-10-12T01:04:21.843 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:04:21.843 DEBUG:boto:Data: 2023-10-12T01:04:21.844 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.845 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.845 DEBUG:boto:Params: {} 2023-10-12T01:04:21.846 DEBUG:boto:Token: None 2023-10-12T01:04:21.846 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj115 2023-10-12T01:04:21.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ZCeW3t8RriXedD8bsaZenUf8Hg= 2023-10-12T01:04:21.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ZCeW3t8RriXedD8bsaZenUf8Hg=', 'Content-Length': '0'} 2023-10-12T01:04:21.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007514b701f1a4acea-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.852 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.853 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.854 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.855 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.855 DEBUG:boto:Method: GET 2023-10-12T01:04:21.856 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.857 DEBUG:boto:Data: 2023-10-12T01:04:21.858 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.859 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.860 DEBUG:boto:Params: {} 2023-10-12T01:04:21.861 DEBUG:boto:Token: None 2023-10-12T01:04:21.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e81704ecde4add7-0065274615-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.868 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.869 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:21.870 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:21.871 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.871 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.872 DEBUG:boto:Method: GET 2023-10-12T01:04:21.873 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:04:21.874 DEBUG:boto:Data: 2023-10-12T01:04:21.875 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.876 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.877 DEBUG:boto:Params: {} 2023-10-12T01:04:21.878 DEBUG:boto:Token: None 2023-10-12T01:04:21.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/ 2023-10-12T01:04:21.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0= 2023-10-12T01:04:21.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y+1Ztb4P7V4jNCKc4zCTgGZAvU0=', 'Content-Length': '0'} 2023-10-12T01:04:21.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c9e4594636dbee-0065274615-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.883 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:21.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:04:21.884 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.885 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.885 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.886 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.886 DEBUG:boto:Method: GET 2023-10-12T01:04:21.887 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.887 DEBUG:boto:Data: 2023-10-12T01:04:21.888 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.889 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.889 DEBUG:boto:Params: {} 2023-10-12T01:04:21.890 DEBUG:boto:Token: None 2023-10-12T01:04:21.890 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc= 2023-10-12T01:04:21.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc=', 'Content-Length': '0'} 2023-10-12T01:04:21.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5624d3bdbf320a6-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.938 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.939 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.940 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.941 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.941 DEBUG:boto:Method: GET 2023-10-12T01:04:21.942 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.943 DEBUG:boto:Data: 2023-10-12T01:04:21.944 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.945 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.946 DEBUG:boto:Params: {} 2023-10-12T01:04:21.947 DEBUG:boto:Token: None 2023-10-12T01:04:21.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc= 2023-10-12T01:04:21.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc=', 'Content-Length': '0'} 2023-10-12T01:04:21.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295225757aeaa225-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.952 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.953 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.954 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.955 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.955 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.956 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.957 DEBUG:boto:Data: 2023-10-12T01:04:21.957 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.958 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.959 DEBUG:boto:Params: {} 2023-10-12T01:04:21.959 DEBUG:boto:Token: None 2023-10-12T01:04:21.960 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1aQIWYVWXViu+1NDyW6nay03Nxw= 2023-10-12T01:04:21.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1aQIWYVWXViu+1NDyW6nay03Nxw=', 'Content-Length': '0'} 2023-10-12T01:04:21.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d068f57973f787e2-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.963 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.964 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.964 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.965 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.965 DEBUG:boto:Method: HEAD 2023-10-12T01:04:21.965 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.966 DEBUG:boto:Data: 2023-10-12T01:04:21.966 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.967 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.968 DEBUG:boto:Params: {} 2023-10-12T01:04:21.968 DEBUG:boto:Token: None 2023-10-12T01:04:21.969 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1aQIWYVWXViu+1NDyW6nay03Nxw= 2023-10-12T01:04:21.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1aQIWYVWXViu+1NDyW6nay03Nxw=', 'Content-Length': '0'} 2023-10-12T01:04:21.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b020d653b72d707c-0065274615-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:04:21.975 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.976 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.977 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.978 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.978 DEBUG:boto:Method: GET 2023-10-12T01:04:21.979 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.980 DEBUG:boto:Data: 2023-10-12T01:04:21.981 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:21.982 DEBUG:boto:Port: 8001 2023-10-12T01:04:21.983 DEBUG:boto:Params: {} 2023-10-12T01:04:21.984 DEBUG:boto:Token: None 2023-10-12T01:04:21.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc= 2023-10-12T01:04:21.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc=', 'Content-Length': '0'} 2023-10-12T01:04:21.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007174048ca9def2ba-0065274615-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:21.990 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:21.991 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:21.991 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.992 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.993 DEBUG:boto:Method: GET 2023-10-12T01:04:21.994 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:21.994 DEBUG:boto:Data: 2023-10-12T01:04:21.995 DEBUG:boto:Headers: {} 2023-10-12T01:04:21.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:21.997 DEBUG:boto:Port: 8000 2023-10-12T01:04:21.998 DEBUG:boto:Params: {} 2023-10-12T01:04:21.999 DEBUG:boto:Token: None 2023-10-12T01:04:21.999 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:21 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:04:22.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc= 2023-10-12T01:04:22.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6C4+mVeL5h83Z7O8OFIJ+YUBapc=', 'Content-Length': '0'} 2023-10-12T01:04:22.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad5ec6979d31fd4-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.046 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.048 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.048 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.049 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.050 DEBUG:boto:Method: GET 2023-10-12T01:04:22.051 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.051 DEBUG:boto:Data: 2023-10-12T01:04:22.052 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.053 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.054 DEBUG:boto:Params: {} 2023-10-12T01:04:22.055 DEBUG:boto:Token: None 2023-10-12T01:04:22.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000403f178adb4bc19-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.063 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.064 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.065 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.066 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.067 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.067 DEBUG:boto:Method: GET 2023-10-12T01:04:22.068 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.068 DEBUG:boto:Data: 2023-10-12T01:04:22.069 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.069 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.070 DEBUG:boto:Params: {} 2023-10-12T01:04:22.070 DEBUG:boto:Token: None 2023-10-12T01:04:22.071 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004788f3cf425ab6ce-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.076 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.077 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:22.077 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:22.078 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.079 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:04:22.080 DEBUG:boto:Data: 2023-10-12T01:04:22.080 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.081 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.082 DEBUG:boto:Params: {} 2023-10-12T01:04:22.083 DEBUG:boto:Token: None 2023-10-12T01:04:22.083 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj17 2023-10-12T01:04:22.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wqEdlssMrtA2ZNcSf+uZIO5Y0T0= 2023-10-12T01:04:22.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wqEdlssMrtA2ZNcSf+uZIO5Y0T0=', 'Content-Length': '0'} 2023-10-12T01:04:22.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f4528410d98a02a-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.087 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:22.088 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:22.088 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.089 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:04:22.089 DEBUG:boto:Data: 2023-10-12T01:04:22.089 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.090 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.091 DEBUG:boto:Params: {} 2023-10-12T01:04:22.091 DEBUG:boto:Token: None 2023-10-12T01:04:22.092 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj17 2023-10-12T01:04:22.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wqEdlssMrtA2ZNcSf+uZIO5Y0T0= 2023-10-12T01:04:22.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wqEdlssMrtA2ZNcSf+uZIO5Y0T0=', 'Content-Length': '0'} 2023-10-12T01:04:22.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8b68272c99645b-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:04:22.096 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:22.096 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:22.097 DEBUG:boto:Method: GET 2023-10-12T01:04:22.097 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:04:22.098 DEBUG:boto:Data: 2023-10-12T01:04:22.098 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.099 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.100 DEBUG:boto:Params: {} 2023-10-12T01:04:22.100 DEBUG:boto:Token: None 2023-10-12T01:04:22.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj17 2023-10-12T01:04:22.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XSYRmGxSLBMpsl/uZvWQY/mGsJ4= 2023-10-12T01:04:22.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XSYRmGxSLBMpsl/uZvWQY/mGsJ4=', 'Content-Length': '0'} 2023-10-12T01:04:22.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca5c3e3bb2ac88a-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.146 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:04:22.147 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:04:22.148 DEBUG:boto:Method: GET 2023-10-12T01:04:22.148 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:04:22.148 DEBUG:boto:Data: 2023-10-12T01:04:22.149 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.150 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.150 DEBUG:boto:Params: {} 2023-10-12T01:04:22.151 DEBUG:boto:Token: None 2023-10-12T01:04:22.151 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj17 2023-10-12T01:04:22.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XSYRmGxSLBMpsl/uZvWQY/mGsJ4= 2023-10-12T01:04:22.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XSYRmGxSLBMpsl/uZvWQY/mGsJ4=', 'Content-Length': '0'} 2023-10-12T01:04:22.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27c31584529a36a-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.155 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.155 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.156 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.156 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.157 DEBUG:boto:Method: GET 2023-10-12T01:04:22.157 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.158 DEBUG:boto:Data: 2023-10-12T01:04:22.158 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.159 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.159 DEBUG:boto:Params: {} 2023-10-12T01:04:22.160 DEBUG:boto:Token: None 2023-10-12T01:04:22.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b50a8354b646e0b-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.165 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.166 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.166 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.167 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.167 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.168 DEBUG:boto:Method: GET 2023-10-12T01:04:22.168 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:04:22.169 DEBUG:boto:Data: 2023-10-12T01:04:22.169 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.170 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.170 DEBUG:boto:Params: {} 2023-10-12T01:04:22.171 DEBUG:boto:Token: None 2023-10-12T01:04:22.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb3888878c38a17d-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.218 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:04:22.220 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.220 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.220 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.221 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.221 DEBUG:boto:Method: GET 2023-10-12T01:04:22.222 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.222 DEBUG:boto:Data: 2023-10-12T01:04:22.222 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.223 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.224 DEBUG:boto:Params: {} 2023-10-12T01:04:22.224 DEBUG:boto:Token: None 2023-10-12T01:04:22.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg= 2023-10-12T01:04:22.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg=', 'Content-Length': '0'} 2023-10-12T01:04:22.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bdd861c49c2042-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.229 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.229 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.230 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.230 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.230 DEBUG:boto:Method: GET 2023-10-12T01:04:22.231 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.231 DEBUG:boto:Data: 2023-10-12T01:04:22.232 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.233 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.233 DEBUG:boto:Params: {} 2023-10-12T01:04:22.233 DEBUG:boto:Token: None 2023-10-12T01:04:22.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.234 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg= 2023-10-12T01:04:22.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg=', 'Content-Length': '0'} 2023-10-12T01:04:22.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001baf17f919e1f8ab-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.238 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.239 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.239 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.239 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.240 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.240 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.241 DEBUG:boto:Data: 2023-10-12T01:04:22.241 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.242 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.242 DEBUG:boto:Params: {} 2023-10-12T01:04:22.243 DEBUG:boto:Token: None 2023-10-12T01:04:22.243 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vnhHincHUSbtqjY5/FF4yTmuMF0= 2023-10-12T01:04:22.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vnhHincHUSbtqjY5/FF4yTmuMF0=', 'Content-Length': '0'} 2023-10-12T01:04:22.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079da5ceb76ad23ce-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.247 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.247 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.248 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.248 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.248 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.249 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.249 DEBUG:boto:Data: 2023-10-12T01:04:22.250 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.250 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.251 DEBUG:boto:Params: {} 2023-10-12T01:04:22.251 DEBUG:boto:Token: None 2023-10-12T01:04:22.252 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vnhHincHUSbtqjY5/FF4yTmuMF0= 2023-10-12T01:04:22.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vnhHincHUSbtqjY5/FF4yTmuMF0=', 'Content-Length': '0'} 2023-10-12T01:04:22.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000759f17b2691ef1fd-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:04:22.256 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.256 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.257 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.257 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.257 DEBUG:boto:Method: GET 2023-10-12T01:04:22.258 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.258 DEBUG:boto:Data: 2023-10-12T01:04:22.259 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.261 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.261 DEBUG:boto:Params: {} 2023-10-12T01:04:22.262 DEBUG:boto:Token: None 2023-10-12T01:04:22.262 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg= 2023-10-12T01:04:22.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg=', 'Content-Length': '0'} 2023-10-12T01:04:22.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daa056273e952d3-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.266 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.267 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.267 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.268 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.268 DEBUG:boto:Method: GET 2023-10-12T01:04:22.269 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.269 DEBUG:boto:Data: 2023-10-12T01:04:22.269 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.270 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.271 DEBUG:boto:Params: {} 2023-10-12T01:04:22.271 DEBUG:boto:Token: None 2023-10-12T01:04:22.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:04:22.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg= 2023-10-12T01:04:22.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y0seTY0CN7lIkpW/B8L2z8W49pg=', 'Content-Length': '0'} 2023-10-12T01:04:22.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f42d7befdbae4fd-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.276 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.276 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.277 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.277 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.277 DEBUG:boto:Method: GET 2023-10-12T01:04:22.278 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.278 DEBUG:boto:Data: 2023-10-12T01:04:22.279 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.280 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.280 DEBUG:boto:Params: {} 2023-10-12T01:04:22.281 DEBUG:boto:Token: None 2023-10-12T01:04:22.281 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e6838033ecb735b-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.285 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.286 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.287 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.287 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.287 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.288 DEBUG:boto:Method: GET 2023-10-12T01:04:22.288 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.289 DEBUG:boto:Data: 2023-10-12T01:04:22.289 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.290 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.291 DEBUG:boto:Params: {} 2023-10-12T01:04:22.291 DEBUG:boto:Token: None 2023-10-12T01:04:22.292 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479fbae470d0439d-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.338 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.339 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.340 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.340 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.341 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:04:22.341 DEBUG:boto:Data: 2023-10-12T01:04:22.342 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.342 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.343 DEBUG:boto:Params: {} 2023-10-12T01:04:22.343 DEBUG:boto:Token: None 2023-10-12T01:04:22.344 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19 2023-10-12T01:04:22.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ad/uq6nZDrGPeBXwkmE4L11yFWA= 2023-10-12T01:04:22.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ad/uq6nZDrGPeBXwkmE4L11yFWA=', 'Content-Length': '0'} 2023-10-12T01:04:22.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003539ddfbff5462a-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.348 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.348 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.348 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.349 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:04:22.349 DEBUG:boto:Data: 2023-10-12T01:04:22.350 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.351 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.352 DEBUG:boto:Params: {} 2023-10-12T01:04:22.352 DEBUG:boto:Token: None 2023-10-12T01:04:22.353 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19 2023-10-12T01:04:22.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ad/uq6nZDrGPeBXwkmE4L11yFWA= 2023-10-12T01:04:22.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ad/uq6nZDrGPeBXwkmE4L11yFWA=', 'Content-Length': '0'} 2023-10-12T01:04:22.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005532c050991626c1-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:04:22.357 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.358 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.358 DEBUG:boto:Method: GET 2023-10-12T01:04:22.359 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:04:22.359 DEBUG:boto:Data: 2023-10-12T01:04:22.360 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.361 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.362 DEBUG:boto:Params: {} 2023-10-12T01:04:22.363 DEBUG:boto:Token: None 2023-10-12T01:04:22.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19 2023-10-12T01:04:22.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JKwRPfkeEjtZbLZ1N13Ji/6f8yM= 2023-10-12T01:04:22.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JKwRPfkeEjtZbLZ1N13Ji/6f8yM=', 'Content-Length': '0'} 2023-10-12T01:04:22.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058014fdd9c40cf64-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.414 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:04:22.415 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:04:22.416 DEBUG:boto:Method: GET 2023-10-12T01:04:22.417 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:04:22.417 DEBUG:boto:Data: 2023-10-12T01:04:22.418 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.419 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.419 DEBUG:boto:Params: {} 2023-10-12T01:04:22.420 DEBUG:boto:Token: None 2023-10-12T01:04:22.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj19 2023-10-12T01:04:22.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JKwRPfkeEjtZbLZ1N13Ji/6f8yM= 2023-10-12T01:04:22.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JKwRPfkeEjtZbLZ1N13Ji/6f8yM=', 'Content-Length': '0'} 2023-10-12T01:04:22.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db0d03d8fae5937-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.424 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.425 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.425 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.425 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.426 DEBUG:boto:Method: GET 2023-10-12T01:04:22.427 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.427 DEBUG:boto:Data: 2023-10-12T01:04:22.428 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.430 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.431 DEBUG:boto:Params: {} 2023-10-12T01:04:22.432 DEBUG:boto:Token: None 2023-10-12T01:04:22.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1e4996606c1727-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.438 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.440 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.440 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.441 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.442 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.443 DEBUG:boto:Method: GET 2023-10-12T01:04:22.443 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:04:22.444 DEBUG:boto:Data: 2023-10-12T01:04:22.445 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.446 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.447 DEBUG:boto:Params: {} 2023-10-12T01:04:22.448 DEBUG:boto:Token: None 2023-10-12T01:04:22.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f744029046fc7f09-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.455 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:04:22.456 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.457 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.457 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.458 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.458 DEBUG:boto:Method: GET 2023-10-12T01:04:22.459 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.459 DEBUG:boto:Data: 2023-10-12T01:04:22.460 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.461 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.461 DEBUG:boto:Params: {} 2023-10-12T01:04:22.462 DEBUG:boto:Token: None 2023-10-12T01:04:22.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA= 2023-10-12T01:04:22.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA=', 'Content-Length': '0'} 2023-10-12T01:04:22.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075201d9dca722a0e-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.468 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.469 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.469 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.470 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.470 DEBUG:boto:Method: GET 2023-10-12T01:04:22.471 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.471 DEBUG:boto:Data: 2023-10-12T01:04:22.472 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.472 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.473 DEBUG:boto:Params: {} 2023-10-12T01:04:22.473 DEBUG:boto:Token: None 2023-10-12T01:04:22.474 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA= 2023-10-12T01:04:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA=', 'Content-Length': '0'} 2023-10-12T01:04:22.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35c6ae9236ee784-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.522 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.523 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.523 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.524 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.524 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.525 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.525 DEBUG:boto:Data: 2023-10-12T01:04:22.525 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.526 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.527 DEBUG:boto:Params: {} 2023-10-12T01:04:22.527 DEBUG:boto:Token: None 2023-10-12T01:04:22.528 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L3eemQ6WU0/3jvWKw3jWPMKcBJg= 2023-10-12T01:04:22.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L3eemQ6WU0/3jvWKw3jWPMKcBJg=', 'Content-Length': '0'} 2023-10-12T01:04:22.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f00305eaecf0a52-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.532 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.532 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.532 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.533 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.533 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.534 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.534 DEBUG:boto:Data: 2023-10-12T01:04:22.535 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.535 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.536 DEBUG:boto:Params: {} 2023-10-12T01:04:22.536 DEBUG:boto:Token: None 2023-10-12T01:04:22.537 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L3eemQ6WU0/3jvWKw3jWPMKcBJg= 2023-10-12T01:04:22.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L3eemQ6WU0/3jvWKw3jWPMKcBJg=', 'Content-Length': '0'} 2023-10-12T01:04:22.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96c1eca2d05f58b-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:04:22.540 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.541 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.541 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.542 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.542 DEBUG:boto:Method: GET 2023-10-12T01:04:22.543 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.543 DEBUG:boto:Data: 2023-10-12T01:04:22.544 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.544 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.545 DEBUG:boto:Params: {} 2023-10-12T01:04:22.545 DEBUG:boto:Token: None 2023-10-12T01:04:22.546 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.546 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA= 2023-10-12T01:04:22.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA=', 'Content-Length': '0'} 2023-10-12T01:04:22.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55f5f415b5326ba-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.549 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.550 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.550 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.551 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.551 DEBUG:boto:Method: GET 2023-10-12T01:04:22.552 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.552 DEBUG:boto:Data: 2023-10-12T01:04:22.552 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.553 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.554 DEBUG:boto:Params: {} 2023-10-12T01:04:22.554 DEBUG:boto:Token: None 2023-10-12T01:04:22.555 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:04:22.555 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA= 2023-10-12T01:04:22.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jfrqUyJdKAfHADU+118LpRWnyyA=', 'Content-Length': '0'} 2023-10-12T01:04:22.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4398cac0ba27a5d-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.559 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.559 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.560 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.560 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.560 DEBUG:boto:Method: GET 2023-10-12T01:04:22.561 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.561 DEBUG:boto:Data: 2023-10-12T01:04:22.562 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.563 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.563 DEBUG:boto:Params: {} 2023-10-12T01:04:22.563 DEBUG:boto:Token: None 2023-10-12T01:04:22.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab12edc8a329121f-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.568 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.569 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.569 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.570 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.570 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.571 DEBUG:boto:Method: GET 2023-10-12T01:04:22.571 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.572 DEBUG:boto:Data: 2023-10-12T01:04:22.572 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.573 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.573 DEBUG:boto:Params: {} 2023-10-12T01:04:22.574 DEBUG:boto:Token: None 2023-10-12T01:04:22.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4ebc31f9105b23-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.579 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.579 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.580 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.580 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.581 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:04:22.581 DEBUG:boto:Data: 2023-10-12T01:04:22.581 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.582 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.583 DEBUG:boto:Params: {} 2023-10-12T01:04:22.583 DEBUG:boto:Token: None 2023-10-12T01:04:22.584 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22 2023-10-12T01:04:22.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M8fXHdDGYCvGdL4YZdh6mIXDB1U= 2023-10-12T01:04:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M8fXHdDGYCvGdL4YZdh6mIXDB1U=', 'Content-Length': '0'} 2023-10-12T01:04:22.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f69ea91a7a8a134-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.587 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.588 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.588 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.589 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:04:22.589 DEBUG:boto:Data: 2023-10-12T01:04:22.589 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.590 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.591 DEBUG:boto:Params: {} 2023-10-12T01:04:22.591 DEBUG:boto:Token: None 2023-10-12T01:04:22.592 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22 2023-10-12T01:04:22.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M8fXHdDGYCvGdL4YZdh6mIXDB1U= 2023-10-12T01:04:22.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M8fXHdDGYCvGdL4YZdh6mIXDB1U=', 'Content-Length': '0'} 2023-10-12T01:04:22.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307fc21e65446d5c-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:04:22.595 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.596 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.596 DEBUG:boto:Method: GET 2023-10-12T01:04:22.597 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:04:22.597 DEBUG:boto:Data: 2023-10-12T01:04:22.598 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.599 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.599 DEBUG:boto:Params: {} 2023-10-12T01:04:22.600 DEBUG:boto:Token: None 2023-10-12T01:04:22.600 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22 2023-10-12T01:04:22.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rMTTCSIiwdxNJ+qVeIcLBoQ77yY= 2023-10-12T01:04:22.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rMTTCSIiwdxNJ+qVeIcLBoQ77yY=', 'Content-Length': '0'} 2023-10-12T01:04:22.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006caebb41d4569ecd-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.650 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:04:22.651 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:04:22.651 DEBUG:boto:Method: GET 2023-10-12T01:04:22.652 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:04:22.652 DEBUG:boto:Data: 2023-10-12T01:04:22.653 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.654 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.654 DEBUG:boto:Params: {} 2023-10-12T01:04:22.655 DEBUG:boto:Token: None 2023-10-12T01:04:22.656 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj22 2023-10-12T01:04:22.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rMTTCSIiwdxNJ+qVeIcLBoQ77yY= 2023-10-12T01:04:22.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rMTTCSIiwdxNJ+qVeIcLBoQ77yY=', 'Content-Length': '0'} 2023-10-12T01:04:22.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015a22797bb2291c7-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.659 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.660 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.660 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.661 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.661 DEBUG:boto:Method: GET 2023-10-12T01:04:22.662 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.662 DEBUG:boto:Data: 2023-10-12T01:04:22.663 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.663 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.664 DEBUG:boto:Params: {} 2023-10-12T01:04:22.664 DEBUG:boto:Token: None 2023-10-12T01:04:22.665 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae727aeb022806e-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.669 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.670 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.670 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.671 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.673 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.674 DEBUG:boto:Method: GET 2023-10-12T01:04:22.674 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:04:22.674 DEBUG:boto:Data: 2023-10-12T01:04:22.675 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.676 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.676 DEBUG:boto:Params: {} 2023-10-12T01:04:22.677 DEBUG:boto:Token: None 2023-10-12T01:04:22.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab92bd9baf741ea0-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.726 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:04:22.729 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.729 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.730 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.730 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.731 DEBUG:boto:Method: GET 2023-10-12T01:04:22.731 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.732 DEBUG:boto:Data: 2023-10-12T01:04:22.732 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.733 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.733 DEBUG:boto:Params: {} 2023-10-12T01:04:22.734 DEBUG:boto:Token: None 2023-10-12T01:04:22.734 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI= 2023-10-12T01:04:22.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI=', 'Content-Length': '0'} 2023-10-12T01:04:22.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9adf2fb2fa2c171-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.740 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.740 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.740 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.741 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.741 DEBUG:boto:Method: GET 2023-10-12T01:04:22.742 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.742 DEBUG:boto:Data: 2023-10-12T01:04:22.743 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.744 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.744 DEBUG:boto:Params: {} 2023-10-12T01:04:22.745 DEBUG:boto:Token: None 2023-10-12T01:04:22.745 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI= 2023-10-12T01:04:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI=', 'Content-Length': '0'} 2023-10-12T01:04:22.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003163cc8758832ba0-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.749 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.750 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.750 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.751 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.751 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.752 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.752 DEBUG:boto:Data: 2023-10-12T01:04:22.752 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.753 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.754 DEBUG:boto:Params: {} 2023-10-12T01:04:22.754 DEBUG:boto:Token: None 2023-10-12T01:04:22.755 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.755 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9IIQXRqkBEyPynnknsxPBfvpzJU= 2023-10-12T01:04:22.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9IIQXRqkBEyPynnknsxPBfvpzJU=', 'Content-Length': '0'} 2023-10-12T01:04:22.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dab342aaf3182b3-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.758 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.759 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.759 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.761 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.761 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.762 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.762 DEBUG:boto:Data: 2023-10-12T01:04:22.763 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.763 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.764 DEBUG:boto:Params: {} 2023-10-12T01:04:22.764 DEBUG:boto:Token: None 2023-10-12T01:04:22.765 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9IIQXRqkBEyPynnknsxPBfvpzJU= 2023-10-12T01:04:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9IIQXRqkBEyPynnknsxPBfvpzJU=', 'Content-Length': '0'} 2023-10-12T01:04:22.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000386792594dd4d786-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:04:22.769 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.770 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.770 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.771 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.771 DEBUG:boto:Method: GET 2023-10-12T01:04:22.772 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.772 DEBUG:boto:Data: 2023-10-12T01:04:22.773 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.773 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.774 DEBUG:boto:Params: {} 2023-10-12T01:04:22.774 DEBUG:boto:Token: None 2023-10-12T01:04:22.775 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI= 2023-10-12T01:04:22.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI=', 'Content-Length': '0'} 2023-10-12T01:04:22.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e79c3911e46074e9-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.778 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.779 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.779 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.780 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.780 DEBUG:boto:Method: GET 2023-10-12T01:04:22.781 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.781 DEBUG:boto:Data: 2023-10-12T01:04:22.781 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.782 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.783 DEBUG:boto:Params: {} 2023-10-12T01:04:22.783 DEBUG:boto:Token: None 2023-10-12T01:04:22.784 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:04:22.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI= 2023-10-12T01:04:22.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KOLHl6Q2gnVwpzk/86HffuNc6gI=', 'Content-Length': '0'} 2023-10-12T01:04:22.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018cb7074455a7a42-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.787 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.788 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.788 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.789 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.789 DEBUG:boto:Method: GET 2023-10-12T01:04:22.789 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.790 DEBUG:boto:Data: 2023-10-12T01:04:22.790 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.791 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.791 DEBUG:boto:Params: {} 2023-10-12T01:04:22.792 DEBUG:boto:Token: None 2023-10-12T01:04:22.792 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029aac175ec5bfc25-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.796 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.797 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.797 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.798 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.798 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.799 DEBUG:boto:Method: GET 2023-10-12T01:04:22.799 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.800 DEBUG:boto:Data: 2023-10-12T01:04:22.800 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.801 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.801 DEBUG:boto:Params: {} 2023-10-12T01:04:22.802 DEBUG:boto:Token: None 2023-10-12T01:04:22.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000802b24732022763-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.807 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.808 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.808 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.808 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.809 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:04:22.809 DEBUG:boto:Data: 2023-10-12T01:04:22.810 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.810 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.811 DEBUG:boto:Params: {} 2023-10-12T01:04:22.811 DEBUG:boto:Token: None 2023-10-12T01:04:22.812 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23 2023-10-12T01:04:22.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mNKuKa1sJJ0yBg7oQWr6SlyKIbk= 2023-10-12T01:04:22.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mNKuKa1sJJ0yBg7oQWr6SlyKIbk=', 'Content-Length': '0'} 2023-10-12T01:04:22.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005c2816bb273b321-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.815 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.815 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.816 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.816 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:04:22.817 DEBUG:boto:Data: 2023-10-12T01:04:22.817 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.818 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.818 DEBUG:boto:Params: {} 2023-10-12T01:04:22.819 DEBUG:boto:Token: None 2023-10-12T01:04:22.819 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23 2023-10-12T01:04:22.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mNKuKa1sJJ0yBg7oQWr6SlyKIbk= 2023-10-12T01:04:22.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mNKuKa1sJJ0yBg7oQWr6SlyKIbk=', 'Content-Length': '0'} 2023-10-12T01:04:22.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca318dbc4ddd8c3a-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:04:22.823 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.823 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.823 DEBUG:boto:Method: GET 2023-10-12T01:04:22.824 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:04:22.824 DEBUG:boto:Data: 2023-10-12T01:04:22.825 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.825 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.826 DEBUG:boto:Params: {} 2023-10-12T01:04:22.826 DEBUG:boto:Token: None 2023-10-12T01:04:22.827 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23 2023-10-12T01:04:22.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l+HUk4QnRnCA7w9jwaZQGmqRUqg= 2023-10-12T01:04:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l+HUk4QnRnCA7w9jwaZQGmqRUqg=', 'Content-Length': '0'} 2023-10-12T01:04:22.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a3a500a6bd913d-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.870 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:04:22.871 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:04:22.871 DEBUG:boto:Method: GET 2023-10-12T01:04:22.872 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:04:22.872 DEBUG:boto:Data: 2023-10-12T01:04:22.873 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.874 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.874 DEBUG:boto:Params: {} 2023-10-12T01:04:22.875 DEBUG:boto:Token: None 2023-10-12T01:04:22.875 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj23 2023-10-12T01:04:22.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l+HUk4QnRnCA7w9jwaZQGmqRUqg= 2023-10-12T01:04:22.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l+HUk4QnRnCA7w9jwaZQGmqRUqg=', 'Content-Length': '0'} 2023-10-12T01:04:22.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6b38a412fd0c87-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.879 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.880 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.880 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.881 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.881 DEBUG:boto:Method: GET 2023-10-12T01:04:22.881 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.882 DEBUG:boto:Data: 2023-10-12T01:04:22.882 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.883 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.884 DEBUG:boto:Params: {} 2023-10-12T01:04:22.884 DEBUG:boto:Token: None 2023-10-12T01:04:22.885 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2a4664e63e2d4af-0065274616-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.889 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.890 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:22.890 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:22.891 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.891 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.892 DEBUG:boto:Method: GET 2023-10-12T01:04:22.892 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:04:22.892 DEBUG:boto:Data: 2023-10-12T01:04:22.893 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.894 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.894 DEBUG:boto:Params: {} 2023-10-12T01:04:22.895 DEBUG:boto:Token: None 2023-10-12T01:04:22.895 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/ 2023-10-12T01:04:22.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM= 2023-10-12T01:04:22.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OvEGYhopFrqXmYmO6UG1KKLLSdM=', 'Content-Length': '0'} 2023-10-12T01:04:22.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019754aa8e11c6b9d-0065274616-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.900 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:22.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:04:22.901 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.902 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.902 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.902 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.903 DEBUG:boto:Method: GET 2023-10-12T01:04:22.903 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.904 DEBUG:boto:Data: 2023-10-12T01:04:22.904 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.905 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.905 DEBUG:boto:Params: {} 2023-10-12T01:04:22.906 DEBUG:boto:Token: None 2023-10-12T01:04:22.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk= 2023-10-12T01:04:22.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk=', 'Content-Length': '0'} 2023-10-12T01:04:22.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073fe75acbdc78be-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.911 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.911 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.911 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.912 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.912 DEBUG:boto:Method: GET 2023-10-12T01:04:22.913 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.913 DEBUG:boto:Data: 2023-10-12T01:04:22.914 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.915 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.915 DEBUG:boto:Params: {} 2023-10-12T01:04:22.916 DEBUG:boto:Token: None 2023-10-12T01:04:22.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk= 2023-10-12T01:04:22.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk=', 'Content-Length': '0'} 2023-10-12T01:04:22.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081751e875a7c385a-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.920 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.920 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.921 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.921 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.922 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.922 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.922 DEBUG:boto:Data: 2023-10-12T01:04:22.923 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.924 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.924 DEBUG:boto:Params: {} 2023-10-12T01:04:22.924 DEBUG:boto:Token: None 2023-10-12T01:04:22.925 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzxyPKUl1y9s2iuQm9QBWhnT1Bo= 2023-10-12T01:04:22.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzxyPKUl1y9s2iuQm9QBWhnT1Bo=', 'Content-Length': '0'} 2023-10-12T01:04:22.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da79ff6d951090f-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.928 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.929 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.929 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.930 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.930 DEBUG:boto:Method: HEAD 2023-10-12T01:04:22.931 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.931 DEBUG:boto:Data: 2023-10-12T01:04:22.932 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.932 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.933 DEBUG:boto:Params: {} 2023-10-12T01:04:22.933 DEBUG:boto:Token: None 2023-10-12T01:04:22.934 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.934 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzxyPKUl1y9s2iuQm9QBWhnT1Bo= 2023-10-12T01:04:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzxyPKUl1y9s2iuQm9QBWhnT1Bo=', 'Content-Length': '0'} 2023-10-12T01:04:22.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14d67fedc9a4fc4-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:04:22.938 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.938 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.939 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.939 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.940 DEBUG:boto:Method: GET 2023-10-12T01:04:22.940 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.941 DEBUG:boto:Data: 2023-10-12T01:04:22.941 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:22.942 DEBUG:boto:Port: 8001 2023-10-12T01:04:22.942 DEBUG:boto:Params: {} 2023-10-12T01:04:22.943 DEBUG:boto:Token: None 2023-10-12T01:04:22.943 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk= 2023-10-12T01:04:22.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk=', 'Content-Length': '0'} 2023-10-12T01:04:22.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009816cf5d22b73113-0065274616-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:22.948 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:22.948 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:22.948 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.949 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.949 DEBUG:boto:Method: GET 2023-10-12T01:04:22.950 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.950 DEBUG:boto:Data: 2023-10-12T01:04:22.951 DEBUG:boto:Headers: {} 2023-10-12T01:04:22.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:22.951 DEBUG:boto:Port: 8000 2023-10-12T01:04:22.952 DEBUG:boto:Params: {} 2023-10-12T01:04:22.952 DEBUG:boto:Token: None 2023-10-12T01:04:22.953 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:22 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:04:22.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk= 2023-10-12T01:04:22.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBrEsRixXqiL9dtrBNXpTf4xTCk=', 'Content-Length': '0'} 2023-10-12T01:04:22.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1eb7f8af08140b-0065274616-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.003 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.003 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.004 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.004 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.005 DEBUG:boto:Method: GET 2023-10-12T01:04:23.005 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.005 DEBUG:boto:Data: 2023-10-12T01:04:23.006 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.007 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.007 DEBUG:boto:Params: {} 2023-10-12T01:04:23.008 DEBUG:boto:Token: None 2023-10-12T01:04:23.008 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2c49222c3b33b1-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.013 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.013 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.014 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.014 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.015 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.015 DEBUG:boto:Method: GET 2023-10-12T01:04:23.016 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.016 DEBUG:boto:Data: 2023-10-12T01:04:23.017 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.017 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.018 DEBUG:boto:Params: {} 2023-10-12T01:04:23.018 DEBUG:boto:Token: None 2023-10-12T01:04:23.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e79ee05740bd1ddd-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.024 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.025 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:23.025 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:23.025 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.026 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:04:23.026 DEBUG:boto:Data: 2023-10-12T01:04:23.027 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.028 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.028 DEBUG:boto:Params: {} 2023-10-12T01:04:23.028 DEBUG:boto:Token: None 2023-10-12T01:04:23.029 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj33 2023-10-12T01:04:23.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aP6Jze5yqXmOjpSne74U9vp0gbk= 2023-10-12T01:04:23.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aP6Jze5yqXmOjpSne74U9vp0gbk=', 'Content-Length': '0'} 2023-10-12T01:04:23.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a66d54f60e30cdd-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.033 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:23.033 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:23.034 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.034 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:04:23.034 DEBUG:boto:Data: 2023-10-12T01:04:23.035 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.036 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.036 DEBUG:boto:Params: {} 2023-10-12T01:04:23.037 DEBUG:boto:Token: None 2023-10-12T01:04:23.037 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj33 2023-10-12T01:04:23.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aP6Jze5yqXmOjpSne74U9vp0gbk= 2023-10-12T01:04:23.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aP6Jze5yqXmOjpSne74U9vp0gbk=', 'Content-Length': '0'} 2023-10-12T01:04:23.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db098267ae7e5b3a-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:04:23.041 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:23.042 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:23.042 DEBUG:boto:Method: GET 2023-10-12T01:04:23.043 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:04:23.043 DEBUG:boto:Data: 2023-10-12T01:04:23.043 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.044 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.045 DEBUG:boto:Params: {} 2023-10-12T01:04:23.045 DEBUG:boto:Token: None 2023-10-12T01:04:23.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj33 2023-10-12T01:04:23.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ud0XoxpERk4pO0YLqeOWy1sMU+I= 2023-10-12T01:04:23.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ud0XoxpERk4pO0YLqeOWy1sMU+I=', 'Content-Length': '0'} 2023-10-12T01:04:23.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536f6ee923f8a687-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.050 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:04:23.050 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:04:23.051 DEBUG:boto:Method: GET 2023-10-12T01:04:23.051 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:04:23.052 DEBUG:boto:Data: 2023-10-12T01:04:23.052 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.053 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.054 DEBUG:boto:Params: {} 2023-10-12T01:04:23.054 DEBUG:boto:Token: None 2023-10-12T01:04:23.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj33 2023-10-12T01:04:23.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ud0XoxpERk4pO0YLqeOWy1sMU+I= 2023-10-12T01:04:23.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ud0XoxpERk4pO0YLqeOWy1sMU+I=', 'Content-Length': '0'} 2023-10-12T01:04:23.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7d55a4130e6063-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.058 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.059 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.059 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.060 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.060 DEBUG:boto:Method: GET 2023-10-12T01:04:23.060 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.061 DEBUG:boto:Data: 2023-10-12T01:04:23.061 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.062 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.063 DEBUG:boto:Params: {} 2023-10-12T01:04:23.063 DEBUG:boto:Token: None 2023-10-12T01:04:23.064 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c7d4bb83a23752-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.068 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.068 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.069 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.069 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.070 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.070 DEBUG:boto:Method: GET 2023-10-12T01:04:23.071 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:04:23.071 DEBUG:boto:Data: 2023-10-12T01:04:23.071 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.072 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.073 DEBUG:boto:Params: {} 2023-10-12T01:04:23.073 DEBUG:boto:Token: None 2023-10-12T01:04:23.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b950e52f76ac5981-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.081 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:04:23.082 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.082 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.083 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.083 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.084 DEBUG:boto:Method: GET 2023-10-12T01:04:23.084 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.084 DEBUG:boto:Data: 2023-10-12T01:04:23.085 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.086 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.086 DEBUG:boto:Params: {} 2023-10-12T01:04:23.087 DEBUG:boto:Token: None 2023-10-12T01:04:23.087 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k= 2023-10-12T01:04:23.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k=', 'Content-Length': '0'} 2023-10-12T01:04:23.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea25b8b07c6d76d4-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.138 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.139 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.139 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.140 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.140 DEBUG:boto:Method: GET 2023-10-12T01:04:23.141 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.141 DEBUG:boto:Data: 2023-10-12T01:04:23.141 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.142 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.143 DEBUG:boto:Params: {} 2023-10-12T01:04:23.143 DEBUG:boto:Token: None 2023-10-12T01:04:23.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k= 2023-10-12T01:04:23.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k=', 'Content-Length': '0'} 2023-10-12T01:04:23.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d67e750eb6812b2-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.148 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.148 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.148 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.149 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.149 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.150 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.150 DEBUG:boto:Data: 2023-10-12T01:04:23.151 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.151 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.152 DEBUG:boto:Params: {} 2023-10-12T01:04:23.152 DEBUG:boto:Token: None 2023-10-12T01:04:23.153 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l4uqxrrVZBHLDUQfPkavjav+2zM= 2023-10-12T01:04:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l4uqxrrVZBHLDUQfPkavjav+2zM=', 'Content-Length': '0'} 2023-10-12T01:04:23.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fe7bef76fb79512-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.157 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.157 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.158 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.158 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.158 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.159 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.159 DEBUG:boto:Data: 2023-10-12T01:04:23.160 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.160 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.161 DEBUG:boto:Params: {} 2023-10-12T01:04:23.161 DEBUG:boto:Token: None 2023-10-12T01:04:23.161 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:l4uqxrrVZBHLDUQfPkavjav+2zM= 2023-10-12T01:04:23.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:l4uqxrrVZBHLDUQfPkavjav+2zM=', 'Content-Length': '0'} 2023-10-12T01:04:23.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f0efff8478e9af-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:04:23.166 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.166 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.167 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.167 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.167 DEBUG:boto:Method: GET 2023-10-12T01:04:23.168 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.168 DEBUG:boto:Data: 2023-10-12T01:04:23.169 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.170 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.170 DEBUG:boto:Params: {} 2023-10-12T01:04:23.170 DEBUG:boto:Token: None 2023-10-12T01:04:23.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.171 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k= 2023-10-12T01:04:23.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k=', 'Content-Length': '0'} 2023-10-12T01:04:23.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044be8d02e13e5754-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.175 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.175 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.176 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.176 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.177 DEBUG:boto:Method: GET 2023-10-12T01:04:23.177 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.177 DEBUG:boto:Data: 2023-10-12T01:04:23.178 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.179 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.179 DEBUG:boto:Params: {} 2023-10-12T01:04:23.180 DEBUG:boto:Token: None 2023-10-12T01:04:23.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:04:23.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k= 2023-10-12T01:04:23.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:01koIi3PDpD4fElwngQCtKBQK9k=', 'Content-Length': '0'} 2023-10-12T01:04:23.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185602438c058c9f-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.226 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.227 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.227 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.229 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.230 DEBUG:boto:Method: GET 2023-10-12T01:04:23.230 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.231 DEBUG:boto:Data: 2023-10-12T01:04:23.231 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.232 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.232 DEBUG:boto:Params: {} 2023-10-12T01:04:23.233 DEBUG:boto:Token: None 2023-10-12T01:04:23.233 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.234 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f304c163ae599cab-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.238 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.239 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.239 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.240 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.240 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.240 DEBUG:boto:Method: GET 2023-10-12T01:04:23.241 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.241 DEBUG:boto:Data: 2023-10-12T01:04:23.242 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.243 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.243 DEBUG:boto:Params: {} 2023-10-12T01:04:23.244 DEBUG:boto:Token: None 2023-10-12T01:04:23.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dbfeb232ecf3e0e-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.249 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.250 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.251 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.251 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.251 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:04:23.252 DEBUG:boto:Data: 2023-10-12T01:04:23.252 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.253 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.254 DEBUG:boto:Params: {} 2023-10-12T01:04:23.254 DEBUG:boto:Token: None 2023-10-12T01:04:23.255 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34 2023-10-12T01:04:23.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZIMO/E/i5iwwsRtbvJcLgz1rNNA= 2023-10-12T01:04:23.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZIMO/E/i5iwwsRtbvJcLgz1rNNA=', 'Content-Length': '0'} 2023-10-12T01:04:23.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f0953f21ea76b06-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.258 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.259 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.259 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.260 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:04:23.260 DEBUG:boto:Data: 2023-10-12T01:04:23.260 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.261 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.262 DEBUG:boto:Params: {} 2023-10-12T01:04:23.262 DEBUG:boto:Token: None 2023-10-12T01:04:23.263 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34 2023-10-12T01:04:23.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZIMO/E/i5iwwsRtbvJcLgz1rNNA= 2023-10-12T01:04:23.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZIMO/E/i5iwwsRtbvJcLgz1rNNA=', 'Content-Length': '0'} 2023-10-12T01:04:23.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005650eceb5ca25db5-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:04:23.267 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.267 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.268 DEBUG:boto:Method: GET 2023-10-12T01:04:23.268 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:04:23.269 DEBUG:boto:Data: 2023-10-12T01:04:23.269 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.270 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.271 DEBUG:boto:Params: {} 2023-10-12T01:04:23.271 DEBUG:boto:Token: None 2023-10-12T01:04:23.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34 2023-10-12T01:04:23.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X0IQpGsgEGvosL1T4foA2TV1csk= 2023-10-12T01:04:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X0IQpGsgEGvosL1T4foA2TV1csk=', 'Content-Length': '0'} 2023-10-12T01:04:23.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576575a32e880c01-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.275 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:04:23.276 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:04:23.276 DEBUG:boto:Method: GET 2023-10-12T01:04:23.277 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:04:23.277 DEBUG:boto:Data: 2023-10-12T01:04:23.278 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.279 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.279 DEBUG:boto:Params: {} 2023-10-12T01:04:23.280 DEBUG:boto:Token: None 2023-10-12T01:04:23.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj34 2023-10-12T01:04:23.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X0IQpGsgEGvosL1T4foA2TV1csk= 2023-10-12T01:04:23.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X0IQpGsgEGvosL1T4foA2TV1csk=', 'Content-Length': '0'} 2023-10-12T01:04:23.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd11a45eae3b801-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.284 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.284 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.285 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.285 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.286 DEBUG:boto:Method: GET 2023-10-12T01:04:23.286 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.287 DEBUG:boto:Data: 2023-10-12T01:04:23.287 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.288 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.288 DEBUG:boto:Params: {} 2023-10-12T01:04:23.289 DEBUG:boto:Token: None 2023-10-12T01:04:23.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8cef067f0da1487-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.294 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.295 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.295 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.296 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.296 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.297 DEBUG:boto:Method: GET 2023-10-12T01:04:23.297 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:04:23.298 DEBUG:boto:Data: 2023-10-12T01:04:23.298 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.299 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.299 DEBUG:boto:Params: {} 2023-10-12T01:04:23.300 DEBUG:boto:Token: None 2023-10-12T01:04:23.300 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dfcbd475feba3c2-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.350 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:04:23.352 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.352 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.353 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.353 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.354 DEBUG:boto:Method: GET 2023-10-12T01:04:23.354 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.355 DEBUG:boto:Data: 2023-10-12T01:04:23.355 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.356 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.356 DEBUG:boto:Params: {} 2023-10-12T01:04:23.357 DEBUG:boto:Token: None 2023-10-12T01:04:23.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E= 2023-10-12T01:04:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E=', 'Content-Length': '0'} 2023-10-12T01:04:23.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594fd0cea8edab0b-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.362 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.362 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.363 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.363 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.364 DEBUG:boto:Method: GET 2023-10-12T01:04:23.364 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.365 DEBUG:boto:Data: 2023-10-12T01:04:23.365 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.366 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.366 DEBUG:boto:Params: {} 2023-10-12T01:04:23.367 DEBUG:boto:Token: None 2023-10-12T01:04:23.367 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E= 2023-10-12T01:04:23.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E=', 'Content-Length': '0'} 2023-10-12T01:04:23.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e49b387b82e4718-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.372 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.372 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.373 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.373 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.374 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.374 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.375 DEBUG:boto:Data: 2023-10-12T01:04:23.375 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.376 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.376 DEBUG:boto:Params: {} 2023-10-12T01:04:23.377 DEBUG:boto:Token: None 2023-10-12T01:04:23.377 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q2Tl4GyCV913BJwOIeTK7rOhWfo= 2023-10-12T01:04:23.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q2Tl4GyCV913BJwOIeTK7rOhWfo=', 'Content-Length': '0'} 2023-10-12T01:04:23.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055356c5cf7e647e1-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.381 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.381 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.382 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.382 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.383 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.383 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.383 DEBUG:boto:Data: 2023-10-12T01:04:23.384 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.385 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.385 DEBUG:boto:Params: {} 2023-10-12T01:04:23.386 DEBUG:boto:Token: None 2023-10-12T01:04:23.386 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q2Tl4GyCV913BJwOIeTK7rOhWfo= 2023-10-12T01:04:23.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q2Tl4GyCV913BJwOIeTK7rOhWfo=', 'Content-Length': '0'} 2023-10-12T01:04:23.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec02de14307c3200-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:04:23.391 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.391 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.392 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.392 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.392 DEBUG:boto:Method: GET 2023-10-12T01:04:23.393 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.393 DEBUG:boto:Data: 2023-10-12T01:04:23.394 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.395 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.395 DEBUG:boto:Params: {} 2023-10-12T01:04:23.396 DEBUG:boto:Token: None 2023-10-12T01:04:23.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E= 2023-10-12T01:04:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E=', 'Content-Length': '0'} 2023-10-12T01:04:23.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee04b27234afb13b-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.400 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.401 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.401 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.402 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.402 DEBUG:boto:Method: GET 2023-10-12T01:04:23.403 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.403 DEBUG:boto:Data: 2023-10-12T01:04:23.403 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.404 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.405 DEBUG:boto:Params: {} 2023-10-12T01:04:23.405 DEBUG:boto:Token: None 2023-10-12T01:04:23.406 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:04:23.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E= 2023-10-12T01:04:23.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:owdioQQO8xOEtvLUl0ROm8Au86E=', 'Content-Length': '0'} 2023-10-12T01:04:23.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb991553703d8ea-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.409 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.410 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.410 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.411 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.411 DEBUG:boto:Method: GET 2023-10-12T01:04:23.412 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.412 DEBUG:boto:Data: 2023-10-12T01:04:23.412 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.413 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.414 DEBUG:boto:Params: {} 2023-10-12T01:04:23.414 DEBUG:boto:Token: None 2023-10-12T01:04:23.415 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1df4a6ccda46036-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.466 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.467 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.468 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.468 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.469 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.469 DEBUG:boto:Method: GET 2023-10-12T01:04:23.469 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.470 DEBUG:boto:Data: 2023-10-12T01:04:23.470 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.471 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.472 DEBUG:boto:Params: {} 2023-10-12T01:04:23.472 DEBUG:boto:Token: None 2023-10-12T01:04:23.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb1477c76d4b6cd-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.478 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.479 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.479 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.479 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.480 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:04:23.480 DEBUG:boto:Data: 2023-10-12T01:04:23.481 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.481 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.482 DEBUG:boto:Params: {} 2023-10-12T01:04:23.482 DEBUG:boto:Token: None 2023-10-12T01:04:23.483 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38 2023-10-12T01:04:23.483 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UYlTsMZFU8odqLfw9skCo+etvtM= 2023-10-12T01:04:23.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UYlTsMZFU8odqLfw9skCo+etvtM=', 'Content-Length': '0'} 2023-10-12T01:04:23.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082d78c421f17eb1-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.486 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.487 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.487 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.488 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:04:23.488 DEBUG:boto:Data: 2023-10-12T01:04:23.488 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.489 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.490 DEBUG:boto:Params: {} 2023-10-12T01:04:23.490 DEBUG:boto:Token: None 2023-10-12T01:04:23.491 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38 2023-10-12T01:04:23.491 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UYlTsMZFU8odqLfw9skCo+etvtM= 2023-10-12T01:04:23.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UYlTsMZFU8odqLfw9skCo+etvtM=', 'Content-Length': '0'} 2023-10-12T01:04:23.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c620996e73f34128-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:04:23.494 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.495 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.495 DEBUG:boto:Method: GET 2023-10-12T01:04:23.496 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:04:23.496 DEBUG:boto:Data: 2023-10-12T01:04:23.497 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.498 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.498 DEBUG:boto:Params: {} 2023-10-12T01:04:23.498 DEBUG:boto:Token: None 2023-10-12T01:04:23.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38 2023-10-12T01:04:23.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8TkecyQ3549qqz/wKEaGeRfIYbI= 2023-10-12T01:04:23.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8TkecyQ3549qqz/wKEaGeRfIYbI=', 'Content-Length': '0'} 2023-10-12T01:04:23.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965ac1b0e75bf5ee-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.503 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:04:23.503 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:04:23.503 DEBUG:boto:Method: GET 2023-10-12T01:04:23.504 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:04:23.504 DEBUG:boto:Data: 2023-10-12T01:04:23.505 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.506 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.506 DEBUG:boto:Params: {} 2023-10-12T01:04:23.507 DEBUG:boto:Token: None 2023-10-12T01:04:23.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj38 2023-10-12T01:04:23.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8TkecyQ3549qqz/wKEaGeRfIYbI= 2023-10-12T01:04:23.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8TkecyQ3549qqz/wKEaGeRfIYbI=', 'Content-Length': '0'} 2023-10-12T01:04:23.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9622c90d34b01c-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.511 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.512 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.512 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.513 DEBUG:boto:Method: GET 2023-10-12T01:04:23.513 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.513 DEBUG:boto:Data: 2023-10-12T01:04:23.514 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.515 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.515 DEBUG:boto:Params: {} 2023-10-12T01:04:23.516 DEBUG:boto:Token: None 2023-10-12T01:04:23.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d27fda7645d6f11-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.520 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.521 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.521 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.522 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.522 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.523 DEBUG:boto:Method: GET 2023-10-12T01:04:23.523 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:04:23.524 DEBUG:boto:Data: 2023-10-12T01:04:23.524 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.525 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.525 DEBUG:boto:Params: {} 2023-10-12T01:04:23.526 DEBUG:boto:Token: None 2023-10-12T01:04:23.526 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2ab92b551c92bb-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.574 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:04:23.576 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.576 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.577 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.577 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.577 DEBUG:boto:Method: GET 2023-10-12T01:04:23.578 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.578 DEBUG:boto:Data: 2023-10-12T01:04:23.579 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.581 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.581 DEBUG:boto:Params: {} 2023-10-12T01:04:23.581 DEBUG:boto:Token: None 2023-10-12T01:04:23.582 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8= 2023-10-12T01:04:23.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8=', 'Content-Length': '0'} 2023-10-12T01:04:23.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f179cf9ca276e290-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.586 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.586 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.587 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.587 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.588 DEBUG:boto:Method: GET 2023-10-12T01:04:23.588 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.588 DEBUG:boto:Data: 2023-10-12T01:04:23.589 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.590 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.590 DEBUG:boto:Params: {} 2023-10-12T01:04:23.591 DEBUG:boto:Token: None 2023-10-12T01:04:23.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8= 2023-10-12T01:04:23.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8=', 'Content-Length': '0'} 2023-10-12T01:04:23.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000733308e5ea5002e4-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.595 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.595 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.596 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.596 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.596 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.597 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.597 DEBUG:boto:Data: 2023-10-12T01:04:23.598 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.599 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.599 DEBUG:boto:Params: {} 2023-10-12T01:04:23.599 DEBUG:boto:Token: None 2023-10-12T01:04:23.600 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+iNWzL6dQ2nv96+RTot7QCGpbpU= 2023-10-12T01:04:23.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+iNWzL6dQ2nv96+RTot7QCGpbpU=', 'Content-Length': '0'} 2023-10-12T01:04:23.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba2ba9ca28a6182-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.604 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.604 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.605 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.605 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.606 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.606 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.606 DEBUG:boto:Data: 2023-10-12T01:04:23.607 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.608 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.608 DEBUG:boto:Params: {} 2023-10-12T01:04:23.608 DEBUG:boto:Token: None 2023-10-12T01:04:23.609 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+iNWzL6dQ2nv96+RTot7QCGpbpU= 2023-10-12T01:04:23.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+iNWzL6dQ2nv96+RTot7QCGpbpU=', 'Content-Length': '0'} 2023-10-12T01:04:23.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf71d2d295d2666-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:04:23.613 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.613 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.614 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.614 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.614 DEBUG:boto:Method: GET 2023-10-12T01:04:23.615 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.615 DEBUG:boto:Data: 2023-10-12T01:04:23.616 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.617 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.617 DEBUG:boto:Params: {} 2023-10-12T01:04:23.618 DEBUG:boto:Token: None 2023-10-12T01:04:23.618 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8= 2023-10-12T01:04:23.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8=', 'Content-Length': '0'} 2023-10-12T01:04:23.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b783baa375d391-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.622 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.622 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.623 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.623 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.623 DEBUG:boto:Method: GET 2023-10-12T01:04:23.624 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.624 DEBUG:boto:Data: 2023-10-12T01:04:23.625 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.626 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.626 DEBUG:boto:Params: {} 2023-10-12T01:04:23.627 DEBUG:boto:Token: None 2023-10-12T01:04:23.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:04:23.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8= 2023-10-12T01:04:23.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+U7x+HiB7ERYI/cxiNTeqWZlii8=', 'Content-Length': '0'} 2023-10-12T01:04:23.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afba47901e7a2110-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.631 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.631 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.632 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.632 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.633 DEBUG:boto:Method: GET 2023-10-12T01:04:23.633 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.633 DEBUG:boto:Data: 2023-10-12T01:04:23.634 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.635 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.635 DEBUG:boto:Params: {} 2023-10-12T01:04:23.636 DEBUG:boto:Token: None 2023-10-12T01:04:23.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f28a3bff58ace146-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.642 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.642 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.643 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.643 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.644 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.644 DEBUG:boto:Method: GET 2023-10-12T01:04:23.645 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.645 DEBUG:boto:Data: 2023-10-12T01:04:23.646 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.646 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.647 DEBUG:boto:Params: {} 2023-10-12T01:04:23.647 DEBUG:boto:Token: None 2023-10-12T01:04:23.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37ca803ac062766-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.698 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.699 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.700 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.700 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.701 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:04:23.701 DEBUG:boto:Data: 2023-10-12T01:04:23.701 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.702 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.702 DEBUG:boto:Params: {} 2023-10-12T01:04:23.703 DEBUG:boto:Token: None 2023-10-12T01:04:23.703 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44 2023-10-12T01:04:23.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sHl0VsGHz6f3kMKBSPBlt6Nwo54= 2023-10-12T01:04:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sHl0VsGHz6f3kMKBSPBlt6Nwo54=', 'Content-Length': '0'} 2023-10-12T01:04:23.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeaed6c0f6719728-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.707 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.707 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.708 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.708 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:04:23.708 DEBUG:boto:Data: 2023-10-12T01:04:23.709 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.710 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.710 DEBUG:boto:Params: {} 2023-10-12T01:04:23.710 DEBUG:boto:Token: None 2023-10-12T01:04:23.711 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44 2023-10-12T01:04:23.711 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sHl0VsGHz6f3kMKBSPBlt6Nwo54= 2023-10-12T01:04:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sHl0VsGHz6f3kMKBSPBlt6Nwo54=', 'Content-Length': '0'} 2023-10-12T01:04:23.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d66a3fd259cf93-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:04:23.715 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.715 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.716 DEBUG:boto:Method: GET 2023-10-12T01:04:23.716 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:04:23.717 DEBUG:boto:Data: 2023-10-12T01:04:23.717 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.718 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.718 DEBUG:boto:Params: {} 2023-10-12T01:04:23.719 DEBUG:boto:Token: None 2023-10-12T01:04:23.719 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44 2023-10-12T01:04:23.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:whmHcTPS7kerUB4IX92VGePYKXY= 2023-10-12T01:04:23.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:whmHcTPS7kerUB4IX92VGePYKXY=', 'Content-Length': '0'} 2023-10-12T01:04:23.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041692776b7985571-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.723 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:04:23.724 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:04:23.724 DEBUG:boto:Method: GET 2023-10-12T01:04:23.725 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:04:23.725 DEBUG:boto:Data: 2023-10-12T01:04:23.725 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.726 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.727 DEBUG:boto:Params: {} 2023-10-12T01:04:23.727 DEBUG:boto:Token: None 2023-10-12T01:04:23.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj44 2023-10-12T01:04:23.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:whmHcTPS7kerUB4IX92VGePYKXY= 2023-10-12T01:04:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:whmHcTPS7kerUB4IX92VGePYKXY=', 'Content-Length': '0'} 2023-10-12T01:04:23.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8883cf18e93fcc7-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.731 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.732 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.732 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.733 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.733 DEBUG:boto:Method: GET 2023-10-12T01:04:23.733 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.734 DEBUG:boto:Data: 2023-10-12T01:04:23.734 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.735 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.736 DEBUG:boto:Params: {} 2023-10-12T01:04:23.736 DEBUG:boto:Token: None 2023-10-12T01:04:23.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057b3e7e0e9aa8a98-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.782 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.783 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.784 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.784 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.784 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.785 DEBUG:boto:Method: GET 2023-10-12T01:04:23.785 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:04:23.786 DEBUG:boto:Data: 2023-10-12T01:04:23.786 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.787 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.788 DEBUG:boto:Params: {} 2023-10-12T01:04:23.788 DEBUG:boto:Token: None 2023-10-12T01:04:23.790 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0ed60189ccd1340-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.795 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:04:23.797 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.798 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.798 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.798 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.799 DEBUG:boto:Method: GET 2023-10-12T01:04:23.799 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.800 DEBUG:boto:Data: 2023-10-12T01:04:23.800 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.801 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.802 DEBUG:boto:Params: {} 2023-10-12T01:04:23.802 DEBUG:boto:Token: None 2023-10-12T01:04:23.803 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8= 2023-10-12T01:04:23.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8=', 'Content-Length': '0'} 2023-10-12T01:04:23.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a606f05d8b17c9f-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.807 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.807 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.808 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.808 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.809 DEBUG:boto:Method: GET 2023-10-12T01:04:23.809 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.810 DEBUG:boto:Data: 2023-10-12T01:04:23.810 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.811 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.811 DEBUG:boto:Params: {} 2023-10-12T01:04:23.812 DEBUG:boto:Token: None 2023-10-12T01:04:23.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8= 2023-10-12T01:04:23.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8=', 'Content-Length': '0'} 2023-10-12T01:04:23.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed01c0ba6830332-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.816 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.817 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.817 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.818 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.818 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.818 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.819 DEBUG:boto:Data: 2023-10-12T01:04:23.819 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.820 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.820 DEBUG:boto:Params: {} 2023-10-12T01:04:23.821 DEBUG:boto:Token: None 2023-10-12T01:04:23.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fEpsz2PtbctkHltSp16fNS5eeVU= 2023-10-12T01:04:23.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fEpsz2PtbctkHltSp16fNS5eeVU=', 'Content-Length': '0'} 2023-10-12T01:04:23.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0453e2602d4beb1-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.825 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.825 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.826 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.826 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.827 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.827 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.827 DEBUG:boto:Data: 2023-10-12T01:04:23.828 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.829 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.829 DEBUG:boto:Params: {} 2023-10-12T01:04:23.829 DEBUG:boto:Token: None 2023-10-12T01:04:23.830 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fEpsz2PtbctkHltSp16fNS5eeVU= 2023-10-12T01:04:23.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fEpsz2PtbctkHltSp16fNS5eeVU=', 'Content-Length': '0'} 2023-10-12T01:04:23.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391cd477c30f9091-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:04:23.834 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.835 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.835 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.835 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.836 DEBUG:boto:Method: GET 2023-10-12T01:04:23.836 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.837 DEBUG:boto:Data: 2023-10-12T01:04:23.838 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.839 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.839 DEBUG:boto:Params: {} 2023-10-12T01:04:23.840 DEBUG:boto:Token: None 2023-10-12T01:04:23.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8= 2023-10-12T01:04:23.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8=', 'Content-Length': '0'} 2023-10-12T01:04:23.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae85b1c8dcd4c5b-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.844 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.844 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.845 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.845 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.846 DEBUG:boto:Method: GET 2023-10-12T01:04:23.846 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.847 DEBUG:boto:Data: 2023-10-12T01:04:23.847 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.848 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.848 DEBUG:boto:Params: {} 2023-10-12T01:04:23.849 DEBUG:boto:Token: None 2023-10-12T01:04:23.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:04:23.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8= 2023-10-12T01:04:23.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:inb4WmW7PFgfC7VdYRGnP+cREz8=', 'Content-Length': '0'} 2023-10-12T01:04:23.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201dc892063f5471-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.855 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.856 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.857 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.858 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.858 DEBUG:boto:Method: GET 2023-10-12T01:04:23.859 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.859 DEBUG:boto:Data: 2023-10-12T01:04:23.860 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.860 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.861 DEBUG:boto:Params: {} 2023-10-12T01:04:23.861 DEBUG:boto:Token: None 2023-10-12T01:04:23.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c01c1e696bb3dae4-0065274617-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.906 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.908 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.908 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.909 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.910 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.910 DEBUG:boto:Method: GET 2023-10-12T01:04:23.911 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.912 DEBUG:boto:Data: 2023-10-12T01:04:23.912 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.914 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.914 DEBUG:boto:Params: {} 2023-10-12T01:04:23.915 DEBUG:boto:Token: None 2023-10-12T01:04:23.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/ 2023-10-12T01:04:23.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw= 2023-10-12T01:04:23.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Tqskb87sITikCDhOLc2ZMRRTUjw=', 'Content-Length': '0'} 2023-10-12T01:04:23.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b0ce37d6a4010c-0065274617-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.921 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:23.921 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.922 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.922 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.923 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:04:23.923 DEBUG:boto:Data: 2023-10-12T01:04:23.924 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.925 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.925 DEBUG:boto:Params: {} 2023-10-12T01:04:23.925 DEBUG:boto:Token: None 2023-10-12T01:04:23.926 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46 2023-10-12T01:04:23.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmufWpP0UGUJDL4ptgBxkMEQ4iw= 2023-10-12T01:04:23.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmufWpP0UGUJDL4ptgBxkMEQ4iw=', 'Content-Length': '0'} 2023-10-12T01:04:23.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bb48326a14ca568-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.929 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.930 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.930 DEBUG:boto:Method: HEAD 2023-10-12T01:04:23.931 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:04:23.931 DEBUG:boto:Data: 2023-10-12T01:04:23.931 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.932 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.933 DEBUG:boto:Params: {} 2023-10-12T01:04:23.933 DEBUG:boto:Token: None 2023-10-12T01:04:23.933 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46 2023-10-12T01:04:23.934 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KmufWpP0UGUJDL4ptgBxkMEQ4iw= 2023-10-12T01:04:23.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KmufWpP0UGUJDL4ptgBxkMEQ4iw=', 'Content-Length': '0'} 2023-10-12T01:04:23.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007732fe998d09f49-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:04:23.937 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.938 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.938 DEBUG:boto:Method: GET 2023-10-12T01:04:23.938 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:04:23.939 DEBUG:boto:Data: 2023-10-12T01:04:23.939 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.940 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.940 DEBUG:boto:Params: {} 2023-10-12T01:04:23.941 DEBUG:boto:Token: None 2023-10-12T01:04:23.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46 2023-10-12T01:04:23.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zFN7xWZe+HJHVJLFpjaaHMrE6F4= 2023-10-12T01:04:23.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zFN7xWZe+HJHVJLFpjaaHMrE6F4=', 'Content-Length': '0'} 2023-10-12T01:04:23.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7ccdd46d680051-0065274617-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.945 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:04:23.945 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:04:23.946 DEBUG:boto:Method: GET 2023-10-12T01:04:23.946 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:04:23.947 DEBUG:boto:Data: 2023-10-12T01:04:23.947 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:23.948 DEBUG:boto:Port: 8000 2023-10-12T01:04:23.948 DEBUG:boto:Params: {} 2023-10-12T01:04:23.949 DEBUG:boto:Token: None 2023-10-12T01:04:23.949 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:23 GMT /wafmoa-59/obj46 2023-10-12T01:04:23.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zFN7xWZe+HJHVJLFpjaaHMrE6F4= 2023-10-12T01:04:23.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zFN7xWZe+HJHVJLFpjaaHMrE6F4=', 'Content-Length': '0'} 2023-10-12T01:04:23.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0b9457284423cf-0065274617-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:23.994 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:23.995 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:23.995 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.996 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.996 DEBUG:boto:Method: GET 2023-10-12T01:04:23.996 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:04:23.997 DEBUG:boto:Data: 2023-10-12T01:04:23.997 DEBUG:boto:Headers: {} 2023-10-12T01:04:23.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:23.998 DEBUG:boto:Port: 8001 2023-10-12T01:04:23.999 DEBUG:boto:Params: {} 2023-10-12T01:04:23.999 DEBUG:boto:Token: None 2023-10-12T01:04:24.000 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acd87a2d3d1b167d-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.004 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.005 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.005 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.006 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:24.006 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:04:24.007 DEBUG:boto:Method: GET 2023-10-12T01:04:24.007 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:04:24.007 DEBUG:boto:Data: 2023-10-12T01:04:24.008 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.009 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.009 DEBUG:boto:Params: {} 2023-10-12T01:04:24.010 DEBUG:boto:Token: None 2023-10-12T01:04:24.010 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b357fe36de88660c-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.015 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:04:24.016 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.017 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.017 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.018 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.018 DEBUG:boto:Method: GET 2023-10-12T01:04:24.019 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.019 DEBUG:boto:Data: 2023-10-12T01:04:24.020 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.021 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.021 DEBUG:boto:Params: {} 2023-10-12T01:04:24.022 DEBUG:boto:Token: None 2023-10-12T01:04:24.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc= 2023-10-12T01:04:24.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc=', 'Content-Length': '0'} 2023-10-12T01:04:24.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091669b6c093391e7-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.026 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.026 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.027 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.027 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.027 DEBUG:boto:Method: GET 2023-10-12T01:04:24.028 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.028 DEBUG:boto:Data: 2023-10-12T01:04:24.029 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.030 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.030 DEBUG:boto:Params: {} 2023-10-12T01:04:24.031 DEBUG:boto:Token: None 2023-10-12T01:04:24.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc= 2023-10-12T01:04:24.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc=', 'Content-Length': '0'} 2023-10-12T01:04:24.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e3507c25e61689-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.035 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.036 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.036 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.037 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.037 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.037 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.038 DEBUG:boto:Data: 2023-10-12T01:04:24.038 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.039 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.040 DEBUG:boto:Params: {} 2023-10-12T01:04:24.040 DEBUG:boto:Token: None 2023-10-12T01:04:24.041 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP2rHxujWg43bKfg2E+NpprkL/U= 2023-10-12T01:04:24.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP2rHxujWg43bKfg2E+NpprkL/U=', 'Content-Length': '0'} 2023-10-12T01:04:24.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5002faccaa19da-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.045 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.045 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.046 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.047 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.048 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.048 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.049 DEBUG:boto:Data: 2023-10-12T01:04:24.050 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.051 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.052 DEBUG:boto:Params: {} 2023-10-12T01:04:24.053 DEBUG:boto:Token: None 2023-10-12T01:04:24.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP2rHxujWg43bKfg2E+NpprkL/U= 2023-10-12T01:04:24.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP2rHxujWg43bKfg2E+NpprkL/U=', 'Content-Length': '0'} 2023-10-12T01:04:24.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcdb4ade0eb6f2d9-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:04:24.060 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.060 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.061 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.062 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.063 DEBUG:boto:Method: GET 2023-10-12T01:04:24.063 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.064 DEBUG:boto:Data: 2023-10-12T01:04:24.065 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.067 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.067 DEBUG:boto:Params: {} 2023-10-12T01:04:24.068 DEBUG:boto:Token: None 2023-10-12T01:04:24.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc= 2023-10-12T01:04:24.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc=', 'Content-Length': '0'} 2023-10-12T01:04:24.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068c0fa5107e3d0f-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.074 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.075 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.076 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.077 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.077 DEBUG:boto:Method: GET 2023-10-12T01:04:24.078 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.079 DEBUG:boto:Data: 2023-10-12T01:04:24.080 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.081 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.082 DEBUG:boto:Params: {} 2023-10-12T01:04:24.083 DEBUG:boto:Token: None 2023-10-12T01:04:24.084 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:04:24.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc= 2023-10-12T01:04:24.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tkPBSTOrdBDIpqHI8NKBRYKttMc=', 'Content-Length': '0'} 2023-10-12T01:04:24.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636b3d049d3a9f05-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.130 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.131 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.132 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.133 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.133 DEBUG:boto:Method: GET 2023-10-12T01:04:24.134 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.135 DEBUG:boto:Data: 2023-10-12T01:04:24.135 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.137 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.137 DEBUG:boto:Params: {} 2023-10-12T01:04:24.138 DEBUG:boto:Token: None 2023-10-12T01:04:24.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3b74f94c0de764-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.145 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.146 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.147 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.148 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.148 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.149 DEBUG:boto:Method: GET 2023-10-12T01:04:24.150 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.151 DEBUG:boto:Data: 2023-10-12T01:04:24.151 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.153 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.154 DEBUG:boto:Params: {} 2023-10-12T01:04:24.155 DEBUG:boto:Token: None 2023-10-12T01:04:24.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c8b0be9e764336-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.162 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.163 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.164 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.165 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.166 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:04:24.166 DEBUG:boto:Data: 2023-10-12T01:04:24.167 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.168 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.169 DEBUG:boto:Params: {} 2023-10-12T01:04:24.170 DEBUG:boto:Token: None 2023-10-12T01:04:24.171 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51 2023-10-12T01:04:24.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IKF24lF/5SzagjAmSPK0WGWweew= 2023-10-12T01:04:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IKF24lF/5SzagjAmSPK0WGWweew=', 'Content-Length': '0'} 2023-10-12T01:04:24.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42abca3f2e53966-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.176 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.177 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.178 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.178 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:04:24.179 DEBUG:boto:Data: 2023-10-12T01:04:24.180 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.182 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.183 DEBUG:boto:Params: {} 2023-10-12T01:04:24.183 DEBUG:boto:Token: None 2023-10-12T01:04:24.184 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51 2023-10-12T01:04:24.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IKF24lF/5SzagjAmSPK0WGWweew= 2023-10-12T01:04:24.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IKF24lF/5SzagjAmSPK0WGWweew=', 'Content-Length': '0'} 2023-10-12T01:04:24.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a648c54d4f2808-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:04:24.188 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.188 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.189 DEBUG:boto:Method: GET 2023-10-12T01:04:24.189 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:04:24.190 DEBUG:boto:Data: 2023-10-12T01:04:24.190 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.191 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.192 DEBUG:boto:Params: {} 2023-10-12T01:04:24.192 DEBUG:boto:Token: None 2023-10-12T01:04:24.193 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51 2023-10-12T01:04:24.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S/jS/QpQ5czGJcNy0avrFB9EMWQ= 2023-10-12T01:04:24.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S/jS/QpQ5czGJcNy0avrFB9EMWQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac890ef6024d0da-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.238 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:04:24.239 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:04:24.239 DEBUG:boto:Method: GET 2023-10-12T01:04:24.240 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:04:24.240 DEBUG:boto:Data: 2023-10-12T01:04:24.241 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.242 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.242 DEBUG:boto:Params: {} 2023-10-12T01:04:24.243 DEBUG:boto:Token: None 2023-10-12T01:04:24.243 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj51 2023-10-12T01:04:24.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S/jS/QpQ5czGJcNy0avrFB9EMWQ= 2023-10-12T01:04:24.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S/jS/QpQ5czGJcNy0avrFB9EMWQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d3afe1d034f46e-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.247 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.248 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.248 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.249 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.249 DEBUG:boto:Method: GET 2023-10-12T01:04:24.250 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.250 DEBUG:boto:Data: 2023-10-12T01:04:24.251 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.251 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.252 DEBUG:boto:Params: {} 2023-10-12T01:04:24.252 DEBUG:boto:Token: None 2023-10-12T01:04:24.253 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d9fb093c7a8070-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.257 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.258 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.258 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.259 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.259 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.260 DEBUG:boto:Method: GET 2023-10-12T01:04:24.260 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:04:24.260 DEBUG:boto:Data: 2023-10-12T01:04:24.261 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.262 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.262 DEBUG:boto:Params: {} 2023-10-12T01:04:24.263 DEBUG:boto:Token: None 2023-10-12T01:04:24.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78ea5070db1127e-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.268 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:04:24.269 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.269 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.270 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.270 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.271 DEBUG:boto:Method: GET 2023-10-12T01:04:24.271 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.271 DEBUG:boto:Data: 2023-10-12T01:04:24.272 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.273 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.273 DEBUG:boto:Params: {} 2023-10-12T01:04:24.274 DEBUG:boto:Token: None 2023-10-12T01:04:24.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY= 2023-10-12T01:04:24.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY=', 'Content-Length': '0'} 2023-10-12T01:04:24.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea5e14afe3f516c-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.278 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.279 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.279 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.279 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.280 DEBUG:boto:Method: GET 2023-10-12T01:04:24.280 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.281 DEBUG:boto:Data: 2023-10-12T01:04:24.281 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.282 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.282 DEBUG:boto:Params: {} 2023-10-12T01:04:24.283 DEBUG:boto:Token: None 2023-10-12T01:04:24.283 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY= 2023-10-12T01:04:24.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY=', 'Content-Length': '0'} 2023-10-12T01:04:24.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008098090f78979ac7-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.330 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.331 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.331 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.332 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.332 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.333 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.333 DEBUG:boto:Data: 2023-10-12T01:04:24.333 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.334 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.335 DEBUG:boto:Params: {} 2023-10-12T01:04:24.335 DEBUG:boto:Token: None 2023-10-12T01:04:24.336 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kWp1eJOhpLpyy6eJkqDNJ+XSwjE= 2023-10-12T01:04:24.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kWp1eJOhpLpyy6eJkqDNJ+XSwjE=', 'Content-Length': '0'} 2023-10-12T01:04:24.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d10c89b3d99a76-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.339 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.340 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.340 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.341 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.341 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.342 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.342 DEBUG:boto:Data: 2023-10-12T01:04:24.343 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.343 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.344 DEBUG:boto:Params: {} 2023-10-12T01:04:24.344 DEBUG:boto:Token: None 2023-10-12T01:04:24.345 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.345 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kWp1eJOhpLpyy6eJkqDNJ+XSwjE= 2023-10-12T01:04:24.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kWp1eJOhpLpyy6eJkqDNJ+XSwjE=', 'Content-Length': '0'} 2023-10-12T01:04:24.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005acf7b526c164117-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:04:24.349 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.349 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.350 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.350 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.350 DEBUG:boto:Method: GET 2023-10-12T01:04:24.351 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.351 DEBUG:boto:Data: 2023-10-12T01:04:24.352 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.353 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.353 DEBUG:boto:Params: {} 2023-10-12T01:04:24.354 DEBUG:boto:Token: None 2023-10-12T01:04:24.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY= 2023-10-12T01:04:24.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY=', 'Content-Length': '0'} 2023-10-12T01:04:24.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058c410ddfd4fdbd-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.358 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.358 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.359 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.359 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.360 DEBUG:boto:Method: GET 2023-10-12T01:04:24.360 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.361 DEBUG:boto:Data: 2023-10-12T01:04:24.361 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.362 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.363 DEBUG:boto:Params: {} 2023-10-12T01:04:24.363 DEBUG:boto:Token: None 2023-10-12T01:04:24.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:04:24.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY= 2023-10-12T01:04:24.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OBFib2vgvsMC+X4fErBcsiOVJrY=', 'Content-Length': '0'} 2023-10-12T01:04:24.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bfa5ea8b4d24363-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.368 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.368 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.368 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.369 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.369 DEBUG:boto:Method: GET 2023-10-12T01:04:24.370 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.370 DEBUG:boto:Data: 2023-10-12T01:04:24.371 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.372 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.372 DEBUG:boto:Params: {} 2023-10-12T01:04:24.372 DEBUG:boto:Token: None 2023-10-12T01:04:24.373 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cf923719d2856cb-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.377 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.378 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.378 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.378 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.379 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.379 DEBUG:boto:Method: GET 2023-10-12T01:04:24.380 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.380 DEBUG:boto:Data: 2023-10-12T01:04:24.381 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.381 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.382 DEBUG:boto:Params: {} 2023-10-12T01:04:24.382 DEBUG:boto:Token: None 2023-10-12T01:04:24.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef143c2af6a72ccc-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.388 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.388 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.389 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.389 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.390 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:04:24.390 DEBUG:boto:Data: 2023-10-12T01:04:24.391 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.391 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.392 DEBUG:boto:Params: {} 2023-10-12T01:04:24.392 DEBUG:boto:Token: None 2023-10-12T01:04:24.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55 2023-10-12T01:04:24.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B4ZHQXXeUP36GBTHTW66dylLWxg= 2023-10-12T01:04:24.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4ZHQXXeUP36GBTHTW66dylLWxg=', 'Content-Length': '0'} 2023-10-12T01:04:24.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0633f17061f481-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.396 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.396 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.397 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.397 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:04:24.398 DEBUG:boto:Data: 2023-10-12T01:04:24.398 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.399 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.399 DEBUG:boto:Params: {} 2023-10-12T01:04:24.400 DEBUG:boto:Token: None 2023-10-12T01:04:24.400 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55 2023-10-12T01:04:24.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B4ZHQXXeUP36GBTHTW66dylLWxg= 2023-10-12T01:04:24.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4ZHQXXeUP36GBTHTW66dylLWxg=', 'Content-Length': '0'} 2023-10-12T01:04:24.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f256670444fe24-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:04:24.405 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.405 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.406 DEBUG:boto:Method: GET 2023-10-12T01:04:24.406 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:04:24.406 DEBUG:boto:Data: 2023-10-12T01:04:24.407 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.408 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.408 DEBUG:boto:Params: {} 2023-10-12T01:04:24.409 DEBUG:boto:Token: None 2023-10-12T01:04:24.409 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55 2023-10-12T01:04:24.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ynZ+zlGBg6inhBGoQkdPks0haQ= 2023-10-12T01:04:24.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ynZ+zlGBg6inhBGoQkdPks0haQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b243c48335289ae8-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.413 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:04:24.413 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:04:24.414 DEBUG:boto:Method: GET 2023-10-12T01:04:24.414 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:04:24.415 DEBUG:boto:Data: 2023-10-12T01:04:24.415 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.416 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.417 DEBUG:boto:Params: {} 2023-10-12T01:04:24.417 DEBUG:boto:Token: None 2023-10-12T01:04:24.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj55 2023-10-12T01:04:24.418 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ynZ+zlGBg6inhBGoQkdPks0haQ= 2023-10-12T01:04:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ynZ+zlGBg6inhBGoQkdPks0haQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c893e613a98ae13-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.462 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.463 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.463 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.464 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.464 DEBUG:boto:Method: GET 2023-10-12T01:04:24.465 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.465 DEBUG:boto:Data: 2023-10-12T01:04:24.466 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.466 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.467 DEBUG:boto:Params: {} 2023-10-12T01:04:24.468 DEBUG:boto:Token: None 2023-10-12T01:04:24.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058cfcf2d83178d58-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.472 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.473 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.473 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.474 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.474 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.475 DEBUG:boto:Method: GET 2023-10-12T01:04:24.475 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:04:24.476 DEBUG:boto:Data: 2023-10-12T01:04:24.476 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.477 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.478 DEBUG:boto:Params: {} 2023-10-12T01:04:24.478 DEBUG:boto:Token: None 2023-10-12T01:04:24.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038654c0b23329af8-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.484 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:04:24.485 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.485 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.486 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.486 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.487 DEBUG:boto:Method: GET 2023-10-12T01:04:24.487 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.487 DEBUG:boto:Data: 2023-10-12T01:04:24.488 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.489 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.489 DEBUG:boto:Params: {} 2023-10-12T01:04:24.490 DEBUG:boto:Token: None 2023-10-12T01:04:24.490 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.491 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ= 2023-10-12T01:04:24.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387a21d3d29e0082-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.534 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.535 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.535 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.536 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.536 DEBUG:boto:Method: GET 2023-10-12T01:04:24.537 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.537 DEBUG:boto:Data: 2023-10-12T01:04:24.537 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.538 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.539 DEBUG:boto:Params: {} 2023-10-12T01:04:24.539 DEBUG:boto:Token: None 2023-10-12T01:04:24.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ= 2023-10-12T01:04:24.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099917b71d63cb64-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.543 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.544 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.544 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.545 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.545 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.545 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.546 DEBUG:boto:Data: 2023-10-12T01:04:24.546 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.547 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.547 DEBUG:boto:Params: {} 2023-10-12T01:04:24.548 DEBUG:boto:Token: None 2023-10-12T01:04:24.548 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.549 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fIHgPKoLpCOO8XhFKsKl1ijGM+M= 2023-10-12T01:04:24.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fIHgPKoLpCOO8XhFKsKl1ijGM+M=', 'Content-Length': '0'} 2023-10-12T01:04:24.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f458f5ee27ad940-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.552 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.552 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.552 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.553 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.553 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.554 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.554 DEBUG:boto:Data: 2023-10-12T01:04:24.554 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.555 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.555 DEBUG:boto:Params: {} 2023-10-12T01:04:24.556 DEBUG:boto:Token: None 2023-10-12T01:04:24.556 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fIHgPKoLpCOO8XhFKsKl1ijGM+M= 2023-10-12T01:04:24.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fIHgPKoLpCOO8XhFKsKl1ijGM+M=', 'Content-Length': '0'} 2023-10-12T01:04:24.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3635ec8345a83d-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:04:24.560 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.561 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.561 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.561 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.562 DEBUG:boto:Method: GET 2023-10-12T01:04:24.562 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.563 DEBUG:boto:Data: 2023-10-12T01:04:24.563 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.564 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.564 DEBUG:boto:Params: {} 2023-10-12T01:04:24.565 DEBUG:boto:Token: None 2023-10-12T01:04:24.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ= 2023-10-12T01:04:24.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a0a61dcb74d9ac-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.569 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.569 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.569 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.570 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.570 DEBUG:boto:Method: GET 2023-10-12T01:04:24.571 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.571 DEBUG:boto:Data: 2023-10-12T01:04:24.571 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.572 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.573 DEBUG:boto:Params: {} 2023-10-12T01:04:24.573 DEBUG:boto:Token: None 2023-10-12T01:04:24.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:04:24.574 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ= 2023-10-12T01:04:24.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tKBIAPu9K+ZiObKIT8uBWgmCtwQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006364319b805a97a4-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.618 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.619 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.620 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.620 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.620 DEBUG:boto:Method: GET 2023-10-12T01:04:24.621 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.621 DEBUG:boto:Data: 2023-10-12T01:04:24.622 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.623 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.623 DEBUG:boto:Params: {} 2023-10-12T01:04:24.623 DEBUG:boto:Token: None 2023-10-12T01:04:24.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000403933ced60ceda9-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.628 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.629 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.629 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.630 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.630 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.631 DEBUG:boto:Method: GET 2023-10-12T01:04:24.631 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.631 DEBUG:boto:Data: 2023-10-12T01:04:24.632 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.633 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.633 DEBUG:boto:Params: {} 2023-10-12T01:04:24.633 DEBUG:boto:Token: None 2023-10-12T01:04:24.634 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec73b4d20efd7f60-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.639 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.640 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.640 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.641 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.641 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:04:24.642 DEBUG:boto:Data: 2023-10-12T01:04:24.642 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.643 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.643 DEBUG:boto:Params: {} 2023-10-12T01:04:24.644 DEBUG:boto:Token: None 2023-10-12T01:04:24.644 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57 2023-10-12T01:04:24.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bYBjK64/V9Qu9f+Jxl7+JOH/cEQ= 2023-10-12T01:04:24.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bYBjK64/V9Qu9f+Jxl7+JOH/cEQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438058093027596f-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.648 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.648 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.649 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.649 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:04:24.650 DEBUG:boto:Data: 2023-10-12T01:04:24.650 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.651 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.651 DEBUG:boto:Params: {} 2023-10-12T01:04:24.652 DEBUG:boto:Token: None 2023-10-12T01:04:24.652 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57 2023-10-12T01:04:24.653 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bYBjK64/V9Qu9f+Jxl7+JOH/cEQ= 2023-10-12T01:04:24.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bYBjK64/V9Qu9f+Jxl7+JOH/cEQ=', 'Content-Length': '0'} 2023-10-12T01:04:24.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b164f16c6ebb1203-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:04:24.656 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.657 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.657 DEBUG:boto:Method: GET 2023-10-12T01:04:24.657 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:04:24.658 DEBUG:boto:Data: 2023-10-12T01:04:24.658 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.659 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.660 DEBUG:boto:Params: {} 2023-10-12T01:04:24.660 DEBUG:boto:Token: None 2023-10-12T01:04:24.661 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57 2023-10-12T01:04:24.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DtCwAvszPQE2OB/oDwuLkyns58s= 2023-10-12T01:04:24.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DtCwAvszPQE2OB/oDwuLkyns58s=', 'Content-Length': '0'} 2023-10-12T01:04:24.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3db8ab2b572245c-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.710 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:04:24.711 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:04:24.711 DEBUG:boto:Method: GET 2023-10-12T01:04:24.712 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:04:24.712 DEBUG:boto:Data: 2023-10-12T01:04:24.713 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.714 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.714 DEBUG:boto:Params: {} 2023-10-12T01:04:24.714 DEBUG:boto:Token: None 2023-10-12T01:04:24.715 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj57 2023-10-12T01:04:24.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DtCwAvszPQE2OB/oDwuLkyns58s= 2023-10-12T01:04:24.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DtCwAvszPQE2OB/oDwuLkyns58s=', 'Content-Length': '0'} 2023-10-12T01:04:24.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2e84df3f73e555-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.722 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.722 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.723 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.723 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.723 DEBUG:boto:Method: GET 2023-10-12T01:04:24.724 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.724 DEBUG:boto:Data: 2023-10-12T01:04:24.725 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.725 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.726 DEBUG:boto:Params: {} 2023-10-12T01:04:24.726 DEBUG:boto:Token: None 2023-10-12T01:04:24.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1eec3489cf55028-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.731 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.732 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.732 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.733 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.733 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.734 DEBUG:boto:Method: GET 2023-10-12T01:04:24.734 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:04:24.734 DEBUG:boto:Data: 2023-10-12T01:04:24.735 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.736 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.736 DEBUG:boto:Params: {} 2023-10-12T01:04:24.736 DEBUG:boto:Token: None 2023-10-12T01:04:24.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b179ba15176b4c4d-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.742 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:04:24.743 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.743 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.744 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.744 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.745 DEBUG:boto:Method: GET 2023-10-12T01:04:24.745 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.745 DEBUG:boto:Data: 2023-10-12T01:04:24.746 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.747 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.747 DEBUG:boto:Params: {} 2023-10-12T01:04:24.748 DEBUG:boto:Token: None 2023-10-12T01:04:24.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo= 2023-10-12T01:04:24.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo=', 'Content-Length': '0'} 2023-10-12T01:04:24.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa7bf3e8b5aa08e-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.751 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.752 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.752 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.753 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.753 DEBUG:boto:Method: GET 2023-10-12T01:04:24.753 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.754 DEBUG:boto:Data: 2023-10-12T01:04:24.754 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.755 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.756 DEBUG:boto:Params: {} 2023-10-12T01:04:24.756 DEBUG:boto:Token: None 2023-10-12T01:04:24.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo= 2023-10-12T01:04:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo=', 'Content-Length': '0'} 2023-10-12T01:04:24.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044013fe44c0f7041-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.761 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.761 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.762 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.762 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.762 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.763 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.763 DEBUG:boto:Data: 2023-10-12T01:04:24.764 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.765 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.765 DEBUG:boto:Params: {} 2023-10-12T01:04:24.765 DEBUG:boto:Token: None 2023-10-12T01:04:24.766 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v+MN6AdJfX0KY5cD8ZNeOGFIge8= 2023-10-12T01:04:24.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v+MN6AdJfX0KY5cD8ZNeOGFIge8=', 'Content-Length': '0'} 2023-10-12T01:04:24.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c79018146c378a-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.769 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.770 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.770 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.771 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.771 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.771 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.772 DEBUG:boto:Data: 2023-10-12T01:04:24.772 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.773 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.773 DEBUG:boto:Params: {} 2023-10-12T01:04:24.774 DEBUG:boto:Token: None 2023-10-12T01:04:24.774 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v+MN6AdJfX0KY5cD8ZNeOGFIge8= 2023-10-12T01:04:24.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v+MN6AdJfX0KY5cD8ZNeOGFIge8=', 'Content-Length': '0'} 2023-10-12T01:04:24.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7be9c66e7286c5-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:04:24.779 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.779 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.779 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.780 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.780 DEBUG:boto:Method: GET 2023-10-12T01:04:24.781 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.781 DEBUG:boto:Data: 2023-10-12T01:04:24.781 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.782 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.783 DEBUG:boto:Params: {} 2023-10-12T01:04:24.783 DEBUG:boto:Token: None 2023-10-12T01:04:24.784 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo= 2023-10-12T01:04:24.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo=', 'Content-Length': '0'} 2023-10-12T01:04:24.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb705e03eed084e-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.830 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.831 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.831 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.832 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.832 DEBUG:boto:Method: GET 2023-10-12T01:04:24.833 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.833 DEBUG:boto:Data: 2023-10-12T01:04:24.833 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.834 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.835 DEBUG:boto:Params: {} 2023-10-12T01:04:24.835 DEBUG:boto:Token: None 2023-10-12T01:04:24.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:04:24.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo= 2023-10-12T01:04:24.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7HjnTzN35vgZNZ1j850LaZOkXo=', 'Content-Length': '0'} 2023-10-12T01:04:24.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6e92eacb256ce1-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.839 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.840 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.840 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.840 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.841 DEBUG:boto:Method: GET 2023-10-12T01:04:24.841 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.842 DEBUG:boto:Data: 2023-10-12T01:04:24.842 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.843 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.843 DEBUG:boto:Params: {} 2023-10-12T01:04:24.844 DEBUG:boto:Token: None 2023-10-12T01:04:24.844 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aced5be972be9f4-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.848 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.849 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.850 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.850 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.850 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.851 DEBUG:boto:Method: GET 2023-10-12T01:04:24.851 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.852 DEBUG:boto:Data: 2023-10-12T01:04:24.852 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.853 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.853 DEBUG:boto:Params: {} 2023-10-12T01:04:24.854 DEBUG:boto:Token: None 2023-10-12T01:04:24.854 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.855 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f1940e0a9956d9-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.906 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.907 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.908 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.908 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.908 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:04:24.909 DEBUG:boto:Data: 2023-10-12T01:04:24.909 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.910 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.911 DEBUG:boto:Params: {} 2023-10-12T01:04:24.911 DEBUG:boto:Token: None 2023-10-12T01:04:24.912 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68 2023-10-12T01:04:24.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QmlQTE0J1k4iyv4hRoSMw/cVp3U= 2023-10-12T01:04:24.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QmlQTE0J1k4iyv4hRoSMw/cVp3U=', 'Content-Length': '0'} 2023-10-12T01:04:24.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000616c172679188d8-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.915 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.916 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.916 DEBUG:boto:Method: HEAD 2023-10-12T01:04:24.917 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:04:24.917 DEBUG:boto:Data: 2023-10-12T01:04:24.917 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.918 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.919 DEBUG:boto:Params: {} 2023-10-12T01:04:24.919 DEBUG:boto:Token: None 2023-10-12T01:04:24.921 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68 2023-10-12T01:04:24.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QmlQTE0J1k4iyv4hRoSMw/cVp3U= 2023-10-12T01:04:24.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QmlQTE0J1k4iyv4hRoSMw/cVp3U=', 'Content-Length': '0'} 2023-10-12T01:04:24.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000befcc6c58ecb9854-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:04:24.925 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.925 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.926 DEBUG:boto:Method: GET 2023-10-12T01:04:24.926 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:04:24.927 DEBUG:boto:Data: 2023-10-12T01:04:24.927 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.928 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.928 DEBUG:boto:Params: {} 2023-10-12T01:04:24.929 DEBUG:boto:Token: None 2023-10-12T01:04:24.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68 2023-10-12T01:04:24.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V9aOKlzYs+ac2ctHbCSneWRuiS8= 2023-10-12T01:04:24.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V9aOKlzYs+ac2ctHbCSneWRuiS8=', 'Content-Length': '0'} 2023-10-12T01:04:24.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cb3aa981c71c41-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.933 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:04:24.933 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:04:24.934 DEBUG:boto:Method: GET 2023-10-12T01:04:24.934 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:04:24.935 DEBUG:boto:Data: 2023-10-12T01:04:24.935 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.936 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.936 DEBUG:boto:Params: {} 2023-10-12T01:04:24.937 DEBUG:boto:Token: None 2023-10-12T01:04:24.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj68 2023-10-12T01:04:24.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V9aOKlzYs+ac2ctHbCSneWRuiS8= 2023-10-12T01:04:24.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V9aOKlzYs+ac2ctHbCSneWRuiS8=', 'Content-Length': '0'} 2023-10-12T01:04:24.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c317b6d7876e7965-0065274618-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.941 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.942 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.942 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.943 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.943 DEBUG:boto:Method: GET 2023-10-12T01:04:24.944 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.944 DEBUG:boto:Data: 2023-10-12T01:04:24.944 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.945 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.946 DEBUG:boto:Params: {} 2023-10-12T01:04:24.946 DEBUG:boto:Token: None 2023-10-12T01:04:24.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1fea0cb7fe60536-0065274618-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.951 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.951 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:24.952 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:24.952 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.953 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.953 DEBUG:boto:Method: GET 2023-10-12T01:04:24.953 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:04:24.954 DEBUG:boto:Data: 2023-10-12T01:04:24.954 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:24.955 DEBUG:boto:Port: 8000 2023-10-12T01:04:24.956 DEBUG:boto:Params: {} 2023-10-12T01:04:24.956 DEBUG:boto:Token: None 2023-10-12T01:04:24.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/ 2023-10-12T01:04:24.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI= 2023-10-12T01:04:24.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNlOsioe5ALKo4dGUiu0oUB8bKI=', 'Content-Length': '0'} 2023-10-12T01:04:24.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b7fdb27a035ba0-0065274618-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:24.962 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:24.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:04:24.963 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:24.963 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:24.964 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:24.964 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:24.964 DEBUG:boto:Method: GET 2023-10-12T01:04:24.965 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:24.965 DEBUG:boto:Data: 2023-10-12T01:04:24.966 DEBUG:boto:Headers: {} 2023-10-12T01:04:24.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:24.967 DEBUG:boto:Port: 8001 2023-10-12T01:04:24.967 DEBUG:boto:Params: {} 2023-10-12T01:04:24.968 DEBUG:boto:Token: None 2023-10-12T01:04:24.968 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:24 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:24.969 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nw/+p0mGSj5vEQD5DguXjqFCQn8= 2023-10-12T01:04:24.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nw/+p0mGSj5vEQD5DguXjqFCQn8=', 'Content-Length': '0'} 2023-10-12T01:04:24.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873bf1f8c4287196-0065274618-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.018 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.019 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.019 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.020 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.020 DEBUG:boto:Method: GET 2023-10-12T01:04:25.021 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.021 DEBUG:boto:Data: 2023-10-12T01:04:25.022 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.022 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.023 DEBUG:boto:Params: {} 2023-10-12T01:04:25.023 DEBUG:boto:Token: None 2023-10-12T01:04:25.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk= 2023-10-12T01:04:25.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk=', 'Content-Length': '0'} 2023-10-12T01:04:25.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000374216decacf757-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.028 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.028 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.029 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.029 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.030 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.030 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.030 DEBUG:boto:Data: 2023-10-12T01:04:25.031 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.032 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.032 DEBUG:boto:Params: {} 2023-10-12T01:04:25.032 DEBUG:boto:Token: None 2023-10-12T01:04:25.033 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HjzoyfC2PxfxtEumUui8cQHdn/I= 2023-10-12T01:04:25.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HjzoyfC2PxfxtEumUui8cQHdn/I=', 'Content-Length': '0'} 2023-10-12T01:04:25.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8e7ef0f8285e54-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.036 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.036 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.037 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.037 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.038 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.038 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.038 DEBUG:boto:Data: 2023-10-12T01:04:25.039 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.040 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.040 DEBUG:boto:Params: {} 2023-10-12T01:04:25.041 DEBUG:boto:Token: None 2023-10-12T01:04:25.041 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HjzoyfC2PxfxtEumUui8cQHdn/I= 2023-10-12T01:04:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HjzoyfC2PxfxtEumUui8cQHdn/I=', 'Content-Length': '0'} 2023-10-12T01:04:25.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c688800d3e4431-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:04:25.045 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.045 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.046 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.046 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.047 DEBUG:boto:Method: GET 2023-10-12T01:04:25.047 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.047 DEBUG:boto:Data: 2023-10-12T01:04:25.048 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.049 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.049 DEBUG:boto:Params: {} 2023-10-12T01:04:25.050 DEBUG:boto:Token: None 2023-10-12T01:04:25.050 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk= 2023-10-12T01:04:25.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk=', 'Content-Length': '0'} 2023-10-12T01:04:25.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c6da1d1b193cd5-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.054 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.054 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.055 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.055 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.056 DEBUG:boto:Method: GET 2023-10-12T01:04:25.056 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.057 DEBUG:boto:Data: 2023-10-12T01:04:25.057 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.058 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.058 DEBUG:boto:Params: {} 2023-10-12T01:04:25.059 DEBUG:boto:Token: None 2023-10-12T01:04:25.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:04:25.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk= 2023-10-12T01:04:25.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HfMJOsJhXDDs+trGAqPUnyEcEgk=', 'Content-Length': '0'} 2023-10-12T01:04:25.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0893d29a1fa9c5a-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.063 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.063 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.064 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.064 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.065 DEBUG:boto:Method: GET 2023-10-12T01:04:25.065 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.065 DEBUG:boto:Data: 2023-10-12T01:04:25.066 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.067 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.067 DEBUG:boto:Params: {} 2023-10-12T01:04:25.068 DEBUG:boto:Token: None 2023-10-12T01:04:25.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5132a44e62ef1fe-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.072 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.073 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.073 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.074 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.074 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.075 DEBUG:boto:Method: GET 2023-10-12T01:04:25.075 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.076 DEBUG:boto:Data: 2023-10-12T01:04:25.076 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.077 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.077 DEBUG:boto:Params: {} 2023-10-12T01:04:25.078 DEBUG:boto:Token: None 2023-10-12T01:04:25.078 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30a693434db5d94-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.126 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.127 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.128 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.128 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.128 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:04:25.129 DEBUG:boto:Data: 2023-10-12T01:04:25.129 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.130 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.130 DEBUG:boto:Params: {} 2023-10-12T01:04:25.131 DEBUG:boto:Token: None 2023-10-12T01:04:25.131 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69 2023-10-12T01:04:25.132 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LRvu8kgwHNQhrsLOo42vA5pWRpk= 2023-10-12T01:04:25.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LRvu8kgwHNQhrsLOo42vA5pWRpk=', 'Content-Length': '0'} 2023-10-12T01:04:25.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb0086f5d3432f1-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.134 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.135 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.135 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.136 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:04:25.136 DEBUG:boto:Data: 2023-10-12T01:04:25.136 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.137 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.138 DEBUG:boto:Params: {} 2023-10-12T01:04:25.138 DEBUG:boto:Token: None 2023-10-12T01:04:25.138 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69 2023-10-12T01:04:25.139 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LRvu8kgwHNQhrsLOo42vA5pWRpk= 2023-10-12T01:04:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LRvu8kgwHNQhrsLOo42vA5pWRpk=', 'Content-Length': '0'} 2023-10-12T01:04:25.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7f7cdd37c08705-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:04:25.142 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.143 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.143 DEBUG:boto:Method: GET 2023-10-12T01:04:25.144 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:04:25.144 DEBUG:boto:Data: 2023-10-12T01:04:25.145 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.145 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.146 DEBUG:boto:Params: {} 2023-10-12T01:04:25.146 DEBUG:boto:Token: None 2023-10-12T01:04:25.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69 2023-10-12T01:04:25.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8EfsJF3ATx3HKgYIDxpEe9ol47w= 2023-10-12T01:04:25.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8EfsJF3ATx3HKgYIDxpEe9ol47w=', 'Content-Length': '0'} 2023-10-12T01:04:25.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dea8e834cbb2ee8-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.151 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:04:25.151 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:04:25.152 DEBUG:boto:Method: GET 2023-10-12T01:04:25.152 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:04:25.152 DEBUG:boto:Data: 2023-10-12T01:04:25.153 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.153 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.154 DEBUG:boto:Params: {} 2023-10-12T01:04:25.154 DEBUG:boto:Token: None 2023-10-12T01:04:25.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj69 2023-10-12T01:04:25.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8EfsJF3ATx3HKgYIDxpEe9ol47w= 2023-10-12T01:04:25.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8EfsJF3ATx3HKgYIDxpEe9ol47w=', 'Content-Length': '0'} 2023-10-12T01:04:25.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312e13fbcb8c7096-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.158 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.159 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.159 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.160 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.160 DEBUG:boto:Method: GET 2023-10-12T01:04:25.161 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.161 DEBUG:boto:Data: 2023-10-12T01:04:25.161 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.162 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.163 DEBUG:boto:Params: {} 2023-10-12T01:04:25.163 DEBUG:boto:Token: None 2023-10-12T01:04:25.163 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fc9713b25600ae-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.167 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.168 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.169 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.169 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.169 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.170 DEBUG:boto:Method: GET 2023-10-12T01:04:25.170 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:04:25.171 DEBUG:boto:Data: 2023-10-12T01:04:25.171 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.172 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.172 DEBUG:boto:Params: {} 2023-10-12T01:04:25.173 DEBUG:boto:Token: None 2023-10-12T01:04:25.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d80c92f4e2dc5bb-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.178 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:04:25.179 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.179 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.180 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.180 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.180 DEBUG:boto:Method: GET 2023-10-12T01:04:25.181 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.181 DEBUG:boto:Data: 2023-10-12T01:04:25.182 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.183 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.183 DEBUG:boto:Params: {} 2023-10-12T01:04:25.184 DEBUG:boto:Token: None 2023-10-12T01:04:25.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc= 2023-10-12T01:04:25.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc=', 'Content-Length': '0'} 2023-10-12T01:04:25.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b90ae9c012eb834-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.188 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.188 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.189 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.189 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.189 DEBUG:boto:Method: GET 2023-10-12T01:04:25.190 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.190 DEBUG:boto:Data: 2023-10-12T01:04:25.191 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.192 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.192 DEBUG:boto:Params: {} 2023-10-12T01:04:25.192 DEBUG:boto:Token: None 2023-10-12T01:04:25.193 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc= 2023-10-12T01:04:25.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc=', 'Content-Length': '0'} 2023-10-12T01:04:25.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e8a64cc45963a00-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.197 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.197 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.198 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.198 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.198 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.199 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.199 DEBUG:boto:Data: 2023-10-12T01:04:25.200 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.200 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.201 DEBUG:boto:Params: {} 2023-10-12T01:04:25.201 DEBUG:boto:Token: None 2023-10-12T01:04:25.202 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.202 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KdBHKo62Spo47uXQW8VrS+L9nfA= 2023-10-12T01:04:25.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KdBHKo62Spo47uXQW8VrS+L9nfA=', 'Content-Length': '0'} 2023-10-12T01:04:25.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600c699d1da38925-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.205 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.206 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.206 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.207 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.207 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.207 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.208 DEBUG:boto:Data: 2023-10-12T01:04:25.208 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.209 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.209 DEBUG:boto:Params: {} 2023-10-12T01:04:25.209 DEBUG:boto:Token: None 2023-10-12T01:04:25.210 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KdBHKo62Spo47uXQW8VrS+L9nfA= 2023-10-12T01:04:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KdBHKo62Spo47uXQW8VrS+L9nfA=', 'Content-Length': '0'} 2023-10-12T01:04:25.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086847a9a2d0a9409-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:04:25.214 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.214 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.215 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.215 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.216 DEBUG:boto:Method: GET 2023-10-12T01:04:25.216 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.217 DEBUG:boto:Data: 2023-10-12T01:04:25.217 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.218 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.218 DEBUG:boto:Params: {} 2023-10-12T01:04:25.219 DEBUG:boto:Token: None 2023-10-12T01:04:25.219 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc= 2023-10-12T01:04:25.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc=', 'Content-Length': '0'} 2023-10-12T01:04:25.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e5a4ec45c6b2c9-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.262 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.263 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.263 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.264 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.264 DEBUG:boto:Method: GET 2023-10-12T01:04:25.265 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.265 DEBUG:boto:Data: 2023-10-12T01:04:25.265 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.266 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.267 DEBUG:boto:Params: {} 2023-10-12T01:04:25.267 DEBUG:boto:Token: None 2023-10-12T01:04:25.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:04:25.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc= 2023-10-12T01:04:25.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oY1zf4ekzB+T/+VSxmulFEorDWc=', 'Content-Length': '0'} 2023-10-12T01:04:25.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ade7571faa148a-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.272 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.272 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.273 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.273 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.273 DEBUG:boto:Method: GET 2023-10-12T01:04:25.274 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.274 DEBUG:boto:Data: 2023-10-12T01:04:25.275 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.276 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.276 DEBUG:boto:Params: {} 2023-10-12T01:04:25.277 DEBUG:boto:Token: None 2023-10-12T01:04:25.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005caccf88b8287e42-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.281 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.282 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.282 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.283 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.283 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.283 DEBUG:boto:Method: GET 2023-10-12T01:04:25.284 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.284 DEBUG:boto:Data: 2023-10-12T01:04:25.285 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.285 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.286 DEBUG:boto:Params: {} 2023-10-12T01:04:25.286 DEBUG:boto:Token: None 2023-10-12T01:04:25.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df8db6b023894ea-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.292 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.292 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.293 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.293 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.294 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:04:25.294 DEBUG:boto:Data: 2023-10-12T01:04:25.294 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.295 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.296 DEBUG:boto:Params: {} 2023-10-12T01:04:25.296 DEBUG:boto:Token: None 2023-10-12T01:04:25.297 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76 2023-10-12T01:04:25.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yA2dto8n186QgaYc5qZxfuLG190= 2023-10-12T01:04:25.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yA2dto8n186QgaYc5qZxfuLG190=', 'Content-Length': '0'} 2023-10-12T01:04:25.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd6731332e2baf7-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.300 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.301 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.301 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.302 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:04:25.302 DEBUG:boto:Data: 2023-10-12T01:04:25.303 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.303 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.304 DEBUG:boto:Params: {} 2023-10-12T01:04:25.304 DEBUG:boto:Token: None 2023-10-12T01:04:25.305 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76 2023-10-12T01:04:25.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yA2dto8n186QgaYc5qZxfuLG190= 2023-10-12T01:04:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yA2dto8n186QgaYc5qZxfuLG190=', 'Content-Length': '0'} 2023-10-12T01:04:25.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e2691eb13d42ec-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:04:25.308 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.309 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.309 DEBUG:boto:Method: GET 2023-10-12T01:04:25.310 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:04:25.310 DEBUG:boto:Data: 2023-10-12T01:04:25.311 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.312 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.312 DEBUG:boto:Params: {} 2023-10-12T01:04:25.313 DEBUG:boto:Token: None 2023-10-12T01:04:25.313 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76 2023-10-12T01:04:25.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rjA/ODDXf+3EcdUrcNDtKkNB5tk= 2023-10-12T01:04:25.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rjA/ODDXf+3EcdUrcNDtKkNB5tk=', 'Content-Length': '0'} 2023-10-12T01:04:25.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a73aff0f879dd81-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.317 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:04:25.317 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:04:25.318 DEBUG:boto:Method: GET 2023-10-12T01:04:25.318 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:04:25.318 DEBUG:boto:Data: 2023-10-12T01:04:25.319 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.320 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.320 DEBUG:boto:Params: {} 2023-10-12T01:04:25.321 DEBUG:boto:Token: None 2023-10-12T01:04:25.321 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj76 2023-10-12T01:04:25.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rjA/ODDXf+3EcdUrcNDtKkNB5tk= 2023-10-12T01:04:25.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rjA/ODDXf+3EcdUrcNDtKkNB5tk=', 'Content-Length': '0'} 2023-10-12T01:04:25.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7450bf6af31bb4-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.366 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.367 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.367 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.368 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.368 DEBUG:boto:Method: GET 2023-10-12T01:04:25.369 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.369 DEBUG:boto:Data: 2023-10-12T01:04:25.370 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.370 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.371 DEBUG:boto:Params: {} 2023-10-12T01:04:25.371 DEBUG:boto:Token: None 2023-10-12T01:04:25.372 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003992535284f00bb2-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.376 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.376 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.377 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.377 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.378 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.378 DEBUG:boto:Method: GET 2023-10-12T01:04:25.379 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:04:25.379 DEBUG:boto:Data: 2023-10-12T01:04:25.379 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.380 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.381 DEBUG:boto:Params: {} 2023-10-12T01:04:25.381 DEBUG:boto:Token: None 2023-10-12T01:04:25.381 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.382 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd90886441ab34a-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.386 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:04:25.387 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.388 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.388 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.388 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.389 DEBUG:boto:Method: GET 2023-10-12T01:04:25.389 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.390 DEBUG:boto:Data: 2023-10-12T01:04:25.390 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.391 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.391 DEBUG:boto:Params: {} 2023-10-12T01:04:25.392 DEBUG:boto:Token: None 2023-10-12T01:04:25.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA= 2023-10-12T01:04:25.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA=', 'Content-Length': '0'} 2023-10-12T01:04:25.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012078aac12246ddc-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.396 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.397 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.397 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.398 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.398 DEBUG:boto:Method: GET 2023-10-12T01:04:25.398 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.399 DEBUG:boto:Data: 2023-10-12T01:04:25.399 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.400 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.401 DEBUG:boto:Params: {} 2023-10-12T01:04:25.401 DEBUG:boto:Token: None 2023-10-12T01:04:25.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA= 2023-10-12T01:04:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA=', 'Content-Length': '0'} 2023-10-12T01:04:25.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e01ba662ee3cc90-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.405 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.406 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.406 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.407 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.407 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.407 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.408 DEBUG:boto:Data: 2023-10-12T01:04:25.408 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.409 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.409 DEBUG:boto:Params: {} 2023-10-12T01:04:25.410 DEBUG:boto:Token: None 2023-10-12T01:04:25.410 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MPK+255DRwkzV89BuCSSHNQOIPY= 2023-10-12T01:04:25.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MPK+255DRwkzV89BuCSSHNQOIPY=', 'Content-Length': '0'} 2023-10-12T01:04:25.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026601cb2c6784dc1-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.413 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.414 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.414 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.415 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.415 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.415 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.416 DEBUG:boto:Data: 2023-10-12T01:04:25.416 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.417 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.417 DEBUG:boto:Params: {} 2023-10-12T01:04:25.418 DEBUG:boto:Token: None 2023-10-12T01:04:25.418 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MPK+255DRwkzV89BuCSSHNQOIPY= 2023-10-12T01:04:25.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MPK+255DRwkzV89BuCSSHNQOIPY=', 'Content-Length': '0'} 2023-10-12T01:04:25.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9ced7ffce8ecdd-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:04:25.422 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.422 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.423 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.423 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.424 DEBUG:boto:Method: GET 2023-10-12T01:04:25.424 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.424 DEBUG:boto:Data: 2023-10-12T01:04:25.425 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.425 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.426 DEBUG:boto:Params: {} 2023-10-12T01:04:25.426 DEBUG:boto:Token: None 2023-10-12T01:04:25.427 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.427 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA= 2023-10-12T01:04:25.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA=', 'Content-Length': '0'} 2023-10-12T01:04:25.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5fde10585afcc6-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.470 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.471 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.471 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.472 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.472 DEBUG:boto:Method: GET 2023-10-12T01:04:25.473 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.473 DEBUG:boto:Data: 2023-10-12T01:04:25.473 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.474 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.475 DEBUG:boto:Params: {} 2023-10-12T01:04:25.475 DEBUG:boto:Token: None 2023-10-12T01:04:25.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:04:25.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA= 2023-10-12T01:04:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5byzNMMQnlwC58sTqAjlgg1EaJA=', 'Content-Length': '0'} 2023-10-12T01:04:25.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db275c897dbe8d87-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.479 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.480 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.480 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.481 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.481 DEBUG:boto:Method: GET 2023-10-12T01:04:25.481 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.482 DEBUG:boto:Data: 2023-10-12T01:04:25.483 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.484 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.484 DEBUG:boto:Params: {} 2023-10-12T01:04:25.485 DEBUG:boto:Token: None 2023-10-12T01:04:25.485 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260544162f5d9aa7-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.489 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.490 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.491 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.491 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.491 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.492 DEBUG:boto:Method: GET 2023-10-12T01:04:25.492 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.493 DEBUG:boto:Data: 2023-10-12T01:04:25.493 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.494 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.494 DEBUG:boto:Params: {} 2023-10-12T01:04:25.495 DEBUG:boto:Token: None 2023-10-12T01:04:25.495 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ecc65f99f44d9eb-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.500 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.501 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.501 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.502 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.502 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:04:25.503 DEBUG:boto:Data: 2023-10-12T01:04:25.503 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.504 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.504 DEBUG:boto:Params: {} 2023-10-12T01:04:25.505 DEBUG:boto:Token: None 2023-10-12T01:04:25.505 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85 2023-10-12T01:04:25.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hfu/rK8CLOCGWI21n788Rw4qQ2k= 2023-10-12T01:04:25.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hfu/rK8CLOCGWI21n788Rw4qQ2k=', 'Content-Length': '0'} 2023-10-12T01:04:25.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ec31b86bb8d166-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.509 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.509 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.510 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.510 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:04:25.510 DEBUG:boto:Data: 2023-10-12T01:04:25.511 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.512 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.512 DEBUG:boto:Params: {} 2023-10-12T01:04:25.513 DEBUG:boto:Token: None 2023-10-12T01:04:25.513 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85 2023-10-12T01:04:25.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hfu/rK8CLOCGWI21n788Rw4qQ2k= 2023-10-12T01:04:25.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hfu/rK8CLOCGWI21n788Rw4qQ2k=', 'Content-Length': '0'} 2023-10-12T01:04:25.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaea8c2d6d8b90c0-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:04:25.517 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.518 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.518 DEBUG:boto:Method: GET 2023-10-12T01:04:25.519 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:04:25.519 DEBUG:boto:Data: 2023-10-12T01:04:25.520 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.521 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.521 DEBUG:boto:Params: {} 2023-10-12T01:04:25.522 DEBUG:boto:Token: None 2023-10-12T01:04:25.523 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85 2023-10-12T01:04:25.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:azL+VRXjTRGu6lp6V4GfFcqunPM= 2023-10-12T01:04:25.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:azL+VRXjTRGu6lp6V4GfFcqunPM=', 'Content-Length': '0'} 2023-10-12T01:04:25.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037efcbaa6089db75-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.528 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:04:25.529 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:04:25.530 DEBUG:boto:Method: GET 2023-10-12T01:04:25.530 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:04:25.533 DEBUG:boto:Data: 2023-10-12T01:04:25.534 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.535 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.536 DEBUG:boto:Params: {} 2023-10-12T01:04:25.536 DEBUG:boto:Token: None 2023-10-12T01:04:25.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj85 2023-10-12T01:04:25.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:azL+VRXjTRGu6lp6V4GfFcqunPM= 2023-10-12T01:04:25.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:azL+VRXjTRGu6lp6V4GfFcqunPM=', 'Content-Length': '0'} 2023-10-12T01:04:25.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007adc53a8c281701e-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.582 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.583 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.584 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.585 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.585 DEBUG:boto:Method: GET 2023-10-12T01:04:25.586 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.587 DEBUG:boto:Data: 2023-10-12T01:04:25.587 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.589 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.589 DEBUG:boto:Params: {} 2023-10-12T01:04:25.590 DEBUG:boto:Token: None 2023-10-12T01:04:25.591 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4f35bc15666659-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.597 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.598 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.599 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.600 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.601 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.601 DEBUG:boto:Method: GET 2023-10-12T01:04:25.602 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:04:25.603 DEBUG:boto:Data: 2023-10-12T01:04:25.604 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.605 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.606 DEBUG:boto:Params: {} 2023-10-12T01:04:25.607 DEBUG:boto:Token: None 2023-10-12T01:04:25.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850909d394a665a3-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.613 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:04:25.614 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.615 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.615 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.616 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.616 DEBUG:boto:Method: GET 2023-10-12T01:04:25.617 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.617 DEBUG:boto:Data: 2023-10-12T01:04:25.617 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.618 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.619 DEBUG:boto:Params: {} 2023-10-12T01:04:25.619 DEBUG:boto:Token: None 2023-10-12T01:04:25.620 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM= 2023-10-12T01:04:25.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM=', 'Content-Length': '0'} 2023-10-12T01:04:25.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f3c0b0f212ae98-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.624 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.624 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.625 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.625 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.626 DEBUG:boto:Method: GET 2023-10-12T01:04:25.626 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.626 DEBUG:boto:Data: 2023-10-12T01:04:25.627 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.628 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.628 DEBUG:boto:Params: {} 2023-10-12T01:04:25.629 DEBUG:boto:Token: None 2023-10-12T01:04:25.629 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.630 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM= 2023-10-12T01:04:25.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM=', 'Content-Length': '0'} 2023-10-12T01:04:25.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008dc5ea4d421d3c9-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.633 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.634 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.634 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.634 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.635 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.635 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.636 DEBUG:boto:Data: 2023-10-12T01:04:25.636 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.637 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.637 DEBUG:boto:Params: {} 2023-10-12T01:04:25.638 DEBUG:boto:Token: None 2023-10-12T01:04:25.638 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:57kD1DcAZdGzw2CCWyNfcsosrJU= 2023-10-12T01:04:25.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:57kD1DcAZdGzw2CCWyNfcsosrJU=', 'Content-Length': '0'} 2023-10-12T01:04:25.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c3d7ec441ad52c-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.642 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.642 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.643 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.643 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.644 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.644 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.644 DEBUG:boto:Data: 2023-10-12T01:04:25.645 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.646 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.646 DEBUG:boto:Params: {} 2023-10-12T01:04:25.646 DEBUG:boto:Token: None 2023-10-12T01:04:25.647 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.647 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:57kD1DcAZdGzw2CCWyNfcsosrJU= 2023-10-12T01:04:25.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:57kD1DcAZdGzw2CCWyNfcsosrJU=', 'Content-Length': '0'} 2023-10-12T01:04:25.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0dbdc55a185cc5-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:04:25.651 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.652 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.652 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.652 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.653 DEBUG:boto:Method: GET 2023-10-12T01:04:25.653 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.654 DEBUG:boto:Data: 2023-10-12T01:04:25.654 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.655 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.655 DEBUG:boto:Params: {} 2023-10-12T01:04:25.656 DEBUG:boto:Token: None 2023-10-12T01:04:25.656 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM= 2023-10-12T01:04:25.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM=', 'Content-Length': '0'} 2023-10-12T01:04:25.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ca0db8c56dfc87-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.702 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.703 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.704 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.705 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.705 DEBUG:boto:Method: GET 2023-10-12T01:04:25.706 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.707 DEBUG:boto:Data: 2023-10-12T01:04:25.707 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.708 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.708 DEBUG:boto:Params: {} 2023-10-12T01:04:25.709 DEBUG:boto:Token: None 2023-10-12T01:04:25.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:04:25.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM= 2023-10-12T01:04:25.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BAtZn393TncVWUY8+zDkkN5TruM=', 'Content-Length': '0'} 2023-10-12T01:04:25.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000decd1f855e6fe21d-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.715 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.716 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.716 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.717 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.717 DEBUG:boto:Method: GET 2023-10-12T01:04:25.718 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.718 DEBUG:boto:Data: 2023-10-12T01:04:25.719 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.719 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.720 DEBUG:boto:Params: {} 2023-10-12T01:04:25.720 DEBUG:boto:Token: None 2023-10-12T01:04:25.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae9f62c409cbb47-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.725 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.726 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.726 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.727 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.727 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.728 DEBUG:boto:Method: GET 2023-10-12T01:04:25.728 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.729 DEBUG:boto:Data: 2023-10-12T01:04:25.729 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.730 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.730 DEBUG:boto:Params: {} 2023-10-12T01:04:25.731 DEBUG:boto:Token: None 2023-10-12T01:04:25.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdc3e1ca976a3f7-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.738 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.739 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.739 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.740 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.741 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:04:25.741 DEBUG:boto:Data: 2023-10-12T01:04:25.742 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.744 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.744 DEBUG:boto:Params: {} 2023-10-12T01:04:25.745 DEBUG:boto:Token: None 2023-10-12T01:04:25.746 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92 2023-10-12T01:04:25.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5/h6jTUGP+4XekmsOD0PZ6u4kM= 2023-10-12T01:04:25.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5/h6jTUGP+4XekmsOD0PZ6u4kM=', 'Content-Length': '0'} 2023-10-12T01:04:25.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a3382b9cf03bbb-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.750 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.751 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.751 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.751 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:04:25.752 DEBUG:boto:Data: 2023-10-12T01:04:25.753 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.754 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.755 DEBUG:boto:Params: {} 2023-10-12T01:04:25.756 DEBUG:boto:Token: None 2023-10-12T01:04:25.756 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92 2023-10-12T01:04:25.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5/h6jTUGP+4XekmsOD0PZ6u4kM= 2023-10-12T01:04:25.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5/h6jTUGP+4XekmsOD0PZ6u4kM=', 'Content-Length': '0'} 2023-10-12T01:04:25.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74b369458af4229-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:04:25.761 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.761 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.762 DEBUG:boto:Method: GET 2023-10-12T01:04:25.762 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:04:25.763 DEBUG:boto:Data: 2023-10-12T01:04:25.763 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.764 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.765 DEBUG:boto:Params: {} 2023-10-12T01:04:25.765 DEBUG:boto:Token: None 2023-10-12T01:04:25.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92 2023-10-12T01:04:25.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UQQsPTHLcN7r/hP/5iBaeywHK8= 2023-10-12T01:04:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UQQsPTHLcN7r/hP/5iBaeywHK8=', 'Content-Length': '0'} 2023-10-12T01:04:25.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f824dd2697d6714b-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.769 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:04:25.770 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:04:25.770 DEBUG:boto:Method: GET 2023-10-12T01:04:25.770 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:04:25.771 DEBUG:boto:Data: 2023-10-12T01:04:25.771 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.772 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.772 DEBUG:boto:Params: {} 2023-10-12T01:04:25.773 DEBUG:boto:Token: None 2023-10-12T01:04:25.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj92 2023-10-12T01:04:25.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UQQsPTHLcN7r/hP/5iBaeywHK8= 2023-10-12T01:04:25.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UQQsPTHLcN7r/hP/5iBaeywHK8=', 'Content-Length': '0'} 2023-10-12T01:04:25.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b28cf265bce4aca-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.818 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.819 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.819 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.820 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.820 DEBUG:boto:Method: GET 2023-10-12T01:04:25.821 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.821 DEBUG:boto:Data: 2023-10-12T01:04:25.822 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.823 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.823 DEBUG:boto:Params: {} 2023-10-12T01:04:25.824 DEBUG:boto:Token: None 2023-10-12T01:04:25.824 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb72c6e1eb17ca6-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.829 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.829 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.830 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.830 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.831 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.831 DEBUG:boto:Method: GET 2023-10-12T01:04:25.832 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:04:25.832 DEBUG:boto:Data: 2023-10-12T01:04:25.832 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.833 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.834 DEBUG:boto:Params: {} 2023-10-12T01:04:25.834 DEBUG:boto:Token: None 2023-10-12T01:04:25.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9cffbd3240f7c9-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.840 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:04:25.842 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.843 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.843 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.844 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.845 DEBUG:boto:Method: GET 2023-10-12T01:04:25.846 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.847 DEBUG:boto:Data: 2023-10-12T01:04:25.847 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.849 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.849 DEBUG:boto:Params: {} 2023-10-12T01:04:25.850 DEBUG:boto:Token: None 2023-10-12T01:04:25.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc= 2023-10-12T01:04:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc=', 'Content-Length': '0'} 2023-10-12T01:04:25.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fed4ebd09e4d236-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.856 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.857 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.858 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.858 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.859 DEBUG:boto:Method: GET 2023-10-12T01:04:25.859 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.860 DEBUG:boto:Data: 2023-10-12T01:04:25.861 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.862 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.863 DEBUG:boto:Params: {} 2023-10-12T01:04:25.863 DEBUG:boto:Token: None 2023-10-12T01:04:25.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc= 2023-10-12T01:04:25.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc=', 'Content-Length': '0'} 2023-10-12T01:04:25.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f819aba4dce47a-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.868 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.868 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.868 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.869 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.869 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.870 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.870 DEBUG:boto:Data: 2023-10-12T01:04:25.871 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.872 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.873 DEBUG:boto:Params: {} 2023-10-12T01:04:25.873 DEBUG:boto:Token: None 2023-10-12T01:04:25.873 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhEuCwxmy4oDBITu4qhnchWlxtc= 2023-10-12T01:04:25.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhEuCwxmy4oDBITu4qhnchWlxtc=', 'Content-Length': '0'} 2023-10-12T01:04:25.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038887e72b02a2387-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.877 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.877 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.878 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.878 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.879 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.879 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.879 DEBUG:boto:Data: 2023-10-12T01:04:25.880 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.881 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.881 DEBUG:boto:Params: {} 2023-10-12T01:04:25.882 DEBUG:boto:Token: None 2023-10-12T01:04:25.882 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhEuCwxmy4oDBITu4qhnchWlxtc= 2023-10-12T01:04:25.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhEuCwxmy4oDBITu4qhnchWlxtc=', 'Content-Length': '0'} 2023-10-12T01:04:25.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb19106108a3bfda-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:04:25.887 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.887 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.888 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.888 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.889 DEBUG:boto:Method: GET 2023-10-12T01:04:25.889 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.890 DEBUG:boto:Data: 2023-10-12T01:04:25.890 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.891 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.891 DEBUG:boto:Params: {} 2023-10-12T01:04:25.892 DEBUG:boto:Token: None 2023-10-12T01:04:25.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc= 2023-10-12T01:04:25.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc=', 'Content-Length': '0'} 2023-10-12T01:04:25.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a017e6db820dbf5a-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.897 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.898 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.899 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.899 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.900 DEBUG:boto:Method: GET 2023-10-12T01:04:25.901 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.901 DEBUG:boto:Data: 2023-10-12T01:04:25.902 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.904 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.904 DEBUG:boto:Params: {} 2023-10-12T01:04:25.905 DEBUG:boto:Token: None 2023-10-12T01:04:25.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:04:25.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc= 2023-10-12T01:04:25.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jL/raYVG3/yYcCUYlBnQgqJKqQc=', 'Content-Length': '0'} 2023-10-12T01:04:25.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7b2ca78174f072-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.954 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.955 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.955 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.956 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.956 DEBUG:boto:Method: GET 2023-10-12T01:04:25.957 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.957 DEBUG:boto:Data: 2023-10-12T01:04:25.958 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.959 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.959 DEBUG:boto:Params: {} 2023-10-12T01:04:25.960 DEBUG:boto:Token: None 2023-10-12T01:04:25.960 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acd322c67fd65f57-0065274619-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.964 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.965 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:25.966 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:25.966 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.966 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.967 DEBUG:boto:Method: GET 2023-10-12T01:04:25.967 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:04:25.968 DEBUG:boto:Data: 2023-10-12T01:04:25.968 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.969 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.969 DEBUG:boto:Params: {} 2023-10-12T01:04:25.970 DEBUG:boto:Token: None 2023-10-12T01:04:25.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/ 2023-10-12T01:04:25.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA= 2023-10-12T01:04:25.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dXsYzx0fVYu46QEaiiCDabxfaxA=', 'Content-Length': '0'} 2023-10-12T01:04:25.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e588b1b14d5823-0065274619-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.976 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:25.976 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.977 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.977 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.978 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:04:25.978 DEBUG:boto:Data: 2023-10-12T01:04:25.979 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.980 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.980 DEBUG:boto:Params: {} 2023-10-12T01:04:25.980 DEBUG:boto:Token: None 2023-10-12T01:04:25.981 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95 2023-10-12T01:04:25.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KBg3BZ80SKQt/Ms4JokHEqd7cmY= 2023-10-12T01:04:25.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KBg3BZ80SKQt/Ms4JokHEqd7cmY=', 'Content-Length': '0'} 2023-10-12T01:04:25.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd2f5c888c37454-0065274619-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.984 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.985 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.985 DEBUG:boto:Method: HEAD 2023-10-12T01:04:25.986 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:04:25.986 DEBUG:boto:Data: 2023-10-12T01:04:25.987 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:25.987 DEBUG:boto:Port: 8000 2023-10-12T01:04:25.988 DEBUG:boto:Params: {} 2023-10-12T01:04:25.988 DEBUG:boto:Token: None 2023-10-12T01:04:25.989 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95 2023-10-12T01:04:25.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KBg3BZ80SKQt/Ms4JokHEqd7cmY= 2023-10-12T01:04:25.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KBg3BZ80SKQt/Ms4JokHEqd7cmY=', 'Content-Length': '0'} 2023-10-12T01:04:25.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002357221dcda59c92-0065274619-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:25.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:04:25.993 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:25.994 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:25.994 DEBUG:boto:Method: GET 2023-10-12T01:04:25.994 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:04:25.995 DEBUG:boto:Data: 2023-10-12T01:04:25.995 DEBUG:boto:Headers: {} 2023-10-12T01:04:25.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:25.996 DEBUG:boto:Port: 8001 2023-10-12T01:04:25.997 DEBUG:boto:Params: {} 2023-10-12T01:04:25.997 DEBUG:boto:Token: None 2023-10-12T01:04:25.998 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:25 GMT /wafmoa-59/obj95 2023-10-12T01:04:25.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LDLDizLdAiskyUgzUYO/U93bESs= 2023-10-12T01:04:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LDLDizLdAiskyUgzUYO/U93bESs=', 'Content-Length': '0'} 2023-10-12T01:04:26.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d978ff67c7af7b9f-006527461a-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.046 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:04:26.047 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:04:26.047 DEBUG:boto:Method: GET 2023-10-12T01:04:26.048 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:04:26.048 DEBUG:boto:Data: 2023-10-12T01:04:26.049 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.050 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.050 DEBUG:boto:Params: {} 2023-10-12T01:04:26.051 DEBUG:boto:Token: None 2023-10-12T01:04:26.051 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj95 2023-10-12T01:04:26.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:198c4PhF3l5jmIjXvmS73uajwtU= 2023-10-12T01:04:26.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:198c4PhF3l5jmIjXvmS73uajwtU=', 'Content-Length': '0'} 2023-10-12T01:04:26.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054dac0ecf7f66924-006527461a-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.055 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.055 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.056 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.056 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.056 DEBUG:boto:Method: GET 2023-10-12T01:04:26.057 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.057 DEBUG:boto:Data: 2023-10-12T01:04:26.058 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:04:26.059 DEBUG:boto:Port: 8001 2023-10-12T01:04:26.059 DEBUG:boto:Params: {} 2023-10-12T01:04:26.060 DEBUG:boto:Token: None 2023-10-12T01:04:26.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/SRxxHvq1GWkAUPb5/+eD1bSpMo= 2023-10-12T01:04:26.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/SRxxHvq1GWkAUPb5/+eD1bSpMo=', 'Content-Length': '0'} 2023-10-12T01:04:26.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d1a22e9296daea0-006527461a-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.064 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:26.065 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.065 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.066 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.066 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.067 DEBUG:boto:Method: GET 2023-10-12T01:04:26.067 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:04:26.067 DEBUG:boto:Data: 2023-10-12T01:04:26.068 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.069 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.069 DEBUG:boto:Params: {} 2023-10-12T01:04:26.070 DEBUG:boto:Token: None 2023-10-12T01:04:26.070 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/SRxxHvq1GWkAUPb5/+eD1bSpMo= 2023-10-12T01:04:26.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/SRxxHvq1GWkAUPb5/+eD1bSpMo=', 'Content-Length': '0'} 2023-10-12T01:04:26.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004622b9468ad9f06a-006527461a-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.075 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:04:26.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:04:26.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:04:26.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:04:26.077 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:04:26.136 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:04:26.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:04:26.186 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:04:26.187 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T01:04:26.194 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-10-12T01:04:26.244 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.244 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.245 DEBUG:boto:Method: HEAD 2023-10-12T01:04:26.246 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:26.246 DEBUG:boto:Data: 2023-10-12T01:04:26.246 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.247 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.248 DEBUG:boto:Params: {} 2023-10-12T01:04:26.248 DEBUG:boto:Token: None 2023-10-12T01:04:26.248 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE= 2023-10-12T01:04:26.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE=', 'Content-Length': '0'} 2023-10-12T01:04:26.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e1ffc433355e6f9-006527461a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.253 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:04:26.254 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:04:26.254 DEBUG:boto:Method: PUT 2023-10-12T01:04:26.255 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:04:26.255 DEBUG:boto:Data: 2023-10-12T01:04:26.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:04:26.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.256 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.256 DEBUG:boto:Params: {} 2023-10-12T01:04:26.257 DEBUG:boto:Token: None 2023-10-12T01:04:26.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj29 2023-10-12T01:04:26.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ZqNjoGY17cY26m4JTHtGdA/XFA= 2023-10-12T01:04:26.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ZqNjoGY17cY26m4JTHtGdA/XFA='} 2023-10-12T01:04:26.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042013fccb80360e9-006527461a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.307 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.307 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.308 DEBUG:boto:Method: HEAD 2023-10-12T01:04:26.308 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:26.309 DEBUG:boto:Data: 2023-10-12T01:04:26.309 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.310 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.310 DEBUG:boto:Params: {} 2023-10-12T01:04:26.311 DEBUG:boto:Token: None 2023-10-12T01:04:26.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE= 2023-10-12T01:04:26.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE=', 'Content-Length': '0'} 2023-10-12T01:04:26.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8834b0c055ea683-006527461a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.315 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:04:26.316 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:04:26.316 DEBUG:boto:Method: PUT 2023-10-12T01:04:26.316 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:04:26.317 DEBUG:boto:Data: 2023-10-12T01:04:26.317 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:04:26.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.318 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.318 DEBUG:boto:Params: {} 2023-10-12T01:04:26.319 DEBUG:boto:Token: None 2023-10-12T01:04:26.319 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj58 2023-10-12T01:04:26.320 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fsotpGfa26n0U3lTZCOFfITw8QI= 2023-10-12T01:04:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fsotpGfa26n0U3lTZCOFfITw8QI='} 2023-10-12T01:04:26.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f296dff7543c2f03-006527461a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.326 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.327 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.327 DEBUG:boto:Method: HEAD 2023-10-12T01:04:26.327 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:26.328 DEBUG:boto:Data: 2023-10-12T01:04:26.328 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.329 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.329 DEBUG:boto:Params: {} 2023-10-12T01:04:26.330 DEBUG:boto:Token: None 2023-10-12T01:04:26.330 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE= 2023-10-12T01:04:26.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE=', 'Content-Length': '0'} 2023-10-12T01:04:26.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1eda5f52aa1af3a-006527461a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.334 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:04:26.335 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:04:26.335 DEBUG:boto:Method: PUT 2023-10-12T01:04:26.336 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:04:26.336 DEBUG:boto:Data: 2023-10-12T01:04:26.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:04:26.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.337 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.337 DEBUG:boto:Params: {} 2023-10-12T01:04:26.338 DEBUG:boto:Token: None 2023-10-12T01:04:26.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj87 2023-10-12T01:04:26.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:98biz9QlDJValAUz5vXMj3O66Cg= 2023-10-12T01:04:26.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:98biz9QlDJValAUz5vXMj3O66Cg='} 2023-10-12T01:04:26.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000096980803540f04d-006527461a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.383 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.384 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.384 DEBUG:boto:Method: HEAD 2023-10-12T01:04:26.384 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:26.385 DEBUG:boto:Data: 2023-10-12T01:04:26.385 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.386 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.386 DEBUG:boto:Params: {} 2023-10-12T01:04:26.387 DEBUG:boto:Token: None 2023-10-12T01:04:26.387 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE= 2023-10-12T01:04:26.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE=', 'Content-Length': '0'} 2023-10-12T01:04:26.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc675c375a14e394-006527461a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.392 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:04:26.392 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:04:26.392 DEBUG:boto:Method: PUT 2023-10-12T01:04:26.393 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:04:26.393 DEBUG:boto:Data: 2023-10-12T01:04:26.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:04:26.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.394 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.395 DEBUG:boto:Params: {} 2023-10-12T01:04:26.395 DEBUG:boto:Token: None 2023-10-12T01:04:26.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj116 2023-10-12T01:04:26.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NfLCCE1GWMBFy+dWSdNx59hRBZY= 2023-10-12T01:04:26.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NfLCCE1GWMBFy+dWSdNx59hRBZY='} 2023-10-12T01:04:26.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ce7bb8beff6405c-006527461a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.402 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:04:26.403 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:04:26.403 DEBUG:boto:Method: HEAD 2023-10-12T01:04:26.403 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:04:26.404 DEBUG:boto:Data: 2023-10-12T01:04:26.404 DEBUG:boto:Headers: {} 2023-10-12T01:04:26.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.405 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.405 DEBUG:boto:Params: {} 2023-10-12T01:04:26.406 DEBUG:boto:Token: None 2023-10-12T01:04:26.406 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/ 2023-10-12T01:04:26.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE= 2023-10-12T01:04:26.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QRfLxhKa+DG4S3yjlXI2CmkDUxE=', 'Content-Length': '0'} 2023-10-12T01:04:26.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be9a32da009b11b6-006527461a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.410 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:04:26.411 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:04:26.411 DEBUG:boto:Method: PUT 2023-10-12T01:04:26.412 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:04:26.412 DEBUG:boto:Data: 2023-10-12T01:04:26.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:04:26.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:04:26.413 DEBUG:boto:Port: 8000 2023-10-12T01:04:26.413 DEBUG:boto:Params: {} 2023-10-12T01:04:26.414 DEBUG:boto:Token: None 2023-10-12T01:04:26.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:04:26 GMT /wafmoa-59/obj145 2023-10-12T01:04:26.415 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LXReRYv8O1cz0AmLTG63Vp8Hkyc= 2023-10-12T01:04:26.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:04:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LXReRYv8O1cz0AmLTG63Vp8Hkyc='} 2023-10-12T01:04:26.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011a46d0b3d54bb92-006527461a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:04:26.459 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:04:26.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:04:26.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:04:26.461 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:04:26.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:04:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:04:26.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:26.549+0000 7f25b9033780 1 waiting to reach latest gen 4, on 3.. 2023-10-12T01:04:31.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:31.550+0000 7f25b9033780 1 waiting to reach latest gen 4, on 3.. 2023-10-12T01:04:36.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:36.552+0000 7f25b9033780 1 waiting to reach latest gen 4, on 3.. 2023-10-12T01:04:41.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:41.552+0000 7f25b9033780 1 waiting to reach latest gen 4, on 3.. 2023-10-12T01:04:46.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:46.555+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:46.555 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:04:46.556 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:04:51.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:51.556+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:51.557 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:04:51.557 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:04:56.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:04:56.558+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:04:56.559 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:04:56.559 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:01.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:01.560+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:01.561 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:01.561 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:06.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:06.562+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:06.563 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:06.563 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:11.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:11.564+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:11.565 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:11.565 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:16.567+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:16.567 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:16.567 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:21.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:21.569+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:21.569 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:21.570 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:26.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:26.571+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:26.571 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:26.571 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:31.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:31.572+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:31.574 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:31.574 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:36.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:36.574+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:36.576 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:36.576 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:41.576+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:41.578 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:41.578 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:46.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:46.578+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:46.579 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:46.580 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:51.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:51.581+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:51.581 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:51.582 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:05:56.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:05:56.583+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:05:56.584 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:05:56.584 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:06:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:01.586+0000 7f25b9033780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:01.586 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:06:01.586 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:06:06.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:06.587+0000 7f25b9033780 1 bucket sync caught up with source: 2023-10-12T01:06:06.588 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:06:06.588 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.80.6, 00000000001.106.6, , , , , 00000000001.81.6, , , , , , , , , , , 00000000002.57.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-12T01:06:06.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:06.587+0000 7f25b9033780 0 bucket checkpoint complete 2023-10-12T01:06:06.605 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:06:06.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:06:06.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:06:06.607 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:06:06.669 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:06:06.669 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:06:06.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:06.707+0000 7fb5ba33e780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:06.708 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000003.221.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:06.708 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000003.223.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:11.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:11.708+0000 7fb5ba33e780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:11.710 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000003.221.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:11.710 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000003.223.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:16.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:16.709+0000 7fb5ba33e780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:16.711 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000002.209.6, 00000000003.221.6, , , 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:16.711 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000003.223.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:21.711+0000 7fb5ba33e780 1 bucket sync caught up with source: 2023-10-12T01:06:21.712 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000003.223.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:21.713 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.328.6, 00000000002.299.6, 00000000001.181.6, 00000000003.223.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000002.287.6, 00000000005.245.6, 00000000004.210.6] 2023-10-12T01:06:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:21.711+0000 7fb5ba33e780 0 bucket checkpoint complete 2023-10-12T01:06:21.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:06:21.729 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.730 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.730 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.731 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:21.731 DEBUG:boto:Data: 2023-10-12T01:06:21.731 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.732 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.732 DEBUG:boto:Params: {} 2023-10-12T01:06:21.733 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:06:21.734 DEBUG:boto:Token: None 2023-10-12T01:06:21.734 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMIdDIyWDKfDpfBrGuAODnUhJLc= 2023-10-12T01:06:21.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMIdDIyWDKfDpfBrGuAODnUhJLc=', 'Content-Length': '0'} 2023-10-12T01:06:21.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b720496e74f4882-006527468d-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.740 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.740 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.741 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.741 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:21.742 DEBUG:boto:Data: 2023-10-12T01:06:21.742 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.743 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.744 DEBUG:boto:Params: {} 2023-10-12T01:06:21.744 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:06:21.745 DEBUG:boto:Token: None 2023-10-12T01:06:21.745 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMIdDIyWDKfDpfBrGuAODnUhJLc= 2023-10-12T01:06:21.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMIdDIyWDKfDpfBrGuAODnUhJLc=', 'Content-Length': '0'} 2023-10-12T01:06:21.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a415ef79a7e51b1e-006527468d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:06:21.752 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.752 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.753 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.753 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.753 DEBUG:boto:Method: GET 2023-10-12T01:06:21.754 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.754 DEBUG:boto:Data: 2023-10-12T01:06:21.755 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.755 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.756 DEBUG:boto:Params: {} 2023-10-12T01:06:21.756 DEBUG:boto:Token: None 2023-10-12T01:06:21.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/?versions 2023-10-12T01:06:21.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A= 2023-10-12T01:06:21.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A=', 'Content-Length': '0'} 2023-10-12T01:06:21.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3eb31261c2610d8-006527468d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.761 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:06:21.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:06:21.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:06:21.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:06:21.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:06:21.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:06:21.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:06:21.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:06:21.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:06:21.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:06:21.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:06:21.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:06:21.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:06:21.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:06:21.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:06:21.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:06:21.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:06:21.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:06:21.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:06:21.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:06:21.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:06:21.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:06:21.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:06:21.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:06:21.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:06:21.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:06:21.772 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.772 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.772 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.773 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.773 DEBUG:boto:Method: GET 2023-10-12T01:06:21.774 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.774 DEBUG:boto:Data: 2023-10-12T01:06:21.775 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.775 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.776 DEBUG:boto:Params: {} 2023-10-12T01:06:21.776 DEBUG:boto:Token: None 2023-10-12T01:06:21.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/?versions 2023-10-12T01:06:21.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A= 2023-10-12T01:06:21.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A=', 'Content-Length': '0'} 2023-10-12T01:06:21.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f3011725cf6ca3-006527468d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.782 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:06:21.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:06:21.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:06:21.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:06:21.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:06:21.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:06:21.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:06:21.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:06:21.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:06:21.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:06:21.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:06:21.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:06:21.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:06:21.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:06:21.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:06:21.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:06:21.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:06:21.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:06:21.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:06:21.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:06:21.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:06:21.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:06:21.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:06:21.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:06:21.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:06:21.792 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.793 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.793 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.794 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.794 DEBUG:boto:Method: GET 2023-10-12T01:06:21.794 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.795 DEBUG:boto:Data: 2023-10-12T01:06:21.795 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.796 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.796 DEBUG:boto:Params: {} 2023-10-12T01:06:21.797 DEBUG:boto:Token: None 2023-10-12T01:06:21.797 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/?versions 2023-10-12T01:06:21.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A= 2023-10-12T01:06:21.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A=', 'Content-Length': '0'} 2023-10-12T01:06:21.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408981b8852231a0-006527468d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.801 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.803 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.804 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.804 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.804 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.805 DEBUG:boto:Method: GET 2023-10-12T01:06:21.805 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:06:21.806 DEBUG:boto:Data: 2023-10-12T01:06:21.806 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.807 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.808 DEBUG:boto:Params: {} 2023-10-12T01:06:21.808 DEBUG:boto:Token: None 2023-10-12T01:06:21.809 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/?versions 2023-10-12T01:06:21.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A= 2023-10-12T01:06:21.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jLajd4X/Zpm4+Sa2a2VKXc+3S0A=', 'Content-Length': '0'} 2023-10-12T01:06:21.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e233425f9dcbc843-006527468d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.814 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:06:21.816 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.817 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.817 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.818 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.818 DEBUG:boto:Method: GET 2023-10-12T01:06:21.819 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.819 DEBUG:boto:Data: 2023-10-12T01:06:21.820 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.820 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.821 DEBUG:boto:Params: {} 2023-10-12T01:06:21.821 DEBUG:boto:Token: None 2023-10-12T01:06:21.822 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck= 2023-10-12T01:06:21.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck=', 'Content-Length': '0'} 2023-10-12T01:06:21.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a61c84329becab9-006527468d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.827 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.828 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.828 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.828 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.829 DEBUG:boto:Method: GET 2023-10-12T01:06:21.829 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.830 DEBUG:boto:Data: 2023-10-12T01:06:21.830 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.831 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.832 DEBUG:boto:Params: {} 2023-10-12T01:06:21.832 DEBUG:boto:Token: None 2023-10-12T01:06:21.833 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck= 2023-10-12T01:06:21.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck=', 'Content-Length': '0'} 2023-10-12T01:06:21.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8095f13be6025de-006527468d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.837 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.837 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.838 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.838 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.838 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.839 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.839 DEBUG:boto:Data: 2023-10-12T01:06:21.840 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.841 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.841 DEBUG:boto:Params: {} 2023-10-12T01:06:21.841 DEBUG:boto:Token: None 2023-10-12T01:06:21.842 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HEDCPN8cio73SUY+L7JBtZKDrvY= 2023-10-12T01:06:21.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HEDCPN8cio73SUY+L7JBtZKDrvY=', 'Content-Length': '0'} 2023-10-12T01:06:21.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664c4e481e1f29fc-006527468d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.846 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.847 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.847 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.848 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.848 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.848 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.849 DEBUG:boto:Data: 2023-10-12T01:06:21.849 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.850 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.850 DEBUG:boto:Params: {} 2023-10-12T01:06:21.851 DEBUG:boto:Token: None 2023-10-12T01:06:21.852 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HEDCPN8cio73SUY+L7JBtZKDrvY= 2023-10-12T01:06:21.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HEDCPN8cio73SUY+L7JBtZKDrvY=', 'Content-Length': '0'} 2023-10-12T01:06:21.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0ad57bc662a5db-006527468d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:06:21.856 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.856 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.857 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.857 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.858 DEBUG:boto:Method: GET 2023-10-12T01:06:21.858 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.859 DEBUG:boto:Data: 2023-10-12T01:06:21.859 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.860 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.861 DEBUG:boto:Params: {} 2023-10-12T01:06:21.861 DEBUG:boto:Token: None 2023-10-12T01:06:21.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck= 2023-10-12T01:06:21.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck=', 'Content-Length': '0'} 2023-10-12T01:06:21.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004378c011cb31c55c-006527468d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.866 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.866 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.867 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.867 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.868 DEBUG:boto:Method: GET 2023-10-12T01:06:21.868 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.869 DEBUG:boto:Data: 2023-10-12T01:06:21.869 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.870 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.871 DEBUG:boto:Params: {} 2023-10-12T01:06:21.871 DEBUG:boto:Token: None 2023-10-12T01:06:21.872 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:06:21.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck= 2023-10-12T01:06:21.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lUa7s1sO7ycmGuFSk8Cu8wmgHck=', 'Content-Length': '0'} 2023-10-12T01:06:21.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000593ffe108b834469-006527468d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.923 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.924 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.924 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.925 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.925 DEBUG:boto:Method: GET 2023-10-12T01:06:21.926 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.926 DEBUG:boto:Data: 2023-10-12T01:06:21.926 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.927 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.928 DEBUG:boto:Params: {} 2023-10-12T01:06:21.928 DEBUG:boto:Token: None 2023-10-12T01:06:21.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI= 2023-10-12T01:06:21.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI=', 'Content-Length': '0'} 2023-10-12T01:06:21.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312019f1a08ec9fe-006527468d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.933 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.934 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.935 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.935 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.935 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.936 DEBUG:boto:Method: GET 2023-10-12T01:06:21.936 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.937 DEBUG:boto:Data: 2023-10-12T01:06:21.937 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.938 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.939 DEBUG:boto:Params: {} 2023-10-12T01:06:21.939 DEBUG:boto:Token: None 2023-10-12T01:06:21.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI= 2023-10-12T01:06:21.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI=', 'Content-Length': '0'} 2023-10-12T01:06:21.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ffafd25ee1863e8-006527468d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.945 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.946 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.946 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.947 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.947 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:06:21.948 DEBUG:boto:Data: 2023-10-12T01:06:21.948 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.949 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.949 DEBUG:boto:Params: {} 2023-10-12T01:06:21.950 DEBUG:boto:Token: None 2023-10-12T01:06:21.950 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11 2023-10-12T01:06:21.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9AsogzE6MwVFEFC7JuXFrBHxqGQ= 2023-10-12T01:06:21.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9AsogzE6MwVFEFC7JuXFrBHxqGQ=', 'Content-Length': '0'} 2023-10-12T01:06:21.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f9a4938b612ce8c-006527468d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.954 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.954 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.955 DEBUG:boto:Method: HEAD 2023-10-12T01:06:21.955 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:06:21.956 DEBUG:boto:Data: 2023-10-12T01:06:21.956 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.957 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.957 DEBUG:boto:Params: {} 2023-10-12T01:06:21.958 DEBUG:boto:Token: None 2023-10-12T01:06:21.958 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11 2023-10-12T01:06:21.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9AsogzE6MwVFEFC7JuXFrBHxqGQ= 2023-10-12T01:06:21.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9AsogzE6MwVFEFC7JuXFrBHxqGQ=', 'Content-Length': '0'} 2023-10-12T01:06:21.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db7a54210a93a0b-006527468d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:06:21.963 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.963 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.964 DEBUG:boto:Method: GET 2023-10-12T01:06:21.964 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:06:21.965 DEBUG:boto:Data: 2023-10-12T01:06:21.965 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.966 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.966 DEBUG:boto:Params: {} 2023-10-12T01:06:21.967 DEBUG:boto:Token: None 2023-10-12T01:06:21.967 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11 2023-10-12T01:06:21.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I7JBFPenBUSbrKt+B6cksoWC+2Y= 2023-10-12T01:06:21.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I7JBFPenBUSbrKt+B6cksoWC+2Y=', 'Content-Length': '0'} 2023-10-12T01:06:21.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f531f0bf6fca3e-006527468d-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.971 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:06:21.971 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:06:21.972 DEBUG:boto:Method: GET 2023-10-12T01:06:21.972 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:06:21.973 DEBUG:boto:Data: 2023-10-12T01:06:21.973 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.974 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.974 DEBUG:boto:Params: {} 2023-10-12T01:06:21.975 DEBUG:boto:Token: None 2023-10-12T01:06:21.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/obj11 2023-10-12T01:06:21.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I7JBFPenBUSbrKt+B6cksoWC+2Y= 2023-10-12T01:06:21.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I7JBFPenBUSbrKt+B6cksoWC+2Y=', 'Content-Length': '0'} 2023-10-12T01:06:21.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4ede4f2891c2e7-006527468d-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.979 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.979 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.980 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.980 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.981 DEBUG:boto:Method: GET 2023-10-12T01:06:21.981 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.982 DEBUG:boto:Data: 2023-10-12T01:06:21.982 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:21.983 DEBUG:boto:Port: 8001 2023-10-12T01:06:21.983 DEBUG:boto:Params: {} 2023-10-12T01:06:21.984 DEBUG:boto:Token: None 2023-10-12T01:06:21.984 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI= 2023-10-12T01:06:21.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI=', 'Content-Length': '0'} 2023-10-12T01:06:21.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b65971387e57ef-006527468d-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:21.989 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:21.990 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:21.990 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:21.991 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.991 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.992 DEBUG:boto:Method: GET 2023-10-12T01:06:21.992 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:06:21.992 DEBUG:boto:Data: 2023-10-12T01:06:21.993 DEBUG:boto:Headers: {} 2023-10-12T01:06:21.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:21.994 DEBUG:boto:Port: 8000 2023-10-12T01:06:21.994 DEBUG:boto:Params: {} 2023-10-12T01:06:21.995 DEBUG:boto:Token: None 2023-10-12T01:06:21.995 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:21 GMT /wafmoa-59/ 2023-10-12T01:06:21.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI= 2023-10-12T01:06:21.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LvpbH6kgF/XTMEqUczK/UtqP8DI=', 'Content-Length': '0'} 2023-10-12T01:06:22.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648f299ed3afd2b4-006527468d-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.042 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:06:22.044 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.044 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.045 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.045 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.046 DEBUG:boto:Method: GET 2023-10-12T01:06:22.046 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.047 DEBUG:boto:Data: 2023-10-12T01:06:22.047 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.048 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.048 DEBUG:boto:Params: {} 2023-10-12T01:06:22.049 DEBUG:boto:Token: None 2023-10-12T01:06:22.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw= 2023-10-12T01:06:22.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw=', 'Content-Length': '0'} 2023-10-12T01:06:22.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dcd713ede57b215-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.053 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.054 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.054 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.055 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.055 DEBUG:boto:Method: GET 2023-10-12T01:06:22.056 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.056 DEBUG:boto:Data: 2023-10-12T01:06:22.056 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.057 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.058 DEBUG:boto:Params: {} 2023-10-12T01:06:22.058 DEBUG:boto:Token: None 2023-10-12T01:06:22.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw= 2023-10-12T01:06:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw=', 'Content-Length': '0'} 2023-10-12T01:06:22.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d608cf3f609cd2e3-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.063 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.063 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.064 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.064 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.064 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.065 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.065 DEBUG:boto:Data: 2023-10-12T01:06:22.066 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.066 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.067 DEBUG:boto:Params: {} 2023-10-12T01:06:22.067 DEBUG:boto:Token: None 2023-10-12T01:06:22.068 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s00pNon7C0gnzuFIBNtvxvi5LY8= 2023-10-12T01:06:22.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s00pNon7C0gnzuFIBNtvxvi5LY8=', 'Content-Length': '0'} 2023-10-12T01:06:22.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108d12567929e321-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.071 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.072 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.072 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.073 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.073 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.073 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.074 DEBUG:boto:Data: 2023-10-12T01:06:22.074 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.075 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.075 DEBUG:boto:Params: {} 2023-10-12T01:06:22.076 DEBUG:boto:Token: None 2023-10-12T01:06:22.076 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.077 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s00pNon7C0gnzuFIBNtvxvi5LY8= 2023-10-12T01:06:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s00pNon7C0gnzuFIBNtvxvi5LY8=', 'Content-Length': '0'} 2023-10-12T01:06:22.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4e5eb53321c4de-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:06:22.080 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.081 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.081 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.082 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.082 DEBUG:boto:Method: GET 2023-10-12T01:06:22.083 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.083 DEBUG:boto:Data: 2023-10-12T01:06:22.083 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.084 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.085 DEBUG:boto:Params: {} 2023-10-12T01:06:22.085 DEBUG:boto:Token: None 2023-10-12T01:06:22.086 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.086 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw= 2023-10-12T01:06:22.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw=', 'Content-Length': '0'} 2023-10-12T01:06:22.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a74793fdfb881f0-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.130 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.131 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.131 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.132 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.132 DEBUG:boto:Method: GET 2023-10-12T01:06:22.133 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.133 DEBUG:boto:Data: 2023-10-12T01:06:22.134 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.134 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.135 DEBUG:boto:Params: {} 2023-10-12T01:06:22.135 DEBUG:boto:Token: None 2023-10-12T01:06:22.136 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:06:22.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw= 2023-10-12T01:06:22.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nQ2J2xRdPTi0rBhdMYLdAON6Fkw=', 'Content-Length': '0'} 2023-10-12T01:06:22.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f837018de5b5edc9-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.140 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.140 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.141 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.141 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.142 DEBUG:boto:Method: GET 2023-10-12T01:06:22.142 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.143 DEBUG:boto:Data: 2023-10-12T01:06:22.143 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.144 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.144 DEBUG:boto:Params: {} 2023-10-12T01:06:22.145 DEBUG:boto:Token: None 2023-10-12T01:06:22.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb064f664aa100d-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.150 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.150 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.151 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.151 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.152 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.152 DEBUG:boto:Method: GET 2023-10-12T01:06:22.152 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.153 DEBUG:boto:Data: 2023-10-12T01:06:22.153 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.154 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.155 DEBUG:boto:Params: {} 2023-10-12T01:06:22.155 DEBUG:boto:Token: None 2023-10-12T01:06:22.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbcde638a6580f3-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.202 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.203 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.204 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.204 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.205 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:06:22.205 DEBUG:boto:Data: 2023-10-12T01:06:22.205 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.206 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.207 DEBUG:boto:Params: {} 2023-10-12T01:06:22.207 DEBUG:boto:Token: None 2023-10-12T01:06:22.208 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115 2023-10-12T01:06:22.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNjkv1WUxq5zVtS0ZRTSMw2IUqY= 2023-10-12T01:06:22.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNjkv1WUxq5zVtS0ZRTSMw2IUqY=', 'Content-Length': '0'} 2023-10-12T01:06:22.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d116ad940b644fbd-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.212 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.213 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.213 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.214 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:06:22.215 DEBUG:boto:Data: 2023-10-12T01:06:22.216 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.217 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.217 DEBUG:boto:Params: {} 2023-10-12T01:06:22.218 DEBUG:boto:Token: None 2023-10-12T01:06:22.219 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115 2023-10-12T01:06:22.219 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VNjkv1WUxq5zVtS0ZRTSMw2IUqY= 2023-10-12T01:06:22.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VNjkv1WUxq5zVtS0ZRTSMw2IUqY=', 'Content-Length': '0'} 2023-10-12T01:06:22.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da807352e42e672-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:06:22.224 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.225 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.226 DEBUG:boto:Method: GET 2023-10-12T01:06:22.227 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:06:22.228 DEBUG:boto:Data: 2023-10-12T01:06:22.229 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.230 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.231 DEBUG:boto:Params: {} 2023-10-12T01:06:22.232 DEBUG:boto:Token: None 2023-10-12T01:06:22.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115 2023-10-12T01:06:22.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTu0GR5n4WGIHosu+zA1u8Vq4no= 2023-10-12T01:06:22.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTu0GR5n4WGIHosu+zA1u8Vq4no=', 'Content-Length': '0'} 2023-10-12T01:06:22.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f37fda98cfd60c1-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.236 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:06:22.236 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:06:22.237 DEBUG:boto:Method: GET 2023-10-12T01:06:22.237 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:06:22.238 DEBUG:boto:Data: 2023-10-12T01:06:22.238 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.239 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.239 DEBUG:boto:Params: {} 2023-10-12T01:06:22.240 DEBUG:boto:Token: None 2023-10-12T01:06:22.240 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj115 2023-10-12T01:06:22.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTu0GR5n4WGIHosu+zA1u8Vq4no= 2023-10-12T01:06:22.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTu0GR5n4WGIHosu+zA1u8Vq4no=', 'Content-Length': '0'} 2023-10-12T01:06:22.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdec8e4b736b751-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.245 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.245 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.245 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.246 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.246 DEBUG:boto:Method: GET 2023-10-12T01:06:22.247 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.247 DEBUG:boto:Data: 2023-10-12T01:06:22.248 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.248 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.249 DEBUG:boto:Params: {} 2023-10-12T01:06:22.249 DEBUG:boto:Token: None 2023-10-12T01:06:22.250 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017ed33154bb6258-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.298 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.299 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.299 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.300 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.300 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.301 DEBUG:boto:Method: GET 2023-10-12T01:06:22.301 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:06:22.301 DEBUG:boto:Data: 2023-10-12T01:06:22.302 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.303 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.303 DEBUG:boto:Params: {} 2023-10-12T01:06:22.304 DEBUG:boto:Token: None 2023-10-12T01:06:22.305 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd3ad0ea4bf260b-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.311 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:06:22.312 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.312 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.313 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.313 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.314 DEBUG:boto:Method: GET 2023-10-12T01:06:22.314 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.315 DEBUG:boto:Data: 2023-10-12T01:06:22.315 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.316 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.316 DEBUG:boto:Params: {} 2023-10-12T01:06:22.317 DEBUG:boto:Token: None 2023-10-12T01:06:22.317 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY= 2023-10-12T01:06:22.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY=', 'Content-Length': '0'} 2023-10-12T01:06:22.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b4e7290c9578e0-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.321 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.322 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.322 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.323 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.323 DEBUG:boto:Method: GET 2023-10-12T01:06:22.324 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.324 DEBUG:boto:Data: 2023-10-12T01:06:22.325 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.325 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.326 DEBUG:boto:Params: {} 2023-10-12T01:06:22.326 DEBUG:boto:Token: None 2023-10-12T01:06:22.327 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY= 2023-10-12T01:06:22.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY=', 'Content-Length': '0'} 2023-10-12T01:06:22.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e9d2cddc786ccd-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.370 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.371 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.371 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.372 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.372 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.372 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.373 DEBUG:boto:Data: 2023-10-12T01:06:22.373 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.374 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.374 DEBUG:boto:Params: {} 2023-10-12T01:06:22.375 DEBUG:boto:Token: None 2023-10-12T01:06:22.375 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TN0d6Mpuhw4hBb6O40dgCq7un6E= 2023-10-12T01:06:22.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TN0d6Mpuhw4hBb6O40dgCq7un6E=', 'Content-Length': '0'} 2023-10-12T01:06:22.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019952c36babf8a7-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.379 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.379 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.379 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.380 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.380 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.381 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.381 DEBUG:boto:Data: 2023-10-12T01:06:22.381 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.382 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.383 DEBUG:boto:Params: {} 2023-10-12T01:06:22.383 DEBUG:boto:Token: None 2023-10-12T01:06:22.383 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TN0d6Mpuhw4hBb6O40dgCq7un6E= 2023-10-12T01:06:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TN0d6Mpuhw4hBb6O40dgCq7un6E=', 'Content-Length': '0'} 2023-10-12T01:06:22.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953a970d1e3db55e-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:06:22.389 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.389 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.390 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.391 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.391 DEBUG:boto:Method: GET 2023-10-12T01:06:22.392 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.393 DEBUG:boto:Data: 2023-10-12T01:06:22.393 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.394 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.395 DEBUG:boto:Params: {} 2023-10-12T01:06:22.395 DEBUG:boto:Token: None 2023-10-12T01:06:22.396 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY= 2023-10-12T01:06:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY=', 'Content-Length': '0'} 2023-10-12T01:06:22.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7382bf694e37ed-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.399 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.399 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.400 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.400 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.401 DEBUG:boto:Method: GET 2023-10-12T01:06:22.401 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.402 DEBUG:boto:Data: 2023-10-12T01:06:22.402 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.403 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.404 DEBUG:boto:Params: {} 2023-10-12T01:06:22.404 DEBUG:boto:Token: None 2023-10-12T01:06:22.404 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:06:22.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY= 2023-10-12T01:06:22.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CaC8/myoavUv/ltwSFeEismEfCY=', 'Content-Length': '0'} 2023-10-12T01:06:22.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e710b2a056615ca-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.408 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.408 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.408 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.409 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.409 DEBUG:boto:Method: GET 2023-10-12T01:06:22.409 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.410 DEBUG:boto:Data: 2023-10-12T01:06:22.410 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.411 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.412 DEBUG:boto:Params: {} 2023-10-12T01:06:22.412 DEBUG:boto:Token: None 2023-10-12T01:06:22.413 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc83562bb34ec04-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.417 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.418 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.419 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.419 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.419 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.420 DEBUG:boto:Method: GET 2023-10-12T01:06:22.420 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.421 DEBUG:boto:Data: 2023-10-12T01:06:22.421 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.422 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.422 DEBUG:boto:Params: {} 2023-10-12T01:06:22.423 DEBUG:boto:Token: None 2023-10-12T01:06:22.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3de4d44dfcfbcb2-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.470 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.471 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.472 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.472 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.473 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:06:22.474 DEBUG:boto:Data: 2023-10-12T01:06:22.475 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.476 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.476 DEBUG:boto:Params: {} 2023-10-12T01:06:22.477 DEBUG:boto:Token: None 2023-10-12T01:06:22.477 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116 2023-10-12T01:06:22.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CjpNowdNtD235+FP0pFcSwcZ7Zs= 2023-10-12T01:06:22.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CjpNowdNtD235+FP0pFcSwcZ7Zs=', 'Content-Length': '0'} 2023-10-12T01:06:22.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b56355764377f12-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.481 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.481 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.482 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.482 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:06:22.483 DEBUG:boto:Data: 2023-10-12T01:06:22.483 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.484 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.484 DEBUG:boto:Params: {} 2023-10-12T01:06:22.485 DEBUG:boto:Token: None 2023-10-12T01:06:22.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116 2023-10-12T01:06:22.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CjpNowdNtD235+FP0pFcSwcZ7Zs= 2023-10-12T01:06:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CjpNowdNtD235+FP0pFcSwcZ7Zs=', 'Content-Length': '0'} 2023-10-12T01:06:22.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5aec0b371235c87-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:06:22.489 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.489 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.490 DEBUG:boto:Method: GET 2023-10-12T01:06:22.490 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:06:22.491 DEBUG:boto:Data: 2023-10-12T01:06:22.491 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.492 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.492 DEBUG:boto:Params: {} 2023-10-12T01:06:22.493 DEBUG:boto:Token: None 2023-10-12T01:06:22.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116 2023-10-12T01:06:22.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sl2X77rl7C6a7tBlO9lwkjxkZYw= 2023-10-12T01:06:22.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sl2X77rl7C6a7tBlO9lwkjxkZYw=', 'Content-Length': '0'} 2023-10-12T01:06:22.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c95f77c70fe6d391-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.497 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:06:22.497 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:06:22.498 DEBUG:boto:Method: GET 2023-10-12T01:06:22.498 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:06:22.499 DEBUG:boto:Data: 2023-10-12T01:06:22.499 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.500 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.501 DEBUG:boto:Params: {} 2023-10-12T01:06:22.501 DEBUG:boto:Token: None 2023-10-12T01:06:22.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj116 2023-10-12T01:06:22.502 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sl2X77rl7C6a7tBlO9lwkjxkZYw= 2023-10-12T01:06:22.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sl2X77rl7C6a7tBlO9lwkjxkZYw=', 'Content-Length': '0'} 2023-10-12T01:06:22.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe816b0a4b3327ba-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.505 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.506 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.506 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.507 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.507 DEBUG:boto:Method: GET 2023-10-12T01:06:22.508 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.508 DEBUG:boto:Data: 2023-10-12T01:06:22.509 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.510 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.511 DEBUG:boto:Params: {} 2023-10-12T01:06:22.511 DEBUG:boto:Token: None 2023-10-12T01:06:22.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ea871398eac0bc-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.516 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.517 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.517 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.518 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.518 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.518 DEBUG:boto:Method: GET 2023-10-12T01:06:22.519 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:06:22.519 DEBUG:boto:Data: 2023-10-12T01:06:22.520 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.521 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.521 DEBUG:boto:Params: {} 2023-10-12T01:06:22.521 DEBUG:boto:Token: None 2023-10-12T01:06:22.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007306d1a83d439270-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.527 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:06:22.528 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.528 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.529 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.529 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.530 DEBUG:boto:Method: GET 2023-10-12T01:06:22.530 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.531 DEBUG:boto:Data: 2023-10-12T01:06:22.531 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.532 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.532 DEBUG:boto:Params: {} 2023-10-12T01:06:22.533 DEBUG:boto:Token: None 2023-10-12T01:06:22.533 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0= 2023-10-12T01:06:22.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0=', 'Content-Length': '0'} 2023-10-12T01:06:22.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e217596d4e4b0a-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.578 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.579 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.579 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.580 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.580 DEBUG:boto:Method: GET 2023-10-12T01:06:22.581 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.581 DEBUG:boto:Data: 2023-10-12T01:06:22.581 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.582 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.583 DEBUG:boto:Params: {} 2023-10-12T01:06:22.583 DEBUG:boto:Token: None 2023-10-12T01:06:22.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0= 2023-10-12T01:06:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0=', 'Content-Length': '0'} 2023-10-12T01:06:22.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d06e7516c98aaf-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.588 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.588 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.588 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.589 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.589 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.590 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.590 DEBUG:boto:Data: 2023-10-12T01:06:22.590 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.591 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.592 DEBUG:boto:Params: {} 2023-10-12T01:06:22.592 DEBUG:boto:Token: None 2023-10-12T01:06:22.593 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tgZro6zjzn+IWicE7L2rGkDzTeQ= 2023-10-12T01:06:22.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tgZro6zjzn+IWicE7L2rGkDzTeQ=', 'Content-Length': '0'} 2023-10-12T01:06:22.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a61229bb4e3014-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.598 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.599 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.599 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.600 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.601 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.601 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.602 DEBUG:boto:Data: 2023-10-12T01:06:22.603 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.604 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.605 DEBUG:boto:Params: {} 2023-10-12T01:06:22.606 DEBUG:boto:Token: None 2023-10-12T01:06:22.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tgZro6zjzn+IWicE7L2rGkDzTeQ= 2023-10-12T01:06:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tgZro6zjzn+IWicE7L2rGkDzTeQ=', 'Content-Length': '0'} 2023-10-12T01:06:22.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008236bc85a64d2032-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:06:22.613 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.613 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.614 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.615 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.616 DEBUG:boto:Method: GET 2023-10-12T01:06:22.616 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.617 DEBUG:boto:Data: 2023-10-12T01:06:22.618 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.619 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.620 DEBUG:boto:Params: {} 2023-10-12T01:06:22.621 DEBUG:boto:Token: None 2023-10-12T01:06:22.622 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0= 2023-10-12T01:06:22.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0=', 'Content-Length': '0'} 2023-10-12T01:06:22.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69b5afb287a7c3b-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.630 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.631 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.631 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.632 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.632 DEBUG:boto:Method: GET 2023-10-12T01:06:22.633 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.633 DEBUG:boto:Data: 2023-10-12T01:06:22.633 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.634 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.635 DEBUG:boto:Params: {} 2023-10-12T01:06:22.635 DEBUG:boto:Token: None 2023-10-12T01:06:22.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:06:22.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0= 2023-10-12T01:06:22.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7U5ux20NEsyQKF7j2yYQKKHTVK0=', 'Content-Length': '0'} 2023-10-12T01:06:22.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b50644808a9a2d-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.682 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.683 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.684 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.684 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.684 DEBUG:boto:Method: GET 2023-10-12T01:06:22.685 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.685 DEBUG:boto:Data: 2023-10-12T01:06:22.686 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.687 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.687 DEBUG:boto:Params: {} 2023-10-12T01:06:22.688 DEBUG:boto:Token: None 2023-10-12T01:06:22.688 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041bb444a5414e568-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.693 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.693 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.694 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.694 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.695 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.695 DEBUG:boto:Method: GET 2023-10-12T01:06:22.696 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.696 DEBUG:boto:Data: 2023-10-12T01:06:22.696 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.697 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.698 DEBUG:boto:Params: {} 2023-10-12T01:06:22.698 DEBUG:boto:Token: None 2023-10-12T01:06:22.699 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9dbdf0e7cbecc0-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.704 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.705 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.705 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.706 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.706 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:06:22.706 DEBUG:boto:Data: 2023-10-12T01:06:22.707 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.708 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.708 DEBUG:boto:Params: {} 2023-10-12T01:06:22.708 DEBUG:boto:Token: None 2023-10-12T01:06:22.709 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145 2023-10-12T01:06:22.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yty/okNR61j+O5NyE/lsHRjSSbg= 2023-10-12T01:06:22.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yty/okNR61j+O5NyE/lsHRjSSbg=', 'Content-Length': '0'} 2023-10-12T01:06:22.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f48756d9f1b0da-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.713 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.713 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.713 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.714 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:06:22.714 DEBUG:boto:Data: 2023-10-12T01:06:22.715 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.716 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.716 DEBUG:boto:Params: {} 2023-10-12T01:06:22.717 DEBUG:boto:Token: None 2023-10-12T01:06:22.717 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145 2023-10-12T01:06:22.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yty/okNR61j+O5NyE/lsHRjSSbg= 2023-10-12T01:06:22.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yty/okNR61j+O5NyE/lsHRjSSbg=', 'Content-Length': '0'} 2023-10-12T01:06:22.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd4afa9955f35cd-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:06:22.721 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.722 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.722 DEBUG:boto:Method: GET 2023-10-12T01:06:22.723 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:06:22.723 DEBUG:boto:Data: 2023-10-12T01:06:22.723 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.724 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.725 DEBUG:boto:Params: {} 2023-10-12T01:06:22.725 DEBUG:boto:Token: None 2023-10-12T01:06:22.726 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145 2023-10-12T01:06:22.726 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EyYdJjQMzJqvPufHo50YyAmKLR0= 2023-10-12T01:06:22.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EyYdJjQMzJqvPufHo50YyAmKLR0=', 'Content-Length': '0'} 2023-10-12T01:06:22.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789926e4bb2e3124-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.729 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:06:22.730 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:06:22.730 DEBUG:boto:Method: GET 2023-10-12T01:06:22.730 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:06:22.731 DEBUG:boto:Data: 2023-10-12T01:06:22.731 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.732 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.733 DEBUG:boto:Params: {} 2023-10-12T01:06:22.733 DEBUG:boto:Token: None 2023-10-12T01:06:22.734 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj145 2023-10-12T01:06:22.734 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EyYdJjQMzJqvPufHo50YyAmKLR0= 2023-10-12T01:06:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EyYdJjQMzJqvPufHo50YyAmKLR0=', 'Content-Length': '0'} 2023-10-12T01:06:22.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c0e76dec318c4d-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.737 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.738 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.738 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.739 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.739 DEBUG:boto:Method: GET 2023-10-12T01:06:22.739 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.740 DEBUG:boto:Data: 2023-10-12T01:06:22.740 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.741 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.741 DEBUG:boto:Params: {} 2023-10-12T01:06:22.742 DEBUG:boto:Token: None 2023-10-12T01:06:22.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.743 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e56b1be0205f877-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.747 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.747 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.748 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.748 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.749 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.749 DEBUG:boto:Method: GET 2023-10-12T01:06:22.750 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:06:22.750 DEBUG:boto:Data: 2023-10-12T01:06:22.750 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.751 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.752 DEBUG:boto:Params: {} 2023-10-12T01:06:22.752 DEBUG:boto:Token: None 2023-10-12T01:06:22.753 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e24f6ed990b311a4-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.758 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:06:22.759 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.760 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.760 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.760 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.761 DEBUG:boto:Method: GET 2023-10-12T01:06:22.761 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.762 DEBUG:boto:Data: 2023-10-12T01:06:22.762 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.763 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.764 DEBUG:boto:Params: {} 2023-10-12T01:06:22.764 DEBUG:boto:Token: None 2023-10-12T01:06:22.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA= 2023-10-12T01:06:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA=', 'Content-Length': '0'} 2023-10-12T01:06:22.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e43e0963b8c05d83-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.769 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.769 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.769 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.770 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.771 DEBUG:boto:Method: GET 2023-10-12T01:06:22.771 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.771 DEBUG:boto:Data: 2023-10-12T01:06:22.772 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.773 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.773 DEBUG:boto:Params: {} 2023-10-12T01:06:22.774 DEBUG:boto:Token: None 2023-10-12T01:06:22.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA= 2023-10-12T01:06:22.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA=', 'Content-Length': '0'} 2023-10-12T01:06:22.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cf76416d7d737e-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.778 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.778 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.779 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.779 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.779 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.780 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.780 DEBUG:boto:Data: 2023-10-12T01:06:22.781 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.782 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.782 DEBUG:boto:Params: {} 2023-10-12T01:06:22.782 DEBUG:boto:Token: None 2023-10-12T01:06:22.783 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Iuxev0Kizr6aP7eFtHDMwaAkyTM= 2023-10-12T01:06:22.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Iuxev0Kizr6aP7eFtHDMwaAkyTM=', 'Content-Length': '0'} 2023-10-12T01:06:22.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb572501bf50a80-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.786 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.787 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.787 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.787 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.788 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.788 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.789 DEBUG:boto:Data: 2023-10-12T01:06:22.789 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.790 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.790 DEBUG:boto:Params: {} 2023-10-12T01:06:22.791 DEBUG:boto:Token: None 2023-10-12T01:06:22.791 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Iuxev0Kizr6aP7eFtHDMwaAkyTM= 2023-10-12T01:06:22.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Iuxev0Kizr6aP7eFtHDMwaAkyTM=', 'Content-Length': '0'} 2023-10-12T01:06:22.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbff6d1c0cd3a01-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:06:22.795 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.795 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.796 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.796 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.797 DEBUG:boto:Method: GET 2023-10-12T01:06:22.797 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.798 DEBUG:boto:Data: 2023-10-12T01:06:22.798 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.799 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.799 DEBUG:boto:Params: {} 2023-10-12T01:06:22.800 DEBUG:boto:Token: None 2023-10-12T01:06:22.800 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA= 2023-10-12T01:06:22.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA=', 'Content-Length': '0'} 2023-10-12T01:06:22.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532ef1fa11663c2f-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.804 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.804 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.805 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.805 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.806 DEBUG:boto:Method: GET 2023-10-12T01:06:22.806 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.807 DEBUG:boto:Data: 2023-10-12T01:06:22.807 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.808 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.808 DEBUG:boto:Params: {} 2023-10-12T01:06:22.809 DEBUG:boto:Token: None 2023-10-12T01:06:22.809 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:06:22.810 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA= 2023-10-12T01:06:22.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kzxt3KT5b52y8vwiewRqTppqSUA=', 'Content-Length': '0'} 2023-10-12T01:06:22.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d426af6920bf5c-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.813 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.814 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.814 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.814 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.815 DEBUG:boto:Method: GET 2023-10-12T01:06:22.815 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.816 DEBUG:boto:Data: 2023-10-12T01:06:22.816 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.817 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.817 DEBUG:boto:Params: {} 2023-10-12T01:06:22.818 DEBUG:boto:Token: None 2023-10-12T01:06:22.818 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d96cf4eb9aec06e-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.822 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.823 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.824 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.824 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.825 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.825 DEBUG:boto:Method: GET 2023-10-12T01:06:22.825 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.826 DEBUG:boto:Data: 2023-10-12T01:06:22.826 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.827 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.828 DEBUG:boto:Params: {} 2023-10-12T01:06:22.828 DEBUG:boto:Token: None 2023-10-12T01:06:22.829 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636bb42f5a57d0ce-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.834 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.835 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.835 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.835 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.836 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:06:22.836 DEBUG:boto:Data: 2023-10-12T01:06:22.837 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.837 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.838 DEBUG:boto:Params: {} 2023-10-12T01:06:22.838 DEBUG:boto:Token: None 2023-10-12T01:06:22.839 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17 2023-10-12T01:06:22.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qLNafHHBLURs/CKTIBCx/ZljBq4= 2023-10-12T01:06:22.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qLNafHHBLURs/CKTIBCx/ZljBq4=', 'Content-Length': '0'} 2023-10-12T01:06:22.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c642b036ef7670c0-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.842 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.842 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.843 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.843 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:06:22.844 DEBUG:boto:Data: 2023-10-12T01:06:22.844 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.845 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.845 DEBUG:boto:Params: {} 2023-10-12T01:06:22.846 DEBUG:boto:Token: None 2023-10-12T01:06:22.846 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17 2023-10-12T01:06:22.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qLNafHHBLURs/CKTIBCx/ZljBq4= 2023-10-12T01:06:22.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qLNafHHBLURs/CKTIBCx/ZljBq4=', 'Content-Length': '0'} 2023-10-12T01:06:22.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf06b0cf446caf53-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:06:22.850 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.850 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.851 DEBUG:boto:Method: GET 2023-10-12T01:06:22.851 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:06:22.852 DEBUG:boto:Data: 2023-10-12T01:06:22.852 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.853 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.854 DEBUG:boto:Params: {} 2023-10-12T01:06:22.854 DEBUG:boto:Token: None 2023-10-12T01:06:22.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17 2023-10-12T01:06:22.855 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5bjNDiDnhUgVp9FjbaD3WnnGYTE= 2023-10-12T01:06:22.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5bjNDiDnhUgVp9FjbaD3WnnGYTE=', 'Content-Length': '0'} 2023-10-12T01:06:22.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f5fde7b00cea77-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.858 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:06:22.859 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:06:22.859 DEBUG:boto:Method: GET 2023-10-12T01:06:22.860 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:06:22.860 DEBUG:boto:Data: 2023-10-12T01:06:22.861 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.862 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.862 DEBUG:boto:Params: {} 2023-10-12T01:06:22.862 DEBUG:boto:Token: None 2023-10-12T01:06:22.863 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj17 2023-10-12T01:06:22.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5bjNDiDnhUgVp9FjbaD3WnnGYTE= 2023-10-12T01:06:22.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5bjNDiDnhUgVp9FjbaD3WnnGYTE=', 'Content-Length': '0'} 2023-10-12T01:06:22.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004974c91bfeacc809-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.866 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.867 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.867 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.868 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.868 DEBUG:boto:Method: GET 2023-10-12T01:06:22.869 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.869 DEBUG:boto:Data: 2023-10-12T01:06:22.870 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.870 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.871 DEBUG:boto:Params: {} 2023-10-12T01:06:22.871 DEBUG:boto:Token: None 2023-10-12T01:06:22.872 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f24dccd1da5bff8-006527468e-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.876 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.877 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:22.877 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:22.878 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.878 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.879 DEBUG:boto:Method: GET 2023-10-12T01:06:22.879 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:06:22.879 DEBUG:boto:Data: 2023-10-12T01:06:22.880 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.881 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.881 DEBUG:boto:Params: {} 2023-10-12T01:06:22.882 DEBUG:boto:Token: None 2023-10-12T01:06:22.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/ 2023-10-12T01:06:22.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA= 2023-10-12T01:06:22.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NDcWBu0gksfZDMVP8y+UQ/ndjCA=', 'Content-Length': '0'} 2023-10-12T01:06:22.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b518ab5bf8d4638a-006527468e-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.887 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:22.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:06:22.888 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.889 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.889 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.889 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.890 DEBUG:boto:Method: GET 2023-10-12T01:06:22.890 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.891 DEBUG:boto:Data: 2023-10-12T01:06:22.891 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.892 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.893 DEBUG:boto:Params: {} 2023-10-12T01:06:22.893 DEBUG:boto:Token: None 2023-10-12T01:06:22.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw= 2023-10-12T01:06:22.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw=', 'Content-Length': '0'} 2023-10-12T01:06:22.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8e1032b15ce4e3-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.938 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.939 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.939 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.940 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.940 DEBUG:boto:Method: GET 2023-10-12T01:06:22.941 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.941 DEBUG:boto:Data: 2023-10-12T01:06:22.941 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.942 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.943 DEBUG:boto:Params: {} 2023-10-12T01:06:22.943 DEBUG:boto:Token: None 2023-10-12T01:06:22.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw= 2023-10-12T01:06:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw=', 'Content-Length': '0'} 2023-10-12T01:06:22.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000384a430745c4f3b4-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.947 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.948 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.948 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.949 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.949 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.950 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.950 DEBUG:boto:Data: 2023-10-12T01:06:22.950 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.951 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.952 DEBUG:boto:Params: {} 2023-10-12T01:06:22.952 DEBUG:boto:Token: None 2023-10-12T01:06:22.952 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZWj5zkcO4DLX3oQzMjs2HEdhJRA= 2023-10-12T01:06:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZWj5zkcO4DLX3oQzMjs2HEdhJRA=', 'Content-Length': '0'} 2023-10-12T01:06:22.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645aef9f0ae95024-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.956 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.956 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.957 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.957 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.957 DEBUG:boto:Method: HEAD 2023-10-12T01:06:22.958 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.958 DEBUG:boto:Data: 2023-10-12T01:06:22.959 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.959 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.960 DEBUG:boto:Params: {} 2023-10-12T01:06:22.960 DEBUG:boto:Token: None 2023-10-12T01:06:22.961 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZWj5zkcO4DLX3oQzMjs2HEdhJRA= 2023-10-12T01:06:22.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZWj5zkcO4DLX3oQzMjs2HEdhJRA=', 'Content-Length': '0'} 2023-10-12T01:06:22.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d64346765fde38-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:06:22.964 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.965 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.965 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.966 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.966 DEBUG:boto:Method: GET 2023-10-12T01:06:22.967 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.967 DEBUG:boto:Data: 2023-10-12T01:06:22.967 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:22.968 DEBUG:boto:Port: 8001 2023-10-12T01:06:22.969 DEBUG:boto:Params: {} 2023-10-12T01:06:22.969 DEBUG:boto:Token: None 2023-10-12T01:06:22.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw= 2023-10-12T01:06:22.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw=', 'Content-Length': '0'} 2023-10-12T01:06:22.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d216b64dd9de0488-006527468e-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:22.973 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:22.974 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:22.974 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.975 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.975 DEBUG:boto:Method: GET 2023-10-12T01:06:22.976 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.976 DEBUG:boto:Data: 2023-10-12T01:06:22.977 DEBUG:boto:Headers: {} 2023-10-12T01:06:22.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:22.977 DEBUG:boto:Port: 8000 2023-10-12T01:06:22.978 DEBUG:boto:Params: {} 2023-10-12T01:06:22.978 DEBUG:boto:Token: None 2023-10-12T01:06:22.979 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:22 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:06:22.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw= 2023-10-12T01:06:22.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z5YtnC16ZP11ViMSbHhNdUa/Mtw=', 'Content-Length': '0'} 2023-10-12T01:06:22.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065075199f2e11463-006527468e-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.022 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.023 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.023 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.024 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.024 DEBUG:boto:Method: GET 2023-10-12T01:06:23.025 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.025 DEBUG:boto:Data: 2023-10-12T01:06:23.026 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.027 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.027 DEBUG:boto:Params: {} 2023-10-12T01:06:23.028 DEBUG:boto:Token: None 2023-10-12T01:06:23.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d3520ae0125f9f-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.033 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.034 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.035 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.035 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.036 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.036 DEBUG:boto:Method: GET 2023-10-12T01:06:23.036 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.037 DEBUG:boto:Data: 2023-10-12T01:06:23.037 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.038 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.039 DEBUG:boto:Params: {} 2023-10-12T01:06:23.039 DEBUG:boto:Token: None 2023-10-12T01:06:23.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b89bc3ab136d7b-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.045 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.045 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:23.046 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:23.046 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.047 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:06:23.047 DEBUG:boto:Data: 2023-10-12T01:06:23.047 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.048 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.049 DEBUG:boto:Params: {} 2023-10-12T01:06:23.049 DEBUG:boto:Token: None 2023-10-12T01:06:23.050 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj19 2023-10-12T01:06:23.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nv83um8LTlkJGdyM92Oy6HWb7o4= 2023-10-12T01:06:23.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nv83um8LTlkJGdyM92Oy6HWb7o4=', 'Content-Length': '0'} 2023-10-12T01:06:23.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd660033f19a8116-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.053 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:23.054 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:23.054 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.054 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:06:23.055 DEBUG:boto:Data: 2023-10-12T01:06:23.055 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.056 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.056 DEBUG:boto:Params: {} 2023-10-12T01:06:23.057 DEBUG:boto:Token: None 2023-10-12T01:06:23.057 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj19 2023-10-12T01:06:23.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nv83um8LTlkJGdyM92Oy6HWb7o4= 2023-10-12T01:06:23.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nv83um8LTlkJGdyM92Oy6HWb7o4=', 'Content-Length': '0'} 2023-10-12T01:06:23.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417c3c7200d918b8-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:06:23.061 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:23.062 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:23.062 DEBUG:boto:Method: GET 2023-10-12T01:06:23.063 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:06:23.063 DEBUG:boto:Data: 2023-10-12T01:06:23.063 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.064 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.065 DEBUG:boto:Params: {} 2023-10-12T01:06:23.065 DEBUG:boto:Token: None 2023-10-12T01:06:23.066 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj19 2023-10-12T01:06:23.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3VgM/y2KBWvPE7wdlkqApV0WHvQ= 2023-10-12T01:06:23.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3VgM/y2KBWvPE7wdlkqApV0WHvQ=', 'Content-Length': '0'} 2023-10-12T01:06:23.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779114944d79b350-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.114 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:06:23.115 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:06:23.115 DEBUG:boto:Method: GET 2023-10-12T01:06:23.116 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:06:23.116 DEBUG:boto:Data: 2023-10-12T01:06:23.116 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.117 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.118 DEBUG:boto:Params: {} 2023-10-12T01:06:23.118 DEBUG:boto:Token: None 2023-10-12T01:06:23.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj19 2023-10-12T01:06:23.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3VgM/y2KBWvPE7wdlkqApV0WHvQ= 2023-10-12T01:06:23.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3VgM/y2KBWvPE7wdlkqApV0WHvQ=', 'Content-Length': '0'} 2023-10-12T01:06:23.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c47f310be38340-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.122 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.123 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.123 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.124 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.124 DEBUG:boto:Method: GET 2023-10-12T01:06:23.124 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.125 DEBUG:boto:Data: 2023-10-12T01:06:23.125 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.126 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.126 DEBUG:boto:Params: {} 2023-10-12T01:06:23.127 DEBUG:boto:Token: None 2023-10-12T01:06:23.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.128 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7d66183d7f816f-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.131 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.132 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.132 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.132 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.133 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.133 DEBUG:boto:Method: GET 2023-10-12T01:06:23.134 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:06:23.134 DEBUG:boto:Data: 2023-10-12T01:06:23.135 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.135 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.136 DEBUG:boto:Params: {} 2023-10-12T01:06:23.136 DEBUG:boto:Token: None 2023-10-12T01:06:23.137 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001af3255c36677f88-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.141 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:06:23.143 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.143 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.143 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.144 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.144 DEBUG:boto:Method: GET 2023-10-12T01:06:23.145 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.145 DEBUG:boto:Data: 2023-10-12T01:06:23.146 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.146 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.147 DEBUG:boto:Params: {} 2023-10-12T01:06:23.147 DEBUG:boto:Token: None 2023-10-12T01:06:23.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc= 2023-10-12T01:06:23.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc=', 'Content-Length': '0'} 2023-10-12T01:06:23.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1bd0c471df1ab9e-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.152 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.153 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.154 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.154 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.155 DEBUG:boto:Method: GET 2023-10-12T01:06:23.156 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.157 DEBUG:boto:Data: 2023-10-12T01:06:23.157 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.158 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.158 DEBUG:boto:Params: {} 2023-10-12T01:06:23.159 DEBUG:boto:Token: None 2023-10-12T01:06:23.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc= 2023-10-12T01:06:23.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc=', 'Content-Length': '0'} 2023-10-12T01:06:23.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd54ecd06fd7792-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.163 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.164 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.164 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.164 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.165 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.165 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.165 DEBUG:boto:Data: 2023-10-12T01:06:23.166 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.167 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.167 DEBUG:boto:Params: {} 2023-10-12T01:06:23.168 DEBUG:boto:Token: None 2023-10-12T01:06:23.168 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F1lNErNg3Pwebmm3SySbsIOVivI= 2023-10-12T01:06:23.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F1lNErNg3Pwebmm3SySbsIOVivI=', 'Content-Length': '0'} 2023-10-12T01:06:23.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0b16a8a9cd11216-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.172 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.173 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.173 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.173 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.174 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.174 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.175 DEBUG:boto:Data: 2023-10-12T01:06:23.175 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.176 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.177 DEBUG:boto:Params: {} 2023-10-12T01:06:23.177 DEBUG:boto:Token: None 2023-10-12T01:06:23.178 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.178 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F1lNErNg3Pwebmm3SySbsIOVivI= 2023-10-12T01:06:23.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F1lNErNg3Pwebmm3SySbsIOVivI=', 'Content-Length': '0'} 2023-10-12T01:06:23.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053670daed7cbddbb-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:06:23.182 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.182 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.183 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.183 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.183 DEBUG:boto:Method: GET 2023-10-12T01:06:23.184 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.184 DEBUG:boto:Data: 2023-10-12T01:06:23.185 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.186 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.186 DEBUG:boto:Params: {} 2023-10-12T01:06:23.187 DEBUG:boto:Token: None 2023-10-12T01:06:23.187 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc= 2023-10-12T01:06:23.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc=', 'Content-Length': '0'} 2023-10-12T01:06:23.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034cd821eecfe336b-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.230 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.231 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.232 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.232 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.233 DEBUG:boto:Method: GET 2023-10-12T01:06:23.233 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.234 DEBUG:boto:Data: 2023-10-12T01:06:23.234 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.235 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.236 DEBUG:boto:Params: {} 2023-10-12T01:06:23.236 DEBUG:boto:Token: None 2023-10-12T01:06:23.237 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:06:23.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc= 2023-10-12T01:06:23.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LAH+K+uqysdolPl0TUWVoCQ6jEc=', 'Content-Length': '0'} 2023-10-12T01:06:23.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ddeafe4d6785b0-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.241 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.242 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.243 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.243 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.243 DEBUG:boto:Method: GET 2023-10-12T01:06:23.244 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.244 DEBUG:boto:Data: 2023-10-12T01:06:23.245 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.246 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.246 DEBUG:boto:Params: {} 2023-10-12T01:06:23.247 DEBUG:boto:Token: None 2023-10-12T01:06:23.248 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048e4a1f790f9c968-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.252 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.253 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.253 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.254 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.254 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.255 DEBUG:boto:Method: GET 2023-10-12T01:06:23.255 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.256 DEBUG:boto:Data: 2023-10-12T01:06:23.256 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.257 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.258 DEBUG:boto:Params: {} 2023-10-12T01:06:23.259 DEBUG:boto:Token: None 2023-10-12T01:06:23.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478fd026c7172ea0-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.264 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.265 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.265 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.266 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.266 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:06:23.267 DEBUG:boto:Data: 2023-10-12T01:06:23.267 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.268 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.269 DEBUG:boto:Params: {} 2023-10-12T01:06:23.270 DEBUG:boto:Token: None 2023-10-12T01:06:23.270 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22 2023-10-12T01:06:23.271 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aMnk6Tpve0NRkKIPT67Wlazzh8M= 2023-10-12T01:06:23.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aMnk6Tpve0NRkKIPT67Wlazzh8M=', 'Content-Length': '0'} 2023-10-12T01:06:23.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52abda75d6aa200-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.274 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.274 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.275 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.275 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:06:23.276 DEBUG:boto:Data: 2023-10-12T01:06:23.276 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.277 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.278 DEBUG:boto:Params: {} 2023-10-12T01:06:23.279 DEBUG:boto:Token: None 2023-10-12T01:06:23.279 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22 2023-10-12T01:06:23.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aMnk6Tpve0NRkKIPT67Wlazzh8M= 2023-10-12T01:06:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aMnk6Tpve0NRkKIPT67Wlazzh8M=', 'Content-Length': '0'} 2023-10-12T01:06:23.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097205a05bd6d7e4b-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:06:23.284 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.284 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.285 DEBUG:boto:Method: GET 2023-10-12T01:06:23.285 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:06:23.286 DEBUG:boto:Data: 2023-10-12T01:06:23.286 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.287 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.288 DEBUG:boto:Params: {} 2023-10-12T01:06:23.288 DEBUG:boto:Token: None 2023-10-12T01:06:23.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22 2023-10-12T01:06:23.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i4g2BPQPs9SndiKBSndbiPoZZM= 2023-10-12T01:06:23.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i4g2BPQPs9SndiKBSndbiPoZZM=', 'Content-Length': '0'} 2023-10-12T01:06:23.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9dfd53964d36eae-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.293 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:06:23.293 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:06:23.294 DEBUG:boto:Method: GET 2023-10-12T01:06:23.294 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:06:23.295 DEBUG:boto:Data: 2023-10-12T01:06:23.295 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.296 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.297 DEBUG:boto:Params: {} 2023-10-12T01:06:23.297 DEBUG:boto:Token: None 2023-10-12T01:06:23.298 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj22 2023-10-12T01:06:23.298 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+i4g2BPQPs9SndiKBSndbiPoZZM= 2023-10-12T01:06:23.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+i4g2BPQPs9SndiKBSndbiPoZZM=', 'Content-Length': '0'} 2023-10-12T01:06:23.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d915036c80d62d1-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.302 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.302 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.303 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.303 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.304 DEBUG:boto:Method: GET 2023-10-12T01:06:23.304 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.305 DEBUG:boto:Data: 2023-10-12T01:06:23.305 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.306 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.307 DEBUG:boto:Params: {} 2023-10-12T01:06:23.307 DEBUG:boto:Token: None 2023-10-12T01:06:23.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971cfe3b847c563e-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.312 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.313 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.313 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.314 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.314 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.315 DEBUG:boto:Method: GET 2023-10-12T01:06:23.315 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:06:23.316 DEBUG:boto:Data: 2023-10-12T01:06:23.316 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.317 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.318 DEBUG:boto:Params: {} 2023-10-12T01:06:23.318 DEBUG:boto:Token: None 2023-10-12T01:06:23.319 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f79f00cdf1762e0-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.324 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:06:23.325 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.325 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.326 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.327 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.327 DEBUG:boto:Method: GET 2023-10-12T01:06:23.328 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.328 DEBUG:boto:Data: 2023-10-12T01:06:23.329 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.329 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.330 DEBUG:boto:Params: {} 2023-10-12T01:06:23.331 DEBUG:boto:Token: None 2023-10-12T01:06:23.331 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A= 2023-10-12T01:06:23.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A=', 'Content-Length': '0'} 2023-10-12T01:06:23.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a272ace8a3a3a886-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.378 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.379 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.379 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.380 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.380 DEBUG:boto:Method: GET 2023-10-12T01:06:23.381 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.381 DEBUG:boto:Data: 2023-10-12T01:06:23.381 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.383 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.383 DEBUG:boto:Params: {} 2023-10-12T01:06:23.383 DEBUG:boto:Token: None 2023-10-12T01:06:23.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A= 2023-10-12T01:06:23.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A=', 'Content-Length': '0'} 2023-10-12T01:06:23.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5921f80c260b14-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.388 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.388 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.389 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.389 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.390 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.390 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.390 DEBUG:boto:Data: 2023-10-12T01:06:23.391 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.391 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.392 DEBUG:boto:Params: {} 2023-10-12T01:06:23.392 DEBUG:boto:Token: None 2023-10-12T01:06:23.393 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SL9+CPVJghAmL4z6HDEs8Xh2idY= 2023-10-12T01:06:23.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SL9+CPVJghAmL4z6HDEs8Xh2idY=', 'Content-Length': '0'} 2023-10-12T01:06:23.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057fdc028c8afe7b4-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.396 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.396 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.397 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.397 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.397 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.398 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.398 DEBUG:boto:Data: 2023-10-12T01:06:23.399 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.399 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.400 DEBUG:boto:Params: {} 2023-10-12T01:06:23.400 DEBUG:boto:Token: None 2023-10-12T01:06:23.401 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SL9+CPVJghAmL4z6HDEs8Xh2idY= 2023-10-12T01:06:23.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SL9+CPVJghAmL4z6HDEs8Xh2idY=', 'Content-Length': '0'} 2023-10-12T01:06:23.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f644b9365aab4fc-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:06:23.405 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.405 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.405 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.406 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.407 DEBUG:boto:Method: GET 2023-10-12T01:06:23.407 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.407 DEBUG:boto:Data: 2023-10-12T01:06:23.408 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.409 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.409 DEBUG:boto:Params: {} 2023-10-12T01:06:23.409 DEBUG:boto:Token: None 2023-10-12T01:06:23.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A= 2023-10-12T01:06:23.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A=', 'Content-Length': '0'} 2023-10-12T01:06:23.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932243811074e8c0-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.413 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.414 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.414 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.415 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.415 DEBUG:boto:Method: GET 2023-10-12T01:06:23.415 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.416 DEBUG:boto:Data: 2023-10-12T01:06:23.416 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.417 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.417 DEBUG:boto:Params: {} 2023-10-12T01:06:23.418 DEBUG:boto:Token: None 2023-10-12T01:06:23.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:06:23.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A= 2023-10-12T01:06:23.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cSdMmKxv/WASBcoecC23EDB3U6A=', 'Content-Length': '0'} 2023-10-12T01:06:23.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ba8a5d34bd4f8e-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.462 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.463 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.463 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.464 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.465 DEBUG:boto:Method: GET 2023-10-12T01:06:23.465 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.465 DEBUG:boto:Data: 2023-10-12T01:06:23.466 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.467 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.467 DEBUG:boto:Params: {} 2023-10-12T01:06:23.468 DEBUG:boto:Token: None 2023-10-12T01:06:23.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c34e8b85a9839a-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.473 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.474 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.474 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.475 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.476 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.476 DEBUG:boto:Method: GET 2023-10-12T01:06:23.477 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.478 DEBUG:boto:Data: 2023-10-12T01:06:23.478 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.480 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.480 DEBUG:boto:Params: {} 2023-10-12T01:06:23.481 DEBUG:boto:Token: None 2023-10-12T01:06:23.482 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63d71dbbf0997c7-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.487 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.487 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.488 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.488 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.489 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:06:23.489 DEBUG:boto:Data: 2023-10-12T01:06:23.490 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.491 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.491 DEBUG:boto:Params: {} 2023-10-12T01:06:23.492 DEBUG:boto:Token: None 2023-10-12T01:06:23.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23 2023-10-12T01:06:23.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CrxN56PNxYKu2xGKa3/Wrxrnp2I= 2023-10-12T01:06:23.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CrxN56PNxYKu2xGKa3/Wrxrnp2I=', 'Content-Length': '0'} 2023-10-12T01:06:23.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f26abeceb29b3c3-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.496 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.496 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.496 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.497 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:06:23.497 DEBUG:boto:Data: 2023-10-12T01:06:23.498 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.499 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.499 DEBUG:boto:Params: {} 2023-10-12T01:06:23.500 DEBUG:boto:Token: None 2023-10-12T01:06:23.501 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23 2023-10-12T01:06:23.502 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CrxN56PNxYKu2xGKa3/Wrxrnp2I= 2023-10-12T01:06:23.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CrxN56PNxYKu2xGKa3/Wrxrnp2I=', 'Content-Length': '0'} 2023-10-12T01:06:23.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db11785ac3a6cc80-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:06:23.506 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.507 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.508 DEBUG:boto:Method: GET 2023-10-12T01:06:23.508 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:06:23.509 DEBUG:boto:Data: 2023-10-12T01:06:23.509 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.510 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.511 DEBUG:boto:Params: {} 2023-10-12T01:06:23.511 DEBUG:boto:Token: None 2023-10-12T01:06:23.512 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23 2023-10-12T01:06:23.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e6u3IIQITG22Fx9bPFb4TXPRuHc= 2023-10-12T01:06:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e6u3IIQITG22Fx9bPFb4TXPRuHc=', 'Content-Length': '0'} 2023-10-12T01:06:23.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d3616eb1f80626-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.515 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:06:23.516 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:06:23.516 DEBUG:boto:Method: GET 2023-10-12T01:06:23.517 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:06:23.517 DEBUG:boto:Data: 2023-10-12T01:06:23.518 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.518 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.519 DEBUG:boto:Params: {} 2023-10-12T01:06:23.519 DEBUG:boto:Token: None 2023-10-12T01:06:23.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj23 2023-10-12T01:06:23.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e6u3IIQITG22Fx9bPFb4TXPRuHc= 2023-10-12T01:06:23.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e6u3IIQITG22Fx9bPFb4TXPRuHc=', 'Content-Length': '0'} 2023-10-12T01:06:23.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cdec88dd9090f7f-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.524 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.524 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.525 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.525 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.526 DEBUG:boto:Method: GET 2023-10-12T01:06:23.526 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.526 DEBUG:boto:Data: 2023-10-12T01:06:23.527 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.528 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.528 DEBUG:boto:Params: {} 2023-10-12T01:06:23.529 DEBUG:boto:Token: None 2023-10-12T01:06:23.529 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd46d4227547546-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.574 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.575 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.576 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.577 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.578 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.578 DEBUG:boto:Method: GET 2023-10-12T01:06:23.579 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:06:23.580 DEBUG:boto:Data: 2023-10-12T01:06:23.580 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.582 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.583 DEBUG:boto:Params: {} 2023-10-12T01:06:23.583 DEBUG:boto:Token: None 2023-10-12T01:06:23.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae6fbfdb9b7d8810-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.589 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:06:23.590 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.591 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.591 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.592 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.592 DEBUG:boto:Method: GET 2023-10-12T01:06:23.593 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.593 DEBUG:boto:Data: 2023-10-12T01:06:23.593 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.594 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.595 DEBUG:boto:Params: {} 2023-10-12T01:06:23.596 DEBUG:boto:Token: None 2023-10-12T01:06:23.597 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI= 2023-10-12T01:06:23.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI=', 'Content-Length': '0'} 2023-10-12T01:06:23.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e844af00497e0eb-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.602 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.603 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.603 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.603 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.604 DEBUG:boto:Method: GET 2023-10-12T01:06:23.604 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.605 DEBUG:boto:Data: 2023-10-12T01:06:23.605 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.606 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.607 DEBUG:boto:Params: {} 2023-10-12T01:06:23.607 DEBUG:boto:Token: None 2023-10-12T01:06:23.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI= 2023-10-12T01:06:23.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI=', 'Content-Length': '0'} 2023-10-12T01:06:23.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005213f0ffee1b189-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.655 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.655 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.656 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.656 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.657 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.657 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.658 DEBUG:boto:Data: 2023-10-12T01:06:23.658 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.659 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.659 DEBUG:boto:Params: {} 2023-10-12T01:06:23.660 DEBUG:boto:Token: None 2023-10-12T01:06:23.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iMcQkfJPDyeGGYKsdKIgYG5OJqA= 2023-10-12T01:06:23.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iMcQkfJPDyeGGYKsdKIgYG5OJqA=', 'Content-Length': '0'} 2023-10-12T01:06:23.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4512aefa872c478-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.664 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.664 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.665 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.665 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.666 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.666 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.667 DEBUG:boto:Data: 2023-10-12T01:06:23.667 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.668 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.668 DEBUG:boto:Params: {} 2023-10-12T01:06:23.669 DEBUG:boto:Token: None 2023-10-12T01:06:23.669 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iMcQkfJPDyeGGYKsdKIgYG5OJqA= 2023-10-12T01:06:23.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iMcQkfJPDyeGGYKsdKIgYG5OJqA=', 'Content-Length': '0'} 2023-10-12T01:06:23.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055fccf793942d34-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:06:23.673 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.673 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.674 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.674 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.675 DEBUG:boto:Method: GET 2023-10-12T01:06:23.675 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.676 DEBUG:boto:Data: 2023-10-12T01:06:23.676 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.677 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.677 DEBUG:boto:Params: {} 2023-10-12T01:06:23.678 DEBUG:boto:Token: None 2023-10-12T01:06:23.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI= 2023-10-12T01:06:23.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI=', 'Content-Length': '0'} 2023-10-12T01:06:23.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ca6ba90f9e5290-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.682 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.683 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.684 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.684 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.685 DEBUG:boto:Method: GET 2023-10-12T01:06:23.685 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.686 DEBUG:boto:Data: 2023-10-12T01:06:23.687 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.687 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.688 DEBUG:boto:Params: {} 2023-10-12T01:06:23.688 DEBUG:boto:Token: None 2023-10-12T01:06:23.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:06:23.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI= 2023-10-12T01:06:23.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fgSqk7Y9eohi4QX8is7Q3inIGI=', 'Content-Length': '0'} 2023-10-12T01:06:23.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e766369c32d10a-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.692 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.693 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.693 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.694 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.694 DEBUG:boto:Method: GET 2023-10-12T01:06:23.694 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.695 DEBUG:boto:Data: 2023-10-12T01:06:23.695 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.696 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.696 DEBUG:boto:Params: {} 2023-10-12T01:06:23.697 DEBUG:boto:Token: None 2023-10-12T01:06:23.697 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.698 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee234bf8e639bf5-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.701 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.702 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.702 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.702 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.703 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.703 DEBUG:boto:Method: GET 2023-10-12T01:06:23.704 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.704 DEBUG:boto:Data: 2023-10-12T01:06:23.704 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.705 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.706 DEBUG:boto:Params: {} 2023-10-12T01:06:23.706 DEBUG:boto:Token: None 2023-10-12T01:06:23.707 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a283a29ad644c40b-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.712 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.712 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.713 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.713 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.713 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:06:23.714 DEBUG:boto:Data: 2023-10-12T01:06:23.714 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.715 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.716 DEBUG:boto:Params: {} 2023-10-12T01:06:23.716 DEBUG:boto:Token: None 2023-10-12T01:06:23.717 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29 2023-10-12T01:06:23.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x1RjC/irQwaKtyvjm55ysxHykIc= 2023-10-12T01:06:23.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x1RjC/irQwaKtyvjm55ysxHykIc=', 'Content-Length': '0'} 2023-10-12T01:06:23.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f79bbab143909bed-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.720 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.720 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.721 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.721 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:06:23.721 DEBUG:boto:Data: 2023-10-12T01:06:23.722 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.723 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.723 DEBUG:boto:Params: {} 2023-10-12T01:06:23.723 DEBUG:boto:Token: None 2023-10-12T01:06:23.724 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29 2023-10-12T01:06:23.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x1RjC/irQwaKtyvjm55ysxHykIc= 2023-10-12T01:06:23.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x1RjC/irQwaKtyvjm55ysxHykIc=', 'Content-Length': '0'} 2023-10-12T01:06:23.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b54ce3d6c24571-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:06:23.729 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.729 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.730 DEBUG:boto:Method: GET 2023-10-12T01:06:23.731 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:06:23.732 DEBUG:boto:Data: 2023-10-12T01:06:23.732 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.734 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.735 DEBUG:boto:Params: {} 2023-10-12T01:06:23.735 DEBUG:boto:Token: None 2023-10-12T01:06:23.736 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29 2023-10-12T01:06:23.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hN0RaKuiD+hhGZWiZaaOyMuuyBw= 2023-10-12T01:06:23.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hN0RaKuiD+hhGZWiZaaOyMuuyBw=', 'Content-Length': '0'} 2023-10-12T01:06:23.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25f86b9319104c0-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.741 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:06:23.742 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:06:23.743 DEBUG:boto:Method: GET 2023-10-12T01:06:23.744 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:06:23.744 DEBUG:boto:Data: 2023-10-12T01:06:23.745 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.746 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.747 DEBUG:boto:Params: {} 2023-10-12T01:06:23.747 DEBUG:boto:Token: None 2023-10-12T01:06:23.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj29 2023-10-12T01:06:23.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hN0RaKuiD+hhGZWiZaaOyMuuyBw= 2023-10-12T01:06:23.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hN0RaKuiD+hhGZWiZaaOyMuuyBw=', 'Content-Length': '0'} 2023-10-12T01:06:23.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0d2fddb94ed1ac-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.751 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.752 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.752 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.753 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.753 DEBUG:boto:Method: GET 2023-10-12T01:06:23.754 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.755 DEBUG:boto:Data: 2023-10-12T01:06:23.756 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.757 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.758 DEBUG:boto:Params: {} 2023-10-12T01:06:23.759 DEBUG:boto:Token: None 2023-10-12T01:06:23.759 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.760 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029c3f8cc5af73ec0-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.764 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.765 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.765 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.766 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.766 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.766 DEBUG:boto:Method: GET 2023-10-12T01:06:23.767 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:06:23.768 DEBUG:boto:Data: 2023-10-12T01:06:23.768 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.769 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.769 DEBUG:boto:Params: {} 2023-10-12T01:06:23.770 DEBUG:boto:Token: None 2023-10-12T01:06:23.770 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00aabacd8f2ef5d-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.818 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:06:23.820 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.821 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.821 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.821 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.822 DEBUG:boto:Method: GET 2023-10-12T01:06:23.822 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.823 DEBUG:boto:Data: 2023-10-12T01:06:23.823 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.824 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.825 DEBUG:boto:Params: {} 2023-10-12T01:06:23.825 DEBUG:boto:Token: None 2023-10-12T01:06:23.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc= 2023-10-12T01:06:23.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc=', 'Content-Length': '0'} 2023-10-12T01:06:23.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5dba9ad59600f8-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.830 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.830 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.831 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.831 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.831 DEBUG:boto:Method: GET 2023-10-12T01:06:23.832 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.832 DEBUG:boto:Data: 2023-10-12T01:06:23.833 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.834 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.834 DEBUG:boto:Params: {} 2023-10-12T01:06:23.835 DEBUG:boto:Token: None 2023-10-12T01:06:23.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc= 2023-10-12T01:06:23.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc=', 'Content-Length': '0'} 2023-10-12T01:06:23.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015231a371290103a-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.838 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.839 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.839 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.840 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.840 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.841 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.841 DEBUG:boto:Data: 2023-10-12T01:06:23.842 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.843 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.843 DEBUG:boto:Params: {} 2023-10-12T01:06:23.843 DEBUG:boto:Token: None 2023-10-12T01:06:23.844 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/hmxsYJHcarNM6GRudZOT4+5Rkc= 2023-10-12T01:06:23.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/hmxsYJHcarNM6GRudZOT4+5Rkc=', 'Content-Length': '0'} 2023-10-12T01:06:23.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e8f05aa05793ee-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.847 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.848 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.848 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.848 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.849 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.849 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.850 DEBUG:boto:Data: 2023-10-12T01:06:23.850 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.851 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.852 DEBUG:boto:Params: {} 2023-10-12T01:06:23.852 DEBUG:boto:Token: None 2023-10-12T01:06:23.853 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/hmxsYJHcarNM6GRudZOT4+5Rkc= 2023-10-12T01:06:23.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/hmxsYJHcarNM6GRudZOT4+5Rkc=', 'Content-Length': '0'} 2023-10-12T01:06:23.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ade6ef5cf1408773-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:06:23.859 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.859 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.860 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.860 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.861 DEBUG:boto:Method: GET 2023-10-12T01:06:23.861 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.862 DEBUG:boto:Data: 2023-10-12T01:06:23.862 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.863 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.864 DEBUG:boto:Params: {} 2023-10-12T01:06:23.864 DEBUG:boto:Token: None 2023-10-12T01:06:23.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc= 2023-10-12T01:06:23.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc=', 'Content-Length': '0'} 2023-10-12T01:06:23.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c7ee4b50bb709c9-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.914 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.915 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.916 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.917 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.917 DEBUG:boto:Method: GET 2023-10-12T01:06:23.918 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.918 DEBUG:boto:Data: 2023-10-12T01:06:23.919 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.920 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.920 DEBUG:boto:Params: {} 2023-10-12T01:06:23.921 DEBUG:boto:Token: None 2023-10-12T01:06:23.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:06:23.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc= 2023-10-12T01:06:23.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:osBpeA9cECOsnvSYHRWyImMPAxc=', 'Content-Length': '0'} 2023-10-12T01:06:23.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f564bb817b26329-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.925 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.926 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.926 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.927 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.927 DEBUG:boto:Method: GET 2023-10-12T01:06:23.927 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.928 DEBUG:boto:Data: 2023-10-12T01:06:23.928 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.929 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.929 DEBUG:boto:Params: {} 2023-10-12T01:06:23.930 DEBUG:boto:Token: None 2023-10-12T01:06:23.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4ab4e9be2e84e2-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.935 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.936 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.936 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.937 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.938 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.938 DEBUG:boto:Method: GET 2023-10-12T01:06:23.939 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.940 DEBUG:boto:Data: 2023-10-12T01:06:23.941 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.942 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.943 DEBUG:boto:Params: {} 2023-10-12T01:06:23.943 DEBUG:boto:Token: None 2023-10-12T01:06:23.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082619e5624401823-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.948 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.949 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.950 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.950 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.951 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:06:23.951 DEBUG:boto:Data: 2023-10-12T01:06:23.951 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.952 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.953 DEBUG:boto:Params: {} 2023-10-12T01:06:23.953 DEBUG:boto:Token: None 2023-10-12T01:06:23.953 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33 2023-10-12T01:06:23.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:REU6Jm/RxG72XvpAcwS3XDc2WjY= 2023-10-12T01:06:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:REU6Jm/RxG72XvpAcwS3XDc2WjY=', 'Content-Length': '0'} 2023-10-12T01:06:23.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005697187759339875-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.957 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.957 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.958 DEBUG:boto:Method: HEAD 2023-10-12T01:06:23.958 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:06:23.958 DEBUG:boto:Data: 2023-10-12T01:06:23.959 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.960 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.961 DEBUG:boto:Params: {} 2023-10-12T01:06:23.961 DEBUG:boto:Token: None 2023-10-12T01:06:23.961 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33 2023-10-12T01:06:23.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:REU6Jm/RxG72XvpAcwS3XDc2WjY= 2023-10-12T01:06:23.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:REU6Jm/RxG72XvpAcwS3XDc2WjY=', 'Content-Length': '0'} 2023-10-12T01:06:23.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e24430744c8973c-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:06:23.965 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.966 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.966 DEBUG:boto:Method: GET 2023-10-12T01:06:23.967 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:06:23.967 DEBUG:boto:Data: 2023-10-12T01:06:23.967 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.968 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.969 DEBUG:boto:Params: {} 2023-10-12T01:06:23.969 DEBUG:boto:Token: None 2023-10-12T01:06:23.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33 2023-10-12T01:06:23.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lD9ZRybxEgXRqaFhGq4ZepO33+U= 2023-10-12T01:06:23.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lD9ZRybxEgXRqaFhGq4ZepO33+U=', 'Content-Length': '0'} 2023-10-12T01:06:23.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f877dcf7292160-006527468f-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.974 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:06:23.974 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:06:23.975 DEBUG:boto:Method: GET 2023-10-12T01:06:23.975 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:06:23.976 DEBUG:boto:Data: 2023-10-12T01:06:23.976 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.977 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.977 DEBUG:boto:Params: {} 2023-10-12T01:06:23.978 DEBUG:boto:Token: None 2023-10-12T01:06:23.978 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/obj33 2023-10-12T01:06:23.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lD9ZRybxEgXRqaFhGq4ZepO33+U= 2023-10-12T01:06:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lD9ZRybxEgXRqaFhGq4ZepO33+U=', 'Content-Length': '0'} 2023-10-12T01:06:23.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007413412967ead1cd-006527468f-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.981 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.982 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.982 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.983 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.983 DEBUG:boto:Method: GET 2023-10-12T01:06:23.984 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.984 DEBUG:boto:Data: 2023-10-12T01:06:23.985 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:23.985 DEBUG:boto:Port: 8001 2023-10-12T01:06:23.986 DEBUG:boto:Params: {} 2023-10-12T01:06:23.986 DEBUG:boto:Token: None 2023-10-12T01:06:23.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:23.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002282237091c176b6-006527468f-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:23.991 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:23.991 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:23.992 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:23.992 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.993 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.993 DEBUG:boto:Method: GET 2023-10-12T01:06:23.993 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:06:23.994 DEBUG:boto:Data: 2023-10-12T01:06:23.994 DEBUG:boto:Headers: {} 2023-10-12T01:06:23.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:23.995 DEBUG:boto:Port: 8000 2023-10-12T01:06:23.995 DEBUG:boto:Params: {} 2023-10-12T01:06:23.996 DEBUG:boto:Token: None 2023-10-12T01:06:23.996 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:23 GMT /wafmoa-59/ 2023-10-12T01:06:23.997 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE= 2023-10-12T01:06:23.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qugOWIbTJP3T3JX3loO1u9Pn9WE=', 'Content-Length': '0'} 2023-10-12T01:06:24.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abfb2987bcdd1543-006527468f-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.001 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:06:24.002 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.003 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.003 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.003 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.004 DEBUG:boto:Method: GET 2023-10-12T01:06:24.004 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.005 DEBUG:boto:Data: 2023-10-12T01:06:24.005 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.006 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.006 DEBUG:boto:Params: {} 2023-10-12T01:06:24.007 DEBUG:boto:Token: None 2023-10-12T01:06:24.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.008 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4= 2023-10-12T01:06:24.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4=', 'Content-Length': '0'} 2023-10-12T01:06:24.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb0bb462eee58bb-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.050 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.051 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.051 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.052 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.052 DEBUG:boto:Method: GET 2023-10-12T01:06:24.052 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.053 DEBUG:boto:Data: 2023-10-12T01:06:24.053 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.054 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.055 DEBUG:boto:Params: {} 2023-10-12T01:06:24.055 DEBUG:boto:Token: None 2023-10-12T01:06:24.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4= 2023-10-12T01:06:24.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4=', 'Content-Length': '0'} 2023-10-12T01:06:24.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005fa660cdc9a2ce4-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.059 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.060 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.060 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.060 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.061 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.061 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.062 DEBUG:boto:Data: 2023-10-12T01:06:24.062 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.063 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.063 DEBUG:boto:Params: {} 2023-10-12T01:06:24.064 DEBUG:boto:Token: None 2023-10-12T01:06:24.064 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1s5lAGA8eSVJlER8c0T5xCPnZJY= 2023-10-12T01:06:24.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1s5lAGA8eSVJlER8c0T5xCPnZJY=', 'Content-Length': '0'} 2023-10-12T01:06:24.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076fd22896dda1a8a-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.067 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.068 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.068 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.068 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.069 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.069 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.070 DEBUG:boto:Data: 2023-10-12T01:06:24.070 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.071 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.071 DEBUG:boto:Params: {} 2023-10-12T01:06:24.072 DEBUG:boto:Token: None 2023-10-12T01:06:24.072 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1s5lAGA8eSVJlER8c0T5xCPnZJY= 2023-10-12T01:06:24.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1s5lAGA8eSVJlER8c0T5xCPnZJY=', 'Content-Length': '0'} 2023-10-12T01:06:24.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a639597d6618a5-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:06:24.076 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.076 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.076 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.077 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.077 DEBUG:boto:Method: GET 2023-10-12T01:06:24.078 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.078 DEBUG:boto:Data: 2023-10-12T01:06:24.079 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.079 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.080 DEBUG:boto:Params: {} 2023-10-12T01:06:24.080 DEBUG:boto:Token: None 2023-10-12T01:06:24.081 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4= 2023-10-12T01:06:24.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4=', 'Content-Length': '0'} 2023-10-12T01:06:24.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ca747a1222dedf-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.084 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.085 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.085 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.086 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.086 DEBUG:boto:Method: GET 2023-10-12T01:06:24.087 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.087 DEBUG:boto:Data: 2023-10-12T01:06:24.087 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.088 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.089 DEBUG:boto:Params: {} 2023-10-12T01:06:24.089 DEBUG:boto:Token: None 2023-10-12T01:06:24.090 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:06:24.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4= 2023-10-12T01:06:24.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dw+MifJekD+yWwUcRSoLIzbQAg4=', 'Content-Length': '0'} 2023-10-12T01:06:24.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9abd10bf7a64cb-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.134 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.135 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.136 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.136 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.136 DEBUG:boto:Method: GET 2023-10-12T01:06:24.137 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.137 DEBUG:boto:Data: 2023-10-12T01:06:24.138 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.139 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.139 DEBUG:boto:Params: {} 2023-10-12T01:06:24.140 DEBUG:boto:Token: None 2023-10-12T01:06:24.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fbfd6f6987653ae-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.145 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.146 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.146 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.147 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.147 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.148 DEBUG:boto:Method: GET 2023-10-12T01:06:24.148 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.148 DEBUG:boto:Data: 2023-10-12T01:06:24.149 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.150 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.151 DEBUG:boto:Params: {} 2023-10-12T01:06:24.151 DEBUG:boto:Token: None 2023-10-12T01:06:24.152 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f88f485f2b90bcb-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.157 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.158 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.159 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.160 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.160 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:06:24.161 DEBUG:boto:Data: 2023-10-12T01:06:24.161 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.163 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.163 DEBUG:boto:Params: {} 2023-10-12T01:06:24.164 DEBUG:boto:Token: None 2023-10-12T01:06:24.164 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34 2023-10-12T01:06:24.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:io5y9vMRbZgz9MuUhwxEApIrn5w= 2023-10-12T01:06:24.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:io5y9vMRbZgz9MuUhwxEApIrn5w=', 'Content-Length': '0'} 2023-10-12T01:06:24.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1454dd3b4e3df4-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.168 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.168 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.168 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.169 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:06:24.169 DEBUG:boto:Data: 2023-10-12T01:06:24.170 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.170 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.171 DEBUG:boto:Params: {} 2023-10-12T01:06:24.172 DEBUG:boto:Token: None 2023-10-12T01:06:24.172 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34 2023-10-12T01:06:24.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:io5y9vMRbZgz9MuUhwxEApIrn5w= 2023-10-12T01:06:24.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:io5y9vMRbZgz9MuUhwxEApIrn5w=', 'Content-Length': '0'} 2023-10-12T01:06:24.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057106afc917f8263-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:06:24.176 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.177 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.177 DEBUG:boto:Method: GET 2023-10-12T01:06:24.178 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:06:24.178 DEBUG:boto:Data: 2023-10-12T01:06:24.178 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.179 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.180 DEBUG:boto:Params: {} 2023-10-12T01:06:24.180 DEBUG:boto:Token: None 2023-10-12T01:06:24.181 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34 2023-10-12T01:06:24.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K87QEbqZO6Zr8PGtGEckzy3vY1I= 2023-10-12T01:06:24.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K87QEbqZO6Zr8PGtGEckzy3vY1I=', 'Content-Length': '0'} 2023-10-12T01:06:24.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001144570b764c5f2f-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.226 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:06:24.227 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:06:24.228 DEBUG:boto:Method: GET 2023-10-12T01:06:24.228 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:06:24.228 DEBUG:boto:Data: 2023-10-12T01:06:24.229 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.230 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.230 DEBUG:boto:Params: {} 2023-10-12T01:06:24.231 DEBUG:boto:Token: None 2023-10-12T01:06:24.231 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj34 2023-10-12T01:06:24.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K87QEbqZO6Zr8PGtGEckzy3vY1I= 2023-10-12T01:06:24.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K87QEbqZO6Zr8PGtGEckzy3vY1I=', 'Content-Length': '0'} 2023-10-12T01:06:24.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e5392e20887d2f-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.235 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.236 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.237 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.238 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.238 DEBUG:boto:Method: GET 2023-10-12T01:06:24.239 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.240 DEBUG:boto:Data: 2023-10-12T01:06:24.240 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.242 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.243 DEBUG:boto:Params: {} 2023-10-12T01:06:24.243 DEBUG:boto:Token: None 2023-10-12T01:06:24.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f76ea9d7270d923-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.248 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.249 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.249 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.250 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.250 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.251 DEBUG:boto:Method: GET 2023-10-12T01:06:24.251 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:06:24.251 DEBUG:boto:Data: 2023-10-12T01:06:24.252 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.253 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.253 DEBUG:boto:Params: {} 2023-10-12T01:06:24.253 DEBUG:boto:Token: None 2023-10-12T01:06:24.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d73eeb836ac7f3a-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.298 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:06:24.299 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.300 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.300 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.300 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.301 DEBUG:boto:Method: GET 2023-10-12T01:06:24.301 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.302 DEBUG:boto:Data: 2023-10-12T01:06:24.302 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.303 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.303 DEBUG:boto:Params: {} 2023-10-12T01:06:24.304 DEBUG:boto:Token: None 2023-10-12T01:06:24.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ= 2023-10-12T01:06:24.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925c9b553243c73d-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.308 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.308 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.308 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.309 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.309 DEBUG:boto:Method: GET 2023-10-12T01:06:24.310 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.310 DEBUG:boto:Data: 2023-10-12T01:06:24.310 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.311 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.312 DEBUG:boto:Params: {} 2023-10-12T01:06:24.312 DEBUG:boto:Token: None 2023-10-12T01:06:24.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ= 2023-10-12T01:06:24.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9fb73536edc123-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.316 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.316 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.316 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.317 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.317 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.318 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.318 DEBUG:boto:Data: 2023-10-12T01:06:24.318 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.319 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.319 DEBUG:boto:Params: {} 2023-10-12T01:06:24.320 DEBUG:boto:Token: None 2023-10-12T01:06:24.320 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zkTXO64CovrWIn0BZWJmd2bmaDQ= 2023-10-12T01:06:24.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zkTXO64CovrWIn0BZWJmd2bmaDQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8e1bccaf505b5c-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.323 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.324 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.324 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.324 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.325 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.325 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.325 DEBUG:boto:Data: 2023-10-12T01:06:24.326 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.327 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.327 DEBUG:boto:Params: {} 2023-10-12T01:06:24.327 DEBUG:boto:Token: None 2023-10-12T01:06:24.328 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zkTXO64CovrWIn0BZWJmd2bmaDQ= 2023-10-12T01:06:24.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zkTXO64CovrWIn0BZWJmd2bmaDQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f8f90dcd6a2c1f8-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:06:24.331 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.332 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.332 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.332 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.333 DEBUG:boto:Method: GET 2023-10-12T01:06:24.333 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.334 DEBUG:boto:Data: 2023-10-12T01:06:24.334 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.335 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.335 DEBUG:boto:Params: {} 2023-10-12T01:06:24.336 DEBUG:boto:Token: None 2023-10-12T01:06:24.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ= 2023-10-12T01:06:24.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a646f88e61cab81-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.382 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.383 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.383 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.384 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.384 DEBUG:boto:Method: GET 2023-10-12T01:06:24.385 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.386 DEBUG:boto:Data: 2023-10-12T01:06:24.386 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.388 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.389 DEBUG:boto:Params: {} 2023-10-12T01:06:24.389 DEBUG:boto:Token: None 2023-10-12T01:06:24.390 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:06:24.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ= 2023-10-12T01:06:24.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ALmrNJtKwSkDhlPxuIq1l3h0mnQ=', 'Content-Length': '0'} 2023-10-12T01:06:24.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045386bcde17a9f7f-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.395 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.395 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.395 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.396 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.396 DEBUG:boto:Method: GET 2023-10-12T01:06:24.397 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.397 DEBUG:boto:Data: 2023-10-12T01:06:24.398 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.399 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.399 DEBUG:boto:Params: {} 2023-10-12T01:06:24.400 DEBUG:boto:Token: None 2023-10-12T01:06:24.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008799ed1c76d6765f-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.404 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.405 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.405 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.405 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.406 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.406 DEBUG:boto:Method: GET 2023-10-12T01:06:24.407 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.408 DEBUG:boto:Data: 2023-10-12T01:06:24.408 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.409 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.410 DEBUG:boto:Params: {} 2023-10-12T01:06:24.410 DEBUG:boto:Token: None 2023-10-12T01:06:24.411 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa0c9db2bc67083-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.415 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.416 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.417 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.417 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.417 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:06:24.418 DEBUG:boto:Data: 2023-10-12T01:06:24.419 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.420 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.420 DEBUG:boto:Params: {} 2023-10-12T01:06:24.420 DEBUG:boto:Token: None 2023-10-12T01:06:24.421 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38 2023-10-12T01:06:24.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bhelCzrtSLt+w6yQRHNb5KvDkec= 2023-10-12T01:06:24.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bhelCzrtSLt+w6yQRHNb5KvDkec=', 'Content-Length': '0'} 2023-10-12T01:06:24.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1094e4f8fe9468-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.424 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.425 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.426 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.427 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:06:24.427 DEBUG:boto:Data: 2023-10-12T01:06:24.428 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.429 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.430 DEBUG:boto:Params: {} 2023-10-12T01:06:24.430 DEBUG:boto:Token: None 2023-10-12T01:06:24.431 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38 2023-10-12T01:06:24.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bhelCzrtSLt+w6yQRHNb5KvDkec= 2023-10-12T01:06:24.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bhelCzrtSLt+w6yQRHNb5KvDkec=', 'Content-Length': '0'} 2023-10-12T01:06:24.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009caa4ce06e9386a2-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:06:24.435 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.435 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.435 DEBUG:boto:Method: GET 2023-10-12T01:06:24.436 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:06:24.436 DEBUG:boto:Data: 2023-10-12T01:06:24.437 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.438 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.438 DEBUG:boto:Params: {} 2023-10-12T01:06:24.439 DEBUG:boto:Token: None 2023-10-12T01:06:24.439 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38 2023-10-12T01:06:24.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZOA2KaoynkhRY3hJziCBbOk01Yc= 2023-10-12T01:06:24.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZOA2KaoynkhRY3hJziCBbOk01Yc=', 'Content-Length': '0'} 2023-10-12T01:06:24.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac47e22015807b27-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.443 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:06:24.444 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:06:24.444 DEBUG:boto:Method: GET 2023-10-12T01:06:24.444 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:06:24.445 DEBUG:boto:Data: 2023-10-12T01:06:24.445 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.446 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.447 DEBUG:boto:Params: {} 2023-10-12T01:06:24.447 DEBUG:boto:Token: None 2023-10-12T01:06:24.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj38 2023-10-12T01:06:24.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZOA2KaoynkhRY3hJziCBbOk01Yc= 2023-10-12T01:06:24.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZOA2KaoynkhRY3hJziCBbOk01Yc=', 'Content-Length': '0'} 2023-10-12T01:06:24.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6acbd3cb2c7b333-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.494 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.495 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.496 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.496 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.497 DEBUG:boto:Method: GET 2023-10-12T01:06:24.497 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.497 DEBUG:boto:Data: 2023-10-12T01:06:24.498 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.499 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.499 DEBUG:boto:Params: {} 2023-10-12T01:06:24.500 DEBUG:boto:Token: None 2023-10-12T01:06:24.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe58af50e75cf771-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.505 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.506 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.507 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.507 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.508 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.509 DEBUG:boto:Method: GET 2023-10-12T01:06:24.509 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:06:24.510 DEBUG:boto:Data: 2023-10-12T01:06:24.511 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.512 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.512 DEBUG:boto:Params: {} 2023-10-12T01:06:24.513 DEBUG:boto:Token: None 2023-10-12T01:06:24.513 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804f04bc86462324-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.518 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:06:24.520 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.520 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.520 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.521 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.521 DEBUG:boto:Method: GET 2023-10-12T01:06:24.522 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.522 DEBUG:boto:Data: 2023-10-12T01:06:24.523 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.524 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.524 DEBUG:boto:Params: {} 2023-10-12T01:06:24.525 DEBUG:boto:Token: None 2023-10-12T01:06:24.525 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0= 2023-10-12T01:06:24.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0=', 'Content-Length': '0'} 2023-10-12T01:06:24.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31f948e50295626-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.529 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.529 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.530 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.530 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.531 DEBUG:boto:Method: GET 2023-10-12T01:06:24.532 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.533 DEBUG:boto:Data: 2023-10-12T01:06:24.533 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.535 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.536 DEBUG:boto:Params: {} 2023-10-12T01:06:24.536 DEBUG:boto:Token: None 2023-10-12T01:06:24.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0= 2023-10-12T01:06:24.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0=', 'Content-Length': '0'} 2023-10-12T01:06:24.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be817b7753a71e0-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.540 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.541 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.541 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.542 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.542 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.543 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.543 DEBUG:boto:Data: 2023-10-12T01:06:24.544 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.544 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.545 DEBUG:boto:Params: {} 2023-10-12T01:06:24.545 DEBUG:boto:Token: None 2023-10-12T01:06:24.546 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sekBxSFPV++qIF053oAL7xzpVMM= 2023-10-12T01:06:24.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sekBxSFPV++qIF053oAL7xzpVMM=', 'Content-Length': '0'} 2023-10-12T01:06:24.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fda3c28e20a9b0-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.551 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.551 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.552 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.553 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.553 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.554 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.554 DEBUG:boto:Data: 2023-10-12T01:06:24.555 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.556 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.556 DEBUG:boto:Params: {} 2023-10-12T01:06:24.556 DEBUG:boto:Token: None 2023-10-12T01:06:24.557 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sekBxSFPV++qIF053oAL7xzpVMM= 2023-10-12T01:06:24.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sekBxSFPV++qIF053oAL7xzpVMM=', 'Content-Length': '0'} 2023-10-12T01:06:24.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e21c1351ba7142-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:06:24.561 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.561 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.561 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.562 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.562 DEBUG:boto:Method: GET 2023-10-12T01:06:24.563 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.563 DEBUG:boto:Data: 2023-10-12T01:06:24.564 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.564 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.565 DEBUG:boto:Params: {} 2023-10-12T01:06:24.565 DEBUG:boto:Token: None 2023-10-12T01:06:24.566 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0= 2023-10-12T01:06:24.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0=', 'Content-Length': '0'} 2023-10-12T01:06:24.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb50316dce7cb32f-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.569 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.570 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.570 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.570 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.571 DEBUG:boto:Method: GET 2023-10-12T01:06:24.571 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.572 DEBUG:boto:Data: 2023-10-12T01:06:24.572 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.573 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.573 DEBUG:boto:Params: {} 2023-10-12T01:06:24.574 DEBUG:boto:Token: None 2023-10-12T01:06:24.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:06:24.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0= 2023-10-12T01:06:24.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2YMlg3dDdtAOSTEBhyHd42tGk+0=', 'Content-Length': '0'} 2023-10-12T01:06:24.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aad075940c84ffa-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.578 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.579 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.579 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.580 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.580 DEBUG:boto:Method: GET 2023-10-12T01:06:24.581 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.581 DEBUG:boto:Data: 2023-10-12T01:06:24.582 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.583 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.583 DEBUG:boto:Params: {} 2023-10-12T01:06:24.584 DEBUG:boto:Token: None 2023-10-12T01:06:24.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd430eb9a5ee9127-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.588 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.588 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.589 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.589 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.590 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.590 DEBUG:boto:Method: GET 2023-10-12T01:06:24.590 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.591 DEBUG:boto:Data: 2023-10-12T01:06:24.591 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.592 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.592 DEBUG:boto:Params: {} 2023-10-12T01:06:24.593 DEBUG:boto:Token: None 2023-10-12T01:06:24.593 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081432126b76098c2-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.598 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.598 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.599 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.599 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.599 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:06:24.600 DEBUG:boto:Data: 2023-10-12T01:06:24.600 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.601 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.601 DEBUG:boto:Params: {} 2023-10-12T01:06:24.602 DEBUG:boto:Token: None 2023-10-12T01:06:24.602 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44 2023-10-12T01:06:24.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uhOnYMJM46OD+M+vRR55+Eg5QjY= 2023-10-12T01:06:24.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uhOnYMJM46OD+M+vRR55+Eg5QjY=', 'Content-Length': '0'} 2023-10-12T01:06:24.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da6ca2e23ac46d82-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.605 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.606 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.606 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.607 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:06:24.607 DEBUG:boto:Data: 2023-10-12T01:06:24.607 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.608 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.608 DEBUG:boto:Params: {} 2023-10-12T01:06:24.609 DEBUG:boto:Token: None 2023-10-12T01:06:24.609 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44 2023-10-12T01:06:24.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uhOnYMJM46OD+M+vRR55+Eg5QjY= 2023-10-12T01:06:24.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uhOnYMJM46OD+M+vRR55+Eg5QjY=', 'Content-Length': '0'} 2023-10-12T01:06:24.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b52a227aeddb53d-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:06:24.613 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.613 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.614 DEBUG:boto:Method: GET 2023-10-12T01:06:24.614 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:06:24.615 DEBUG:boto:Data: 2023-10-12T01:06:24.615 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.616 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.616 DEBUG:boto:Params: {} 2023-10-12T01:06:24.616 DEBUG:boto:Token: None 2023-10-12T01:06:24.617 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44 2023-10-12T01:06:24.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+M3I63YdHO72/FHjxaYzYLHaZkw= 2023-10-12T01:06:24.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+M3I63YdHO72/FHjxaYzYLHaZkw=', 'Content-Length': '0'} 2023-10-12T01:06:24.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056973d5fec213d2a-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.662 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:06:24.663 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:06:24.663 DEBUG:boto:Method: GET 2023-10-12T01:06:24.664 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:06:24.664 DEBUG:boto:Data: 2023-10-12T01:06:24.665 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.665 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.666 DEBUG:boto:Params: {} 2023-10-12T01:06:24.666 DEBUG:boto:Token: None 2023-10-12T01:06:24.667 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj44 2023-10-12T01:06:24.667 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+M3I63YdHO72/FHjxaYzYLHaZkw= 2023-10-12T01:06:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+M3I63YdHO72/FHjxaYzYLHaZkw=', 'Content-Length': '0'} 2023-10-12T01:06:24.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091270c1df07f3b1a-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.670 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.671 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.671 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.672 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.672 DEBUG:boto:Method: GET 2023-10-12T01:06:24.673 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.673 DEBUG:boto:Data: 2023-10-12T01:06:24.673 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.674 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.675 DEBUG:boto:Params: {} 2023-10-12T01:06:24.677 DEBUG:boto:Token: None 2023-10-12T01:06:24.678 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60351552f96b90b-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.682 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.683 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.684 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.685 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.685 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.686 DEBUG:boto:Method: GET 2023-10-12T01:06:24.687 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:06:24.688 DEBUG:boto:Data: 2023-10-12T01:06:24.688 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.690 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.691 DEBUG:boto:Params: {} 2023-10-12T01:06:24.691 DEBUG:boto:Token: None 2023-10-12T01:06:24.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6402679223e65d1-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.697 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:06:24.698 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.699 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.699 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.700 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.700 DEBUG:boto:Method: GET 2023-10-12T01:06:24.701 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.701 DEBUG:boto:Data: 2023-10-12T01:06:24.702 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.702 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.703 DEBUG:boto:Params: {} 2023-10-12T01:06:24.703 DEBUG:boto:Token: None 2023-10-12T01:06:24.704 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c= 2023-10-12T01:06:24.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c=', 'Content-Length': '0'} 2023-10-12T01:06:24.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034cbb3a9ffd2a697-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.708 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.708 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.709 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.709 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.710 DEBUG:boto:Method: GET 2023-10-12T01:06:24.710 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.711 DEBUG:boto:Data: 2023-10-12T01:06:24.711 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.712 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.712 DEBUG:boto:Params: {} 2023-10-12T01:06:24.713 DEBUG:boto:Token: None 2023-10-12T01:06:24.713 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.714 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c= 2023-10-12T01:06:24.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c=', 'Content-Length': '0'} 2023-10-12T01:06:24.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010efce339bd64868-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.717 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.717 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.718 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.718 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.719 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.720 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.720 DEBUG:boto:Data: 2023-10-12T01:06:24.721 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.723 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.723 DEBUG:boto:Params: {} 2023-10-12T01:06:24.724 DEBUG:boto:Token: None 2023-10-12T01:06:24.725 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.725 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qAix57vbggT/i/pmbTlFqk1S3rA= 2023-10-12T01:06:24.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qAix57vbggT/i/pmbTlFqk1S3rA=', 'Content-Length': '0'} 2023-10-12T01:06:24.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066e74a4e19e2d4b-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.728 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.729 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.729 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.730 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.730 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.731 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.731 DEBUG:boto:Data: 2023-10-12T01:06:24.732 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.733 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.733 DEBUG:boto:Params: {} 2023-10-12T01:06:24.734 DEBUG:boto:Token: None 2023-10-12T01:06:24.734 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qAix57vbggT/i/pmbTlFqk1S3rA= 2023-10-12T01:06:24.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qAix57vbggT/i/pmbTlFqk1S3rA=', 'Content-Length': '0'} 2023-10-12T01:06:24.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986a0bc7d2b127b1-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:06:24.738 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.738 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.739 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.739 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.740 DEBUG:boto:Method: GET 2023-10-12T01:06:24.740 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.740 DEBUG:boto:Data: 2023-10-12T01:06:24.741 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.742 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.742 DEBUG:boto:Params: {} 2023-10-12T01:06:24.743 DEBUG:boto:Token: None 2023-10-12T01:06:24.743 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c= 2023-10-12T01:06:24.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c=', 'Content-Length': '0'} 2023-10-12T01:06:24.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009485cf4e67c57c14-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.747 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.747 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.748 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.748 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.749 DEBUG:boto:Method: GET 2023-10-12T01:06:24.749 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.750 DEBUG:boto:Data: 2023-10-12T01:06:24.750 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.751 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.752 DEBUG:boto:Params: {} 2023-10-12T01:06:24.752 DEBUG:boto:Token: None 2023-10-12T01:06:24.753 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:06:24.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c= 2023-10-12T01:06:24.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jF/qiwM4HBbogiy8BxOJlpQCX6c=', 'Content-Length': '0'} 2023-10-12T01:06:24.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768d62519e2fe1e7-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.757 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.757 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.758 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.759 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.760 DEBUG:boto:Method: GET 2023-10-12T01:06:24.761 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.761 DEBUG:boto:Data: 2023-10-12T01:06:24.762 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.764 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.764 DEBUG:boto:Params: {} 2023-10-12T01:06:24.765 DEBUG:boto:Token: None 2023-10-12T01:06:24.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51859242ea9e6f0-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.810 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.811 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.812 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.812 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.813 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.813 DEBUG:boto:Method: GET 2023-10-12T01:06:24.813 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.814 DEBUG:boto:Data: 2023-10-12T01:06:24.814 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.815 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.816 DEBUG:boto:Params: {} 2023-10-12T01:06:24.816 DEBUG:boto:Token: None 2023-10-12T01:06:24.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecaa676a9c31292d-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.821 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.822 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.822 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.823 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.823 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:06:24.824 DEBUG:boto:Data: 2023-10-12T01:06:24.824 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.825 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.825 DEBUG:boto:Params: {} 2023-10-12T01:06:24.826 DEBUG:boto:Token: None 2023-10-12T01:06:24.826 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46 2023-10-12T01:06:24.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BH5NuEP2fqE5GQXTAmXYHFDR+Ks= 2023-10-12T01:06:24.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BH5NuEP2fqE5GQXTAmXYHFDR+Ks=', 'Content-Length': '0'} 2023-10-12T01:06:24.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46e5c27e11285f3-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.829 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.830 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.830 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.831 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:06:24.831 DEBUG:boto:Data: 2023-10-12T01:06:24.831 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.832 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.833 DEBUG:boto:Params: {} 2023-10-12T01:06:24.833 DEBUG:boto:Token: None 2023-10-12T01:06:24.834 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46 2023-10-12T01:06:24.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BH5NuEP2fqE5GQXTAmXYHFDR+Ks= 2023-10-12T01:06:24.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BH5NuEP2fqE5GQXTAmXYHFDR+Ks=', 'Content-Length': '0'} 2023-10-12T01:06:24.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0b30b04928c1c3-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:06:24.837 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.838 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.838 DEBUG:boto:Method: GET 2023-10-12T01:06:24.839 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:06:24.839 DEBUG:boto:Data: 2023-10-12T01:06:24.839 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.840 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.841 DEBUG:boto:Params: {} 2023-10-12T01:06:24.841 DEBUG:boto:Token: None 2023-10-12T01:06:24.842 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46 2023-10-12T01:06:24.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A1iywKekBqPVEsgJu+zNl1/t+7Y= 2023-10-12T01:06:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A1iywKekBqPVEsgJu+zNl1/t+7Y=', 'Content-Length': '0'} 2023-10-12T01:06:24.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667618cb8cd4feaa-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.845 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:06:24.845 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:06:24.846 DEBUG:boto:Method: GET 2023-10-12T01:06:24.846 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:06:24.847 DEBUG:boto:Data: 2023-10-12T01:06:24.847 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.848 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.848 DEBUG:boto:Params: {} 2023-10-12T01:06:24.849 DEBUG:boto:Token: None 2023-10-12T01:06:24.849 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj46 2023-10-12T01:06:24.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A1iywKekBqPVEsgJu+zNl1/t+7Y= 2023-10-12T01:06:24.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A1iywKekBqPVEsgJu+zNl1/t+7Y=', 'Content-Length': '0'} 2023-10-12T01:06:24.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e653e87184f54a-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.853 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.853 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.853 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.854 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.854 DEBUG:boto:Method: GET 2023-10-12T01:06:24.855 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.855 DEBUG:boto:Data: 2023-10-12T01:06:24.855 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.856 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.857 DEBUG:boto:Params: {} 2023-10-12T01:06:24.857 DEBUG:boto:Token: None 2023-10-12T01:06:24.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d29587cf63ce0e0-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.861 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.862 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.862 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.863 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.863 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.864 DEBUG:boto:Method: GET 2023-10-12T01:06:24.864 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:06:24.864 DEBUG:boto:Data: 2023-10-12T01:06:24.865 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.866 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.866 DEBUG:boto:Params: {} 2023-10-12T01:06:24.866 DEBUG:boto:Token: None 2023-10-12T01:06:24.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760fbec4af2db54a-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.871 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:06:24.872 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.873 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.873 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.874 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.874 DEBUG:boto:Method: GET 2023-10-12T01:06:24.875 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.875 DEBUG:boto:Data: 2023-10-12T01:06:24.875 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.876 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.877 DEBUG:boto:Params: {} 2023-10-12T01:06:24.877 DEBUG:boto:Token: None 2023-10-12T01:06:24.878 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.878 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY= 2023-10-12T01:06:24.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY=', 'Content-Length': '0'} 2023-10-12T01:06:24.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031fa8468242c349f-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.883 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.884 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.884 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.885 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.885 DEBUG:boto:Method: GET 2023-10-12T01:06:24.885 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.886 DEBUG:boto:Data: 2023-10-12T01:06:24.886 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.887 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.887 DEBUG:boto:Params: {} 2023-10-12T01:06:24.888 DEBUG:boto:Token: None 2023-10-12T01:06:24.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY= 2023-10-12T01:06:24.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY=', 'Content-Length': '0'} 2023-10-12T01:06:24.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3f4f5c45eaaa6e-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.934 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.935 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.935 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.936 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.936 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.937 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.937 DEBUG:boto:Data: 2023-10-12T01:06:24.937 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.938 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.939 DEBUG:boto:Params: {} 2023-10-12T01:06:24.939 DEBUG:boto:Token: None 2023-10-12T01:06:24.940 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0I6YrYKJUaslYhxZNV7aPs4NK6s= 2023-10-12T01:06:24.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0I6YrYKJUaslYhxZNV7aPs4NK6s=', 'Content-Length': '0'} 2023-10-12T01:06:24.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13a8f0b1223b357-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.943 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.943 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.944 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.944 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.944 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.945 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.945 DEBUG:boto:Data: 2023-10-12T01:06:24.946 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.946 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.947 DEBUG:boto:Params: {} 2023-10-12T01:06:24.947 DEBUG:boto:Token: None 2023-10-12T01:06:24.948 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0I6YrYKJUaslYhxZNV7aPs4NK6s= 2023-10-12T01:06:24.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0I6YrYKJUaslYhxZNV7aPs4NK6s=', 'Content-Length': '0'} 2023-10-12T01:06:24.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18347ff5dce0629-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:06:24.951 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.952 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.952 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.953 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.953 DEBUG:boto:Method: GET 2023-10-12T01:06:24.954 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.954 DEBUG:boto:Data: 2023-10-12T01:06:24.954 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.955 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.956 DEBUG:boto:Params: {} 2023-10-12T01:06:24.956 DEBUG:boto:Token: None 2023-10-12T01:06:24.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY= 2023-10-12T01:06:24.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY=', 'Content-Length': '0'} 2023-10-12T01:06:24.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ae014faefb6abc-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.960 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.961 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.961 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.961 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.962 DEBUG:boto:Method: GET 2023-10-12T01:06:24.962 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.963 DEBUG:boto:Data: 2023-10-12T01:06:24.963 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.964 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.964 DEBUG:boto:Params: {} 2023-10-12T01:06:24.965 DEBUG:boto:Token: None 2023-10-12T01:06:24.965 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:06:24.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY= 2023-10-12T01:06:24.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z2kXT9LJIdtIVnNqOPK/g/CiXEY=', 'Content-Length': '0'} 2023-10-12T01:06:24.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055041082a7a0eb05-0065274690-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.969 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.969 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.969 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.970 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.970 DEBUG:boto:Method: GET 2023-10-12T01:06:24.971 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.971 DEBUG:boto:Data: 2023-10-12T01:06:24.972 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.972 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.973 DEBUG:boto:Params: {} 2023-10-12T01:06:24.973 DEBUG:boto:Token: None 2023-10-12T01:06:24.974 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.974 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d5b5570b9444f8-0065274690-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.978 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.979 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:24.979 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:24.979 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.980 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.980 DEBUG:boto:Method: GET 2023-10-12T01:06:24.981 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:06:24.981 DEBUG:boto:Data: 2023-10-12T01:06:24.981 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:24.982 DEBUG:boto:Port: 8000 2023-10-12T01:06:24.983 DEBUG:boto:Params: {} 2023-10-12T01:06:24.983 DEBUG:boto:Token: None 2023-10-12T01:06:24.984 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/ 2023-10-12T01:06:24.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A= 2023-10-12T01:06:24.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSyb38U/SZVrdQ+L5Utith46F3A=', 'Content-Length': '0'} 2023-10-12T01:06:24.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc2c759d2536b63-0065274690-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.988 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:24.989 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.990 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.990 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.990 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:06:24.991 DEBUG:boto:Data: 2023-10-12T01:06:24.991 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:24.992 DEBUG:boto:Port: 8001 2023-10-12T01:06:24.993 DEBUG:boto:Params: {} 2023-10-12T01:06:24.993 DEBUG:boto:Token: None 2023-10-12T01:06:24.994 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:24 GMT /wafmoa-59/obj51 2023-10-12T01:06:24.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJsAfkDaZJEHNjnElPV/4A5aJX4= 2023-10-12T01:06:24.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJsAfkDaZJEHNjnElPV/4A5aJX4=', 'Content-Length': '0'} 2023-10-12T01:06:24.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce3dc7333b0a96c-0065274690-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:24.997 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:24.997 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:24.998 DEBUG:boto:Method: HEAD 2023-10-12T01:06:24.998 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:06:24.998 DEBUG:boto:Data: 2023-10-12T01:06:24.999 DEBUG:boto:Headers: {} 2023-10-12T01:06:24.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.000 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.000 DEBUG:boto:Params: {} 2023-10-12T01:06:25.000 DEBUG:boto:Token: None 2023-10-12T01:06:25.001 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj51 2023-10-12T01:06:25.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRWNMAnSH4T6xfNxS19YrjkrO38= 2023-10-12T01:06:25.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRWNMAnSH4T6xfNxS19YrjkrO38=', 'Content-Length': '0'} 2023-10-12T01:06:25.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea8ee98eef10a2a-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:06:25.005 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:25.005 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:25.005 DEBUG:boto:Method: GET 2023-10-12T01:06:25.006 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:06:25.006 DEBUG:boto:Data: 2023-10-12T01:06:25.007 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.008 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.008 DEBUG:boto:Params: {} 2023-10-12T01:06:25.009 DEBUG:boto:Token: None 2023-10-12T01:06:25.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj51 2023-10-12T01:06:25.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YM0OVSTcMsIohfB9GK5JMEDHqco= 2023-10-12T01:06:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YM0OVSTcMsIohfB9GK5JMEDHqco=', 'Content-Length': '0'} 2023-10-12T01:06:25.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004502f61476c9b1c-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.012 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:06:25.013 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:06:25.013 DEBUG:boto:Method: GET 2023-10-12T01:06:25.014 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:06:25.014 DEBUG:boto:Data: 2023-10-12T01:06:25.015 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.015 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.016 DEBUG:boto:Params: {} 2023-10-12T01:06:25.016 DEBUG:boto:Token: None 2023-10-12T01:06:25.017 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj51 2023-10-12T01:06:25.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YM0OVSTcMsIohfB9GK5JMEDHqco= 2023-10-12T01:06:25.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YM0OVSTcMsIohfB9GK5JMEDHqco=', 'Content-Length': '0'} 2023-10-12T01:06:25.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9c13aecd308d32-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.062 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.063 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.063 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.064 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.064 DEBUG:boto:Method: GET 2023-10-12T01:06:25.065 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.065 DEBUG:boto:Data: 2023-10-12T01:06:25.066 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.067 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.067 DEBUG:boto:Params: {} 2023-10-12T01:06:25.068 DEBUG:boto:Token: None 2023-10-12T01:06:25.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008467a6e4505f3d42-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.072 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.073 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.073 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.074 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.074 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.075 DEBUG:boto:Method: GET 2023-10-12T01:06:25.075 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:06:25.076 DEBUG:boto:Data: 2023-10-12T01:06:25.076 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.077 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.077 DEBUG:boto:Params: {} 2023-10-12T01:06:25.078 DEBUG:boto:Token: None 2023-10-12T01:06:25.078 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0e77a7636ba81f-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.083 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:06:25.084 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.085 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.085 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.085 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.086 DEBUG:boto:Method: GET 2023-10-12T01:06:25.086 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.087 DEBUG:boto:Data: 2023-10-12T01:06:25.087 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.088 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.089 DEBUG:boto:Params: {} 2023-10-12T01:06:25.089 DEBUG:boto:Token: None 2023-10-12T01:06:25.090 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q= 2023-10-12T01:06:25.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087647067088f8931-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.134 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.135 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.135 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.136 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.136 DEBUG:boto:Method: GET 2023-10-12T01:06:25.137 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.137 DEBUG:boto:Data: 2023-10-12T01:06:25.138 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.138 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.139 DEBUG:boto:Params: {} 2023-10-12T01:06:25.139 DEBUG:boto:Token: None 2023-10-12T01:06:25.140 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q= 2023-10-12T01:06:25.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098705bc68f7382c8-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.144 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.144 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.145 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.145 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.146 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.146 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.146 DEBUG:boto:Data: 2023-10-12T01:06:25.147 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.148 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.148 DEBUG:boto:Params: {} 2023-10-12T01:06:25.148 DEBUG:boto:Token: None 2023-10-12T01:06:25.149 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uYokrQpO84eX60I351wfYZoC+ds= 2023-10-12T01:06:25.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uYokrQpO84eX60I351wfYZoC+ds=', 'Content-Length': '0'} 2023-10-12T01:06:25.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab86822a69801cea-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.152 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.152 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.153 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.153 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.154 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.154 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.155 DEBUG:boto:Data: 2023-10-12T01:06:25.155 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.156 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.156 DEBUG:boto:Params: {} 2023-10-12T01:06:25.157 DEBUG:boto:Token: None 2023-10-12T01:06:25.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uYokrQpO84eX60I351wfYZoC+ds= 2023-10-12T01:06:25.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uYokrQpO84eX60I351wfYZoC+ds=', 'Content-Length': '0'} 2023-10-12T01:06:25.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc419fed7fd29809-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:06:25.161 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.161 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.162 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.162 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.163 DEBUG:boto:Method: GET 2023-10-12T01:06:25.163 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.164 DEBUG:boto:Data: 2023-10-12T01:06:25.164 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.165 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.166 DEBUG:boto:Params: {} 2023-10-12T01:06:25.166 DEBUG:boto:Token: None 2023-10-12T01:06:25.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q= 2023-10-12T01:06:25.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc63a08d6ff79fd4-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.171 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.171 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.172 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.172 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.172 DEBUG:boto:Method: GET 2023-10-12T01:06:25.173 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.173 DEBUG:boto:Data: 2023-10-12T01:06:25.174 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.175 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.175 DEBUG:boto:Params: {} 2023-10-12T01:06:25.176 DEBUG:boto:Token: None 2023-10-12T01:06:25.176 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:06:25.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q= 2023-10-12T01:06:25.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aaG9qTTANnNkxSG/tXpCdEsgK9Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a7d8d7ae7c46f0-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.180 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.180 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.180 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.181 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.181 DEBUG:boto:Method: GET 2023-10-12T01:06:25.182 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.182 DEBUG:boto:Data: 2023-10-12T01:06:25.182 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.183 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.184 DEBUG:boto:Params: {} 2023-10-12T01:06:25.184 DEBUG:boto:Token: None 2023-10-12T01:06:25.185 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000324bcb7b5309bb16-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.189 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.189 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.190 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.190 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.191 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.191 DEBUG:boto:Method: GET 2023-10-12T01:06:25.192 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.192 DEBUG:boto:Data: 2023-10-12T01:06:25.193 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.193 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.194 DEBUG:boto:Params: {} 2023-10-12T01:06:25.194 DEBUG:boto:Token: None 2023-10-12T01:06:25.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d9debd43f614c9-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.242 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.243 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.244 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.244 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.244 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:06:25.245 DEBUG:boto:Data: 2023-10-12T01:06:25.245 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.246 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.247 DEBUG:boto:Params: {} 2023-10-12T01:06:25.247 DEBUG:boto:Token: None 2023-10-12T01:06:25.248 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55 2023-10-12T01:06:25.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PLOOqPnjocewcBMx2RF89/7G1Tg= 2023-10-12T01:06:25.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PLOOqPnjocewcBMx2RF89/7G1Tg=', 'Content-Length': '0'} 2023-10-12T01:06:25.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc18a0083917bff-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.251 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.252 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.252 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.252 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:06:25.253 DEBUG:boto:Data: 2023-10-12T01:06:25.253 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.254 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.254 DEBUG:boto:Params: {} 2023-10-12T01:06:25.255 DEBUG:boto:Token: None 2023-10-12T01:06:25.255 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55 2023-10-12T01:06:25.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PLOOqPnjocewcBMx2RF89/7G1Tg= 2023-10-12T01:06:25.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PLOOqPnjocewcBMx2RF89/7G1Tg=', 'Content-Length': '0'} 2023-10-12T01:06:25.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026c6a07c598a82e9-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:06:25.259 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.260 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.260 DEBUG:boto:Method: GET 2023-10-12T01:06:25.261 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:06:25.261 DEBUG:boto:Data: 2023-10-12T01:06:25.261 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.262 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.263 DEBUG:boto:Params: {} 2023-10-12T01:06:25.263 DEBUG:boto:Token: None 2023-10-12T01:06:25.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55 2023-10-12T01:06:25.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NXXJiImN5QvdIIdvaTDS3tCQwPw= 2023-10-12T01:06:25.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NXXJiImN5QvdIIdvaTDS3tCQwPw=', 'Content-Length': '0'} 2023-10-12T01:06:25.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4522b578c152bf-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.267 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:06:25.268 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:06:25.268 DEBUG:boto:Method: GET 2023-10-12T01:06:25.268 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:06:25.269 DEBUG:boto:Data: 2023-10-12T01:06:25.269 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.270 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.271 DEBUG:boto:Params: {} 2023-10-12T01:06:25.271 DEBUG:boto:Token: None 2023-10-12T01:06:25.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj55 2023-10-12T01:06:25.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NXXJiImN5QvdIIdvaTDS3tCQwPw= 2023-10-12T01:06:25.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NXXJiImN5QvdIIdvaTDS3tCQwPw=', 'Content-Length': '0'} 2023-10-12T01:06:25.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000450d09e021d3c82c-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.275 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.276 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.276 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.277 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.278 DEBUG:boto:Method: GET 2023-10-12T01:06:25.278 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.278 DEBUG:boto:Data: 2023-10-12T01:06:25.279 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.280 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.280 DEBUG:boto:Params: {} 2023-10-12T01:06:25.281 DEBUG:boto:Token: None 2023-10-12T01:06:25.281 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006837043b47131d31-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.330 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.331 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.332 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.332 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.333 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.333 DEBUG:boto:Method: GET 2023-10-12T01:06:25.333 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:06:25.334 DEBUG:boto:Data: 2023-10-12T01:06:25.334 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.335 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.336 DEBUG:boto:Params: {} 2023-10-12T01:06:25.336 DEBUG:boto:Token: None 2023-10-12T01:06:25.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b40d2cacc9b8fc1-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.342 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:06:25.343 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.343 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.343 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.344 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.344 DEBUG:boto:Method: GET 2023-10-12T01:06:25.345 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.345 DEBUG:boto:Data: 2023-10-12T01:06:25.346 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.347 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.347 DEBUG:boto:Params: {} 2023-10-12T01:06:25.347 DEBUG:boto:Token: None 2023-10-12T01:06:25.348 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc= 2023-10-12T01:06:25.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc=', 'Content-Length': '0'} 2023-10-12T01:06:25.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198eb193681e2b00-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.351 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.352 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.352 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.353 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.353 DEBUG:boto:Method: GET 2023-10-12T01:06:25.354 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.354 DEBUG:boto:Data: 2023-10-12T01:06:25.355 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.356 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.356 DEBUG:boto:Params: {} 2023-10-12T01:06:25.357 DEBUG:boto:Token: None 2023-10-12T01:06:25.357 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc= 2023-10-12T01:06:25.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc=', 'Content-Length': '0'} 2023-10-12T01:06:25.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e29d23a9eac5d3d-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.360 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.361 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.361 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.362 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.362 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.363 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.363 DEBUG:boto:Data: 2023-10-12T01:06:25.364 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.364 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.365 DEBUG:boto:Params: {} 2023-10-12T01:06:25.365 DEBUG:boto:Token: None 2023-10-12T01:06:25.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xJoxzdP8Bsng0Wj/lk1h5kCe8Zc= 2023-10-12T01:06:25.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xJoxzdP8Bsng0Wj/lk1h5kCe8Zc=', 'Content-Length': '0'} 2023-10-12T01:06:25.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e225a7f0a3e2a63-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.369 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.370 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.370 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.371 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.371 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.371 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.372 DEBUG:boto:Data: 2023-10-12T01:06:25.372 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.373 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.374 DEBUG:boto:Params: {} 2023-10-12T01:06:25.374 DEBUG:boto:Token: None 2023-10-12T01:06:25.375 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xJoxzdP8Bsng0Wj/lk1h5kCe8Zc= 2023-10-12T01:06:25.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xJoxzdP8Bsng0Wj/lk1h5kCe8Zc=', 'Content-Length': '0'} 2023-10-12T01:06:25.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f0095f622ff1d3-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:06:25.379 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.379 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.379 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.380 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.380 DEBUG:boto:Method: GET 2023-10-12T01:06:25.381 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.381 DEBUG:boto:Data: 2023-10-12T01:06:25.382 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.383 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.383 DEBUG:boto:Params: {} 2023-10-12T01:06:25.383 DEBUG:boto:Token: None 2023-10-12T01:06:25.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc= 2023-10-12T01:06:25.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc=', 'Content-Length': '0'} 2023-10-12T01:06:25.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018196ea2faf7be7c-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.388 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.388 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.389 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.389 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.390 DEBUG:boto:Method: GET 2023-10-12T01:06:25.390 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.390 DEBUG:boto:Data: 2023-10-12T01:06:25.391 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.392 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.392 DEBUG:boto:Params: {} 2023-10-12T01:06:25.393 DEBUG:boto:Token: None 2023-10-12T01:06:25.393 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:06:25.394 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc= 2023-10-12T01:06:25.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:G0RM15mAJCa6Cv8ygLYKX/qbFpc=', 'Content-Length': '0'} 2023-10-12T01:06:25.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b542ee25a538ce74-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.397 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.397 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.397 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.398 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.398 DEBUG:boto:Method: GET 2023-10-12T01:06:25.399 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.399 DEBUG:boto:Data: 2023-10-12T01:06:25.400 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.401 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.401 DEBUG:boto:Params: {} 2023-10-12T01:06:25.401 DEBUG:boto:Token: None 2023-10-12T01:06:25.402 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e001dab2b3764701-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.446 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.447 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.447 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.448 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.448 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.449 DEBUG:boto:Method: GET 2023-10-12T01:06:25.449 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.450 DEBUG:boto:Data: 2023-10-12T01:06:25.450 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.451 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.451 DEBUG:boto:Params: {} 2023-10-12T01:06:25.452 DEBUG:boto:Token: None 2023-10-12T01:06:25.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b877954487dbcb-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.457 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.457 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.458 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.458 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.458 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:06:25.459 DEBUG:boto:Data: 2023-10-12T01:06:25.459 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.460 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.460 DEBUG:boto:Params: {} 2023-10-12T01:06:25.461 DEBUG:boto:Token: None 2023-10-12T01:06:25.461 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57 2023-10-12T01:06:25.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V2HCLTF+gUu8Aqamyp5TzNXN3Hw= 2023-10-12T01:06:25.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V2HCLTF+gUu8Aqamyp5TzNXN3Hw=', 'Content-Length': '0'} 2023-10-12T01:06:25.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526c0ecc9ed047c0-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.464 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.465 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.465 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.465 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:06:25.466 DEBUG:boto:Data: 2023-10-12T01:06:25.466 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.467 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.467 DEBUG:boto:Params: {} 2023-10-12T01:06:25.468 DEBUG:boto:Token: None 2023-10-12T01:06:25.468 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57 2023-10-12T01:06:25.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V2HCLTF+gUu8Aqamyp5TzNXN3Hw= 2023-10-12T01:06:25.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V2HCLTF+gUu8Aqamyp5TzNXN3Hw=', 'Content-Length': '0'} 2023-10-12T01:06:25.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8eb8f1efe5706d4-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:06:25.471 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.472 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.472 DEBUG:boto:Method: GET 2023-10-12T01:06:25.473 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:06:25.473 DEBUG:boto:Data: 2023-10-12T01:06:25.473 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.474 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.475 DEBUG:boto:Params: {} 2023-10-12T01:06:25.475 DEBUG:boto:Token: None 2023-10-12T01:06:25.475 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57 2023-10-12T01:06:25.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8eA0qLZ7/0/e3mFyoXRgMjK9ThI= 2023-10-12T01:06:25.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8eA0qLZ7/0/e3mFyoXRgMjK9ThI=', 'Content-Length': '0'} 2023-10-12T01:06:25.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60aa30b7085c1ec-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.479 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:06:25.479 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:06:25.480 DEBUG:boto:Method: GET 2023-10-12T01:06:25.480 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:06:25.481 DEBUG:boto:Data: 2023-10-12T01:06:25.481 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.482 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.482 DEBUG:boto:Params: {} 2023-10-12T01:06:25.483 DEBUG:boto:Token: None 2023-10-12T01:06:25.483 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj57 2023-10-12T01:06:25.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8eA0qLZ7/0/e3mFyoXRgMjK9ThI= 2023-10-12T01:06:25.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8eA0qLZ7/0/e3mFyoXRgMjK9ThI=', 'Content-Length': '0'} 2023-10-12T01:06:25.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d7184a399b1a9da-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.526 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.527 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.527 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.528 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.528 DEBUG:boto:Method: GET 2023-10-12T01:06:25.529 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.529 DEBUG:boto:Data: 2023-10-12T01:06:25.529 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.530 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.531 DEBUG:boto:Params: {} 2023-10-12T01:06:25.531 DEBUG:boto:Token: None 2023-10-12T01:06:25.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1303230419fc466-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.536 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.536 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.537 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.537 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.538 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.538 DEBUG:boto:Method: GET 2023-10-12T01:06:25.539 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:06:25.539 DEBUG:boto:Data: 2023-10-12T01:06:25.539 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.540 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.541 DEBUG:boto:Params: {} 2023-10-12T01:06:25.541 DEBUG:boto:Token: None 2023-10-12T01:06:25.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056115863413fd624-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.546 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:06:25.547 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.548 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.548 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.549 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.549 DEBUG:boto:Method: GET 2023-10-12T01:06:25.550 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.550 DEBUG:boto:Data: 2023-10-12T01:06:25.550 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.551 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.552 DEBUG:boto:Params: {} 2023-10-12T01:06:25.552 DEBUG:boto:Token: None 2023-10-12T01:06:25.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI= 2023-10-12T01:06:25.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI=', 'Content-Length': '0'} 2023-10-12T01:06:25.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27b6baf1fcb6808-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.556 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.557 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.557 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.557 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.558 DEBUG:boto:Method: GET 2023-10-12T01:06:25.558 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.559 DEBUG:boto:Data: 2023-10-12T01:06:25.559 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.560 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.560 DEBUG:boto:Params: {} 2023-10-12T01:06:25.561 DEBUG:boto:Token: None 2023-10-12T01:06:25.561 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.562 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI= 2023-10-12T01:06:25.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI=', 'Content-Length': '0'} 2023-10-12T01:06:25.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000984b003666cc5b02-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.565 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.565 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.566 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.566 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.567 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.567 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.567 DEBUG:boto:Data: 2023-10-12T01:06:25.568 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.569 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.569 DEBUG:boto:Params: {} 2023-10-12T01:06:25.570 DEBUG:boto:Token: None 2023-10-12T01:06:25.570 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C1vabsrVcor/7cqOJah78ETHG+8= 2023-10-12T01:06:25.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C1vabsrVcor/7cqOJah78ETHG+8=', 'Content-Length': '0'} 2023-10-12T01:06:25.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13a7137a2d7ea1d-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.573 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.574 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.574 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.575 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.575 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.575 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.576 DEBUG:boto:Data: 2023-10-12T01:06:25.576 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.577 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.578 DEBUG:boto:Params: {} 2023-10-12T01:06:25.578 DEBUG:boto:Token: None 2023-10-12T01:06:25.578 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.579 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C1vabsrVcor/7cqOJah78ETHG+8= 2023-10-12T01:06:25.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C1vabsrVcor/7cqOJah78ETHG+8=', 'Content-Length': '0'} 2023-10-12T01:06:25.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e72f94cc2e0dbf-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:06:25.582 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.582 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.583 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.583 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.584 DEBUG:boto:Method: GET 2023-10-12T01:06:25.584 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.585 DEBUG:boto:Data: 2023-10-12T01:06:25.585 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.586 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.586 DEBUG:boto:Params: {} 2023-10-12T01:06:25.587 DEBUG:boto:Token: None 2023-10-12T01:06:25.587 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.588 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI= 2023-10-12T01:06:25.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI=', 'Content-Length': '0'} 2023-10-12T01:06:25.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff473c353e9f738-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.591 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.591 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.592 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.592 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.593 DEBUG:boto:Method: GET 2023-10-12T01:06:25.593 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.593 DEBUG:boto:Data: 2023-10-12T01:06:25.594 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.595 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.595 DEBUG:boto:Params: {} 2023-10-12T01:06:25.596 DEBUG:boto:Token: None 2023-10-12T01:06:25.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:06:25.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI= 2023-10-12T01:06:25.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ISpPZIZQy8RjyWYIeXhYCG+7RWI=', 'Content-Length': '0'} 2023-10-12T01:06:25.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000471cfad2e2bb87e2-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.600 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.600 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.601 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.601 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.601 DEBUG:boto:Method: GET 2023-10-12T01:06:25.602 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.602 DEBUG:boto:Data: 2023-10-12T01:06:25.603 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.604 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.604 DEBUG:boto:Params: {} 2023-10-12T01:06:25.605 DEBUG:boto:Token: None 2023-10-12T01:06:25.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ebb12867c9a8c9-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.654 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.655 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.656 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.656 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.656 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.657 DEBUG:boto:Method: GET 2023-10-12T01:06:25.657 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.658 DEBUG:boto:Data: 2023-10-12T01:06:25.658 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.659 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.660 DEBUG:boto:Params: {} 2023-10-12T01:06:25.660 DEBUG:boto:Token: None 2023-10-12T01:06:25.661 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039327bec5e0295d4-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.665 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.666 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.666 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.667 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.667 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:06:25.667 DEBUG:boto:Data: 2023-10-12T01:06:25.668 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.669 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.669 DEBUG:boto:Params: {} 2023-10-12T01:06:25.670 DEBUG:boto:Token: None 2023-10-12T01:06:25.670 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58 2023-10-12T01:06:25.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vSYqg4WBxmFR2ZiXkorKPCC5gOQ= 2023-10-12T01:06:25.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vSYqg4WBxmFR2ZiXkorKPCC5gOQ=', 'Content-Length': '0'} 2023-10-12T01:06:25.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87f79558ff29da6-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.673 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.674 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.674 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.675 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:06:25.675 DEBUG:boto:Data: 2023-10-12T01:06:25.676 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.676 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.677 DEBUG:boto:Params: {} 2023-10-12T01:06:25.677 DEBUG:boto:Token: None 2023-10-12T01:06:25.678 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58 2023-10-12T01:06:25.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vSYqg4WBxmFR2ZiXkorKPCC5gOQ= 2023-10-12T01:06:25.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vSYqg4WBxmFR2ZiXkorKPCC5gOQ=', 'Content-Length': '0'} 2023-10-12T01:06:25.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf692a68251c8d3c-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:06:25.681 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.682 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.682 DEBUG:boto:Method: GET 2023-10-12T01:06:25.683 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:06:25.683 DEBUG:boto:Data: 2023-10-12T01:06:25.684 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.684 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.685 DEBUG:boto:Params: {} 2023-10-12T01:06:25.685 DEBUG:boto:Token: None 2023-10-12T01:06:25.686 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58 2023-10-12T01:06:25.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DUzr+KIgX3Z6AdOnz2e4Z5QIbsM= 2023-10-12T01:06:25.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DUzr+KIgX3Z6AdOnz2e4Z5QIbsM=', 'Content-Length': '0'} 2023-10-12T01:06:25.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f89f121d44ece2-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.689 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:06:25.690 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:06:25.690 DEBUG:boto:Method: GET 2023-10-12T01:06:25.690 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:06:25.691 DEBUG:boto:Data: 2023-10-12T01:06:25.691 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.692 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.693 DEBUG:boto:Params: {} 2023-10-12T01:06:25.693 DEBUG:boto:Token: None 2023-10-12T01:06:25.694 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj58 2023-10-12T01:06:25.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DUzr+KIgX3Z6AdOnz2e4Z5QIbsM= 2023-10-12T01:06:25.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DUzr+KIgX3Z6AdOnz2e4Z5QIbsM=', 'Content-Length': '0'} 2023-10-12T01:06:25.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b0c51671f39e900-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.697 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.697 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.698 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.698 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.699 DEBUG:boto:Method: GET 2023-10-12T01:06:25.699 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.700 DEBUG:boto:Data: 2023-10-12T01:06:25.700 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.701 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.701 DEBUG:boto:Params: {} 2023-10-12T01:06:25.702 DEBUG:boto:Token: None 2023-10-12T01:06:25.702 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0c2bf90c0684f6-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.706 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.707 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.707 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.708 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.708 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.708 DEBUG:boto:Method: GET 2023-10-12T01:06:25.709 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:06:25.709 DEBUG:boto:Data: 2023-10-12T01:06:25.710 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.711 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.711 DEBUG:boto:Params: {} 2023-10-12T01:06:25.711 DEBUG:boto:Token: None 2023-10-12T01:06:25.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d532900e00583938-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.717 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:06:25.717 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.718 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.718 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.719 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.719 DEBUG:boto:Method: GET 2023-10-12T01:06:25.720 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.720 DEBUG:boto:Data: 2023-10-12T01:06:25.721 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.721 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.722 DEBUG:boto:Params: {} 2023-10-12T01:06:25.722 DEBUG:boto:Token: None 2023-10-12T01:06:25.723 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.723 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0= 2023-10-12T01:06:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0=', 'Content-Length': '0'} 2023-10-12T01:06:25.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fee9bb7d4a9942f-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.726 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.727 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.727 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.728 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.728 DEBUG:boto:Method: GET 2023-10-12T01:06:25.728 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.729 DEBUG:boto:Data: 2023-10-12T01:06:25.729 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.730 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.730 DEBUG:boto:Params: {} 2023-10-12T01:06:25.731 DEBUG:boto:Token: None 2023-10-12T01:06:25.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0= 2023-10-12T01:06:25.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0=', 'Content-Length': '0'} 2023-10-12T01:06:25.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6c7bdceec1ab94-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.735 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.735 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.736 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.736 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.736 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.737 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.737 DEBUG:boto:Data: 2023-10-12T01:06:25.738 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.738 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.739 DEBUG:boto:Params: {} 2023-10-12T01:06:25.739 DEBUG:boto:Token: None 2023-10-12T01:06:25.740 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TfvNHv9AUkSPDE07W148fhw9t7Q= 2023-10-12T01:06:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TfvNHv9AUkSPDE07W148fhw9t7Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ffbc10792b9624-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.743 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.743 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.744 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.744 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.744 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.745 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.745 DEBUG:boto:Data: 2023-10-12T01:06:25.746 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.746 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.747 DEBUG:boto:Params: {} 2023-10-12T01:06:25.747 DEBUG:boto:Token: None 2023-10-12T01:06:25.748 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.748 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TfvNHv9AUkSPDE07W148fhw9t7Q= 2023-10-12T01:06:25.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TfvNHv9AUkSPDE07W148fhw9t7Q=', 'Content-Length': '0'} 2023-10-12T01:06:25.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d369e070ca781d02-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:06:25.752 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.752 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.752 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.753 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.753 DEBUG:boto:Method: GET 2023-10-12T01:06:25.754 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.754 DEBUG:boto:Data: 2023-10-12T01:06:25.755 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.755 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.756 DEBUG:boto:Params: {} 2023-10-12T01:06:25.756 DEBUG:boto:Token: None 2023-10-12T01:06:25.757 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0= 2023-10-12T01:06:25.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0=', 'Content-Length': '0'} 2023-10-12T01:06:25.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e267ed22a072365-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.760 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.761 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.761 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.762 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.762 DEBUG:boto:Method: GET 2023-10-12T01:06:25.763 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.763 DEBUG:boto:Data: 2023-10-12T01:06:25.763 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.764 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.765 DEBUG:boto:Params: {} 2023-10-12T01:06:25.765 DEBUG:boto:Token: None 2023-10-12T01:06:25.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:06:25.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0= 2023-10-12T01:06:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OU50J+EH1v7vlDF4nj9xRMeIsO0=', 'Content-Length': '0'} 2023-10-12T01:06:25.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f511e18efdad62e-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.814 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.815 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.816 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.816 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.816 DEBUG:boto:Method: GET 2023-10-12T01:06:25.817 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.817 DEBUG:boto:Data: 2023-10-12T01:06:25.818 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.819 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.819 DEBUG:boto:Params: {} 2023-10-12T01:06:25.819 DEBUG:boto:Token: None 2023-10-12T01:06:25.820 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0f2572dc8e1e74-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.824 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.825 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.825 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.825 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.826 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.826 DEBUG:boto:Method: GET 2023-10-12T01:06:25.827 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.827 DEBUG:boto:Data: 2023-10-12T01:06:25.828 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.829 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.829 DEBUG:boto:Params: {} 2023-10-12T01:06:25.830 DEBUG:boto:Token: None 2023-10-12T01:06:25.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ad28e6b3fed15e-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.835 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.836 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.836 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.836 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.837 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:06:25.837 DEBUG:boto:Data: 2023-10-12T01:06:25.838 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.839 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.839 DEBUG:boto:Params: {} 2023-10-12T01:06:25.839 DEBUG:boto:Token: None 2023-10-12T01:06:25.840 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68 2023-10-12T01:06:25.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bmB88MtvrxUWZMSl24VtOGkQfng= 2023-10-12T01:06:25.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bmB88MtvrxUWZMSl24VtOGkQfng=', 'Content-Length': '0'} 2023-10-12T01:06:25.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001726671f57df470-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.843 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.844 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.844 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.844 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:06:25.845 DEBUG:boto:Data: 2023-10-12T01:06:25.845 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.846 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.846 DEBUG:boto:Params: {} 2023-10-12T01:06:25.847 DEBUG:boto:Token: None 2023-10-12T01:06:25.847 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68 2023-10-12T01:06:25.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bmB88MtvrxUWZMSl24VtOGkQfng= 2023-10-12T01:06:25.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bmB88MtvrxUWZMSl24VtOGkQfng=', 'Content-Length': '0'} 2023-10-12T01:06:25.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc397c41d33ca6e-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:06:25.851 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.852 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.852 DEBUG:boto:Method: GET 2023-10-12T01:06:25.852 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:06:25.853 DEBUG:boto:Data: 2023-10-12T01:06:25.853 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.854 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.855 DEBUG:boto:Params: {} 2023-10-12T01:06:25.855 DEBUG:boto:Token: None 2023-10-12T01:06:25.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68 2023-10-12T01:06:25.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6r2CQLajdhcb0g91VQCBilpQseY= 2023-10-12T01:06:25.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6r2CQLajdhcb0g91VQCBilpQseY=', 'Content-Length': '0'} 2023-10-12T01:06:25.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1886ddc01d03e1-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.902 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:06:25.903 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:06:25.903 DEBUG:boto:Method: GET 2023-10-12T01:06:25.904 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:06:25.904 DEBUG:boto:Data: 2023-10-12T01:06:25.905 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.906 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.906 DEBUG:boto:Params: {} 2023-10-12T01:06:25.907 DEBUG:boto:Token: None 2023-10-12T01:06:25.907 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj68 2023-10-12T01:06:25.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6r2CQLajdhcb0g91VQCBilpQseY= 2023-10-12T01:06:25.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6r2CQLajdhcb0g91VQCBilpQseY=', 'Content-Length': '0'} 2023-10-12T01:06:25.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e7f9a1bdce8699-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.911 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.911 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.912 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.912 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.912 DEBUG:boto:Method: GET 2023-10-12T01:06:25.913 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.913 DEBUG:boto:Data: 2023-10-12T01:06:25.914 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.915 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.915 DEBUG:boto:Params: {} 2023-10-12T01:06:25.915 DEBUG:boto:Token: None 2023-10-12T01:06:25.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000605e69bc2b68346-0065274691-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.920 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.920 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:25.921 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:25.921 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.922 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.922 DEBUG:boto:Method: GET 2023-10-12T01:06:25.923 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:06:25.923 DEBUG:boto:Data: 2023-10-12T01:06:25.924 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.924 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.925 DEBUG:boto:Params: {} 2023-10-12T01:06:25.925 DEBUG:boto:Token: None 2023-10-12T01:06:25.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/ 2023-10-12T01:06:25.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA= 2023-10-12T01:06:25.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nCEpU/kN+ZbtVm1E0OM9+GIlZjA=', 'Content-Length': '0'} 2023-10-12T01:06:25.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3b3d535783e382-0065274691-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.931 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:25.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:06:25.932 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:25.932 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:25.933 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.933 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.934 DEBUG:boto:Method: GET 2023-10-12T01:06:25.934 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.935 DEBUG:boto:Data: 2023-10-12T01:06:25.935 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.936 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.936 DEBUG:boto:Params: {} 2023-10-12T01:06:25.937 DEBUG:boto:Token: None 2023-10-12T01:06:25.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q5TXE/TiLbnSj7Vqb3yUIJCDIr8= 2023-10-12T01:06:25.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q5TXE/TiLbnSj7Vqb3yUIJCDIr8=', 'Content-Length': '0'} 2023-10-12T01:06:25.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2394c9576e8a72-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.941 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:25.941 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:25.942 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.942 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.943 DEBUG:boto:Method: GET 2023-10-12T01:06:25.943 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.944 DEBUG:boto:Data: 2023-10-12T01:06:25.944 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:25.945 DEBUG:boto:Port: 8000 2023-10-12T01:06:25.945 DEBUG:boto:Params: {} 2023-10-12T01:06:25.946 DEBUG:boto:Token: None 2023-10-12T01:06:25.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q5TXE/TiLbnSj7Vqb3yUIJCDIr8= 2023-10-12T01:06:25.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q5TXE/TiLbnSj7Vqb3yUIJCDIr8=', 'Content-Length': '0'} 2023-10-12T01:06:25.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bbcaf0bb21f3293-0065274691-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.990 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:25.991 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:25.991 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.992 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.992 DEBUG:boto:Method: HEAD 2023-10-12T01:06:25.993 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.993 DEBUG:boto:Data: 2023-10-12T01:06:25.993 DEBUG:boto:Headers: {} 2023-10-12T01:06:25.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:25.994 DEBUG:boto:Port: 8001 2023-10-12T01:06:25.995 DEBUG:boto:Params: {} 2023-10-12T01:06:25.995 DEBUG:boto:Token: None 2023-10-12T01:06:25.995 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:25 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:25.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cGrEtK0367lhsMSae2WeqYATvrg= 2023-10-12T01:06:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cGrEtK0367lhsMSae2WeqYATvrg=', 'Content-Length': '0'} 2023-10-12T01:06:25.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936d756e20bd9e8f-0065274691-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:25.999 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:25.999 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.000 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.000 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.000 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.001 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.001 DEBUG:boto:Data: 2023-10-12T01:06:26.002 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.003 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.003 DEBUG:boto:Params: {} 2023-10-12T01:06:26.003 DEBUG:boto:Token: None 2023-10-12T01:06:26.004 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:07695SlVp1RYwARGTrfvO/oyqkc= 2023-10-12T01:06:26.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:07695SlVp1RYwARGTrfvO/oyqkc=', 'Content-Length': '0'} 2023-10-12T01:06:26.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008a5c44d535e6fc6-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:06:26.008 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.008 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.009 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.009 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.010 DEBUG:boto:Method: GET 2023-10-12T01:06:26.010 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.010 DEBUG:boto:Data: 2023-10-12T01:06:26.011 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.012 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.012 DEBUG:boto:Params: {} 2023-10-12T01:06:26.012 DEBUG:boto:Token: None 2023-10-12T01:06:26.013 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eIaGUhd6vRqQXWGhNb7Fs7g4TXc= 2023-10-12T01:06:26.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eIaGUhd6vRqQXWGhNb7Fs7g4TXc=', 'Content-Length': '0'} 2023-10-12T01:06:26.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014418f236732e130-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.016 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.016 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.017 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.017 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.018 DEBUG:boto:Method: GET 2023-10-12T01:06:26.018 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.019 DEBUG:boto:Data: 2023-10-12T01:06:26.019 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.020 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.020 DEBUG:boto:Params: {} 2023-10-12T01:06:26.020 DEBUG:boto:Token: None 2023-10-12T01:06:26.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:06:26.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eIaGUhd6vRqQXWGhNb7Fs7g4TXc= 2023-10-12T01:06:26.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eIaGUhd6vRqQXWGhNb7Fs7g4TXc=', 'Content-Length': '0'} 2023-10-12T01:06:26.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf900ebab741c1f-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.024 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.025 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.025 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.026 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.026 DEBUG:boto:Method: GET 2023-10-12T01:06:26.027 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.027 DEBUG:boto:Data: 2023-10-12T01:06:26.027 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.028 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.029 DEBUG:boto:Params: {} 2023-10-12T01:06:26.029 DEBUG:boto:Token: None 2023-10-12T01:06:26.029 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002974f0cb5306297f-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.033 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.034 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.034 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.035 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.035 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.035 DEBUG:boto:Method: GET 2023-10-12T01:06:26.036 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.036 DEBUG:boto:Data: 2023-10-12T01:06:26.037 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.038 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.038 DEBUG:boto:Params: {} 2023-10-12T01:06:26.039 DEBUG:boto:Token: None 2023-10-12T01:06:26.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000938ae811baeaae16-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.044 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.045 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.045 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.046 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.046 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:06:26.047 DEBUG:boto:Data: 2023-10-12T01:06:26.047 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.048 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.048 DEBUG:boto:Params: {} 2023-10-12T01:06:26.049 DEBUG:boto:Token: None 2023-10-12T01:06:26.049 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69 2023-10-12T01:06:26.050 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QfHv3t5Y/6CAvBxe7fSbSYg3APE= 2023-10-12T01:06:26.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QfHv3t5Y/6CAvBxe7fSbSYg3APE=', 'Content-Length': '0'} 2023-10-12T01:06:26.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d61d04f0bdee2cc7-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.053 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.053 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.053 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.054 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:06:26.054 DEBUG:boto:Data: 2023-10-12T01:06:26.055 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.055 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.056 DEBUG:boto:Params: {} 2023-10-12T01:06:26.056 DEBUG:boto:Token: None 2023-10-12T01:06:26.057 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69 2023-10-12T01:06:26.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QfHv3t5Y/6CAvBxe7fSbSYg3APE= 2023-10-12T01:06:26.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QfHv3t5Y/6CAvBxe7fSbSYg3APE=', 'Content-Length': '0'} 2023-10-12T01:06:26.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e35ab6e8a3d5940c-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:06:26.060 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.061 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.061 DEBUG:boto:Method: GET 2023-10-12T01:06:26.062 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:06:26.062 DEBUG:boto:Data: 2023-10-12T01:06:26.063 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.063 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.064 DEBUG:boto:Params: {} 2023-10-12T01:06:26.064 DEBUG:boto:Token: None 2023-10-12T01:06:26.065 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69 2023-10-12T01:06:26.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6TPHza/xmPDNX23+lEkPaQbNFc8= 2023-10-12T01:06:26.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6TPHza/xmPDNX23+lEkPaQbNFc8=', 'Content-Length': '0'} 2023-10-12T01:06:26.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f8ea4bf9d958a0-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.068 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:06:26.069 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:06:26.069 DEBUG:boto:Method: GET 2023-10-12T01:06:26.069 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:06:26.070 DEBUG:boto:Data: 2023-10-12T01:06:26.070 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.071 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.071 DEBUG:boto:Params: {} 2023-10-12T01:06:26.072 DEBUG:boto:Token: None 2023-10-12T01:06:26.072 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj69 2023-10-12T01:06:26.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6TPHza/xmPDNX23+lEkPaQbNFc8= 2023-10-12T01:06:26.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6TPHza/xmPDNX23+lEkPaQbNFc8=', 'Content-Length': '0'} 2023-10-12T01:06:26.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3126250aba46ed-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.118 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.119 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.119 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.120 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.120 DEBUG:boto:Method: GET 2023-10-12T01:06:26.121 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.121 DEBUG:boto:Data: 2023-10-12T01:06:26.121 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.122 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.123 DEBUG:boto:Params: {} 2023-10-12T01:06:26.123 DEBUG:boto:Token: None 2023-10-12T01:06:26.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed2d14902091670a-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.128 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.128 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.129 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.129 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.130 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.130 DEBUG:boto:Method: GET 2023-10-12T01:06:26.131 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:06:26.131 DEBUG:boto:Data: 2023-10-12T01:06:26.131 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.132 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.133 DEBUG:boto:Params: {} 2023-10-12T01:06:26.133 DEBUG:boto:Token: None 2023-10-12T01:06:26.134 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.134 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ae3be34b05226a-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.139 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:06:26.140 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.140 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.141 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.141 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.141 DEBUG:boto:Method: GET 2023-10-12T01:06:26.142 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.142 DEBUG:boto:Data: 2023-10-12T01:06:26.143 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.144 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.144 DEBUG:boto:Params: {} 2023-10-12T01:06:26.144 DEBUG:boto:Token: None 2023-10-12T01:06:26.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4= 2023-10-12T01:06:26.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4=', 'Content-Length': '0'} 2023-10-12T01:06:26.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf78ba6ac07503c-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.190 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.191 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.191 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.192 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.192 DEBUG:boto:Method: GET 2023-10-12T01:06:26.193 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.193 DEBUG:boto:Data: 2023-10-12T01:06:26.194 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.194 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.195 DEBUG:boto:Params: {} 2023-10-12T01:06:26.195 DEBUG:boto:Token: None 2023-10-12T01:06:26.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4= 2023-10-12T01:06:26.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4=', 'Content-Length': '0'} 2023-10-12T01:06:26.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009415de51ff189d24-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.199 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.200 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.200 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.201 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.201 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.202 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.202 DEBUG:boto:Data: 2023-10-12T01:06:26.202 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.203 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.204 DEBUG:boto:Params: {} 2023-10-12T01:06:26.205 DEBUG:boto:Token: None 2023-10-12T01:06:26.205 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I0MOIPwAeBF5NylLAGstYjeM/Gg= 2023-10-12T01:06:26.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I0MOIPwAeBF5NylLAGstYjeM/Gg=', 'Content-Length': '0'} 2023-10-12T01:06:26.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005216d0756745bd43-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.209 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.209 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.209 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.210 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.210 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.211 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.211 DEBUG:boto:Data: 2023-10-12T01:06:26.212 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.212 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.213 DEBUG:boto:Params: {} 2023-10-12T01:06:26.213 DEBUG:boto:Token: None 2023-10-12T01:06:26.214 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I0MOIPwAeBF5NylLAGstYjeM/Gg= 2023-10-12T01:06:26.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I0MOIPwAeBF5NylLAGstYjeM/Gg=', 'Content-Length': '0'} 2023-10-12T01:06:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000231443a58279e59e-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:06:26.218 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.218 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.218 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.219 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.219 DEBUG:boto:Method: GET 2023-10-12T01:06:26.220 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.220 DEBUG:boto:Data: 2023-10-12T01:06:26.221 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.222 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.222 DEBUG:boto:Params: {} 2023-10-12T01:06:26.222 DEBUG:boto:Token: None 2023-10-12T01:06:26.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4= 2023-10-12T01:06:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4=', 'Content-Length': '0'} 2023-10-12T01:06:26.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a13915b2d0b4a5a-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.226 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.227 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.227 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.228 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.228 DEBUG:boto:Method: GET 2023-10-12T01:06:26.228 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.229 DEBUG:boto:Data: 2023-10-12T01:06:26.229 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.230 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.231 DEBUG:boto:Params: {} 2023-10-12T01:06:26.231 DEBUG:boto:Token: None 2023-10-12T01:06:26.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:06:26.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4= 2023-10-12T01:06:26.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yuT0KgX5MTjOuJPrAAAvQCn8Kf4=', 'Content-Length': '0'} 2023-10-12T01:06:26.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259c9ded8416d907-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.235 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.236 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.236 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.236 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.237 DEBUG:boto:Method: GET 2023-10-12T01:06:26.237 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.238 DEBUG:boto:Data: 2023-10-12T01:06:26.238 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.239 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.240 DEBUG:boto:Params: {} 2023-10-12T01:06:26.240 DEBUG:boto:Token: None 2023-10-12T01:06:26.241 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4c148f5c8527b8-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.244 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.245 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.246 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.246 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.246 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.247 DEBUG:boto:Method: GET 2023-10-12T01:06:26.247 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.248 DEBUG:boto:Data: 2023-10-12T01:06:26.248 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.249 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.249 DEBUG:boto:Params: {} 2023-10-12T01:06:26.250 DEBUG:boto:Token: None 2023-10-12T01:06:26.250 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.251 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83956b917f686a1-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.302 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.303 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.304 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.304 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.304 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:06:26.305 DEBUG:boto:Data: 2023-10-12T01:06:26.305 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.306 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.307 DEBUG:boto:Params: {} 2023-10-12T01:06:26.307 DEBUG:boto:Token: None 2023-10-12T01:06:26.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76 2023-10-12T01:06:26.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GnP1RuSVFptjOHpcdRCgoQ/hgRY= 2023-10-12T01:06:26.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GnP1RuSVFptjOHpcdRCgoQ/hgRY=', 'Content-Length': '0'} 2023-10-12T01:06:26.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d07f2f775a7690-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.311 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.311 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.312 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.312 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:06:26.313 DEBUG:boto:Data: 2023-10-12T01:06:26.313 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.314 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.314 DEBUG:boto:Params: {} 2023-10-12T01:06:26.314 DEBUG:boto:Token: None 2023-10-12T01:06:26.315 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76 2023-10-12T01:06:26.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GnP1RuSVFptjOHpcdRCgoQ/hgRY= 2023-10-12T01:06:26.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GnP1RuSVFptjOHpcdRCgoQ/hgRY=', 'Content-Length': '0'} 2023-10-12T01:06:26.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162f2f9d8f01b46e-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:06:26.319 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.319 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.320 DEBUG:boto:Method: GET 2023-10-12T01:06:26.320 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:06:26.320 DEBUG:boto:Data: 2023-10-12T01:06:26.321 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.322 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.322 DEBUG:boto:Params: {} 2023-10-12T01:06:26.322 DEBUG:boto:Token: None 2023-10-12T01:06:26.323 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76 2023-10-12T01:06:26.323 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L27kFoqliTBRhcut+1rD819vyW0= 2023-10-12T01:06:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L27kFoqliTBRhcut+1rD819vyW0=', 'Content-Length': '0'} 2023-10-12T01:06:26.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9c2998586f6c61-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.326 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:06:26.327 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:06:26.327 DEBUG:boto:Method: GET 2023-10-12T01:06:26.328 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:06:26.328 DEBUG:boto:Data: 2023-10-12T01:06:26.328 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.329 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.329 DEBUG:boto:Params: {} 2023-10-12T01:06:26.330 DEBUG:boto:Token: None 2023-10-12T01:06:26.330 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj76 2023-10-12T01:06:26.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L27kFoqliTBRhcut+1rD819vyW0= 2023-10-12T01:06:26.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L27kFoqliTBRhcut+1rD819vyW0=', 'Content-Length': '0'} 2023-10-12T01:06:26.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11e130c3673fc0b-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.334 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.334 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.334 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.335 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.335 DEBUG:boto:Method: GET 2023-10-12T01:06:26.336 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.336 DEBUG:boto:Data: 2023-10-12T01:06:26.336 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.337 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.338 DEBUG:boto:Params: {} 2023-10-12T01:06:26.338 DEBUG:boto:Token: None 2023-10-12T01:06:26.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8666a0e35e6f86-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.342 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.343 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.343 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.344 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.344 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.344 DEBUG:boto:Method: GET 2023-10-12T01:06:26.345 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:06:26.345 DEBUG:boto:Data: 2023-10-12T01:06:26.346 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.346 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.347 DEBUG:boto:Params: {} 2023-10-12T01:06:26.347 DEBUG:boto:Token: None 2023-10-12T01:06:26.348 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b87921eb4f625ca-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.352 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:06:26.353 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.354 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.354 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.355 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.355 DEBUG:boto:Method: GET 2023-10-12T01:06:26.356 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.356 DEBUG:boto:Data: 2023-10-12T01:06:26.357 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.357 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.358 DEBUG:boto:Params: {} 2023-10-12T01:06:26.358 DEBUG:boto:Token: None 2023-10-12T01:06:26.359 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0= 2023-10-12T01:06:26.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0=', 'Content-Length': '0'} 2023-10-12T01:06:26.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8a6fd9f744454f-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.362 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.363 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.363 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.364 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.364 DEBUG:boto:Method: GET 2023-10-12T01:06:26.365 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.365 DEBUG:boto:Data: 2023-10-12T01:06:26.365 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.366 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.367 DEBUG:boto:Params: {} 2023-10-12T01:06:26.367 DEBUG:boto:Token: None 2023-10-12T01:06:26.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0= 2023-10-12T01:06:26.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0=', 'Content-Length': '0'} 2023-10-12T01:06:26.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d38d6752844262-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.371 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.371 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.372 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.372 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.372 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.373 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.373 DEBUG:boto:Data: 2023-10-12T01:06:26.374 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.374 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.375 DEBUG:boto:Params: {} 2023-10-12T01:06:26.375 DEBUG:boto:Token: None 2023-10-12T01:06:26.376 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rbb4SHVHKNISQ9dw5r/m3q6scEs= 2023-10-12T01:06:26.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rbb4SHVHKNISQ9dw5r/m3q6scEs=', 'Content-Length': '0'} 2023-10-12T01:06:26.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099884b221fdac880-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.379 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.379 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.380 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.380 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.380 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.381 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.381 DEBUG:boto:Data: 2023-10-12T01:06:26.382 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.383 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.383 DEBUG:boto:Params: {} 2023-10-12T01:06:26.383 DEBUG:boto:Token: None 2023-10-12T01:06:26.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rbb4SHVHKNISQ9dw5r/m3q6scEs= 2023-10-12T01:06:26.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rbb4SHVHKNISQ9dw5r/m3q6scEs=', 'Content-Length': '0'} 2023-10-12T01:06:26.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0d0cc4274e9fcd-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:06:26.387 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.388 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.388 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.389 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.389 DEBUG:boto:Method: GET 2023-10-12T01:06:26.389 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.390 DEBUG:boto:Data: 2023-10-12T01:06:26.390 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.391 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.391 DEBUG:boto:Params: {} 2023-10-12T01:06:26.392 DEBUG:boto:Token: None 2023-10-12T01:06:26.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0= 2023-10-12T01:06:26.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0=', 'Content-Length': '0'} 2023-10-12T01:06:26.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a28f3ed84b63a0-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.442 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.443 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.443 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.444 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.444 DEBUG:boto:Method: GET 2023-10-12T01:06:26.445 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.445 DEBUG:boto:Data: 2023-10-12T01:06:26.446 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.446 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.447 DEBUG:boto:Params: {} 2023-10-12T01:06:26.447 DEBUG:boto:Token: None 2023-10-12T01:06:26.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:06:26.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0= 2023-10-12T01:06:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MDrlzAiN6gx3t99d5DAZsPh8Fb0=', 'Content-Length': '0'} 2023-10-12T01:06:26.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004135752734681bc1-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.452 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.452 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.453 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.453 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.453 DEBUG:boto:Method: GET 2023-10-12T01:06:26.454 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.454 DEBUG:boto:Data: 2023-10-12T01:06:26.455 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.456 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.456 DEBUG:boto:Params: {} 2023-10-12T01:06:26.457 DEBUG:boto:Token: None 2023-10-12T01:06:26.457 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.458 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e388405efe56f3fc-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.461 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.462 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.462 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.463 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.463 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.464 DEBUG:boto:Method: GET 2023-10-12T01:06:26.464 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.465 DEBUG:boto:Data: 2023-10-12T01:06:26.465 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.466 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.466 DEBUG:boto:Params: {} 2023-10-12T01:06:26.467 DEBUG:boto:Token: None 2023-10-12T01:06:26.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194e1b7c48a50318-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.472 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.473 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.473 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.474 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.474 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:06:26.475 DEBUG:boto:Data: 2023-10-12T01:06:26.475 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.476 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.476 DEBUG:boto:Params: {} 2023-10-12T01:06:26.477 DEBUG:boto:Token: None 2023-10-12T01:06:26.477 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85 2023-10-12T01:06:26.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTDTY3jprz9Qa+//yQbERDxyoRM= 2023-10-12T01:06:26.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTDTY3jprz9Qa+//yQbERDxyoRM=', 'Content-Length': '0'} 2023-10-12T01:06:26.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076faf274ce9db71b-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.480 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.481 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.481 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.482 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:06:26.482 DEBUG:boto:Data: 2023-10-12T01:06:26.483 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.483 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.484 DEBUG:boto:Params: {} 2023-10-12T01:06:26.484 DEBUG:boto:Token: None 2023-10-12T01:06:26.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85 2023-10-12T01:06:26.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTDTY3jprz9Qa+//yQbERDxyoRM= 2023-10-12T01:06:26.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTDTY3jprz9Qa+//yQbERDxyoRM=', 'Content-Length': '0'} 2023-10-12T01:06:26.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008909ba77973ff5a-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:06:26.489 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.489 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.490 DEBUG:boto:Method: GET 2023-10-12T01:06:26.490 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:06:26.490 DEBUG:boto:Data: 2023-10-12T01:06:26.491 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.492 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.492 DEBUG:boto:Params: {} 2023-10-12T01:06:26.493 DEBUG:boto:Token: None 2023-10-12T01:06:26.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85 2023-10-12T01:06:26.494 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RbHoglQ2lbVxdNv6EEZJPad6C54= 2023-10-12T01:06:26.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RbHoglQ2lbVxdNv6EEZJPad6C54=', 'Content-Length': '0'} 2023-10-12T01:06:26.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093dca2a52503d5e7-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.496 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:06:26.497 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:06:26.497 DEBUG:boto:Method: GET 2023-10-12T01:06:26.498 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:06:26.498 DEBUG:boto:Data: 2023-10-12T01:06:26.499 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.500 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.500 DEBUG:boto:Params: {} 2023-10-12T01:06:26.500 DEBUG:boto:Token: None 2023-10-12T01:06:26.501 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj85 2023-10-12T01:06:26.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RbHoglQ2lbVxdNv6EEZJPad6C54= 2023-10-12T01:06:26.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RbHoglQ2lbVxdNv6EEZJPad6C54=', 'Content-Length': '0'} 2023-10-12T01:06:26.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8197f3d037e6cc8-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.505 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.505 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.506 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.506 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.506 DEBUG:boto:Method: GET 2023-10-12T01:06:26.507 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.507 DEBUG:boto:Data: 2023-10-12T01:06:26.508 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.509 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.509 DEBUG:boto:Params: {} 2023-10-12T01:06:26.509 DEBUG:boto:Token: None 2023-10-12T01:06:26.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9beb5b58844828-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.514 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.515 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.515 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.516 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.516 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.517 DEBUG:boto:Method: GET 2023-10-12T01:06:26.517 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:06:26.518 DEBUG:boto:Data: 2023-10-12T01:06:26.518 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.519 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.519 DEBUG:boto:Params: {} 2023-10-12T01:06:26.520 DEBUG:boto:Token: None 2023-10-12T01:06:26.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0a6de54541fe5e-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.525 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:06:26.526 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.526 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.526 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.527 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.527 DEBUG:boto:Method: GET 2023-10-12T01:06:26.528 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.528 DEBUG:boto:Data: 2023-10-12T01:06:26.529 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.529 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.530 DEBUG:boto:Params: {} 2023-10-12T01:06:26.530 DEBUG:boto:Token: None 2023-10-12T01:06:26.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc= 2023-10-12T01:06:26.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc=', 'Content-Length': '0'} 2023-10-12T01:06:26.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4305f4cb6de193-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.534 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.535 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.535 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.536 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.536 DEBUG:boto:Method: GET 2023-10-12T01:06:26.537 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.537 DEBUG:boto:Data: 2023-10-12T01:06:26.538 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.538 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.539 DEBUG:boto:Params: {} 2023-10-12T01:06:26.539 DEBUG:boto:Token: None 2023-10-12T01:06:26.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc= 2023-10-12T01:06:26.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc=', 'Content-Length': '0'} 2023-10-12T01:06:26.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc8b4affc7b0b7e-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.586 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.587 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.587 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.587 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.588 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.588 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.589 DEBUG:boto:Data: 2023-10-12T01:06:26.589 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.590 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.590 DEBUG:boto:Params: {} 2023-10-12T01:06:26.591 DEBUG:boto:Token: None 2023-10-12T01:06:26.591 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BjCWY32qWly3/7uwDauDL/hGE/E= 2023-10-12T01:06:26.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjCWY32qWly3/7uwDauDL/hGE/E=', 'Content-Length': '0'} 2023-10-12T01:06:26.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ab047748b97a60-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.594 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.595 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.595 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.596 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.596 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.597 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.597 DEBUG:boto:Data: 2023-10-12T01:06:26.597 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.598 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.599 DEBUG:boto:Params: {} 2023-10-12T01:06:26.599 DEBUG:boto:Token: None 2023-10-12T01:06:26.600 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BjCWY32qWly3/7uwDauDL/hGE/E= 2023-10-12T01:06:26.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjCWY32qWly3/7uwDauDL/hGE/E=', 'Content-Length': '0'} 2023-10-12T01:06:26.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007dc22ffeb8b67b6-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:06:26.603 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.603 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.604 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.604 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.605 DEBUG:boto:Method: GET 2023-10-12T01:06:26.605 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.605 DEBUG:boto:Data: 2023-10-12T01:06:26.606 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.607 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.607 DEBUG:boto:Params: {} 2023-10-12T01:06:26.608 DEBUG:boto:Token: None 2023-10-12T01:06:26.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc= 2023-10-12T01:06:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc=', 'Content-Length': '0'} 2023-10-12T01:06:26.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000976f2599a6625ba-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.612 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.612 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.613 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.613 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.613 DEBUG:boto:Method: GET 2023-10-12T01:06:26.614 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.614 DEBUG:boto:Data: 2023-10-12T01:06:26.615 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.615 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.616 DEBUG:boto:Params: {} 2023-10-12T01:06:26.616 DEBUG:boto:Token: None 2023-10-12T01:06:26.617 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:06:26.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc= 2023-10-12T01:06:26.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:suC8fLk4qtNNA+eqZeGVd2TUTSc=', 'Content-Length': '0'} 2023-10-12T01:06:26.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e50fd72151adcfa1-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.620 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.621 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.621 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.622 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.622 DEBUG:boto:Method: GET 2023-10-12T01:06:26.622 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.623 DEBUG:boto:Data: 2023-10-12T01:06:26.623 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.624 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.624 DEBUG:boto:Params: {} 2023-10-12T01:06:26.625 DEBUG:boto:Token: None 2023-10-12T01:06:26.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b729a8494352470-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.629 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.630 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.630 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.631 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.631 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.631 DEBUG:boto:Method: GET 2023-10-12T01:06:26.632 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.632 DEBUG:boto:Data: 2023-10-12T01:06:26.633 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.633 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.634 DEBUG:boto:Params: {} 2023-10-12T01:06:26.634 DEBUG:boto:Token: None 2023-10-12T01:06:26.635 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f675366b38be8c83-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.640 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.640 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.641 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.641 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.641 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:06:26.642 DEBUG:boto:Data: 2023-10-12T01:06:26.642 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.643 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.643 DEBUG:boto:Params: {} 2023-10-12T01:06:26.644 DEBUG:boto:Token: None 2023-10-12T01:06:26.644 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87 2023-10-12T01:06:26.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UrgzPluknbcIEOTmszFc5b3cUP8= 2023-10-12T01:06:26.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UrgzPluknbcIEOTmszFc5b3cUP8=', 'Content-Length': '0'} 2023-10-12T01:06:26.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a640df57410ccbde-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.648 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.648 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.649 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.649 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:06:26.649 DEBUG:boto:Data: 2023-10-12T01:06:26.650 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.651 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.651 DEBUG:boto:Params: {} 2023-10-12T01:06:26.651 DEBUG:boto:Token: None 2023-10-12T01:06:26.652 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87 2023-10-12T01:06:26.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UrgzPluknbcIEOTmszFc5b3cUP8= 2023-10-12T01:06:26.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UrgzPluknbcIEOTmszFc5b3cUP8=', 'Content-Length': '0'} 2023-10-12T01:06:26.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b248a207b3eb70ba-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:06:26.655 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.656 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.656 DEBUG:boto:Method: GET 2023-10-12T01:06:26.657 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:06:26.657 DEBUG:boto:Data: 2023-10-12T01:06:26.658 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.659 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.659 DEBUG:boto:Params: {} 2023-10-12T01:06:26.660 DEBUG:boto:Token: None 2023-10-12T01:06:26.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87 2023-10-12T01:06:26.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:08dtRQHuCHyRxb1k29xAaieveXY= 2023-10-12T01:06:26.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:08dtRQHuCHyRxb1k29xAaieveXY=', 'Content-Length': '0'} 2023-10-12T01:06:26.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9c24c315f62f3a-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.706 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:06:26.707 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:06:26.707 DEBUG:boto:Method: GET 2023-10-12T01:06:26.708 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:06:26.708 DEBUG:boto:Data: 2023-10-12T01:06:26.708 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.709 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.710 DEBUG:boto:Params: {} 2023-10-12T01:06:26.710 DEBUG:boto:Token: None 2023-10-12T01:06:26.711 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj87 2023-10-12T01:06:26.711 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:08dtRQHuCHyRxb1k29xAaieveXY= 2023-10-12T01:06:26.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:08dtRQHuCHyRxb1k29xAaieveXY=', 'Content-Length': '0'} 2023-10-12T01:06:26.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3989b7281ab4c01-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.714 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.715 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.715 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.715 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.716 DEBUG:boto:Method: GET 2023-10-12T01:06:26.716 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.717 DEBUG:boto:Data: 2023-10-12T01:06:26.717 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.718 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.718 DEBUG:boto:Params: {} 2023-10-12T01:06:26.719 DEBUG:boto:Token: None 2023-10-12T01:06:26.719 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd5daf4c4df163c-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.723 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.724 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.724 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.725 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.725 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.726 DEBUG:boto:Method: GET 2023-10-12T01:06:26.726 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:06:26.726 DEBUG:boto:Data: 2023-10-12T01:06:26.727 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.728 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.728 DEBUG:boto:Params: {} 2023-10-12T01:06:26.729 DEBUG:boto:Token: None 2023-10-12T01:06:26.729 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5ecd3baa6e1cae-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.734 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:06:26.735 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.735 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.736 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.736 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.737 DEBUG:boto:Method: GET 2023-10-12T01:06:26.737 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.737 DEBUG:boto:Data: 2023-10-12T01:06:26.738 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.739 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.739 DEBUG:boto:Params: {} 2023-10-12T01:06:26.740 DEBUG:boto:Token: None 2023-10-12T01:06:26.740 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I= 2023-10-12T01:06:26.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I=', 'Content-Length': '0'} 2023-10-12T01:06:26.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770418657facc029-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.744 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.744 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.745 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.745 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.745 DEBUG:boto:Method: GET 2023-10-12T01:06:26.746 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.746 DEBUG:boto:Data: 2023-10-12T01:06:26.747 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.748 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.748 DEBUG:boto:Params: {} 2023-10-12T01:06:26.749 DEBUG:boto:Token: None 2023-10-12T01:06:26.749 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I= 2023-10-12T01:06:26.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I=', 'Content-Length': '0'} 2023-10-12T01:06:26.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db6b070931b40fb-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.753 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.753 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.754 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.754 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.754 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.755 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.755 DEBUG:boto:Data: 2023-10-12T01:06:26.756 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.756 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.757 DEBUG:boto:Params: {} 2023-10-12T01:06:26.757 DEBUG:boto:Token: None 2023-10-12T01:06:26.758 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.758 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CEsjhISJtgfthOBuL0Hvjy8CQks= 2023-10-12T01:06:26.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CEsjhISJtgfthOBuL0Hvjy8CQks=', 'Content-Length': '0'} 2023-10-12T01:06:26.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac50e8acb676454d-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.761 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.762 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.762 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.763 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.763 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.763 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.764 DEBUG:boto:Data: 2023-10-12T01:06:26.764 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.765 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.765 DEBUG:boto:Params: {} 2023-10-12T01:06:26.766 DEBUG:boto:Token: None 2023-10-12T01:06:26.766 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CEsjhISJtgfthOBuL0Hvjy8CQks= 2023-10-12T01:06:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CEsjhISJtgfthOBuL0Hvjy8CQks=', 'Content-Length': '0'} 2023-10-12T01:06:26.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5988ce5ae358602-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:06:26.770 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.770 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.771 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.771 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.772 DEBUG:boto:Method: GET 2023-10-12T01:06:26.772 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.773 DEBUG:boto:Data: 2023-10-12T01:06:26.773 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.774 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.775 DEBUG:boto:Params: {} 2023-10-12T01:06:26.775 DEBUG:boto:Token: None 2023-10-12T01:06:26.776 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I= 2023-10-12T01:06:26.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I=', 'Content-Length': '0'} 2023-10-12T01:06:26.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ca7a52cb23a02b-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.826 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.827 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.827 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.828 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.828 DEBUG:boto:Method: GET 2023-10-12T01:06:26.829 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.829 DEBUG:boto:Data: 2023-10-12T01:06:26.830 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.831 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.831 DEBUG:boto:Params: {} 2023-10-12T01:06:26.831 DEBUG:boto:Token: None 2023-10-12T01:06:26.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:06:26.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I= 2023-10-12T01:06:26.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j0qjoeU/SNxc5OR6AsttUuL5T7I=', 'Content-Length': '0'} 2023-10-12T01:06:26.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e413e8b01dd70ef5-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.836 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.836 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.837 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.837 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.838 DEBUG:boto:Method: GET 2023-10-12T01:06:26.838 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.839 DEBUG:boto:Data: 2023-10-12T01:06:26.839 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.840 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.840 DEBUG:boto:Params: {} 2023-10-12T01:06:26.841 DEBUG:boto:Token: None 2023-10-12T01:06:26.841 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008024a10ec90348ac-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.845 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.846 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.846 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.847 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.847 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.848 DEBUG:boto:Method: GET 2023-10-12T01:06:26.848 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.849 DEBUG:boto:Data: 2023-10-12T01:06:26.849 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.850 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.850 DEBUG:boto:Params: {} 2023-10-12T01:06:26.851 DEBUG:boto:Token: None 2023-10-12T01:06:26.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f4a2d9920c72d07-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.898 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.899 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.899 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.900 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.900 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:06:26.901 DEBUG:boto:Data: 2023-10-12T01:06:26.901 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.902 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.902 DEBUG:boto:Params: {} 2023-10-12T01:06:26.903 DEBUG:boto:Token: None 2023-10-12T01:06:26.903 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92 2023-10-12T01:06:26.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vl9RXlmapLCVdLkRq0F9unab3yo= 2023-10-12T01:06:26.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vl9RXlmapLCVdLkRq0F9unab3yo=', 'Content-Length': '0'} 2023-10-12T01:06:26.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77ff4eb25546a0d-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.906 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.907 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.907 DEBUG:boto:Method: HEAD 2023-10-12T01:06:26.908 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:06:26.908 DEBUG:boto:Data: 2023-10-12T01:06:26.909 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.909 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.910 DEBUG:boto:Params: {} 2023-10-12T01:06:26.910 DEBUG:boto:Token: None 2023-10-12T01:06:26.911 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92 2023-10-12T01:06:26.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vl9RXlmapLCVdLkRq0F9unab3yo= 2023-10-12T01:06:26.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vl9RXlmapLCVdLkRq0F9unab3yo=', 'Content-Length': '0'} 2023-10-12T01:06:26.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a41f296563335bba-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:06:26.914 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.915 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.915 DEBUG:boto:Method: GET 2023-10-12T01:06:26.916 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:06:26.916 DEBUG:boto:Data: 2023-10-12T01:06:26.916 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.917 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.918 DEBUG:boto:Params: {} 2023-10-12T01:06:26.918 DEBUG:boto:Token: None 2023-10-12T01:06:26.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92 2023-10-12T01:06:26.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XTvnKLmJMUeX6Cs1BGBGmr8FcXY= 2023-10-12T01:06:26.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XTvnKLmJMUeX6Cs1BGBGmr8FcXY=', 'Content-Length': '0'} 2023-10-12T01:06:26.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff4e6a46ed43ed9-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.922 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:06:26.923 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:06:26.923 DEBUG:boto:Method: GET 2023-10-12T01:06:26.923 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:06:26.924 DEBUG:boto:Data: 2023-10-12T01:06:26.924 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.925 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.925 DEBUG:boto:Params: {} 2023-10-12T01:06:26.926 DEBUG:boto:Token: None 2023-10-12T01:06:26.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj92 2023-10-12T01:06:26.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XTvnKLmJMUeX6Cs1BGBGmr8FcXY= 2023-10-12T01:06:26.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XTvnKLmJMUeX6Cs1BGBGmr8FcXY=', 'Content-Length': '0'} 2023-10-12T01:06:26.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000404554970e55012b-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.931 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.931 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.932 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.932 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.933 DEBUG:boto:Method: GET 2023-10-12T01:06:26.933 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.933 DEBUG:boto:Data: 2023-10-12T01:06:26.934 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.935 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.935 DEBUG:boto:Params: {} 2023-10-12T01:06:26.936 DEBUG:boto:Token: None 2023-10-12T01:06:26.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001982c3004995fb76-0065274692-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.940 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.941 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:26.941 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:26.942 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.942 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.942 DEBUG:boto:Method: GET 2023-10-12T01:06:26.943 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:06:26.943 DEBUG:boto:Data: 2023-10-12T01:06:26.944 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.945 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.945 DEBUG:boto:Params: {} 2023-10-12T01:06:26.945 DEBUG:boto:Token: None 2023-10-12T01:06:26.946 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/ 2023-10-12T01:06:26.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70= 2023-10-12T01:06:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XP3iHS3iPSm7Q+ROwBOnFrFft70=', 'Content-Length': '0'} 2023-10-12T01:06:26.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0db0d2e771401a9-0065274692-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.951 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:26.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:06:26.952 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:26.952 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:26.953 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.953 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.953 DEBUG:boto:Method: GET 2023-10-12T01:06:26.954 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.954 DEBUG:boto:Data: 2023-10-12T01:06:26.955 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:26.955 DEBUG:boto:Port: 8001 2023-10-12T01:06:26.956 DEBUG:boto:Params: {} 2023-10-12T01:06:26.956 DEBUG:boto:Token: None 2023-10-12T01:06:26.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z71Gd4Oz+8+xpAO14RBD+adljX8= 2023-10-12T01:06:26.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z71Gd4Oz+8+xpAO14RBD+adljX8=', 'Content-Length': '0'} 2023-10-12T01:06:26.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8acc631ab8cfe6-0065274692-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:26.960 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:26.961 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:26.961 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.961 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.962 DEBUG:boto:Method: GET 2023-10-12T01:06:26.962 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.963 DEBUG:boto:Data: 2023-10-12T01:06:26.963 DEBUG:boto:Headers: {} 2023-10-12T01:06:26.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:26.964 DEBUG:boto:Port: 8000 2023-10-12T01:06:26.964 DEBUG:boto:Params: {} 2023-10-12T01:06:26.965 DEBUG:boto:Token: None 2023-10-12T01:06:26.965 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:26 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:26.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z71Gd4Oz+8+xpAO14RBD+adljX8= 2023-10-12T01:06:26.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z71Gd4Oz+8+xpAO14RBD+adljX8=', 'Content-Length': '0'} 2023-10-12T01:06:26.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef869dfb1fc4e361-0065274692-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.010 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.011 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.011 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.012 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.012 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.012 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.013 DEBUG:boto:Data: 2023-10-12T01:06:27.013 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.014 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.015 DEBUG:boto:Params: {} 2023-10-12T01:06:27.015 DEBUG:boto:Token: None 2023-10-12T01:06:27.015 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iXhIDGNMWCxpPrbdo/5/orCse4E= 2023-10-12T01:06:27.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iXhIDGNMWCxpPrbdo/5/orCse4E=', 'Content-Length': '0'} 2023-10-12T01:06:27.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086194a4db45777fe-0065274693-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.019 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.019 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.020 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.020 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.020 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.021 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.021 DEBUG:boto:Data: 2023-10-12T01:06:27.022 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.022 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.023 DEBUG:boto:Params: {} 2023-10-12T01:06:27.023 DEBUG:boto:Token: None 2023-10-12T01:06:27.024 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iXhIDGNMWCxpPrbdo/5/orCse4E= 2023-10-12T01:06:27.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iXhIDGNMWCxpPrbdo/5/orCse4E=', 'Content-Length': '0'} 2023-10-12T01:06:27.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c94da5a6ee836b-0065274693-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:06:27.028 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.028 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.028 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.029 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.029 DEBUG:boto:Method: GET 2023-10-12T01:06:27.030 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.030 DEBUG:boto:Data: 2023-10-12T01:06:27.030 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.031 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.032 DEBUG:boto:Params: {} 2023-10-12T01:06:27.032 DEBUG:boto:Token: None 2023-10-12T01:06:27.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9b8rQddtS62bPFoISi8gL/nPE5A= 2023-10-12T01:06:27.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9b8rQddtS62bPFoISi8gL/nPE5A=', 'Content-Length': '0'} 2023-10-12T01:06:27.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b3bc04e72b4475-0065274693-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.036 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.037 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.037 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.037 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.038 DEBUG:boto:Method: GET 2023-10-12T01:06:27.038 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.039 DEBUG:boto:Data: 2023-10-12T01:06:27.039 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.040 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.040 DEBUG:boto:Params: {} 2023-10-12T01:06:27.041 DEBUG:boto:Token: None 2023-10-12T01:06:27.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:06:27.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9b8rQddtS62bPFoISi8gL/nPE5A= 2023-10-12T01:06:27.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9b8rQddtS62bPFoISi8gL/nPE5A=', 'Content-Length': '0'} 2023-10-12T01:06:27.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa7892a219f2a60-0065274693-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.045 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.046 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.046 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.046 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.047 DEBUG:boto:Method: GET 2023-10-12T01:06:27.047 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.048 DEBUG:boto:Data: 2023-10-12T01:06:27.048 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.049 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.050 DEBUG:boto:Params: {} 2023-10-12T01:06:27.050 DEBUG:boto:Token: None 2023-10-12T01:06:27.051 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4= 2023-10-12T01:06:27.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4=', 'Content-Length': '0'} 2023-10-12T01:06:27.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab8d7cf0fdf14f5-0065274693-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.054 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:27.055 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.055 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.056 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.056 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.057 DEBUG:boto:Method: GET 2023-10-12T01:06:27.057 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.058 DEBUG:boto:Data: 2023-10-12T01:06:27.058 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.059 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.060 DEBUG:boto:Params: {} 2023-10-12T01:06:27.060 DEBUG:boto:Token: None 2023-10-12T01:06:27.061 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4= 2023-10-12T01:06:27.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4=', 'Content-Length': '0'} 2023-10-12T01:06:27.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f40cd57d1cf87999-0065274693-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.065 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:27.066 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.067 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.067 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.067 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:06:27.068 DEBUG:boto:Data: 2023-10-12T01:06:27.068 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.069 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.070 DEBUG:boto:Params: {} 2023-10-12T01:06:27.070 DEBUG:boto:Token: None 2023-10-12T01:06:27.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95 2023-10-12T01:06:27.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4nnobWzEd4CvaGcQXX0pQVpYZtA= 2023-10-12T01:06:27.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4nnobWzEd4CvaGcQXX0pQVpYZtA=', 'Content-Length': '0'} 2023-10-12T01:06:27.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a1195b088b3909-0065274693-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.074 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.074 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.075 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.075 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:06:27.076 DEBUG:boto:Data: 2023-10-12T01:06:27.076 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.077 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.077 DEBUG:boto:Params: {} 2023-10-12T01:06:27.078 DEBUG:boto:Token: None 2023-10-12T01:06:27.078 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95 2023-10-12T01:06:27.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4nnobWzEd4CvaGcQXX0pQVpYZtA= 2023-10-12T01:06:27.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4nnobWzEd4CvaGcQXX0pQVpYZtA=', 'Content-Length': '0'} 2023-10-12T01:06:27.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7aa97f18313dd9a-0065274693-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:06:27.082 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.083 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.083 DEBUG:boto:Method: GET 2023-10-12T01:06:27.084 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:06:27.084 DEBUG:boto:Data: 2023-10-12T01:06:27.085 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.086 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.086 DEBUG:boto:Params: {} 2023-10-12T01:06:27.087 DEBUG:boto:Token: None 2023-10-12T01:06:27.087 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95 2023-10-12T01:06:27.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EICeN2BPOd0j5iFwL6EahgLw/fA= 2023-10-12T01:06:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EICeN2BPOd0j5iFwL6EahgLw/fA=', 'Content-Length': '0'} 2023-10-12T01:06:27.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daac3c51d8aad67e-0065274693-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.091 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:06:27.091 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:06:27.091 DEBUG:boto:Method: GET 2023-10-12T01:06:27.092 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:06:27.092 DEBUG:boto:Data: 2023-10-12T01:06:27.093 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.094 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.094 DEBUG:boto:Params: {} 2023-10-12T01:06:27.095 DEBUG:boto:Token: None 2023-10-12T01:06:27.095 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj95 2023-10-12T01:06:27.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EICeN2BPOd0j5iFwL6EahgLw/fA= 2023-10-12T01:06:27.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EICeN2BPOd0j5iFwL6EahgLw/fA=', 'Content-Length': '0'} 2023-10-12T01:06:27.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3b51ed9013ec5c-0065274693-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.098 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.099 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.099 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.100 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.100 DEBUG:boto:Method: GET 2023-10-12T01:06:27.101 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.101 DEBUG:boto:Data: 2023-10-12T01:06:27.102 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:06:27.103 DEBUG:boto:Port: 8001 2023-10-12T01:06:27.103 DEBUG:boto:Params: {} 2023-10-12T01:06:27.103 DEBUG:boto:Token: None 2023-10-12T01:06:27.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4= 2023-10-12T01:06:27.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4=', 'Content-Length': '0'} 2023-10-12T01:06:27.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe259bdcb5cfcef-0065274693-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.107 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:27.108 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.108 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.109 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.109 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.110 DEBUG:boto:Method: GET 2023-10-12T01:06:27.110 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:06:27.111 DEBUG:boto:Data: 2023-10-12T01:06:27.111 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.112 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.113 DEBUG:boto:Params: {} 2023-10-12T01:06:27.113 DEBUG:boto:Token: None 2023-10-12T01:06:27.114 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4= 2023-10-12T01:06:27.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6tDaPSrJgjon1VHxM0F7GyAz9C4=', 'Content-Length': '0'} 2023-10-12T01:06:27.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b0dc489bd96f63-0065274693-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.118 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:06:27.119 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:06:27.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:06:27.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:06:27.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:06:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:06:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:06:27.225 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:06:27.225 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T01:06:27.235 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-10-12T01:06:27.292 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.293 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.293 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.294 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:27.294 DEBUG:boto:Data: 2023-10-12T01:06:27.295 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.295 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.296 DEBUG:boto:Params: {} 2023-10-12T01:06:27.296 DEBUG:boto:Token: None 2023-10-12T01:06:27.297 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk= 2023-10-12T01:06:27.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk=', 'Content-Length': '0'} 2023-10-12T01:06:27.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c8a35ccf97de05e-0065274693-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.302 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:06:27.302 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:06:27.303 DEBUG:boto:Method: PUT 2023-10-12T01:06:27.303 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:06:27.304 DEBUG:boto:Data: 2023-10-12T01:06:27.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:06:27.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.305 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.305 DEBUG:boto:Params: {} 2023-10-12T01:06:27.306 DEBUG:boto:Token: None 2023-10-12T01:06:27.306 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj31 2023-10-12T01:06:27.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TCvSXPh+4M9LBqwPA0uAyEYdEAI= 2023-10-12T01:06:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TCvSXPh+4M9LBqwPA0uAyEYdEAI='} 2023-10-12T01:06:27.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015ea92db5ef51e14-0065274693-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.313 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.313 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.314 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.314 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:27.315 DEBUG:boto:Data: 2023-10-12T01:06:27.315 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.316 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.316 DEBUG:boto:Params: {} 2023-10-12T01:06:27.317 DEBUG:boto:Token: None 2023-10-12T01:06:27.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk= 2023-10-12T01:06:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk=', 'Content-Length': '0'} 2023-10-12T01:06:27.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a3026ab15b707b7-0065274693-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.322 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:06:27.322 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:06:27.323 DEBUG:boto:Method: PUT 2023-10-12T01:06:27.323 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:06:27.324 DEBUG:boto:Data: 2023-10-12T01:06:27.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:06:27.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.325 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.325 DEBUG:boto:Params: {} 2023-10-12T01:06:27.326 DEBUG:boto:Token: None 2023-10-12T01:06:27.326 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj62 2023-10-12T01:06:27.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LBsQ4cIO+1HY5xT49YOj0hpioGk= 2023-10-12T01:06:27.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LBsQ4cIO+1HY5xT49YOj0hpioGk='} 2023-10-12T01:06:27.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006777570e7a0d4810-0065274693-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.371 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.371 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.372 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.372 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:27.373 DEBUG:boto:Data: 2023-10-12T01:06:27.373 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.374 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.374 DEBUG:boto:Params: {} 2023-10-12T01:06:27.375 DEBUG:boto:Token: None 2023-10-12T01:06:27.375 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk= 2023-10-12T01:06:27.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk=', 'Content-Length': '0'} 2023-10-12T01:06:27.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000872ffaa4a122f742-0065274693-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.380 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:06:27.381 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:06:27.381 DEBUG:boto:Method: PUT 2023-10-12T01:06:27.382 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:06:27.382 DEBUG:boto:Data: 2023-10-12T01:06:27.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:06:27.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.383 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.384 DEBUG:boto:Params: {} 2023-10-12T01:06:27.384 DEBUG:boto:Token: None 2023-10-12T01:06:27.384 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj93 2023-10-12T01:06:27.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nF8d86szpnaslXsH2SJmngXrHJo= 2023-10-12T01:06:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nF8d86szpnaslXsH2SJmngXrHJo='} 2023-10-12T01:06:27.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7533583727381ed-0065274693-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.391 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.392 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.392 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.393 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:27.393 DEBUG:boto:Data: 2023-10-12T01:06:27.394 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.394 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.395 DEBUG:boto:Params: {} 2023-10-12T01:06:27.395 DEBUG:boto:Token: None 2023-10-12T01:06:27.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk= 2023-10-12T01:06:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk=', 'Content-Length': '0'} 2023-10-12T01:06:27.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc92c9329f6716f4-0065274693-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.400 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:06:27.401 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:06:27.401 DEBUG:boto:Method: PUT 2023-10-12T01:06:27.402 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:06:27.402 DEBUG:boto:Data: 2023-10-12T01:06:27.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:06:27.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.403 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.404 DEBUG:boto:Params: {} 2023-10-12T01:06:27.404 DEBUG:boto:Token: None 2023-10-12T01:06:27.405 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj124 2023-10-12T01:06:27.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cK0aec1YRzocrA8ZroJ7el8XMTg= 2023-10-12T01:06:27.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cK0aec1YRzocrA8ZroJ7el8XMTg='} 2023-10-12T01:06:27.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b40c03773141ecea-0065274693-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.451 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:06:27.452 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:06:27.452 DEBUG:boto:Method: HEAD 2023-10-12T01:06:27.452 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:06:27.453 DEBUG:boto:Data: 2023-10-12T01:06:27.453 DEBUG:boto:Headers: {} 2023-10-12T01:06:27.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.454 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.454 DEBUG:boto:Params: {} 2023-10-12T01:06:27.455 DEBUG:boto:Token: None 2023-10-12T01:06:27.455 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/ 2023-10-12T01:06:27.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk= 2023-10-12T01:06:27.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y2xQ4bGKnwXyUYAhvD6+gXlSfgk=', 'Content-Length': '0'} 2023-10-12T01:06:27.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afd041dc63e9a0c4-0065274693-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.460 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:06:27.460 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:06:27.461 DEBUG:boto:Method: PUT 2023-10-12T01:06:27.461 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:06:27.462 DEBUG:boto:Data: 2023-10-12T01:06:27.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:06:27.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:06:27.463 DEBUG:boto:Port: 8000 2023-10-12T01:06:27.463 DEBUG:boto:Params: {} 2023-10-12T01:06:27.464 DEBUG:boto:Token: None 2023-10-12T01:06:27.464 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:06:27 GMT /wafmoa-59/obj155 2023-10-12T01:06:27.465 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OORUnaiYPBr3l9MOtijwdUuAyak= 2023-10-12T01:06:27.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:06:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OORUnaiYPBr3l9MOtijwdUuAyak='} 2023-10-12T01:06:27.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a9647ebd63f4d9c-0065274693-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:06:27.471 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:06:27.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:06:27.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:06:27.472 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:06:27.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:06:27.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:06:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:27.566+0000 7fedb5f7f780 1 waiting to reach latest gen 5, on 4.. 2023-10-12T01:06:32.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:32.568+0000 7fedb5f7f780 1 waiting to reach latest gen 5, on 4.. 2023-10-12T01:06:37.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:37.568+0000 7fedb5f7f780 1 waiting to reach latest gen 5, on 4.. 2023-10-12T01:06:42.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:42.571+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:42.571 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:06:42.571 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:47.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:47.572+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:47.573 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:47.573 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:52.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:52.574+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:52.575 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:52.576 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:57.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:06:57.576+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:06:57.578 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:06:57.578 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:02.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:02.579+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:02.580 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:02.580 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:07.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:07.582+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:07.582 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:07.582 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:12.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:12.584+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:12.584 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:12.584 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:17.586+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:17.586 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:17.587 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:22.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:22.587+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:22.589 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:22.589 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:27.589+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:27.591 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:27.591 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:32.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:32.592+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:32.593 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:32.593 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:37.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:37.594+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:37.595 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:37.596 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:42.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:42.596+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:42.597 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:42.598 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:47.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:47.599+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:47.599 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:47.600 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:52.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:52.602+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:52.602 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:52.602 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:57.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:07:57.604+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:07:57.604 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , , , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:07:57.604 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:02.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:02.605+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:02.606 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:02.607 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:07.607+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:07.608 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:07.609 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:12.609+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:12.610 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:12.611 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:17.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:17.611+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:22.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:22.614+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:22.615 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:22.616 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:27.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:27.617+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:27.617 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:27.617 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:32.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:32.619+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:32.619 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:32.619 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:37.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:37.620+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:37.622 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:37.622 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:42.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:42.623+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:42.624 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:42.624 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:47.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:47.625+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:47.626 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:47.627 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:52.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:52.627+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:52.629 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:52.629 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:57.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:08:57.629+0000 7fedb5f7f780 1 waiting for incremental sync to catch up: 2023-10-12T01:08:57.631 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:08:57.631 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:09:02.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:02.633+0000 7fedb5f7f780 1 bucket sync caught up with source: 2023-10-12T01:09:02.633 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:09:02.633 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.107.6, , 00000000001.87.6, , 00000000001.57.6, , , , , , , , , , 00000000001.89.6, , , , , , , , , , , , , , 00000000001.71.6] 2023-10-12T01:09:02.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:02.633+0000 7fedb5f7f780 0 bucket checkpoint complete 2023-10-12T01:09:02.650 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:09:02.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:09:02.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:09:02.652 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:09:02.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:09:02.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:09:02.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:02.753+0000 7f5bce15f780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:02.753 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000001.181.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:02.754 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:07.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:07.754+0000 7f5bce15f780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:07.754 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000001.181.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:07.754 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:12.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:12.755+0000 7f5bce15f780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:12.755 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000001.181.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:12.756 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:17.755+0000 7f5bce15f780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:17.756 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000001.181.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:17.757 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:21.001 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:09:21.001+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:09:22.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:22.756+0000 7f5bce15f780 1 bucket sync caught up with source: 2023-10-12T01:09:22.758 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:22.758 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000004.233.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000003.311.6, 00000000006.252.6, 00000000004.210.6] 2023-10-12T01:09:22.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:22.756+0000 7f5bce15f780 0 bucket checkpoint complete 2023-10-12T01:09:22.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:09:22.775 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.775 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.776 DEBUG:boto:Method: HEAD 2023-10-12T01:09:22.776 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:22.777 DEBUG:boto:Data: 2023-10-12T01:09:22.778 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.779 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.779 DEBUG:boto:Params: {} 2023-10-12T01:09:22.780 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:09:22.780 DEBUG:boto:Token: None 2023-10-12T01:09:22.781 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/ 2023-10-12T01:09:22.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KK86fNdNj+4fhj7TQqaaxCNzMWs= 2023-10-12T01:09:22.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KK86fNdNj+4fhj7TQqaaxCNzMWs=', 'Content-Length': '0'} 2023-10-12T01:09:22.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000badd8bb9e28d5ea1-0065274742-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.787 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.787 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.788 DEBUG:boto:Method: HEAD 2023-10-12T01:09:22.788 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:22.789 DEBUG:boto:Data: 2023-10-12T01:09:22.789 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:22.790 DEBUG:boto:Port: 8000 2023-10-12T01:09:22.790 DEBUG:boto:Params: {} 2023-10-12T01:09:22.791 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:09:22.792 DEBUG:boto:Token: None 2023-10-12T01:09:22.792 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/ 2023-10-12T01:09:22.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KK86fNdNj+4fhj7TQqaaxCNzMWs= 2023-10-12T01:09:22.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KK86fNdNj+4fhj7TQqaaxCNzMWs=', 'Content-Length': '0'} 2023-10-12T01:09:22.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb3f558e0919324d-0065274742-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:09:22.801 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.801 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.801 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.802 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.802 DEBUG:boto:Method: GET 2023-10-12T01:09:22.803 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.803 DEBUG:boto:Data: 2023-10-12T01:09:22.804 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.804 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.805 DEBUG:boto:Params: {} 2023-10-12T01:09:22.805 DEBUG:boto:Token: None 2023-10-12T01:09:22.806 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/?versions 2023-10-12T01:09:22.807 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk= 2023-10-12T01:09:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk=', 'Content-Length': '0'} 2023-10-12T01:09:22.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ede1dc055a3b28-0065274742-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.810 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:22.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:09:22.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:09:22.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:09:22.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:09:22.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:09:22.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:09:22.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:09:22.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:09:22.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:09:22.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:09:22.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:09:22.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:09:22.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:09:22.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:09:22.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:09:22.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:09:22.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:09:22.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:09:22.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:09:22.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:09:22.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:09:22.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:09:22.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:09:22.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:09:22.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:09:22.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:09:22.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:09:22.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:09:22.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:09:22.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:09:22.824 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:09:22.825 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.825 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.825 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.826 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.826 DEBUG:boto:Method: GET 2023-10-12T01:09:22.827 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.827 DEBUG:boto:Data: 2023-10-12T01:09:22.828 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:22.828 DEBUG:boto:Port: 8000 2023-10-12T01:09:22.829 DEBUG:boto:Params: {} 2023-10-12T01:09:22.829 DEBUG:boto:Token: None 2023-10-12T01:09:22.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/?versions 2023-10-12T01:09:22.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk= 2023-10-12T01:09:22.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk=', 'Content-Length': '0'} 2023-10-12T01:09:22.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bbe1e22ae74287-0065274742-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.836 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:22.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:09:22.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:09:22.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:09:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:09:22.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:09:22.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:09:22.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:09:22.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:09:22.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:09:22.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:09:22.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:09:22.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:09:22.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:09:22.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:09:22.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:09:22.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:09:22.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:09:22.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:09:22.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:09:22.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:09:22.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:09:22.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:09:22.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:09:22.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:09:22.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:09:22.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:09:22.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:09:22.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:09:22.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:09:22.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:09:22.850 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.850 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.851 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.851 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.852 DEBUG:boto:Method: GET 2023-10-12T01:09:22.852 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.853 DEBUG:boto:Data: 2023-10-12T01:09:22.853 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.854 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.854 DEBUG:boto:Params: {} 2023-10-12T01:09:22.855 DEBUG:boto:Token: None 2023-10-12T01:09:22.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/?versions 2023-10-12T01:09:22.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk= 2023-10-12T01:09:22.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk=', 'Content-Length': '0'} 2023-10-12T01:09:22.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d81beaeec3abdd-0065274742-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.860 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:22.862 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:22.862 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:22.863 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.863 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.864 DEBUG:boto:Method: GET 2023-10-12T01:09:22.864 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:09:22.865 DEBUG:boto:Data: 2023-10-12T01:09:22.865 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:22.866 DEBUG:boto:Port: 8000 2023-10-12T01:09:22.866 DEBUG:boto:Params: {} 2023-10-12T01:09:22.867 DEBUG:boto:Token: None 2023-10-12T01:09:22.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/?versions 2023-10-12T01:09:22.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk= 2023-10-12T01:09:22.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fydkUSoMmJsbGvkMMfyObwzTiGk=', 'Content-Length': '0'} 2023-10-12T01:09:22.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d713ae6b3df4daad-0065274742-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.873 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:22.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:09:22.875 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:22.876 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:22.876 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.876 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.877 DEBUG:boto:Method: GET 2023-10-12T01:09:22.877 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.878 DEBUG:boto:Data: 2023-10-12T01:09:22.878 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.879 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.879 DEBUG:boto:Params: {} 2023-10-12T01:09:22.880 DEBUG:boto:Token: None 2023-10-12T01:09:22.880 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.881 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ= 2023-10-12T01:09:22.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ=', 'Content-Length': '0'} 2023-10-12T01:09:22.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1093e9767b767ab-0065274742-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.884 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:22.885 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:22.885 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.886 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.886 DEBUG:boto:Method: GET 2023-10-12T01:09:22.887 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.887 DEBUG:boto:Data: 2023-10-12T01:09:22.887 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:22.888 DEBUG:boto:Port: 8000 2023-10-12T01:09:22.889 DEBUG:boto:Params: {} 2023-10-12T01:09:22.889 DEBUG:boto:Token: None 2023-10-12T01:09:22.890 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ= 2023-10-12T01:09:22.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ=', 'Content-Length': '0'} 2023-10-12T01:09:22.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885a32ef92259f22-0065274742-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.938 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:22.939 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:22.939 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.940 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.940 DEBUG:boto:Method: HEAD 2023-10-12T01:09:22.941 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.941 DEBUG:boto:Data: 2023-10-12T01:09:22.941 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.942 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.943 DEBUG:boto:Params: {} 2023-10-12T01:09:22.943 DEBUG:boto:Token: None 2023-10-12T01:09:22.944 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g7F9ObTb0xH5SVFhLL+UyoGIsVo= 2023-10-12T01:09:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g7F9ObTb0xH5SVFhLL+UyoGIsVo=', 'Content-Length': '0'} 2023-10-12T01:09:22.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c508c09672c2dc-0065274742-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.947 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:22.948 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:22.948 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.949 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.949 DEBUG:boto:Method: HEAD 2023-10-12T01:09:22.950 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.950 DEBUG:boto:Data: 2023-10-12T01:09:22.951 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:22.951 DEBUG:boto:Port: 8000 2023-10-12T01:09:22.952 DEBUG:boto:Params: {} 2023-10-12T01:09:22.952 DEBUG:boto:Token: None 2023-10-12T01:09:22.953 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g7F9ObTb0xH5SVFhLL+UyoGIsVo= 2023-10-12T01:09:22.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g7F9ObTb0xH5SVFhLL+UyoGIsVo=', 'Content-Length': '0'} 2023-10-12T01:09:22.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c2c0cc26f7ded4-0065274742-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:22.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:09:22.957 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:22.957 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:22.958 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.958 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.959 DEBUG:boto:Method: GET 2023-10-12T01:09:22.959 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.959 DEBUG:boto:Data: 2023-10-12T01:09:22.960 DEBUG:boto:Headers: {} 2023-10-12T01:09:22.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:22.961 DEBUG:boto:Port: 8001 2023-10-12T01:09:22.961 DEBUG:boto:Params: {} 2023-10-12T01:09:22.962 DEBUG:boto:Token: None 2023-10-12T01:09:22.962 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:22 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:22.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ= 2023-10-12T01:09:22.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:22 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:egReraLewKrneARiZXMHY8hFPkQ=', 'Content-Length': '0'} 2023-10-12T01:09:22.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc443805d480ff1-0065274742-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.006 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:23.007 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:23.008 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:23.008 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:09:23.009 DEBUG:boto:Method: GET 2023-10-12T01:09:23.009 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:23.009 DEBUG:boto:Data: 2023-10-12T01:09:23.010 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.011 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.011 DEBUG:boto:Params: {} 2023-10-12T01:09:23.012 DEBUG:boto:Token: None 2023-10-12T01:09:23.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:09:23.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I8Ix3gZos0pTUW0IcL86OY/7UeA= 2023-10-12T01:09:23.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I8Ix3gZos0pTUW0IcL86OY/7UeA=', 'Content-Length': '0'} 2023-10-12T01:09:23.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d76557a11f4d1f8-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.017 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.017 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.018 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.018 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.019 DEBUG:boto:Method: GET 2023-10-12T01:09:23.019 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.019 DEBUG:boto:Data: 2023-10-12T01:09:23.020 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.021 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.021 DEBUG:boto:Params: {} 2023-10-12T01:09:23.022 DEBUG:boto:Token: None 2023-10-12T01:09:23.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92b511f910c52e9-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.027 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.027 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.028 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.028 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.029 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.029 DEBUG:boto:Method: GET 2023-10-12T01:09:23.030 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.030 DEBUG:boto:Data: 2023-10-12T01:09:23.030 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.031 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.032 DEBUG:boto:Params: {} 2023-10-12T01:09:23.032 DEBUG:boto:Token: None 2023-10-12T01:09:23.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc318429bd00cf7-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.038 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.041 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:23.042 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:23.042 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.043 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:09:23.043 DEBUG:boto:Data: 2023-10-12T01:09:23.044 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.044 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.045 DEBUG:boto:Params: {} 2023-10-12T01:09:23.045 DEBUG:boto:Token: None 2023-10-12T01:09:23.046 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj11 2023-10-12T01:09:23.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ze7CXTFqkgTc/FUxz3Fq7XOKACU= 2023-10-12T01:09:23.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ze7CXTFqkgTc/FUxz3Fq7XOKACU=', 'Content-Length': '0'} 2023-10-12T01:09:23.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe50dc30e4384fa-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.049 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:23.050 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:23.050 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.051 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:09:23.051 DEBUG:boto:Data: 2023-10-12T01:09:23.051 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.052 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.053 DEBUG:boto:Params: {} 2023-10-12T01:09:23.053 DEBUG:boto:Token: None 2023-10-12T01:09:23.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj11 2023-10-12T01:09:23.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ze7CXTFqkgTc/FUxz3Fq7XOKACU= 2023-10-12T01:09:23.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ze7CXTFqkgTc/FUxz3Fq7XOKACU=', 'Content-Length': '0'} 2023-10-12T01:09:23.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecaa9920b5afa9c9-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:09:23.058 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:23.058 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:23.059 DEBUG:boto:Method: GET 2023-10-12T01:09:23.059 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:09:23.060 DEBUG:boto:Data: 2023-10-12T01:09:23.060 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.061 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.061 DEBUG:boto:Params: {} 2023-10-12T01:09:23.062 DEBUG:boto:Token: None 2023-10-12T01:09:23.062 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj11 2023-10-12T01:09:23.063 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WRMa/WS44aElG1G3BZxqRwI4poQ= 2023-10-12T01:09:23.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WRMa/WS44aElG1G3BZxqRwI4poQ=', 'Content-Length': '0'} 2023-10-12T01:09:23.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54c1ccedac20e95-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.066 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:09:23.066 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:09:23.067 DEBUG:boto:Method: GET 2023-10-12T01:09:23.067 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:09:23.067 DEBUG:boto:Data: 2023-10-12T01:09:23.068 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.069 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.069 DEBUG:boto:Params: {} 2023-10-12T01:09:23.070 DEBUG:boto:Token: None 2023-10-12T01:09:23.070 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj11 2023-10-12T01:09:23.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WRMa/WS44aElG1G3BZxqRwI4poQ= 2023-10-12T01:09:23.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WRMa/WS44aElG1G3BZxqRwI4poQ=', 'Content-Length': '0'} 2023-10-12T01:09:23.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5dc181bb1f91a6b-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.074 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.074 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.075 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.075 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.075 DEBUG:boto:Method: GET 2023-10-12T01:09:23.076 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.076 DEBUG:boto:Data: 2023-10-12T01:09:23.077 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.078 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.078 DEBUG:boto:Params: {} 2023-10-12T01:09:23.079 DEBUG:boto:Token: None 2023-10-12T01:09:23.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a05d1497fc98546-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.084 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.084 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.085 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.085 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.086 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.086 DEBUG:boto:Method: GET 2023-10-12T01:09:23.087 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:09:23.087 DEBUG:boto:Data: 2023-10-12T01:09:23.088 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.089 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.089 DEBUG:boto:Params: {} 2023-10-12T01:09:23.089 DEBUG:boto:Token: None 2023-10-12T01:09:23.090 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8f2c82f92b3e0b-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.095 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:09:23.096 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.097 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.097 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.098 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.098 DEBUG:boto:Method: GET 2023-10-12T01:09:23.099 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.099 DEBUG:boto:Data: 2023-10-12T01:09:23.100 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.100 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.101 DEBUG:boto:Params: {} 2023-10-12T01:09:23.101 DEBUG:boto:Token: None 2023-10-12T01:09:23.102 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI= 2023-10-12T01:09:23.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI=', 'Content-Length': '0'} 2023-10-12T01:09:23.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeafce9c758e8f1c-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.150 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.151 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.151 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.152 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.152 DEBUG:boto:Method: GET 2023-10-12T01:09:23.153 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.153 DEBUG:boto:Data: 2023-10-12T01:09:23.154 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.155 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.155 DEBUG:boto:Params: {} 2023-10-12T01:09:23.155 DEBUG:boto:Token: None 2023-10-12T01:09:23.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI= 2023-10-12T01:09:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI=', 'Content-Length': '0'} 2023-10-12T01:09:23.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf0664117d27856-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.160 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.160 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.160 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.161 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.161 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.162 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.162 DEBUG:boto:Data: 2023-10-12T01:09:23.162 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.163 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.164 DEBUG:boto:Params: {} 2023-10-12T01:09:23.164 DEBUG:boto:Token: None 2023-10-12T01:09:23.165 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hk4SAKXK+g0V827tGqGITqcPB8U= 2023-10-12T01:09:23.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hk4SAKXK+g0V827tGqGITqcPB8U=', 'Content-Length': '0'} 2023-10-12T01:09:23.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956a55ab1a95a93b-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.168 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.168 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.169 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.169 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.170 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.170 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.170 DEBUG:boto:Data: 2023-10-12T01:09:23.171 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.172 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.172 DEBUG:boto:Params: {} 2023-10-12T01:09:23.173 DEBUG:boto:Token: None 2023-10-12T01:09:23.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hk4SAKXK+g0V827tGqGITqcPB8U= 2023-10-12T01:09:23.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hk4SAKXK+g0V827tGqGITqcPB8U=', 'Content-Length': '0'} 2023-10-12T01:09:23.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005882f41298c1663b-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:09:23.176 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.177 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.177 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.178 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.178 DEBUG:boto:Method: GET 2023-10-12T01:09:23.179 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.179 DEBUG:boto:Data: 2023-10-12T01:09:23.179 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.180 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.181 DEBUG:boto:Params: {} 2023-10-12T01:09:23.181 DEBUG:boto:Token: None 2023-10-12T01:09:23.182 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI= 2023-10-12T01:09:23.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI=', 'Content-Length': '0'} 2023-10-12T01:09:23.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b784c723fe4bc5d6-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.185 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.186 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.186 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.187 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.187 DEBUG:boto:Method: GET 2023-10-12T01:09:23.188 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.188 DEBUG:boto:Data: 2023-10-12T01:09:23.189 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.189 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.190 DEBUG:boto:Params: {} 2023-10-12T01:09:23.190 DEBUG:boto:Token: None 2023-10-12T01:09:23.191 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:09:23.191 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI= 2023-10-12T01:09:23.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bEPJTxq4hx7f+zDEtyzHTGKJ4SI=', 'Content-Length': '0'} 2023-10-12T01:09:23.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf07d251b8e592d7-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.194 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.195 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.195 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.196 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.196 DEBUG:boto:Method: GET 2023-10-12T01:09:23.196 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.197 DEBUG:boto:Data: 2023-10-12T01:09:23.197 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.198 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.199 DEBUG:boto:Params: {} 2023-10-12T01:09:23.199 DEBUG:boto:Token: None 2023-10-12T01:09:23.200 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc2d47cc10ebb8b-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.204 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.204 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.205 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.205 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.206 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.206 DEBUG:boto:Method: GET 2023-10-12T01:09:23.207 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.207 DEBUG:boto:Data: 2023-10-12T01:09:23.207 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.208 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.209 DEBUG:boto:Params: {} 2023-10-12T01:09:23.209 DEBUG:boto:Token: None 2023-10-12T01:09:23.210 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c259bb7918e273d1-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.215 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.215 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.216 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.216 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.216 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:09:23.217 DEBUG:boto:Data: 2023-10-12T01:09:23.217 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.218 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.219 DEBUG:boto:Params: {} 2023-10-12T01:09:23.219 DEBUG:boto:Token: None 2023-10-12T01:09:23.219 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115 2023-10-12T01:09:23.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eBlh2FPjcF4lB5aAQ1gkFL1afuc= 2023-10-12T01:09:23.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eBlh2FPjcF4lB5aAQ1gkFL1afuc=', 'Content-Length': '0'} 2023-10-12T01:09:23.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052aca8f2405e48ff-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.223 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.223 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.224 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.224 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:09:23.225 DEBUG:boto:Data: 2023-10-12T01:09:23.225 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.226 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.226 DEBUG:boto:Params: {} 2023-10-12T01:09:23.227 DEBUG:boto:Token: None 2023-10-12T01:09:23.227 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115 2023-10-12T01:09:23.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eBlh2FPjcF4lB5aAQ1gkFL1afuc= 2023-10-12T01:09:23.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eBlh2FPjcF4lB5aAQ1gkFL1afuc=', 'Content-Length': '0'} 2023-10-12T01:09:23.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a8394b1ff4abe4-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:09:23.231 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.232 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.232 DEBUG:boto:Method: GET 2023-10-12T01:09:23.232 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:09:23.233 DEBUG:boto:Data: 2023-10-12T01:09:23.233 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.234 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.235 DEBUG:boto:Params: {} 2023-10-12T01:09:23.235 DEBUG:boto:Token: None 2023-10-12T01:09:23.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115 2023-10-12T01:09:23.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkMTUUyqWbixACLQ0ksffE4xaU8= 2023-10-12T01:09:23.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkMTUUyqWbixACLQ0ksffE4xaU8=', 'Content-Length': '0'} 2023-10-12T01:09:23.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1f8ade833164c2-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.282 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:09:23.283 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:09:23.284 DEBUG:boto:Method: GET 2023-10-12T01:09:23.284 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:09:23.284 DEBUG:boto:Data: 2023-10-12T01:09:23.285 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.286 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.286 DEBUG:boto:Params: {} 2023-10-12T01:09:23.287 DEBUG:boto:Token: None 2023-10-12T01:09:23.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj115 2023-10-12T01:09:23.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DkMTUUyqWbixACLQ0ksffE4xaU8= 2023-10-12T01:09:23.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DkMTUUyqWbixACLQ0ksffE4xaU8=', 'Content-Length': '0'} 2023-10-12T01:09:23.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea65b93dcf38575-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.291 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.292 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.292 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.292 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.293 DEBUG:boto:Method: GET 2023-10-12T01:09:23.293 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.294 DEBUG:boto:Data: 2023-10-12T01:09:23.294 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.295 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.295 DEBUG:boto:Params: {} 2023-10-12T01:09:23.296 DEBUG:boto:Token: None 2023-10-12T01:09:23.296 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a52d2259d7a9e0c-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.300 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.301 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.302 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.302 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.302 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.303 DEBUG:boto:Method: GET 2023-10-12T01:09:23.303 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:09:23.304 DEBUG:boto:Data: 2023-10-12T01:09:23.304 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.305 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.306 DEBUG:boto:Params: {} 2023-10-12T01:09:23.306 DEBUG:boto:Token: None 2023-10-12T01:09:23.307 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9dec8ac3b1a19cd-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.312 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:09:23.313 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.313 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.314 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.314 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.315 DEBUG:boto:Method: GET 2023-10-12T01:09:23.315 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.316 DEBUG:boto:Data: 2023-10-12T01:09:23.316 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.317 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.317 DEBUG:boto:Params: {} 2023-10-12T01:09:23.318 DEBUG:boto:Token: None 2023-10-12T01:09:23.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4= 2023-10-12T01:09:23.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4=', 'Content-Length': '0'} 2023-10-12T01:09:23.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ea11ebf2d8af29-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.322 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.322 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.323 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.323 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.324 DEBUG:boto:Method: GET 2023-10-12T01:09:23.324 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.324 DEBUG:boto:Data: 2023-10-12T01:09:23.325 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.326 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.326 DEBUG:boto:Params: {} 2023-10-12T01:09:23.327 DEBUG:boto:Token: None 2023-10-12T01:09:23.327 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4= 2023-10-12T01:09:23.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4=', 'Content-Length': '0'} 2023-10-12T01:09:23.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2dc1f7ad7ecafa-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.331 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.331 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.332 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.332 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.332 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.333 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.333 DEBUG:boto:Data: 2023-10-12T01:09:23.334 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.334 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.335 DEBUG:boto:Params: {} 2023-10-12T01:09:23.335 DEBUG:boto:Token: None 2023-10-12T01:09:23.336 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XfY7i3TSOVz56g/yccJWKDrgNqU= 2023-10-12T01:09:23.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XfY7i3TSOVz56g/yccJWKDrgNqU=', 'Content-Length': '0'} 2023-10-12T01:09:23.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059781afd5288e825-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.339 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.339 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.340 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.340 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.341 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.341 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.341 DEBUG:boto:Data: 2023-10-12T01:09:23.342 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.343 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.343 DEBUG:boto:Params: {} 2023-10-12T01:09:23.343 DEBUG:boto:Token: None 2023-10-12T01:09:23.344 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XfY7i3TSOVz56g/yccJWKDrgNqU= 2023-10-12T01:09:23.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XfY7i3TSOVz56g/yccJWKDrgNqU=', 'Content-Length': '0'} 2023-10-12T01:09:23.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055971c93914f1b86-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:09:23.348 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.348 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.349 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.349 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.349 DEBUG:boto:Method: GET 2023-10-12T01:09:23.350 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.350 DEBUG:boto:Data: 2023-10-12T01:09:23.351 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.352 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.352 DEBUG:boto:Params: {} 2023-10-12T01:09:23.353 DEBUG:boto:Token: None 2023-10-12T01:09:23.353 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4= 2023-10-12T01:09:23.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4=', 'Content-Length': '0'} 2023-10-12T01:09:23.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0b8e0391048c87-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.357 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.357 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.357 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.358 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.358 DEBUG:boto:Method: GET 2023-10-12T01:09:23.359 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.359 DEBUG:boto:Data: 2023-10-12T01:09:23.360 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.360 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.361 DEBUG:boto:Params: {} 2023-10-12T01:09:23.361 DEBUG:boto:Token: None 2023-10-12T01:09:23.362 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:09:23.362 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4= 2023-10-12T01:09:23.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Lv8MAW6HgbsIdZq3/audG/0Huy4=', 'Content-Length': '0'} 2023-10-12T01:09:23.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a880ac3a8d780a6-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.406 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.407 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.407 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.408 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.408 DEBUG:boto:Method: GET 2023-10-12T01:09:23.409 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.409 DEBUG:boto:Data: 2023-10-12T01:09:23.410 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.410 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.411 DEBUG:boto:Params: {} 2023-10-12T01:09:23.411 DEBUG:boto:Token: None 2023-10-12T01:09:23.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014eeacacaf5ac821-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.416 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.417 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.417 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.418 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.418 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.418 DEBUG:boto:Method: GET 2023-10-12T01:09:23.419 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.419 DEBUG:boto:Data: 2023-10-12T01:09:23.420 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.421 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.421 DEBUG:boto:Params: {} 2023-10-12T01:09:23.422 DEBUG:boto:Token: None 2023-10-12T01:09:23.422 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e2cdfe00fc56d9-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.427 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.428 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.428 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.429 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.429 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:09:23.430 DEBUG:boto:Data: 2023-10-12T01:09:23.430 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.431 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.431 DEBUG:boto:Params: {} 2023-10-12T01:09:23.432 DEBUG:boto:Token: None 2023-10-12T01:09:23.432 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116 2023-10-12T01:09:23.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EVgPRWiV/9QzORMEBABZXvJWFok= 2023-10-12T01:09:23.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVgPRWiV/9QzORMEBABZXvJWFok=', 'Content-Length': '0'} 2023-10-12T01:09:23.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fc14a12d706bb0-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.436 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.436 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.437 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.437 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:09:23.437 DEBUG:boto:Data: 2023-10-12T01:09:23.438 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.439 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.439 DEBUG:boto:Params: {} 2023-10-12T01:09:23.440 DEBUG:boto:Token: None 2023-10-12T01:09:23.440 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116 2023-10-12T01:09:23.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EVgPRWiV/9QzORMEBABZXvJWFok= 2023-10-12T01:09:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVgPRWiV/9QzORMEBABZXvJWFok=', 'Content-Length': '0'} 2023-10-12T01:09:23.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c54b6ae09f93d99-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:09:23.444 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.444 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.445 DEBUG:boto:Method: GET 2023-10-12T01:09:23.445 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:09:23.446 DEBUG:boto:Data: 2023-10-12T01:09:23.446 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.447 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.448 DEBUG:boto:Params: {} 2023-10-12T01:09:23.450 DEBUG:boto:Token: None 2023-10-12T01:09:23.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116 2023-10-12T01:09:23.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Bp8X7X0uPv5+0bl0vIMbtBMoDE= 2023-10-12T01:09:23.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Bp8X7X0uPv5+0bl0vIMbtBMoDE=', 'Content-Length': '0'} 2023-10-12T01:09:23.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244016b2c6248f93-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.494 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:09:23.495 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:09:23.496 DEBUG:boto:Method: GET 2023-10-12T01:09:23.496 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:09:23.496 DEBUG:boto:Data: 2023-10-12T01:09:23.497 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.498 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.498 DEBUG:boto:Params: {} 2023-10-12T01:09:23.499 DEBUG:boto:Token: None 2023-10-12T01:09:23.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj116 2023-10-12T01:09:23.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7Bp8X7X0uPv5+0bl0vIMbtBMoDE= 2023-10-12T01:09:23.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7Bp8X7X0uPv5+0bl0vIMbtBMoDE=', 'Content-Length': '0'} 2023-10-12T01:09:23.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990f1dd8faa71444-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.503 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.504 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.504 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.505 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.505 DEBUG:boto:Method: GET 2023-10-12T01:09:23.506 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.506 DEBUG:boto:Data: 2023-10-12T01:09:23.506 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.507 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.508 DEBUG:boto:Params: {} 2023-10-12T01:09:23.508 DEBUG:boto:Token: None 2023-10-12T01:09:23.509 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092305c0c011720f6-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.514 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.515 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.516 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.516 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.517 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.518 DEBUG:boto:Method: GET 2023-10-12T01:09:23.519 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:09:23.519 DEBUG:boto:Data: 2023-10-12T01:09:23.520 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.521 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.522 DEBUG:boto:Params: {} 2023-10-12T01:09:23.522 DEBUG:boto:Token: None 2023-10-12T01:09:23.523 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abca14bc91093163-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.574 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:09:23.576 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.576 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.577 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.577 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.578 DEBUG:boto:Method: GET 2023-10-12T01:09:23.579 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.579 DEBUG:boto:Data: 2023-10-12T01:09:23.580 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.582 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.582 DEBUG:boto:Params: {} 2023-10-12T01:09:23.583 DEBUG:boto:Token: None 2023-10-12T01:09:23.584 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.585 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU= 2023-10-12T01:09:23.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU=', 'Content-Length': '0'} 2023-10-12T01:09:23.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f219f4fcc3e2129a-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.588 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.589 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.589 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.589 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.590 DEBUG:boto:Method: GET 2023-10-12T01:09:23.591 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.591 DEBUG:boto:Data: 2023-10-12T01:09:23.591 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.592 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.593 DEBUG:boto:Params: {} 2023-10-12T01:09:23.594 DEBUG:boto:Token: None 2023-10-12T01:09:23.594 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU= 2023-10-12T01:09:23.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU=', 'Content-Length': '0'} 2023-10-12T01:09:23.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e601fdbd82eb1a-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.598 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.599 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.599 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.599 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.600 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.600 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.601 DEBUG:boto:Data: 2023-10-12T01:09:23.601 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.602 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.602 DEBUG:boto:Params: {} 2023-10-12T01:09:23.603 DEBUG:boto:Token: None 2023-10-12T01:09:23.604 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhguZXZbIghErz329Szcl0grDCU= 2023-10-12T01:09:23.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhguZXZbIghErz329Szcl0grDCU=', 'Content-Length': '0'} 2023-10-12T01:09:23.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306969aaef8bc162-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.607 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.608 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.608 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.609 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.609 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.610 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.610 DEBUG:boto:Data: 2023-10-12T01:09:23.611 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.613 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.613 DEBUG:boto:Params: {} 2023-10-12T01:09:23.614 DEBUG:boto:Token: None 2023-10-12T01:09:23.615 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BhguZXZbIghErz329Szcl0grDCU= 2023-10-12T01:09:23.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BhguZXZbIghErz329Szcl0grDCU=', 'Content-Length': '0'} 2023-10-12T01:09:23.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df61873d7a6905e5-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:09:23.619 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.619 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.620 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.620 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.621 DEBUG:boto:Method: GET 2023-10-12T01:09:23.621 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.622 DEBUG:boto:Data: 2023-10-12T01:09:23.622 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.623 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.624 DEBUG:boto:Params: {} 2023-10-12T01:09:23.624 DEBUG:boto:Token: None 2023-10-12T01:09:23.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU= 2023-10-12T01:09:23.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU=', 'Content-Length': '0'} 2023-10-12T01:09:23.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb465535a026a2d-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.670 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.671 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.671 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.672 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.672 DEBUG:boto:Method: GET 2023-10-12T01:09:23.673 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.673 DEBUG:boto:Data: 2023-10-12T01:09:23.674 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.675 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.675 DEBUG:boto:Params: {} 2023-10-12T01:09:23.676 DEBUG:boto:Token: None 2023-10-12T01:09:23.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:09:23.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU= 2023-10-12T01:09:23.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OFjZ2lIlc2ooVCKOVhjERKBPpdU=', 'Content-Length': '0'} 2023-10-12T01:09:23.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b093a5a29bb687-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.680 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.681 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.682 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.682 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.683 DEBUG:boto:Method: GET 2023-10-12T01:09:23.683 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.683 DEBUG:boto:Data: 2023-10-12T01:09:23.684 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.685 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.685 DEBUG:boto:Params: {} 2023-10-12T01:09:23.686 DEBUG:boto:Token: None 2023-10-12T01:09:23.686 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000316e3619af8e380-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.690 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.691 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.692 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.693 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.693 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.693 DEBUG:boto:Method: GET 2023-10-12T01:09:23.694 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.694 DEBUG:boto:Data: 2023-10-12T01:09:23.695 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.696 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.696 DEBUG:boto:Params: {} 2023-10-12T01:09:23.697 DEBUG:boto:Token: None 2023-10-12T01:09:23.697 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.698 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637db64f96a30fc6-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.703 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.704 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.705 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.705 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.706 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:09:23.706 DEBUG:boto:Data: 2023-10-12T01:09:23.707 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.708 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.709 DEBUG:boto:Params: {} 2023-10-12T01:09:23.709 DEBUG:boto:Token: None 2023-10-12T01:09:23.710 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124 2023-10-12T01:09:23.711 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FjaE0txl7UQzDIiD+m33tTkZXYA= 2023-10-12T01:09:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FjaE0txl7UQzDIiD+m33tTkZXYA=', 'Content-Length': '0'} 2023-10-12T01:09:23.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe211dda6819cc7-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.716 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.717 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.717 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.718 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:09:23.719 DEBUG:boto:Data: 2023-10-12T01:09:23.720 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.721 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.722 DEBUG:boto:Params: {} 2023-10-12T01:09:23.723 DEBUG:boto:Token: None 2023-10-12T01:09:23.724 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124 2023-10-12T01:09:23.725 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FjaE0txl7UQzDIiD+m33tTkZXYA= 2023-10-12T01:09:23.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FjaE0txl7UQzDIiD+m33tTkZXYA=', 'Content-Length': '0'} 2023-10-12T01:09:23.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac9198ca6348883c-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:09:23.730 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.731 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.731 DEBUG:boto:Method: GET 2023-10-12T01:09:23.732 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:09:23.733 DEBUG:boto:Data: 2023-10-12T01:09:23.733 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.735 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.736 DEBUG:boto:Params: {} 2023-10-12T01:09:23.736 DEBUG:boto:Token: None 2023-10-12T01:09:23.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124 2023-10-12T01:09:23.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L2z6XYeBVR+8opCQ9kKwfoUww9k= 2023-10-12T01:09:23.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L2z6XYeBVR+8opCQ9kKwfoUww9k=', 'Content-Length': '0'} 2023-10-12T01:09:23.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea96bbbaf1ac9db-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.741 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:09:23.742 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:09:23.743 DEBUG:boto:Method: GET 2023-10-12T01:09:23.744 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:09:23.744 DEBUG:boto:Data: 2023-10-12T01:09:23.745 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.747 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.747 DEBUG:boto:Params: {} 2023-10-12T01:09:23.748 DEBUG:boto:Token: None 2023-10-12T01:09:23.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj124 2023-10-12T01:09:23.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L2z6XYeBVR+8opCQ9kKwfoUww9k= 2023-10-12T01:09:23.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L2z6XYeBVR+8opCQ9kKwfoUww9k=', 'Content-Length': '0'} 2023-10-12T01:09:23.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ef0f16ffd6f219-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.794 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.795 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.796 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.796 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.797 DEBUG:boto:Method: GET 2023-10-12T01:09:23.797 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.798 DEBUG:boto:Data: 2023-10-12T01:09:23.798 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.799 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.799 DEBUG:boto:Params: {} 2023-10-12T01:09:23.800 DEBUG:boto:Token: None 2023-10-12T01:09:23.800 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c19ff9fed52ed629-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.805 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.805 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.806 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.806 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.807 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.807 DEBUG:boto:Method: GET 2023-10-12T01:09:23.808 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:09:23.808 DEBUG:boto:Data: 2023-10-12T01:09:23.809 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.810 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.810 DEBUG:boto:Params: {} 2023-10-12T01:09:23.811 DEBUG:boto:Token: None 2023-10-12T01:09:23.811 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326ca0f029c20c64-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.818 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:09:23.819 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.820 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.820 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.821 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.821 DEBUG:boto:Method: GET 2023-10-12T01:09:23.822 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.822 DEBUG:boto:Data: 2023-10-12T01:09:23.823 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.824 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.824 DEBUG:boto:Params: {} 2023-10-12T01:09:23.825 DEBUG:boto:Token: None 2023-10-12T01:09:23.825 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q= 2023-10-12T01:09:23.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q=', 'Content-Length': '0'} 2023-10-12T01:09:23.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5fa82550b3214d-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.829 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.830 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.830 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.831 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.831 DEBUG:boto:Method: GET 2023-10-12T01:09:23.831 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.832 DEBUG:boto:Data: 2023-10-12T01:09:23.832 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.833 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.834 DEBUG:boto:Params: {} 2023-10-12T01:09:23.834 DEBUG:boto:Token: None 2023-10-12T01:09:23.835 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.835 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q= 2023-10-12T01:09:23.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q=', 'Content-Length': '0'} 2023-10-12T01:09:23.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f55727c486bcdd-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.839 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.839 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.840 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.840 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.840 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.841 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.841 DEBUG:boto:Data: 2023-10-12T01:09:23.842 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.843 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.843 DEBUG:boto:Params: {} 2023-10-12T01:09:23.844 DEBUG:boto:Token: None 2023-10-12T01:09:23.844 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YETo0z/TxEIFD/5vH1wCK4tgld4= 2023-10-12T01:09:23.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YETo0z/TxEIFD/5vH1wCK4tgld4=', 'Content-Length': '0'} 2023-10-12T01:09:23.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e1b9297d022be1-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.848 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.848 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.849 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.849 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.849 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.850 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.850 DEBUG:boto:Data: 2023-10-12T01:09:23.851 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.852 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.852 DEBUG:boto:Params: {} 2023-10-12T01:09:23.852 DEBUG:boto:Token: None 2023-10-12T01:09:23.853 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.854 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YETo0z/TxEIFD/5vH1wCK4tgld4= 2023-10-12T01:09:23.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YETo0z/TxEIFD/5vH1wCK4tgld4=', 'Content-Length': '0'} 2023-10-12T01:09:23.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008898c26f27a958bd-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:09:23.857 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.857 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.858 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.858 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.861 DEBUG:boto:Method: GET 2023-10-12T01:09:23.862 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.862 DEBUG:boto:Data: 2023-10-12T01:09:23.863 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.863 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.864 DEBUG:boto:Params: {} 2023-10-12T01:09:23.864 DEBUG:boto:Token: None 2023-10-12T01:09:23.865 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q= 2023-10-12T01:09:23.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q=', 'Content-Length': '0'} 2023-10-12T01:09:23.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a98b62dd564762-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.914 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.915 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.916 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.916 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.917 DEBUG:boto:Method: GET 2023-10-12T01:09:23.917 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.918 DEBUG:boto:Data: 2023-10-12T01:09:23.918 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.919 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.920 DEBUG:boto:Params: {} 2023-10-12T01:09:23.920 DEBUG:boto:Token: None 2023-10-12T01:09:23.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:09:23.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q= 2023-10-12T01:09:23.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2mnIDtv9Bfnk5klDw+Yz6wtP2Q=', 'Content-Length': '0'} 2023-10-12T01:09:23.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d819151f4d2b045-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.925 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.925 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.926 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.926 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.927 DEBUG:boto:Method: GET 2023-10-12T01:09:23.927 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.928 DEBUG:boto:Data: 2023-10-12T01:09:23.928 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.929 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.929 DEBUG:boto:Params: {} 2023-10-12T01:09:23.930 DEBUG:boto:Token: None 2023-10-12T01:09:23.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f24d431e93b1ccfd-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.935 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.935 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.936 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.936 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.937 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.937 DEBUG:boto:Method: GET 2023-10-12T01:09:23.938 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.938 DEBUG:boto:Data: 2023-10-12T01:09:23.939 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.939 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.940 DEBUG:boto:Params: {} 2023-10-12T01:09:23.940 DEBUG:boto:Token: None 2023-10-12T01:09:23.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c08b535cf3fc47c-0065274743-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.946 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:23.947 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.947 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.948 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.948 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:09:23.949 DEBUG:boto:Data: 2023-10-12T01:09:23.949 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.950 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.950 DEBUG:boto:Params: {} 2023-10-12T01:09:23.951 DEBUG:boto:Token: None 2023-10-12T01:09:23.951 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145 2023-10-12T01:09:23.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dFgXp1ZNduyX92U9C5AstxD7QLg= 2023-10-12T01:09:23.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dFgXp1ZNduyX92U9C5AstxD7QLg=', 'Content-Length': '0'} 2023-10-12T01:09:23.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002350db271584f5cb-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.955 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.955 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.956 DEBUG:boto:Method: HEAD 2023-10-12T01:09:23.956 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:09:23.956 DEBUG:boto:Data: 2023-10-12T01:09:23.957 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.958 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.958 DEBUG:boto:Params: {} 2023-10-12T01:09:23.959 DEBUG:boto:Token: None 2023-10-12T01:09:23.959 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145 2023-10-12T01:09:23.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dFgXp1ZNduyX92U9C5AstxD7QLg= 2023-10-12T01:09:23.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dFgXp1ZNduyX92U9C5AstxD7QLg=', 'Content-Length': '0'} 2023-10-12T01:09:23.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a4e73b37fe41215-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:09:23.963 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.964 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.964 DEBUG:boto:Method: GET 2023-10-12T01:09:23.964 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:09:23.965 DEBUG:boto:Data: 2023-10-12T01:09:23.965 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.966 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.967 DEBUG:boto:Params: {} 2023-10-12T01:09:23.967 DEBUG:boto:Token: None 2023-10-12T01:09:23.968 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145 2023-10-12T01:09:23.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADhjBmElSf2N0kFIblsOGuzCYJE= 2023-10-12T01:09:23.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADhjBmElSf2N0kFIblsOGuzCYJE=', 'Content-Length': '0'} 2023-10-12T01:09:23.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319d61ade07afc7c-0065274743-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.971 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:09:23.972 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:09:23.972 DEBUG:boto:Method: GET 2023-10-12T01:09:23.973 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:09:23.973 DEBUG:boto:Data: 2023-10-12T01:09:23.974 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:23.975 DEBUG:boto:Port: 8000 2023-10-12T01:09:23.975 DEBUG:boto:Params: {} 2023-10-12T01:09:23.975 DEBUG:boto:Token: None 2023-10-12T01:09:23.976 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/obj145 2023-10-12T01:09:23.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ADhjBmElSf2N0kFIblsOGuzCYJE= 2023-10-12T01:09:23.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ADhjBmElSf2N0kFIblsOGuzCYJE=', 'Content-Length': '0'} 2023-10-12T01:09:23.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073abe70a66ece76d-0065274743-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:23.980 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:23.980 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:23.981 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.982 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.982 DEBUG:boto:Method: GET 2023-10-12T01:09:23.982 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:09:23.983 DEBUG:boto:Data: 2023-10-12T01:09:23.983 DEBUG:boto:Headers: {} 2023-10-12T01:09:23.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:23.984 DEBUG:boto:Port: 8001 2023-10-12T01:09:23.985 DEBUG:boto:Params: {} 2023-10-12T01:09:23.985 DEBUG:boto:Token: None 2023-10-12T01:09:23.986 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:23 GMT /wafmoa-59/ 2023-10-12T01:09:23.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc= 2023-10-12T01:09:23.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GbCj+DfCzYGtc6bdC51sTbClgKc=', 'Content-Length': '0'} 2023-10-12T01:09:23.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d44675a66a959fdb-0065274743-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.030 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.031 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.032 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.032 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:24.033 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:09:24.033 DEBUG:boto:Method: GET 2023-10-12T01:09:24.033 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:09:24.034 DEBUG:boto:Data: 2023-10-12T01:09:24.034 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.035 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.036 DEBUG:boto:Params: {} 2023-10-12T01:09:24.036 DEBUG:boto:Token: None 2023-10-12T01:09:24.037 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348a863612b926cc-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.044 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:09:24.045 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.046 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.046 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.047 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.047 DEBUG:boto:Method: GET 2023-10-12T01:09:24.048 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.048 DEBUG:boto:Data: 2023-10-12T01:09:24.049 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.050 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.050 DEBUG:boto:Params: {} 2023-10-12T01:09:24.051 DEBUG:boto:Token: None 2023-10-12T01:09:24.051 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI= 2023-10-12T01:09:24.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI=', 'Content-Length': '0'} 2023-10-12T01:09:24.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077d64bb5ccb1c7c6-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.056 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.057 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.058 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.059 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.060 DEBUG:boto:Method: GET 2023-10-12T01:09:24.060 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.061 DEBUG:boto:Data: 2023-10-12T01:09:24.062 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.063 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.065 DEBUG:boto:Params: {} 2023-10-12T01:09:24.066 DEBUG:boto:Token: None 2023-10-12T01:09:24.066 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI= 2023-10-12T01:09:24.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI=', 'Content-Length': '0'} 2023-10-12T01:09:24.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c2c98c8171e394-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.110 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.111 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.111 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.112 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.112 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.113 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.113 DEBUG:boto:Data: 2023-10-12T01:09:24.114 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.115 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.115 DEBUG:boto:Params: {} 2023-10-12T01:09:24.115 DEBUG:boto:Token: None 2023-10-12T01:09:24.116 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I5LZp5klMpSFezf343W+oWfEjXo= 2023-10-12T01:09:24.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I5LZp5klMpSFezf343W+oWfEjXo=', 'Content-Length': '0'} 2023-10-12T01:09:24.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea70487539906c2-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.120 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.120 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.121 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.121 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.122 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.122 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.122 DEBUG:boto:Data: 2023-10-12T01:09:24.123 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.124 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.124 DEBUG:boto:Params: {} 2023-10-12T01:09:24.125 DEBUG:boto:Token: None 2023-10-12T01:09:24.125 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I5LZp5klMpSFezf343W+oWfEjXo= 2023-10-12T01:09:24.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I5LZp5klMpSFezf343W+oWfEjXo=', 'Content-Length': '0'} 2023-10-12T01:09:24.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002334c3c2ad6dc521-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:09:24.129 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.130 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.130 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.131 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.131 DEBUG:boto:Method: GET 2023-10-12T01:09:24.132 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.132 DEBUG:boto:Data: 2023-10-12T01:09:24.133 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.134 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.134 DEBUG:boto:Params: {} 2023-10-12T01:09:24.135 DEBUG:boto:Token: None 2023-10-12T01:09:24.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI= 2023-10-12T01:09:24.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI=', 'Content-Length': '0'} 2023-10-12T01:09:24.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2561d753dfabaa5-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.139 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.139 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.140 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.140 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.141 DEBUG:boto:Method: GET 2023-10-12T01:09:24.141 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.142 DEBUG:boto:Data: 2023-10-12T01:09:24.142 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.143 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.144 DEBUG:boto:Params: {} 2023-10-12T01:09:24.144 DEBUG:boto:Token: None 2023-10-12T01:09:24.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:09:24.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI= 2023-10-12T01:09:24.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pESzOWPkXy3pPPKD0GKxxRrZtdI=', 'Content-Length': '0'} 2023-10-12T01:09:24.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a7b0f1c6087a0f-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.149 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.149 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.150 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.150 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.151 DEBUG:boto:Method: GET 2023-10-12T01:09:24.151 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.152 DEBUG:boto:Data: 2023-10-12T01:09:24.152 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.153 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.153 DEBUG:boto:Params: {} 2023-10-12T01:09:24.154 DEBUG:boto:Token: None 2023-10-12T01:09:24.154 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c64af01b2a8a94b0-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.202 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.203 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.204 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.204 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.205 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.205 DEBUG:boto:Method: GET 2023-10-12T01:09:24.206 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.206 DEBUG:boto:Data: 2023-10-12T01:09:24.206 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.207 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.208 DEBUG:boto:Params: {} 2023-10-12T01:09:24.208 DEBUG:boto:Token: None 2023-10-12T01:09:24.209 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2fcbda8f220cdc-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.215 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.215 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.216 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.216 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.217 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:09:24.217 DEBUG:boto:Data: 2023-10-12T01:09:24.218 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.219 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.219 DEBUG:boto:Params: {} 2023-10-12T01:09:24.220 DEBUG:boto:Token: None 2023-10-12T01:09:24.220 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155 2023-10-12T01:09:24.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uEPFLaNYyJ2y7FvoDCkvr65bJxM= 2023-10-12T01:09:24.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uEPFLaNYyJ2y7FvoDCkvr65bJxM=', 'Content-Length': '0'} 2023-10-12T01:09:24.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002807f296236f8db7-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.224 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.225 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.225 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.225 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:09:24.226 DEBUG:boto:Data: 2023-10-12T01:09:24.226 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.227 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.228 DEBUG:boto:Params: {} 2023-10-12T01:09:24.228 DEBUG:boto:Token: None 2023-10-12T01:09:24.228 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155 2023-10-12T01:09:24.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uEPFLaNYyJ2y7FvoDCkvr65bJxM= 2023-10-12T01:09:24.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uEPFLaNYyJ2y7FvoDCkvr65bJxM=', 'Content-Length': '0'} 2023-10-12T01:09:24.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa0fbd3c56667b9f-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:09:24.233 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.233 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.233 DEBUG:boto:Method: GET 2023-10-12T01:09:24.234 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:09:24.234 DEBUG:boto:Data: 2023-10-12T01:09:24.235 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.236 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.236 DEBUG:boto:Params: {} 2023-10-12T01:09:24.237 DEBUG:boto:Token: None 2023-10-12T01:09:24.237 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155 2023-10-12T01:09:24.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7FsMjRyliIIfgSPD/+YfzgvFoaY= 2023-10-12T01:09:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7FsMjRyliIIfgSPD/+YfzgvFoaY=', 'Content-Length': '0'} 2023-10-12T01:09:24.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000494dd17082ac6dce-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.241 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:09:24.241 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:09:24.242 DEBUG:boto:Method: GET 2023-10-12T01:09:24.242 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:09:24.242 DEBUG:boto:Data: 2023-10-12T01:09:24.243 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.244 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.244 DEBUG:boto:Params: {} 2023-10-12T01:09:24.245 DEBUG:boto:Token: None 2023-10-12T01:09:24.245 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj155 2023-10-12T01:09:24.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7FsMjRyliIIfgSPD/+YfzgvFoaY= 2023-10-12T01:09:24.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7FsMjRyliIIfgSPD/+YfzgvFoaY=', 'Content-Length': '0'} 2023-10-12T01:09:24.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c3279709ae7b33-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.249 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.249 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.250 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.250 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.251 DEBUG:boto:Method: GET 2023-10-12T01:09:24.251 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.251 DEBUG:boto:Data: 2023-10-12T01:09:24.252 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.253 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.253 DEBUG:boto:Params: {} 2023-10-12T01:09:24.254 DEBUG:boto:Token: None 2023-10-12T01:09:24.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159b95ef5cb33d8b-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.258 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.259 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.260 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.260 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.260 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.261 DEBUG:boto:Method: GET 2023-10-12T01:09:24.261 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:09:24.262 DEBUG:boto:Data: 2023-10-12T01:09:24.262 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.263 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.263 DEBUG:boto:Params: {} 2023-10-12T01:09:24.264 DEBUG:boto:Token: None 2023-10-12T01:09:24.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d67a1af37b19d15f-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.270 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:09:24.271 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.271 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.272 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.272 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.273 DEBUG:boto:Method: GET 2023-10-12T01:09:24.273 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.273 DEBUG:boto:Data: 2023-10-12T01:09:24.274 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.275 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.275 DEBUG:boto:Params: {} 2023-10-12T01:09:24.276 DEBUG:boto:Token: None 2023-10-12T01:09:24.276 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0= 2023-10-12T01:09:24.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0=', 'Content-Length': '0'} 2023-10-12T01:09:24.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4ff1330d8a9878-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.280 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.281 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.281 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.282 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.282 DEBUG:boto:Method: GET 2023-10-12T01:09:24.283 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.283 DEBUG:boto:Data: 2023-10-12T01:09:24.283 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.284 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.285 DEBUG:boto:Params: {} 2023-10-12T01:09:24.285 DEBUG:boto:Token: None 2023-10-12T01:09:24.286 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0= 2023-10-12T01:09:24.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0=', 'Content-Length': '0'} 2023-10-12T01:09:24.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb16fdb75880a1b-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.290 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.290 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.291 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.291 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.291 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.292 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.292 DEBUG:boto:Data: 2023-10-12T01:09:24.293 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.293 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.294 DEBUG:boto:Params: {} 2023-10-12T01:09:24.294 DEBUG:boto:Token: None 2023-10-12T01:09:24.295 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.295 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8Y+5TqEAiricfeRpRx12gSt4WU= 2023-10-12T01:09:24.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8Y+5TqEAiricfeRpRx12gSt4WU=', 'Content-Length': '0'} 2023-10-12T01:09:24.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005746cd5a758e0bc7-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.298 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.299 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.299 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.300 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.300 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.300 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.301 DEBUG:boto:Data: 2023-10-12T01:09:24.301 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.302 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.302 DEBUG:boto:Params: {} 2023-10-12T01:09:24.303 DEBUG:boto:Token: None 2023-10-12T01:09:24.303 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.304 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u8Y+5TqEAiricfeRpRx12gSt4WU= 2023-10-12T01:09:24.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u8Y+5TqEAiricfeRpRx12gSt4WU=', 'Content-Length': '0'} 2023-10-12T01:09:24.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006431cf0df35bb310-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:09:24.307 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.308 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.308 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.309 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.309 DEBUG:boto:Method: GET 2023-10-12T01:09:24.309 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.310 DEBUG:boto:Data: 2023-10-12T01:09:24.310 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.311 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.312 DEBUG:boto:Params: {} 2023-10-12T01:09:24.312 DEBUG:boto:Token: None 2023-10-12T01:09:24.313 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0= 2023-10-12T01:09:24.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0=', 'Content-Length': '0'} 2023-10-12T01:09:24.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbfba33d3b48c85a-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.358 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.359 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.360 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.360 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.361 DEBUG:boto:Method: GET 2023-10-12T01:09:24.361 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.362 DEBUG:boto:Data: 2023-10-12T01:09:24.362 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.363 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.364 DEBUG:boto:Params: {} 2023-10-12T01:09:24.364 DEBUG:boto:Token: None 2023-10-12T01:09:24.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:09:24.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0= 2023-10-12T01:09:24.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TV9dy/f0L7LGoa4kN0/89rHD5r0=', 'Content-Length': '0'} 2023-10-12T01:09:24.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705f5fbcc9911f24-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.369 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.369 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.370 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.371 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.371 DEBUG:boto:Method: GET 2023-10-12T01:09:24.372 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.372 DEBUG:boto:Data: 2023-10-12T01:09:24.373 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.374 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.374 DEBUG:boto:Params: {} 2023-10-12T01:09:24.375 DEBUG:boto:Token: None 2023-10-12T01:09:24.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a771de39af45f9-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.380 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.380 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.381 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.381 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.382 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.382 DEBUG:boto:Method: GET 2023-10-12T01:09:24.383 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.384 DEBUG:boto:Data: 2023-10-12T01:09:24.385 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.386 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.387 DEBUG:boto:Params: {} 2023-10-12T01:09:24.388 DEBUG:boto:Token: None 2023-10-12T01:09:24.389 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f8f27daa0fbab1-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.396 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.397 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.398 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.399 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.399 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:09:24.400 DEBUG:boto:Data: 2023-10-12T01:09:24.401 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.402 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.402 DEBUG:boto:Params: {} 2023-10-12T01:09:24.403 DEBUG:boto:Token: None 2023-10-12T01:09:24.404 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17 2023-10-12T01:09:24.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0sL+bGg0wY8eVoAJkQaDki3zB9E= 2023-10-12T01:09:24.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0sL+bGg0wY8eVoAJkQaDki3zB9E=', 'Content-Length': '0'} 2023-10-12T01:09:24.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026ce4059c28d1178-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.409 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.409 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.410 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.410 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:09:24.411 DEBUG:boto:Data: 2023-10-12T01:09:24.411 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.412 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.413 DEBUG:boto:Params: {} 2023-10-12T01:09:24.413 DEBUG:boto:Token: None 2023-10-12T01:09:24.414 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17 2023-10-12T01:09:24.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0sL+bGg0wY8eVoAJkQaDki3zB9E= 2023-10-12T01:09:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0sL+bGg0wY8eVoAJkQaDki3zB9E=', 'Content-Length': '0'} 2023-10-12T01:09:24.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a242771cebde5a-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:09:24.418 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.419 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.419 DEBUG:boto:Method: GET 2023-10-12T01:09:24.420 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:09:24.420 DEBUG:boto:Data: 2023-10-12T01:09:24.421 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.421 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.422 DEBUG:boto:Params: {} 2023-10-12T01:09:24.422 DEBUG:boto:Token: None 2023-10-12T01:09:24.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17 2023-10-12T01:09:24.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yj9zhqG5sexZ8f8UjEhoM2QLbHQ= 2023-10-12T01:09:24.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yj9zhqG5sexZ8f8UjEhoM2QLbHQ=', 'Content-Length': '0'} 2023-10-12T01:09:24.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db285c2680d5c7ea-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.427 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:09:24.427 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:09:24.428 DEBUG:boto:Method: GET 2023-10-12T01:09:24.428 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:09:24.429 DEBUG:boto:Data: 2023-10-12T01:09:24.429 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.430 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.431 DEBUG:boto:Params: {} 2023-10-12T01:09:24.432 DEBUG:boto:Token: None 2023-10-12T01:09:24.433 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj17 2023-10-12T01:09:24.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yj9zhqG5sexZ8f8UjEhoM2QLbHQ= 2023-10-12T01:09:24.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yj9zhqG5sexZ8f8UjEhoM2QLbHQ=', 'Content-Length': '0'} 2023-10-12T01:09:24.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f636826c1082c5e7-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.438 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.438 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.439 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.440 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.440 DEBUG:boto:Method: GET 2023-10-12T01:09:24.441 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.442 DEBUG:boto:Data: 2023-10-12T01:09:24.442 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.443 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.444 DEBUG:boto:Params: {} 2023-10-12T01:09:24.445 DEBUG:boto:Token: None 2023-10-12T01:09:24.445 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a38ce86e61b870a-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.449 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.450 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.451 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.452 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.452 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.453 DEBUG:boto:Method: GET 2023-10-12T01:09:24.454 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:09:24.454 DEBUG:boto:Data: 2023-10-12T01:09:24.455 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.456 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.457 DEBUG:boto:Params: {} 2023-10-12T01:09:24.457 DEBUG:boto:Token: None 2023-10-12T01:09:24.458 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cdb16d354032279-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.463 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:09:24.465 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.465 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.466 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.466 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.467 DEBUG:boto:Method: GET 2023-10-12T01:09:24.467 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.468 DEBUG:boto:Data: 2023-10-12T01:09:24.468 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.470 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.470 DEBUG:boto:Params: {} 2023-10-12T01:09:24.473 DEBUG:boto:Token: None 2023-10-12T01:09:24.474 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.475 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg= 2023-10-12T01:09:24.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg=', 'Content-Length': '0'} 2023-10-12T01:09:24.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6623872fb4af91a-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.479 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.479 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.480 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.480 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.480 DEBUG:boto:Method: GET 2023-10-12T01:09:24.481 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.481 DEBUG:boto:Data: 2023-10-12T01:09:24.482 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.483 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.483 DEBUG:boto:Params: {} 2023-10-12T01:09:24.484 DEBUG:boto:Token: None 2023-10-12T01:09:24.484 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg= 2023-10-12T01:09:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg=', 'Content-Length': '0'} 2023-10-12T01:09:24.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1edc0bab888d9f-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.530 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.531 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.532 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.533 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.534 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.534 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.535 DEBUG:boto:Data: 2023-10-12T01:09:24.536 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.537 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.538 DEBUG:boto:Params: {} 2023-10-12T01:09:24.539 DEBUG:boto:Token: None 2023-10-12T01:09:24.539 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KMsbea70h2S49K8XtXQsIc9F9d0= 2023-10-12T01:09:24.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KMsbea70h2S49K8XtXQsIc9F9d0=', 'Content-Length': '0'} 2023-10-12T01:09:24.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed54c144f9845c4-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.544 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.545 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.545 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.546 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.546 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.546 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.547 DEBUG:boto:Data: 2023-10-12T01:09:24.548 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.549 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.550 DEBUG:boto:Params: {} 2023-10-12T01:09:24.550 DEBUG:boto:Token: None 2023-10-12T01:09:24.551 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KMsbea70h2S49K8XtXQsIc9F9d0= 2023-10-12T01:09:24.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KMsbea70h2S49K8XtXQsIc9F9d0=', 'Content-Length': '0'} 2023-10-12T01:09:24.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4208e4e1a41dc7-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:09:24.555 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.555 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.555 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.556 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.556 DEBUG:boto:Method: GET 2023-10-12T01:09:24.557 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.558 DEBUG:boto:Data: 2023-10-12T01:09:24.558 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.560 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.561 DEBUG:boto:Params: {} 2023-10-12T01:09:24.562 DEBUG:boto:Token: None 2023-10-12T01:09:24.563 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg= 2023-10-12T01:09:24.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg=', 'Content-Length': '0'} 2023-10-12T01:09:24.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c529ffd97c1a889-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.614 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.615 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.616 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.617 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.617 DEBUG:boto:Method: GET 2023-10-12T01:09:24.618 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.619 DEBUG:boto:Data: 2023-10-12T01:09:24.620 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.621 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.622 DEBUG:boto:Params: {} 2023-10-12T01:09:24.622 DEBUG:boto:Token: None 2023-10-12T01:09:24.623 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:09:24.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg= 2023-10-12T01:09:24.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Pa27kHeHcgBB1cZ/YqlNqFma2mg=', 'Content-Length': '0'} 2023-10-12T01:09:24.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc69a0a343bf8c56-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.627 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.628 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.628 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.628 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.629 DEBUG:boto:Method: GET 2023-10-12T01:09:24.630 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.630 DEBUG:boto:Data: 2023-10-12T01:09:24.630 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.631 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.632 DEBUG:boto:Params: {} 2023-10-12T01:09:24.632 DEBUG:boto:Token: None 2023-10-12T01:09:24.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000604c959d86b4830c-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.637 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.638 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.638 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.639 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.639 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.640 DEBUG:boto:Method: GET 2023-10-12T01:09:24.640 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.641 DEBUG:boto:Data: 2023-10-12T01:09:24.641 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.642 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.642 DEBUG:boto:Params: {} 2023-10-12T01:09:24.643 DEBUG:boto:Token: None 2023-10-12T01:09:24.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cc170824c7fb98e-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.649 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.650 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.650 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.651 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.651 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:09:24.652 DEBUG:boto:Data: 2023-10-12T01:09:24.652 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.653 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.653 DEBUG:boto:Params: {} 2023-10-12T01:09:24.654 DEBUG:boto:Token: None 2023-10-12T01:09:24.654 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19 2023-10-12T01:09:24.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyCsYyo1kpv0eIW69RxjWhCOZew= 2023-10-12T01:09:24.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyCsYyo1kpv0eIW69RxjWhCOZew=', 'Content-Length': '0'} 2023-10-12T01:09:24.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a08d81b0d4e71d-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.658 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.659 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.659 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.659 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:09:24.660 DEBUG:boto:Data: 2023-10-12T01:09:24.660 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.661 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.662 DEBUG:boto:Params: {} 2023-10-12T01:09:24.662 DEBUG:boto:Token: None 2023-10-12T01:09:24.662 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19 2023-10-12T01:09:24.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyCsYyo1kpv0eIW69RxjWhCOZew= 2023-10-12T01:09:24.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyCsYyo1kpv0eIW69RxjWhCOZew=', 'Content-Length': '0'} 2023-10-12T01:09:24.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e164d085bdf6788-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:09:24.666 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.667 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.667 DEBUG:boto:Method: GET 2023-10-12T01:09:24.668 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:09:24.668 DEBUG:boto:Data: 2023-10-12T01:09:24.669 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.669 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.670 DEBUG:boto:Params: {} 2023-10-12T01:09:24.670 DEBUG:boto:Token: None 2023-10-12T01:09:24.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19 2023-10-12T01:09:24.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y5LpzY+YxBlGUYh/9ufixaRoYUs= 2023-10-12T01:09:24.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y5LpzY+YxBlGUYh/9ufixaRoYUs=', 'Content-Length': '0'} 2023-10-12T01:09:24.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036dcd75f5e0cdbad-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.675 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:09:24.675 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:09:24.676 DEBUG:boto:Method: GET 2023-10-12T01:09:24.676 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:09:24.677 DEBUG:boto:Data: 2023-10-12T01:09:24.677 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.678 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.679 DEBUG:boto:Params: {} 2023-10-12T01:09:24.679 DEBUG:boto:Token: None 2023-10-12T01:09:24.680 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj19 2023-10-12T01:09:24.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y5LpzY+YxBlGUYh/9ufixaRoYUs= 2023-10-12T01:09:24.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y5LpzY+YxBlGUYh/9ufixaRoYUs=', 'Content-Length': '0'} 2023-10-12T01:09:24.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080abe552a80bd0df-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.726 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.727 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.727 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.728 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.728 DEBUG:boto:Method: GET 2023-10-12T01:09:24.729 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.729 DEBUG:boto:Data: 2023-10-12T01:09:24.729 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.730 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.731 DEBUG:boto:Params: {} 2023-10-12T01:09:24.731 DEBUG:boto:Token: None 2023-10-12T01:09:24.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9053d8f0a07b97d-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.783 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.783 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.784 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.784 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.785 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.785 DEBUG:boto:Method: GET 2023-10-12T01:09:24.785 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:09:24.786 DEBUG:boto:Data: 2023-10-12T01:09:24.786 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.787 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.788 DEBUG:boto:Params: {} 2023-10-12T01:09:24.788 DEBUG:boto:Token: None 2023-10-12T01:09:24.789 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba9ab0c71948dca-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.795 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:09:24.796 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.796 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.798 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.798 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.799 DEBUG:boto:Method: GET 2023-10-12T01:09:24.799 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.799 DEBUG:boto:Data: 2023-10-12T01:09:24.800 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.801 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.801 DEBUG:boto:Params: {} 2023-10-12T01:09:24.802 DEBUG:boto:Token: None 2023-10-12T01:09:24.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw= 2023-10-12T01:09:24.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw=', 'Content-Length': '0'} 2023-10-12T01:09:24.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192c80fe8cdfe089-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.806 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.807 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.807 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.807 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.808 DEBUG:boto:Method: GET 2023-10-12T01:09:24.808 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.809 DEBUG:boto:Data: 2023-10-12T01:09:24.809 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.810 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.810 DEBUG:boto:Params: {} 2023-10-12T01:09:24.811 DEBUG:boto:Token: None 2023-10-12T01:09:24.811 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw= 2023-10-12T01:09:24.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw=', 'Content-Length': '0'} 2023-10-12T01:09:24.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033dedce70976c846-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.815 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.816 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.816 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.817 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.817 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.817 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.818 DEBUG:boto:Data: 2023-10-12T01:09:24.818 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.819 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.820 DEBUG:boto:Params: {} 2023-10-12T01:09:24.820 DEBUG:boto:Token: None 2023-10-12T01:09:24.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YII5QctVR1m9FadhXnbuRgzE2nA= 2023-10-12T01:09:24.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YII5QctVR1m9FadhXnbuRgzE2nA=', 'Content-Length': '0'} 2023-10-12T01:09:24.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6d9dd2a37461eb-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.824 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.825 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.825 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.826 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.826 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.826 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.827 DEBUG:boto:Data: 2023-10-12T01:09:24.827 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.828 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.828 DEBUG:boto:Params: {} 2023-10-12T01:09:24.829 DEBUG:boto:Token: None 2023-10-12T01:09:24.829 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YII5QctVR1m9FadhXnbuRgzE2nA= 2023-10-12T01:09:24.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YII5QctVR1m9FadhXnbuRgzE2nA=', 'Content-Length': '0'} 2023-10-12T01:09:24.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bca9081b082b6ec-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:09:24.833 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.834 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.834 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.835 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.835 DEBUG:boto:Method: GET 2023-10-12T01:09:24.836 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.836 DEBUG:boto:Data: 2023-10-12T01:09:24.836 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.837 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.838 DEBUG:boto:Params: {} 2023-10-12T01:09:24.838 DEBUG:boto:Token: None 2023-10-12T01:09:24.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw= 2023-10-12T01:09:24.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw=', 'Content-Length': '0'} 2023-10-12T01:09:24.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6d70af6e4cbf03-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.843 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.843 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.844 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.844 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.845 DEBUG:boto:Method: GET 2023-10-12T01:09:24.845 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.845 DEBUG:boto:Data: 2023-10-12T01:09:24.846 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.847 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.847 DEBUG:boto:Params: {} 2023-10-12T01:09:24.848 DEBUG:boto:Token: None 2023-10-12T01:09:24.848 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:09:24.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw= 2023-10-12T01:09:24.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PocpTStY40p0moLflbesjArXujw=', 'Content-Length': '0'} 2023-10-12T01:09:24.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006293a46b0bd1f549-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.852 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.852 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.853 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.853 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.854 DEBUG:boto:Method: GET 2023-10-12T01:09:24.854 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.855 DEBUG:boto:Data: 2023-10-12T01:09:24.855 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.856 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.857 DEBUG:boto:Params: {} 2023-10-12T01:09:24.857 DEBUG:boto:Token: None 2023-10-12T01:09:24.858 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de20c3c9f9ac6ab9-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.862 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.863 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.863 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.864 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.864 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.864 DEBUG:boto:Method: GET 2023-10-12T01:09:24.865 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.865 DEBUG:boto:Data: 2023-10-12T01:09:24.866 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.867 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.867 DEBUG:boto:Params: {} 2023-10-12T01:09:24.868 DEBUG:boto:Token: None 2023-10-12T01:09:24.868 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021cf394dd039dee0-0065274744-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.914 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:24.915 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.916 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.916 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.916 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:09:24.917 DEBUG:boto:Data: 2023-10-12T01:09:24.917 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.918 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.919 DEBUG:boto:Params: {} 2023-10-12T01:09:24.919 DEBUG:boto:Token: None 2023-10-12T01:09:24.919 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22 2023-10-12T01:09:24.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XWuU1O99SZI9kuX9L7aimuGHIyc= 2023-10-12T01:09:24.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XWuU1O99SZI9kuX9L7aimuGHIyc=', 'Content-Length': '0'} 2023-10-12T01:09:24.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b886964035cd3a9c-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.923 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.924 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.924 DEBUG:boto:Method: HEAD 2023-10-12T01:09:24.924 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:09:24.925 DEBUG:boto:Data: 2023-10-12T01:09:24.925 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.926 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.926 DEBUG:boto:Params: {} 2023-10-12T01:09:24.927 DEBUG:boto:Token: None 2023-10-12T01:09:24.927 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22 2023-10-12T01:09:24.928 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XWuU1O99SZI9kuX9L7aimuGHIyc= 2023-10-12T01:09:24.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XWuU1O99SZI9kuX9L7aimuGHIyc=', 'Content-Length': '0'} 2023-10-12T01:09:24.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a42e755b3b9f893-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:09:24.931 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.932 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.932 DEBUG:boto:Method: GET 2023-10-12T01:09:24.933 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:09:24.933 DEBUG:boto:Data: 2023-10-12T01:09:24.933 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.934 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.935 DEBUG:boto:Params: {} 2023-10-12T01:09:24.935 DEBUG:boto:Token: None 2023-10-12T01:09:24.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22 2023-10-12T01:09:24.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sr1krt38Gl+xhG11lRDuM0gwl20= 2023-10-12T01:09:24.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sr1krt38Gl+xhG11lRDuM0gwl20=', 'Content-Length': '0'} 2023-10-12T01:09:24.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c27f9db98a8fff-0065274744-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.982 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:09:24.983 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:09:24.983 DEBUG:boto:Method: GET 2023-10-12T01:09:24.984 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:09:24.984 DEBUG:boto:Data: 2023-10-12T01:09:24.985 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:24.986 DEBUG:boto:Port: 8000 2023-10-12T01:09:24.986 DEBUG:boto:Params: {} 2023-10-12T01:09:24.987 DEBUG:boto:Token: None 2023-10-12T01:09:24.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/obj22 2023-10-12T01:09:24.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sr1krt38Gl+xhG11lRDuM0gwl20= 2023-10-12T01:09:24.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sr1krt38Gl+xhG11lRDuM0gwl20=', 'Content-Length': '0'} 2023-10-12T01:09:24.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af322b42d0a68b43-0065274744-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:24.991 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:24.991 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:24.992 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.992 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.993 DEBUG:boto:Method: GET 2023-10-12T01:09:24.993 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:09:24.993 DEBUG:boto:Data: 2023-10-12T01:09:24.994 DEBUG:boto:Headers: {} 2023-10-12T01:09:24.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:24.995 DEBUG:boto:Port: 8001 2023-10-12T01:09:24.995 DEBUG:boto:Params: {} 2023-10-12T01:09:24.996 DEBUG:boto:Token: None 2023-10-12T01:09:24.996 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:24 GMT /wafmoa-59/ 2023-10-12T01:09:24.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4= 2023-10-12T01:09:24.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:24 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hnItTt92aTS7IIwcOMJzeRhw3c4=', 'Content-Length': '0'} 2023-10-12T01:09:24.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e331c3bb238f003b-0065274744-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.000 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.001 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.001 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.002 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:25.002 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:09:25.002 DEBUG:boto:Method: GET 2023-10-12T01:09:25.003 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:09:25.003 DEBUG:boto:Data: 2023-10-12T01:09:25.004 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.004 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.005 DEBUG:boto:Params: {} 2023-10-12T01:09:25.005 DEBUG:boto:Token: None 2023-10-12T01:09:25.006 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002205465108826c9b-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.011 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:09:25.012 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.013 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.013 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.014 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.014 DEBUG:boto:Method: GET 2023-10-12T01:09:25.014 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.015 DEBUG:boto:Data: 2023-10-12T01:09:25.015 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.016 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.017 DEBUG:boto:Params: {} 2023-10-12T01:09:25.017 DEBUG:boto:Token: None 2023-10-12T01:09:25.018 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo= 2023-10-12T01:09:25.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo=', 'Content-Length': '0'} 2023-10-12T01:09:25.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724864c2147e238b-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.021 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.022 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.022 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.023 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.023 DEBUG:boto:Method: GET 2023-10-12T01:09:25.024 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.024 DEBUG:boto:Data: 2023-10-12T01:09:25.024 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.025 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.026 DEBUG:boto:Params: {} 2023-10-12T01:09:25.026 DEBUG:boto:Token: None 2023-10-12T01:09:25.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo= 2023-10-12T01:09:25.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo=', 'Content-Length': '0'} 2023-10-12T01:09:25.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3c2a91912fd58d-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.070 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.071 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.071 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.072 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.072 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.073 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.073 DEBUG:boto:Data: 2023-10-12T01:09:25.074 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.074 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.075 DEBUG:boto:Params: {} 2023-10-12T01:09:25.075 DEBUG:boto:Token: None 2023-10-12T01:09:25.076 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ppmKhWFZiApgLVSjSj/GIZ6sGmQ= 2023-10-12T01:09:25.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ppmKhWFZiApgLVSjSj/GIZ6sGmQ=', 'Content-Length': '0'} 2023-10-12T01:09:25.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fdbdd9fd7f8ab3b-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.079 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.080 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.080 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.081 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.081 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.082 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.082 DEBUG:boto:Data: 2023-10-12T01:09:25.082 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.083 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.084 DEBUG:boto:Params: {} 2023-10-12T01:09:25.084 DEBUG:boto:Token: None 2023-10-12T01:09:25.085 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ppmKhWFZiApgLVSjSj/GIZ6sGmQ= 2023-10-12T01:09:25.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ppmKhWFZiApgLVSjSj/GIZ6sGmQ=', 'Content-Length': '0'} 2023-10-12T01:09:25.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b8b79f615e19c6-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:09:25.090 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.090 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.091 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.091 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.092 DEBUG:boto:Method: GET 2023-10-12T01:09:25.092 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.093 DEBUG:boto:Data: 2023-10-12T01:09:25.093 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.094 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.094 DEBUG:boto:Params: {} 2023-10-12T01:09:25.095 DEBUG:boto:Token: None 2023-10-12T01:09:25.095 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo= 2023-10-12T01:09:25.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo=', 'Content-Length': '0'} 2023-10-12T01:09:25.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f59bd25bbbbe343-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.142 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.143 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.144 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.144 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.144 DEBUG:boto:Method: GET 2023-10-12T01:09:25.145 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.145 DEBUG:boto:Data: 2023-10-12T01:09:25.146 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.147 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.147 DEBUG:boto:Params: {} 2023-10-12T01:09:25.148 DEBUG:boto:Token: None 2023-10-12T01:09:25.148 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:09:25.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo= 2023-10-12T01:09:25.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cskA0IRospW7Blg96qc5+I36moo=', 'Content-Length': '0'} 2023-10-12T01:09:25.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3dc3d3b1128109-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.152 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.152 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.153 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.153 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.154 DEBUG:boto:Method: GET 2023-10-12T01:09:25.154 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.155 DEBUG:boto:Data: 2023-10-12T01:09:25.155 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.156 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.156 DEBUG:boto:Params: {} 2023-10-12T01:09:25.157 DEBUG:boto:Token: None 2023-10-12T01:09:25.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c57273dca462ae41-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.162 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.162 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.163 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.163 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.164 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.164 DEBUG:boto:Method: GET 2023-10-12T01:09:25.165 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.165 DEBUG:boto:Data: 2023-10-12T01:09:25.166 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.167 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.167 DEBUG:boto:Params: {} 2023-10-12T01:09:25.167 DEBUG:boto:Token: None 2023-10-12T01:09:25.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b795d975bf9b079c-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.173 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.174 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.175 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.175 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.176 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:09:25.176 DEBUG:boto:Data: 2023-10-12T01:09:25.177 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.177 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.178 DEBUG:boto:Params: {} 2023-10-12T01:09:25.178 DEBUG:boto:Token: None 2023-10-12T01:09:25.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23 2023-10-12T01:09:25.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lRrPzM7uK4VaOQvvHXlgf5X0Nqk= 2023-10-12T01:09:25.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lRrPzM7uK4VaOQvvHXlgf5X0Nqk=', 'Content-Length': '0'} 2023-10-12T01:09:25.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae2aa1fa2949f03-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.182 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.183 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.183 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.184 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:09:25.184 DEBUG:boto:Data: 2023-10-12T01:09:25.185 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.185 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.186 DEBUG:boto:Params: {} 2023-10-12T01:09:25.186 DEBUG:boto:Token: None 2023-10-12T01:09:25.187 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23 2023-10-12T01:09:25.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lRrPzM7uK4VaOQvvHXlgf5X0Nqk= 2023-10-12T01:09:25.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lRrPzM7uK4VaOQvvHXlgf5X0Nqk=', 'Content-Length': '0'} 2023-10-12T01:09:25.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66140eedc5a8ee2-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:09:25.191 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.191 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.192 DEBUG:boto:Method: GET 2023-10-12T01:09:25.192 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:09:25.193 DEBUG:boto:Data: 2023-10-12T01:09:25.193 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.194 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.194 DEBUG:boto:Params: {} 2023-10-12T01:09:25.195 DEBUG:boto:Token: None 2023-10-12T01:09:25.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23 2023-10-12T01:09:25.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PobMMpuOcNPmglTCfEzzkVD08qM= 2023-10-12T01:09:25.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PobMMpuOcNPmglTCfEzzkVD08qM=', 'Content-Length': '0'} 2023-10-12T01:09:25.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066f6731de02710e-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.199 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:09:25.199 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:09:25.200 DEBUG:boto:Method: GET 2023-10-12T01:09:25.200 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:09:25.201 DEBUG:boto:Data: 2023-10-12T01:09:25.201 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.202 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.202 DEBUG:boto:Params: {} 2023-10-12T01:09:25.203 DEBUG:boto:Token: None 2023-10-12T01:09:25.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj23 2023-10-12T01:09:25.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PobMMpuOcNPmglTCfEzzkVD08qM= 2023-10-12T01:09:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PobMMpuOcNPmglTCfEzzkVD08qM=', 'Content-Length': '0'} 2023-10-12T01:09:25.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002025ba391933a3bf-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.250 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.251 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.251 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.252 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.252 DEBUG:boto:Method: GET 2023-10-12T01:09:25.253 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.253 DEBUG:boto:Data: 2023-10-12T01:09:25.254 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.254 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.255 DEBUG:boto:Params: {} 2023-10-12T01:09:25.255 DEBUG:boto:Token: None 2023-10-12T01:09:25.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552805c018844771-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.260 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.261 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.262 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.262 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.263 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.263 DEBUG:boto:Method: GET 2023-10-12T01:09:25.263 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:09:25.264 DEBUG:boto:Data: 2023-10-12T01:09:25.264 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.265 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.265 DEBUG:boto:Params: {} 2023-10-12T01:09:25.266 DEBUG:boto:Token: None 2023-10-12T01:09:25.266 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.267 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b9370316b0bb7d-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.272 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:09:25.273 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.274 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.274 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.274 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.275 DEBUG:boto:Method: GET 2023-10-12T01:09:25.276 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.276 DEBUG:boto:Data: 2023-10-12T01:09:25.277 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.278 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.278 DEBUG:boto:Params: {} 2023-10-12T01:09:25.279 DEBUG:boto:Token: None 2023-10-12T01:09:25.279 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE= 2023-10-12T01:09:25.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE=', 'Content-Length': '0'} 2023-10-12T01:09:25.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029015ccb8ce1a2ae-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.283 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.284 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.284 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.284 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.285 DEBUG:boto:Method: GET 2023-10-12T01:09:25.285 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.286 DEBUG:boto:Data: 2023-10-12T01:09:25.286 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.287 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.287 DEBUG:boto:Params: {} 2023-10-12T01:09:25.288 DEBUG:boto:Token: None 2023-10-12T01:09:25.288 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.289 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE= 2023-10-12T01:09:25.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE=', 'Content-Length': '0'} 2023-10-12T01:09:25.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1a0c1ea9bcc201-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.292 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.292 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.293 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.293 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.294 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.294 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.295 DEBUG:boto:Data: 2023-10-12T01:09:25.295 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.296 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.296 DEBUG:boto:Params: {} 2023-10-12T01:09:25.297 DEBUG:boto:Token: None 2023-10-12T01:09:25.297 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.298 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uyHpqI5vjfVyzw3WQ/S7HTKJ4S4= 2023-10-12T01:09:25.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uyHpqI5vjfVyzw3WQ/S7HTKJ4S4=', 'Content-Length': '0'} 2023-10-12T01:09:25.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6fc2633ec726fea-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.300 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.301 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.301 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.302 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.302 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.302 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.303 DEBUG:boto:Data: 2023-10-12T01:09:25.303 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.304 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.304 DEBUG:boto:Params: {} 2023-10-12T01:09:25.305 DEBUG:boto:Token: None 2023-10-12T01:09:25.305 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.306 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uyHpqI5vjfVyzw3WQ/S7HTKJ4S4= 2023-10-12T01:09:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uyHpqI5vjfVyzw3WQ/S7HTKJ4S4=', 'Content-Length': '0'} 2023-10-12T01:09:25.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736d5eb9d5184e68-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:09:25.309 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.310 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.310 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.310 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.311 DEBUG:boto:Method: GET 2023-10-12T01:09:25.311 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.312 DEBUG:boto:Data: 2023-10-12T01:09:25.312 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.313 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.313 DEBUG:boto:Params: {} 2023-10-12T01:09:25.314 DEBUG:boto:Token: None 2023-10-12T01:09:25.314 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE= 2023-10-12T01:09:25.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE=', 'Content-Length': '0'} 2023-10-12T01:09:25.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc4f1278e56b31d-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.358 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.359 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.359 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.360 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.360 DEBUG:boto:Method: GET 2023-10-12T01:09:25.361 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.361 DEBUG:boto:Data: 2023-10-12T01:09:25.362 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.362 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.363 DEBUG:boto:Params: {} 2023-10-12T01:09:25.363 DEBUG:boto:Token: None 2023-10-12T01:09:25.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:09:25.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE= 2023-10-12T01:09:25.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t+VMpoI1EjMjC914U20QGjcvvjE=', 'Content-Length': '0'} 2023-10-12T01:09:25.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b289920c022531-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.368 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.368 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.369 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.369 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.370 DEBUG:boto:Method: GET 2023-10-12T01:09:25.370 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.371 DEBUG:boto:Data: 2023-10-12T01:09:25.371 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.372 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.372 DEBUG:boto:Params: {} 2023-10-12T01:09:25.373 DEBUG:boto:Token: None 2023-10-12T01:09:25.373 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019dd7d96514aad2f-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.378 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.378 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.379 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.379 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.380 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.380 DEBUG:boto:Method: GET 2023-10-12T01:09:25.381 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.381 DEBUG:boto:Data: 2023-10-12T01:09:25.381 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.382 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.383 DEBUG:boto:Params: {} 2023-10-12T01:09:25.383 DEBUG:boto:Token: None 2023-10-12T01:09:25.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96d91d82b51c411-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.389 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.390 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.390 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.390 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.391 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:09:25.391 DEBUG:boto:Data: 2023-10-12T01:09:25.392 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.393 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.393 DEBUG:boto:Params: {} 2023-10-12T01:09:25.393 DEBUG:boto:Token: None 2023-10-12T01:09:25.394 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29 2023-10-12T01:09:25.394 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dGE3MZE1ECauKFZfq3zv8Mh+0QE= 2023-10-12T01:09:25.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dGE3MZE1ECauKFZfq3zv8Mh+0QE=', 'Content-Length': '0'} 2023-10-12T01:09:25.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6202b9cfbb4ec3-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.398 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.398 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.399 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.399 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:09:25.399 DEBUG:boto:Data: 2023-10-12T01:09:25.400 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.401 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.401 DEBUG:boto:Params: {} 2023-10-12T01:09:25.401 DEBUG:boto:Token: None 2023-10-12T01:09:25.402 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29 2023-10-12T01:09:25.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dGE3MZE1ECauKFZfq3zv8Mh+0QE= 2023-10-12T01:09:25.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dGE3MZE1ECauKFZfq3zv8Mh+0QE=', 'Content-Length': '0'} 2023-10-12T01:09:25.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24d34a22c43749e-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:09:25.406 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.406 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.407 DEBUG:boto:Method: GET 2023-10-12T01:09:25.407 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:09:25.407 DEBUG:boto:Data: 2023-10-12T01:09:25.408 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.409 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.409 DEBUG:boto:Params: {} 2023-10-12T01:09:25.410 DEBUG:boto:Token: None 2023-10-12T01:09:25.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29 2023-10-12T01:09:25.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGTJmaUp7GEUK+3NXUypc1fg5z8= 2023-10-12T01:09:25.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGTJmaUp7GEUK+3NXUypc1fg5z8=', 'Content-Length': '0'} 2023-10-12T01:09:25.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359ce00b45ff601d-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.414 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:09:25.414 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:09:25.415 DEBUG:boto:Method: GET 2023-10-12T01:09:25.415 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:09:25.415 DEBUG:boto:Data: 2023-10-12T01:09:25.416 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.417 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.417 DEBUG:boto:Params: {} 2023-10-12T01:09:25.418 DEBUG:boto:Token: None 2023-10-12T01:09:25.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj29 2023-10-12T01:09:25.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGTJmaUp7GEUK+3NXUypc1fg5z8= 2023-10-12T01:09:25.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGTJmaUp7GEUK+3NXUypc1fg5z8=', 'Content-Length': '0'} 2023-10-12T01:09:25.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b01860c960acab4-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.422 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.422 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.423 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.423 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.424 DEBUG:boto:Method: GET 2023-10-12T01:09:25.424 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.425 DEBUG:boto:Data: 2023-10-12T01:09:25.425 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.426 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.426 DEBUG:boto:Params: {} 2023-10-12T01:09:25.427 DEBUG:boto:Token: None 2023-10-12T01:09:25.427 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e594d1e78471e10b-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.432 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.432 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.433 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.433 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.434 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.434 DEBUG:boto:Method: GET 2023-10-12T01:09:25.435 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:09:25.435 DEBUG:boto:Data: 2023-10-12T01:09:25.436 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.436 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.437 DEBUG:boto:Params: {} 2023-10-12T01:09:25.437 DEBUG:boto:Token: None 2023-10-12T01:09:25.438 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e77a9e9cab5724-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.443 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:09:25.444 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.445 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.445 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.446 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.446 DEBUG:boto:Method: GET 2023-10-12T01:09:25.446 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.447 DEBUG:boto:Data: 2023-10-12T01:09:25.447 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.448 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.449 DEBUG:boto:Params: {} 2023-10-12T01:09:25.449 DEBUG:boto:Token: None 2023-10-12T01:09:25.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc= 2023-10-12T01:09:25.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc=', 'Content-Length': '0'} 2023-10-12T01:09:25.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b48cec194d901242-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.454 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.454 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.455 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.455 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.455 DEBUG:boto:Method: GET 2023-10-12T01:09:25.456 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.456 DEBUG:boto:Data: 2023-10-12T01:09:25.457 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.458 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.458 DEBUG:boto:Params: {} 2023-10-12T01:09:25.458 DEBUG:boto:Token: None 2023-10-12T01:09:25.459 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.459 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc= 2023-10-12T01:09:25.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc=', 'Content-Length': '0'} 2023-10-12T01:09:25.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cb84a509b57d11-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.502 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.503 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.503 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.504 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.504 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.505 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.505 DEBUG:boto:Data: 2023-10-12T01:09:25.505 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.506 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.507 DEBUG:boto:Params: {} 2023-10-12T01:09:25.509 DEBUG:boto:Token: None 2023-10-12T01:09:25.509 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K4gbdchEZrCSMP+FFtWadVnOzOY= 2023-10-12T01:09:25.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K4gbdchEZrCSMP+FFtWadVnOzOY=', 'Content-Length': '0'} 2023-10-12T01:09:25.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011fdb240858cdc3f-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.513 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.514 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.514 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.515 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.515 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.516 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.516 DEBUG:boto:Data: 2023-10-12T01:09:25.516 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.517 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.518 DEBUG:boto:Params: {} 2023-10-12T01:09:25.518 DEBUG:boto:Token: None 2023-10-12T01:09:25.519 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.519 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K4gbdchEZrCSMP+FFtWadVnOzOY= 2023-10-12T01:09:25.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K4gbdchEZrCSMP+FFtWadVnOzOY=', 'Content-Length': '0'} 2023-10-12T01:09:25.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff3eaffcdc4b97d-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:09:25.523 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.523 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.524 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.524 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.525 DEBUG:boto:Method: GET 2023-10-12T01:09:25.525 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.526 DEBUG:boto:Data: 2023-10-12T01:09:25.526 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.527 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.527 DEBUG:boto:Params: {} 2023-10-12T01:09:25.528 DEBUG:boto:Token: None 2023-10-12T01:09:25.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc= 2023-10-12T01:09:25.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc=', 'Content-Length': '0'} 2023-10-12T01:09:25.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd4958d53fd0ec9-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.532 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.532 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.533 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.533 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.534 DEBUG:boto:Method: GET 2023-10-12T01:09:25.534 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.535 DEBUG:boto:Data: 2023-10-12T01:09:25.535 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.536 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.536 DEBUG:boto:Params: {} 2023-10-12T01:09:25.537 DEBUG:boto:Token: None 2023-10-12T01:09:25.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:09:25.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc= 2023-10-12T01:09:25.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nBVhv7/QW3hCPFQA8erlGkdcRQc=', 'Content-Length': '0'} 2023-10-12T01:09:25.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da5384c1d91cbac-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.541 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.542 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.542 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.543 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.543 DEBUG:boto:Method: GET 2023-10-12T01:09:25.543 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.544 DEBUG:boto:Data: 2023-10-12T01:09:25.544 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.545 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.546 DEBUG:boto:Params: {} 2023-10-12T01:09:25.546 DEBUG:boto:Token: None 2023-10-12T01:09:25.546 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bc008f08a40b79-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.590 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.591 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.592 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.592 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.593 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.593 DEBUG:boto:Method: GET 2023-10-12T01:09:25.593 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.594 DEBUG:boto:Data: 2023-10-12T01:09:25.594 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.595 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.596 DEBUG:boto:Params: {} 2023-10-12T01:09:25.596 DEBUG:boto:Token: None 2023-10-12T01:09:25.597 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005abaf01ae2e89c77-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.602 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.603 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.603 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.604 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.604 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:09:25.605 DEBUG:boto:Data: 2023-10-12T01:09:25.605 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.606 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.606 DEBUG:boto:Params: {} 2023-10-12T01:09:25.607 DEBUG:boto:Token: None 2023-10-12T01:09:25.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31 2023-10-12T01:09:25.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fSCLEzV/etiwHF2jjmljXWA13dI= 2023-10-12T01:09:25.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fSCLEzV/etiwHF2jjmljXWA13dI=', 'Content-Length': '0'} 2023-10-12T01:09:25.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007139e211d96c4b96-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.611 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.611 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.612 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.612 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:09:25.613 DEBUG:boto:Data: 2023-10-12T01:09:25.613 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.614 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.614 DEBUG:boto:Params: {} 2023-10-12T01:09:25.615 DEBUG:boto:Token: None 2023-10-12T01:09:25.615 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31 2023-10-12T01:09:25.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fSCLEzV/etiwHF2jjmljXWA13dI= 2023-10-12T01:09:25.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fSCLEzV/etiwHF2jjmljXWA13dI=', 'Content-Length': '0'} 2023-10-12T01:09:25.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca682b40d041e23f-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:09:25.620 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.621 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.621 DEBUG:boto:Method: GET 2023-10-12T01:09:25.622 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:09:25.622 DEBUG:boto:Data: 2023-10-12T01:09:25.623 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.624 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.624 DEBUG:boto:Params: {} 2023-10-12T01:09:25.625 DEBUG:boto:Token: None 2023-10-12T01:09:25.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31 2023-10-12T01:09:25.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XFHWmm+lnr2PMBKDyc/ceN428Ao= 2023-10-12T01:09:25.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XFHWmm+lnr2PMBKDyc/ceN428Ao=', 'Content-Length': '0'} 2023-10-12T01:09:25.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea235b9546197a5e-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.631 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:09:25.631 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:09:25.632 DEBUG:boto:Method: GET 2023-10-12T01:09:25.632 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:09:25.633 DEBUG:boto:Data: 2023-10-12T01:09:25.633 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.634 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.635 DEBUG:boto:Params: {} 2023-10-12T01:09:25.636 DEBUG:boto:Token: None 2023-10-12T01:09:25.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj31 2023-10-12T01:09:25.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XFHWmm+lnr2PMBKDyc/ceN428Ao= 2023-10-12T01:09:25.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XFHWmm+lnr2PMBKDyc/ceN428Ao=', 'Content-Length': '0'} 2023-10-12T01:09:25.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eaac3ecbf9f4a9d-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.640 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.640 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.641 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.641 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.641 DEBUG:boto:Method: GET 2023-10-12T01:09:25.642 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.642 DEBUG:boto:Data: 2023-10-12T01:09:25.643 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.644 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.644 DEBUG:boto:Params: {} 2023-10-12T01:09:25.645 DEBUG:boto:Token: None 2023-10-12T01:09:25.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71494c2f4159259-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.690 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.691 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.692 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.692 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.693 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.693 DEBUG:boto:Method: GET 2023-10-12T01:09:25.693 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:09:25.694 DEBUG:boto:Data: 2023-10-12T01:09:25.694 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.695 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.696 DEBUG:boto:Params: {} 2023-10-12T01:09:25.696 DEBUG:boto:Token: None 2023-10-12T01:09:25.697 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf77c6e5c36a95c8-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.702 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:09:25.704 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.704 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.704 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.705 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.705 DEBUG:boto:Method: GET 2023-10-12T01:09:25.706 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.706 DEBUG:boto:Data: 2023-10-12T01:09:25.707 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.707 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.708 DEBUG:boto:Params: {} 2023-10-12T01:09:25.708 DEBUG:boto:Token: None 2023-10-12T01:09:25.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak= 2023-10-12T01:09:25.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak=', 'Content-Length': '0'} 2023-10-12T01:09:25.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b72b96fec0e0ebc9-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.713 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.713 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.714 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.714 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.714 DEBUG:boto:Method: GET 2023-10-12T01:09:25.715 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.715 DEBUG:boto:Data: 2023-10-12T01:09:25.716 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.717 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.717 DEBUG:boto:Params: {} 2023-10-12T01:09:25.717 DEBUG:boto:Token: None 2023-10-12T01:09:25.718 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak= 2023-10-12T01:09:25.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak=', 'Content-Length': '0'} 2023-10-12T01:09:25.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d2232aa2bf3c8a-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.722 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.722 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.723 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.723 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.724 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.724 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.724 DEBUG:boto:Data: 2023-10-12T01:09:25.725 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.726 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.726 DEBUG:boto:Params: {} 2023-10-12T01:09:25.727 DEBUG:boto:Token: None 2023-10-12T01:09:25.727 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D9V25gCNoywO9HU7AttJlCEJp4Y= 2023-10-12T01:09:25.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D9V25gCNoywO9HU7AttJlCEJp4Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d766e13de903435b-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.730 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.731 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.731 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.732 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.732 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.732 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.733 DEBUG:boto:Data: 2023-10-12T01:09:25.733 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.734 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.734 DEBUG:boto:Params: {} 2023-10-12T01:09:25.735 DEBUG:boto:Token: None 2023-10-12T01:09:25.735 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:D9V25gCNoywO9HU7AttJlCEJp4Y= 2023-10-12T01:09:25.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:D9V25gCNoywO9HU7AttJlCEJp4Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b9c0af214b850e-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:09:25.739 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.740 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.740 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.740 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.741 DEBUG:boto:Method: GET 2023-10-12T01:09:25.741 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.742 DEBUG:boto:Data: 2023-10-12T01:09:25.742 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.743 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.743 DEBUG:boto:Params: {} 2023-10-12T01:09:25.744 DEBUG:boto:Token: None 2023-10-12T01:09:25.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak= 2023-10-12T01:09:25.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak=', 'Content-Length': '0'} 2023-10-12T01:09:25.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d756de8db30c2b0-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.748 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.748 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.749 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.749 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.749 DEBUG:boto:Method: GET 2023-10-12T01:09:25.750 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.750 DEBUG:boto:Data: 2023-10-12T01:09:25.751 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.752 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.752 DEBUG:boto:Params: {} 2023-10-12T01:09:25.752 DEBUG:boto:Token: None 2023-10-12T01:09:25.753 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:09:25.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak= 2023-10-12T01:09:25.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/6PeplO5QjChTnEs1pJtOLRiPak=', 'Content-Length': '0'} 2023-10-12T01:09:25.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8564bd171c2319-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.798 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.799 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.799 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.800 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.800 DEBUG:boto:Method: GET 2023-10-12T01:09:25.801 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.801 DEBUG:boto:Data: 2023-10-12T01:09:25.802 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.802 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.803 DEBUG:boto:Params: {} 2023-10-12T01:09:25.803 DEBUG:boto:Token: None 2023-10-12T01:09:25.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076613010f3b25995-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.808 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.809 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.809 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.810 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.810 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.811 DEBUG:boto:Method: GET 2023-10-12T01:09:25.811 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.812 DEBUG:boto:Data: 2023-10-12T01:09:25.812 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.813 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.813 DEBUG:boto:Params: {} 2023-10-12T01:09:25.814 DEBUG:boto:Token: None 2023-10-12T01:09:25.814 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc95d44beef5b4b8-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.820 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.820 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.821 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.821 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.822 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:09:25.822 DEBUG:boto:Data: 2023-10-12T01:09:25.823 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.823 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.824 DEBUG:boto:Params: {} 2023-10-12T01:09:25.824 DEBUG:boto:Token: None 2023-10-12T01:09:25.825 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33 2023-10-12T01:09:25.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AzUl5c/knmAkH05hFCj7UV7CtY4= 2023-10-12T01:09:25.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AzUl5c/knmAkH05hFCj7UV7CtY4=', 'Content-Length': '0'} 2023-10-12T01:09:25.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8aef384e93d034-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.828 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.828 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.829 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.829 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:09:25.830 DEBUG:boto:Data: 2023-10-12T01:09:25.830 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.831 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.831 DEBUG:boto:Params: {} 2023-10-12T01:09:25.832 DEBUG:boto:Token: None 2023-10-12T01:09:25.832 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33 2023-10-12T01:09:25.833 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AzUl5c/knmAkH05hFCj7UV7CtY4= 2023-10-12T01:09:25.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AzUl5c/knmAkH05hFCj7UV7CtY4=', 'Content-Length': '0'} 2023-10-12T01:09:25.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c3f516fb135c7d-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:09:25.836 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.836 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.837 DEBUG:boto:Method: GET 2023-10-12T01:09:25.837 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:09:25.838 DEBUG:boto:Data: 2023-10-12T01:09:25.838 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.839 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.839 DEBUG:boto:Params: {} 2023-10-12T01:09:25.840 DEBUG:boto:Token: None 2023-10-12T01:09:25.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33 2023-10-12T01:09:25.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nWpAHp3RyLhtrrxCFwnnuXwvLdk= 2023-10-12T01:09:25.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nWpAHp3RyLhtrrxCFwnnuXwvLdk=', 'Content-Length': '0'} 2023-10-12T01:09:25.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043400fbfb894d76e-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.844 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:09:25.844 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:09:25.844 DEBUG:boto:Method: GET 2023-10-12T01:09:25.845 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:09:25.845 DEBUG:boto:Data: 2023-10-12T01:09:25.846 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.847 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.847 DEBUG:boto:Params: {} 2023-10-12T01:09:25.847 DEBUG:boto:Token: None 2023-10-12T01:09:25.848 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj33 2023-10-12T01:09:25.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nWpAHp3RyLhtrrxCFwnnuXwvLdk= 2023-10-12T01:09:25.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nWpAHp3RyLhtrrxCFwnnuXwvLdk=', 'Content-Length': '0'} 2023-10-12T01:09:25.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcec30a1793812e-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.894 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.895 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.895 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.896 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.896 DEBUG:boto:Method: GET 2023-10-12T01:09:25.897 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.897 DEBUG:boto:Data: 2023-10-12T01:09:25.898 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.899 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.899 DEBUG:boto:Params: {} 2023-10-12T01:09:25.899 DEBUG:boto:Token: None 2023-10-12T01:09:25.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9a1c225ef1f632d-0065274745-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.904 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.905 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:25.905 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:25.906 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.906 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.907 DEBUG:boto:Method: GET 2023-10-12T01:09:25.907 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:09:25.908 DEBUG:boto:Data: 2023-10-12T01:09:25.908 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.909 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.909 DEBUG:boto:Params: {} 2023-10-12T01:09:25.910 DEBUG:boto:Token: None 2023-10-12T01:09:25.910 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/ 2023-10-12T01:09:25.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y= 2023-10-12T01:09:25.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JW+W29ZOfaN+0DEXIHI/WQBKZ0Y=', 'Content-Length': '0'} 2023-10-12T01:09:25.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a52fe39f41eb25-0065274745-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.916 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:25.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:09:25.917 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:25.917 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:25.918 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.918 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.919 DEBUG:boto:Method: GET 2023-10-12T01:09:25.919 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.919 DEBUG:boto:Data: 2023-10-12T01:09:25.920 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.921 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.921 DEBUG:boto:Params: {} 2023-10-12T01:09:25.922 DEBUG:boto:Token: None 2023-10-12T01:09:25.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:28+PhlkzWTh7LIMyFCryEjC4kDs= 2023-10-12T01:09:25.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:28+PhlkzWTh7LIMyFCryEjC4kDs=', 'Content-Length': '0'} 2023-10-12T01:09:25.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a821af91d6a89ff-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.970 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:25.971 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:25.971 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.972 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.973 DEBUG:boto:Method: GET 2023-10-12T01:09:25.973 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.973 DEBUG:boto:Data: 2023-10-12T01:09:25.974 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.975 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.975 DEBUG:boto:Params: {} 2023-10-12T01:09:25.976 DEBUG:boto:Token: None 2023-10-12T01:09:25.976 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.977 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:28+PhlkzWTh7LIMyFCryEjC4kDs= 2023-10-12T01:09:25.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:28+PhlkzWTh7LIMyFCryEjC4kDs=', 'Content-Length': '0'} 2023-10-12T01:09:25.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624620bfcb0922e6-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.980 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:25.980 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:25.981 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.981 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.982 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.982 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.982 DEBUG:boto:Data: 2023-10-12T01:09:25.983 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:25.984 DEBUG:boto:Port: 8001 2023-10-12T01:09:25.984 DEBUG:boto:Params: {} 2023-10-12T01:09:25.985 DEBUG:boto:Token: None 2023-10-12T01:09:25.985 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7lHjTX/MhTTxxDufgDzT0N0yrkE= 2023-10-12T01:09:25.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7lHjTX/MhTTxxDufgDzT0N0yrkE=', 'Content-Length': '0'} 2023-10-12T01:09:25.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d623c9c9b69e1e88-0065274745-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.989 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:25.989 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:25.990 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.990 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.990 DEBUG:boto:Method: HEAD 2023-10-12T01:09:25.991 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.991 DEBUG:boto:Data: 2023-10-12T01:09:25.992 DEBUG:boto:Headers: {} 2023-10-12T01:09:25.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:25.992 DEBUG:boto:Port: 8000 2023-10-12T01:09:25.993 DEBUG:boto:Params: {} 2023-10-12T01:09:25.993 DEBUG:boto:Token: None 2023-10-12T01:09:25.994 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:25 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7lHjTX/MhTTxxDufgDzT0N0yrkE= 2023-10-12T01:09:25.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:25 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7lHjTX/MhTTxxDufgDzT0N0yrkE=', 'Content-Length': '0'} 2023-10-12T01:09:25.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6e45f559af9993-0065274745-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:25.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:09:25.998 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:25.998 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:25.999 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.999 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:25.999 DEBUG:boto:Method: GET 2023-10-12T01:09:26.000 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.000 DEBUG:boto:Data: 2023-10-12T01:09:26.001 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.001 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.002 DEBUG:boto:Params: {} 2023-10-12T01:09:26.002 DEBUG:boto:Token: None 2023-10-12T01:09:26.003 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tA8iTJL4EwRO9xKbV6qn8ai3gS4= 2023-10-12T01:09:26.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tA8iTJL4EwRO9xKbV6qn8ai3gS4=', 'Content-Length': '0'} 2023-10-12T01:09:26.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b040ff30356215da-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.006 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:26.007 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:26.007 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.008 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.008 DEBUG:boto:Method: GET 2023-10-12T01:09:26.008 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.009 DEBUG:boto:Data: 2023-10-12T01:09:26.009 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.010 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.011 DEBUG:boto:Params: {} 2023-10-12T01:09:26.011 DEBUG:boto:Token: None 2023-10-12T01:09:26.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:09:26.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tA8iTJL4EwRO9xKbV6qn8ai3gS4= 2023-10-12T01:09:26.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tA8iTJL4EwRO9xKbV6qn8ai3gS4=', 'Content-Length': '0'} 2023-10-12T01:09:26.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f237566ed53d93ac-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.015 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.015 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.016 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.016 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.017 DEBUG:boto:Method: GET 2023-10-12T01:09:26.017 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.017 DEBUG:boto:Data: 2023-10-12T01:09:26.018 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.019 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.019 DEBUG:boto:Params: {} 2023-10-12T01:09:26.019 DEBUG:boto:Token: None 2023-10-12T01:09:26.020 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78776595ebbd268-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.024 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.025 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.025 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.025 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.026 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.026 DEBUG:boto:Method: GET 2023-10-12T01:09:26.027 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.027 DEBUG:boto:Data: 2023-10-12T01:09:26.027 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.028 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.029 DEBUG:boto:Params: {} 2023-10-12T01:09:26.029 DEBUG:boto:Token: None 2023-10-12T01:09:26.029 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408b7a93080bf04b-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.074 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.075 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:26.076 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:26.076 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.076 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:09:26.077 DEBUG:boto:Data: 2023-10-12T01:09:26.077 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.078 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.078 DEBUG:boto:Params: {} 2023-10-12T01:09:26.079 DEBUG:boto:Token: None 2023-10-12T01:09:26.079 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34 2023-10-12T01:09:26.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WpYbTPsKtRtpbF0UBAzufJ3f2D0= 2023-10-12T01:09:26.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WpYbTPsKtRtpbF0UBAzufJ3f2D0=', 'Content-Length': '0'} 2023-10-12T01:09:26.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a94023eebf14d4f-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.083 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:26.084 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:26.084 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.084 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:09:26.085 DEBUG:boto:Data: 2023-10-12T01:09:26.085 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.086 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.086 DEBUG:boto:Params: {} 2023-10-12T01:09:26.087 DEBUG:boto:Token: None 2023-10-12T01:09:26.087 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34 2023-10-12T01:09:26.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WpYbTPsKtRtpbF0UBAzufJ3f2D0= 2023-10-12T01:09:26.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WpYbTPsKtRtpbF0UBAzufJ3f2D0=', 'Content-Length': '0'} 2023-10-12T01:09:26.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea38d086c737537-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:09:26.091 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:26.091 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:26.092 DEBUG:boto:Method: GET 2023-10-12T01:09:26.092 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:09:26.093 DEBUG:boto:Data: 2023-10-12T01:09:26.093 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.094 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.094 DEBUG:boto:Params: {} 2023-10-12T01:09:26.095 DEBUG:boto:Token: None 2023-10-12T01:09:26.095 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34 2023-10-12T01:09:26.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71zW3akBURwvpPe/tjH5fi8qjhE= 2023-10-12T01:09:26.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71zW3akBURwvpPe/tjH5fi8qjhE=', 'Content-Length': '0'} 2023-10-12T01:09:26.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099efba9fabce3bb1-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.099 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:09:26.099 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:09:26.100 DEBUG:boto:Method: GET 2023-10-12T01:09:26.100 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:09:26.101 DEBUG:boto:Data: 2023-10-12T01:09:26.101 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.102 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.102 DEBUG:boto:Params: {} 2023-10-12T01:09:26.103 DEBUG:boto:Token: None 2023-10-12T01:09:26.103 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj34 2023-10-12T01:09:26.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71zW3akBURwvpPe/tjH5fi8qjhE= 2023-10-12T01:09:26.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71zW3akBURwvpPe/tjH5fi8qjhE=', 'Content-Length': '0'} 2023-10-12T01:09:26.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b4e56ebfb2dd93-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.107 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.107 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.108 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.108 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.109 DEBUG:boto:Method: GET 2023-10-12T01:09:26.109 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.109 DEBUG:boto:Data: 2023-10-12T01:09:26.110 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.111 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.111 DEBUG:boto:Params: {} 2023-10-12T01:09:26.112 DEBUG:boto:Token: None 2023-10-12T01:09:26.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a6add45b21fd984-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.158 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.159 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.160 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.160 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.160 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.161 DEBUG:boto:Method: GET 2023-10-12T01:09:26.161 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:09:26.162 DEBUG:boto:Data: 2023-10-12T01:09:26.162 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.163 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.163 DEBUG:boto:Params: {} 2023-10-12T01:09:26.164 DEBUG:boto:Token: None 2023-10-12T01:09:26.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007055182f83fb07b2-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.170 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:09:26.171 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.172 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.172 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.173 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.173 DEBUG:boto:Method: GET 2023-10-12T01:09:26.174 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.174 DEBUG:boto:Data: 2023-10-12T01:09:26.174 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.175 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.176 DEBUG:boto:Params: {} 2023-10-12T01:09:26.176 DEBUG:boto:Token: None 2023-10-12T01:09:26.177 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ= 2023-10-12T01:09:26.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6b8f9c9596a44d-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.181 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.181 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.182 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.182 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.182 DEBUG:boto:Method: GET 2023-10-12T01:09:26.183 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.183 DEBUG:boto:Data: 2023-10-12T01:09:26.184 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.185 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.185 DEBUG:boto:Params: {} 2023-10-12T01:09:26.185 DEBUG:boto:Token: None 2023-10-12T01:09:26.186 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.186 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ= 2023-10-12T01:09:26.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629588aa042a2c81-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.190 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.190 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.191 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.191 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.191 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.192 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.192 DEBUG:boto:Data: 2023-10-12T01:09:26.193 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.193 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.194 DEBUG:boto:Params: {} 2023-10-12T01:09:26.194 DEBUG:boto:Token: None 2023-10-12T01:09:26.195 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0r5kK64zdRYgakfzB+FRfFwAwhw= 2023-10-12T01:09:26.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0r5kK64zdRYgakfzB+FRfFwAwhw=', 'Content-Length': '0'} 2023-10-12T01:09:26.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1f1e75a3953b85-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.199 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.199 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.199 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.200 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.200 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.201 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.201 DEBUG:boto:Data: 2023-10-12T01:09:26.201 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.202 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.203 DEBUG:boto:Params: {} 2023-10-12T01:09:26.203 DEBUG:boto:Token: None 2023-10-12T01:09:26.204 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0r5kK64zdRYgakfzB+FRfFwAwhw= 2023-10-12T01:09:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0r5kK64zdRYgakfzB+FRfFwAwhw=', 'Content-Length': '0'} 2023-10-12T01:09:26.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c73d85c81364c42-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:09:26.208 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.208 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.208 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.209 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.209 DEBUG:boto:Method: GET 2023-10-12T01:09:26.210 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.210 DEBUG:boto:Data: 2023-10-12T01:09:26.211 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.211 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.212 DEBUG:boto:Params: {} 2023-10-12T01:09:26.212 DEBUG:boto:Token: None 2023-10-12T01:09:26.213 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ= 2023-10-12T01:09:26.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b32c825f931529-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.217 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.217 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.218 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.218 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.219 DEBUG:boto:Method: GET 2023-10-12T01:09:26.219 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.219 DEBUG:boto:Data: 2023-10-12T01:09:26.220 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.221 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.221 DEBUG:boto:Params: {} 2023-10-12T01:09:26.222 DEBUG:boto:Token: None 2023-10-12T01:09:26.222 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:09:26.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ= 2023-10-12T01:09:26.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IT1dy+9bJXbq9WNA7pA5YvUwZhQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f140dbefed4f2cf9-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.226 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.226 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.227 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.227 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.228 DEBUG:boto:Method: GET 2023-10-12T01:09:26.228 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.228 DEBUG:boto:Data: 2023-10-12T01:09:26.229 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.230 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.230 DEBUG:boto:Params: {} 2023-10-12T01:09:26.231 DEBUG:boto:Token: None 2023-10-12T01:09:26.231 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5aa7ec0224b816-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.235 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.236 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.236 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.237 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.237 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.238 DEBUG:boto:Method: GET 2023-10-12T01:09:26.238 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.239 DEBUG:boto:Data: 2023-10-12T01:09:26.239 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.240 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.240 DEBUG:boto:Params: {} 2023-10-12T01:09:26.241 DEBUG:boto:Token: None 2023-10-12T01:09:26.241 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.242 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005adda321ae8efed-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.247 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.247 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.248 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.248 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.249 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:09:26.249 DEBUG:boto:Data: 2023-10-12T01:09:26.249 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.250 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.251 DEBUG:boto:Params: {} 2023-10-12T01:09:26.251 DEBUG:boto:Token: None 2023-10-12T01:09:26.252 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38 2023-10-12T01:09:26.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KfjK6Ou8ybLKx+CrMtQYJIT4A+g= 2023-10-12T01:09:26.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KfjK6Ou8ybLKx+CrMtQYJIT4A+g=', 'Content-Length': '0'} 2023-10-12T01:09:26.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb11a6de9b812689-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.255 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.256 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.256 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.257 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:09:26.257 DEBUG:boto:Data: 2023-10-12T01:09:26.257 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.258 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.259 DEBUG:boto:Params: {} 2023-10-12T01:09:26.259 DEBUG:boto:Token: None 2023-10-12T01:09:26.260 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38 2023-10-12T01:09:26.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KfjK6Ou8ybLKx+CrMtQYJIT4A+g= 2023-10-12T01:09:26.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KfjK6Ou8ybLKx+CrMtQYJIT4A+g=', 'Content-Length': '0'} 2023-10-12T01:09:26.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdcd4549e7cdb544-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:09:26.264 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.264 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.264 DEBUG:boto:Method: GET 2023-10-12T01:09:26.265 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:09:26.265 DEBUG:boto:Data: 2023-10-12T01:09:26.266 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.267 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.267 DEBUG:boto:Params: {} 2023-10-12T01:09:26.267 DEBUG:boto:Token: None 2023-10-12T01:09:26.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38 2023-10-12T01:09:26.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dMiNOXRXHdb4cAh476l8rhPtQDQ= 2023-10-12T01:09:26.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dMiNOXRXHdb4cAh476l8rhPtQDQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2f6c78e83b0d74-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.314 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:09:26.315 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:09:26.315 DEBUG:boto:Method: GET 2023-10-12T01:09:26.316 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:09:26.316 DEBUG:boto:Data: 2023-10-12T01:09:26.317 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.317 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.318 DEBUG:boto:Params: {} 2023-10-12T01:09:26.318 DEBUG:boto:Token: None 2023-10-12T01:09:26.319 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj38 2023-10-12T01:09:26.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dMiNOXRXHdb4cAh476l8rhPtQDQ= 2023-10-12T01:09:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dMiNOXRXHdb4cAh476l8rhPtQDQ=', 'Content-Length': '0'} 2023-10-12T01:09:26.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744f8d66545c12ad-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.323 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.325 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.325 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.326 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.327 DEBUG:boto:Method: GET 2023-10-12T01:09:26.327 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.328 DEBUG:boto:Data: 2023-10-12T01:09:26.328 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.329 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.329 DEBUG:boto:Params: {} 2023-10-12T01:09:26.330 DEBUG:boto:Token: None 2023-10-12T01:09:26.330 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.331 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa420b0d82c1d63b-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.335 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.336 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.336 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.337 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.337 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.338 DEBUG:boto:Method: GET 2023-10-12T01:09:26.338 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:09:26.339 DEBUG:boto:Data: 2023-10-12T01:09:26.339 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.340 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.340 DEBUG:boto:Params: {} 2023-10-12T01:09:26.341 DEBUG:boto:Token: None 2023-10-12T01:09:26.341 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.342 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ca7b131385a8e3-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.347 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:09:26.348 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.348 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.349 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.349 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.350 DEBUG:boto:Method: GET 2023-10-12T01:09:26.350 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.351 DEBUG:boto:Data: 2023-10-12T01:09:26.351 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.352 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.352 DEBUG:boto:Params: {} 2023-10-12T01:09:26.353 DEBUG:boto:Token: None 2023-10-12T01:09:26.353 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI= 2023-10-12T01:09:26.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI=', 'Content-Length': '0'} 2023-10-12T01:09:26.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776168c5dac75dee-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.357 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.358 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.358 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.359 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.359 DEBUG:boto:Method: GET 2023-10-12T01:09:26.359 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.360 DEBUG:boto:Data: 2023-10-12T01:09:26.360 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.361 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.362 DEBUG:boto:Params: {} 2023-10-12T01:09:26.362 DEBUG:boto:Token: None 2023-10-12T01:09:26.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI= 2023-10-12T01:09:26.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI=', 'Content-Length': '0'} 2023-10-12T01:09:26.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d7f9251ceda881-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.367 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.367 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.367 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.368 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.368 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.369 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.369 DEBUG:boto:Data: 2023-10-12T01:09:26.370 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.370 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.371 DEBUG:boto:Params: {} 2023-10-12T01:09:26.371 DEBUG:boto:Token: None 2023-10-12T01:09:26.372 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O1r2tkZEJqc+PXsFNdHZX3r8MLo= 2023-10-12T01:09:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O1r2tkZEJqc+PXsFNdHZX3r8MLo=', 'Content-Length': '0'} 2023-10-12T01:09:26.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c129234162f1ca68-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.375 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.376 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.376 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.377 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.377 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.378 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.378 DEBUG:boto:Data: 2023-10-12T01:09:26.378 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.379 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.380 DEBUG:boto:Params: {} 2023-10-12T01:09:26.380 DEBUG:boto:Token: None 2023-10-12T01:09:26.381 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O1r2tkZEJqc+PXsFNdHZX3r8MLo= 2023-10-12T01:09:26.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O1r2tkZEJqc+PXsFNdHZX3r8MLo=', 'Content-Length': '0'} 2023-10-12T01:09:26.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c5cc43b0b54184-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:09:26.385 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.385 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.386 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.386 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.387 DEBUG:boto:Method: GET 2023-10-12T01:09:26.387 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.387 DEBUG:boto:Data: 2023-10-12T01:09:26.388 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.389 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.389 DEBUG:boto:Params: {} 2023-10-12T01:09:26.390 DEBUG:boto:Token: None 2023-10-12T01:09:26.390 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI= 2023-10-12T01:09:26.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI=', 'Content-Length': '0'} 2023-10-12T01:09:26.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005541afb941ec650-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.434 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.435 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.435 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.436 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.436 DEBUG:boto:Method: GET 2023-10-12T01:09:26.437 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.437 DEBUG:boto:Data: 2023-10-12T01:09:26.438 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.439 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.439 DEBUG:boto:Params: {} 2023-10-12T01:09:26.440 DEBUG:boto:Token: None 2023-10-12T01:09:26.440 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:09:26.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI= 2023-10-12T01:09:26.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sifBBFsRNhn/Yk8tLZprrSQiNXI=', 'Content-Length': '0'} 2023-10-12T01:09:26.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9179caeb603f4b2-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.445 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.445 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.446 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.446 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.447 DEBUG:boto:Method: GET 2023-10-12T01:09:26.447 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.448 DEBUG:boto:Data: 2023-10-12T01:09:26.448 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.449 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.449 DEBUG:boto:Params: {} 2023-10-12T01:09:26.450 DEBUG:boto:Token: None 2023-10-12T01:09:26.450 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063762788b663f538-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.456 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.457 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.457 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.458 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.458 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.459 DEBUG:boto:Method: GET 2023-10-12T01:09:26.459 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.460 DEBUG:boto:Data: 2023-10-12T01:09:26.460 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.461 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.461 DEBUG:boto:Params: {} 2023-10-12T01:09:26.462 DEBUG:boto:Token: None 2023-10-12T01:09:26.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98b77cee0a35087-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.468 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.468 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.469 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.469 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.470 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:09:26.470 DEBUG:boto:Data: 2023-10-12T01:09:26.471 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.471 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.472 DEBUG:boto:Params: {} 2023-10-12T01:09:26.472 DEBUG:boto:Token: None 2023-10-12T01:09:26.473 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44 2023-10-12T01:09:26.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UwBSjg1wZyYB3xbZFAQs3DHkORc= 2023-10-12T01:09:26.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UwBSjg1wZyYB3xbZFAQs3DHkORc=', 'Content-Length': '0'} 2023-10-12T01:09:26.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f34293b564b8ef-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.477 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.477 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.477 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.478 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:09:26.478 DEBUG:boto:Data: 2023-10-12T01:09:26.479 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.480 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.480 DEBUG:boto:Params: {} 2023-10-12T01:09:26.480 DEBUG:boto:Token: None 2023-10-12T01:09:26.481 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44 2023-10-12T01:09:26.481 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UwBSjg1wZyYB3xbZFAQs3DHkORc= 2023-10-12T01:09:26.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UwBSjg1wZyYB3xbZFAQs3DHkORc=', 'Content-Length': '0'} 2023-10-12T01:09:26.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000057f78c151a3f213-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:09:26.485 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.486 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.486 DEBUG:boto:Method: GET 2023-10-12T01:09:26.486 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:09:26.487 DEBUG:boto:Data: 2023-10-12T01:09:26.487 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.488 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.489 DEBUG:boto:Params: {} 2023-10-12T01:09:26.489 DEBUG:boto:Token: None 2023-10-12T01:09:26.490 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44 2023-10-12T01:09:26.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OR/OnQyRP3d9QVqZL12adHMs5zA= 2023-10-12T01:09:26.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OR/OnQyRP3d9QVqZL12adHMs5zA=', 'Content-Length': '0'} 2023-10-12T01:09:26.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebef4039a11fcf0-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.493 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:09:26.494 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:09:26.494 DEBUG:boto:Method: GET 2023-10-12T01:09:26.495 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:09:26.495 DEBUG:boto:Data: 2023-10-12T01:09:26.496 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.497 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.497 DEBUG:boto:Params: {} 2023-10-12T01:09:26.498 DEBUG:boto:Token: None 2023-10-12T01:09:26.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj44 2023-10-12T01:09:26.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OR/OnQyRP3d9QVqZL12adHMs5zA= 2023-10-12T01:09:26.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OR/OnQyRP3d9QVqZL12adHMs5zA=', 'Content-Length': '0'} 2023-10-12T01:09:26.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b5c7586b90a03f-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.502 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.502 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.503 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.503 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.504 DEBUG:boto:Method: GET 2023-10-12T01:09:26.504 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.505 DEBUG:boto:Data: 2023-10-12T01:09:26.505 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.506 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.506 DEBUG:boto:Params: {} 2023-10-12T01:09:26.507 DEBUG:boto:Token: None 2023-10-12T01:09:26.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009581afa79461e1d0-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.512 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.512 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.513 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.513 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.514 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.514 DEBUG:boto:Method: GET 2023-10-12T01:09:26.515 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:09:26.515 DEBUG:boto:Data: 2023-10-12T01:09:26.516 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.517 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.517 DEBUG:boto:Params: {} 2023-10-12T01:09:26.517 DEBUG:boto:Token: None 2023-10-12T01:09:26.518 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072540c5f8ea22f9e-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.524 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:09:26.525 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.525 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.526 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.526 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.526 DEBUG:boto:Method: GET 2023-10-12T01:09:26.527 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.527 DEBUG:boto:Data: 2023-10-12T01:09:26.528 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.529 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.529 DEBUG:boto:Params: {} 2023-10-12T01:09:26.530 DEBUG:boto:Token: None 2023-10-12T01:09:26.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI= 2023-10-12T01:09:26.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI=', 'Content-Length': '0'} 2023-10-12T01:09:26.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9eba1da922486d-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.583 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.583 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.584 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.584 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.585 DEBUG:boto:Method: GET 2023-10-12T01:09:26.585 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.586 DEBUG:boto:Data: 2023-10-12T01:09:26.586 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.587 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.587 DEBUG:boto:Params: {} 2023-10-12T01:09:26.588 DEBUG:boto:Token: None 2023-10-12T01:09:26.588 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI= 2023-10-12T01:09:26.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI=', 'Content-Length': '0'} 2023-10-12T01:09:26.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027fe60cd3b118e93-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.592 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.593 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.593 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.594 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.594 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.595 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.595 DEBUG:boto:Data: 2023-10-12T01:09:26.595 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.596 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.597 DEBUG:boto:Params: {} 2023-10-12T01:09:26.597 DEBUG:boto:Token: None 2023-10-12T01:09:26.598 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cvNGIKWfwA6JHAxPm5Fogp+vB5s= 2023-10-12T01:09:26.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cvNGIKWfwA6JHAxPm5Fogp+vB5s=', 'Content-Length': '0'} 2023-10-12T01:09:26.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006101375b29af3d0c-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.601 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.602 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.602 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.603 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.603 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.603 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.604 DEBUG:boto:Data: 2023-10-12T01:09:26.604 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.605 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.606 DEBUG:boto:Params: {} 2023-10-12T01:09:26.606 DEBUG:boto:Token: None 2023-10-12T01:09:26.606 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cvNGIKWfwA6JHAxPm5Fogp+vB5s= 2023-10-12T01:09:26.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cvNGIKWfwA6JHAxPm5Fogp+vB5s=', 'Content-Length': '0'} 2023-10-12T01:09:26.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a49b9a7335278a9-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:09:26.610 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.611 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.612 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.612 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.613 DEBUG:boto:Method: GET 2023-10-12T01:09:26.613 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.613 DEBUG:boto:Data: 2023-10-12T01:09:26.614 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.615 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.615 DEBUG:boto:Params: {} 2023-10-12T01:09:26.616 DEBUG:boto:Token: None 2023-10-12T01:09:26.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI= 2023-10-12T01:09:26.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI=', 'Content-Length': '0'} 2023-10-12T01:09:26.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9007ec93b09944b-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.620 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.620 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.621 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.621 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.622 DEBUG:boto:Method: GET 2023-10-12T01:09:26.622 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.623 DEBUG:boto:Data: 2023-10-12T01:09:26.623 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.624 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.625 DEBUG:boto:Params: {} 2023-10-12T01:09:26.625 DEBUG:boto:Token: None 2023-10-12T01:09:26.626 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:09:26.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI= 2023-10-12T01:09:26.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ElwCg9YCvKUq1zbJ8qZ1KEvqUrI=', 'Content-Length': '0'} 2023-10-12T01:09:26.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef480ebd0a7d2af-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.629 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.630 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.630 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.630 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.631 DEBUG:boto:Method: GET 2023-10-12T01:09:26.631 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.632 DEBUG:boto:Data: 2023-10-12T01:09:26.632 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.633 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.633 DEBUG:boto:Params: {} 2023-10-12T01:09:26.634 DEBUG:boto:Token: None 2023-10-12T01:09:26.634 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030b038fa40195d26-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.639 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.639 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.640 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.640 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.641 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.641 DEBUG:boto:Method: GET 2023-10-12T01:09:26.642 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.642 DEBUG:boto:Data: 2023-10-12T01:09:26.642 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.643 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.644 DEBUG:boto:Params: {} 2023-10-12T01:09:26.644 DEBUG:boto:Token: None 2023-10-12T01:09:26.645 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eec24e0c7243a98-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.650 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.651 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.651 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.652 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.652 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:09:26.652 DEBUG:boto:Data: 2023-10-12T01:09:26.653 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.654 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.654 DEBUG:boto:Params: {} 2023-10-12T01:09:26.655 DEBUG:boto:Token: None 2023-10-12T01:09:26.655 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46 2023-10-12T01:09:26.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8eJytr4qLndMxhTL9hdefwhcdNA= 2023-10-12T01:09:26.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8eJytr4qLndMxhTL9hdefwhcdNA=', 'Content-Length': '0'} 2023-10-12T01:09:26.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcfaca71f61ab094-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.659 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.659 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.659 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.660 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:09:26.660 DEBUG:boto:Data: 2023-10-12T01:09:26.661 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.662 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.662 DEBUG:boto:Params: {} 2023-10-12T01:09:26.663 DEBUG:boto:Token: None 2023-10-12T01:09:26.663 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46 2023-10-12T01:09:26.664 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8eJytr4qLndMxhTL9hdefwhcdNA= 2023-10-12T01:09:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8eJytr4qLndMxhTL9hdefwhcdNA=', 'Content-Length': '0'} 2023-10-12T01:09:26.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6b8efbb7451647-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:09:26.667 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.668 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.668 DEBUG:boto:Method: GET 2023-10-12T01:09:26.669 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:09:26.669 DEBUG:boto:Data: 2023-10-12T01:09:26.670 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.670 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.671 DEBUG:boto:Params: {} 2023-10-12T01:09:26.671 DEBUG:boto:Token: None 2023-10-12T01:09:26.672 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46 2023-10-12T01:09:26.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tD2EkYJcm2Lc7nonLouKvL8qljM= 2023-10-12T01:09:26.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tD2EkYJcm2Lc7nonLouKvL8qljM=', 'Content-Length': '0'} 2023-10-12T01:09:26.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af25b6c4ed7064ca-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.718 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:09:26.719 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:09:26.719 DEBUG:boto:Method: GET 2023-10-12T01:09:26.720 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:09:26.720 DEBUG:boto:Data: 2023-10-12T01:09:26.721 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.721 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.722 DEBUG:boto:Params: {} 2023-10-12T01:09:26.722 DEBUG:boto:Token: None 2023-10-12T01:09:26.723 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj46 2023-10-12T01:09:26.723 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tD2EkYJcm2Lc7nonLouKvL8qljM= 2023-10-12T01:09:26.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tD2EkYJcm2Lc7nonLouKvL8qljM=', 'Content-Length': '0'} 2023-10-12T01:09:26.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9b4ee3e845d6b3-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.727 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.727 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.727 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.728 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.728 DEBUG:boto:Method: GET 2023-10-12T01:09:26.728 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.729 DEBUG:boto:Data: 2023-10-12T01:09:26.729 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.730 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.730 DEBUG:boto:Params: {} 2023-10-12T01:09:26.731 DEBUG:boto:Token: None 2023-10-12T01:09:26.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000075f018b3f40dc63-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.738 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.738 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.739 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.739 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.740 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.740 DEBUG:boto:Method: GET 2023-10-12T01:09:26.741 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:09:26.741 DEBUG:boto:Data: 2023-10-12T01:09:26.741 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.742 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.743 DEBUG:boto:Params: {} 2023-10-12T01:09:26.743 DEBUG:boto:Token: None 2023-10-12T01:09:26.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005618caeb9302897b-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.749 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:09:26.750 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.750 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.751 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.751 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.752 DEBUG:boto:Method: GET 2023-10-12T01:09:26.752 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.753 DEBUG:boto:Data: 2023-10-12T01:09:26.753 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.754 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.754 DEBUG:boto:Params: {} 2023-10-12T01:09:26.755 DEBUG:boto:Token: None 2023-10-12T01:09:26.755 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4= 2023-10-12T01:09:26.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4=', 'Content-Length': '0'} 2023-10-12T01:09:26.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b83eda436e723d-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.759 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.759 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.760 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.760 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.760 DEBUG:boto:Method: GET 2023-10-12T01:09:26.761 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.761 DEBUG:boto:Data: 2023-10-12T01:09:26.762 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.763 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.763 DEBUG:boto:Params: {} 2023-10-12T01:09:26.764 DEBUG:boto:Token: None 2023-10-12T01:09:26.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4= 2023-10-12T01:09:26.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4=', 'Content-Length': '0'} 2023-10-12T01:09:26.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa74dabc31b98307-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.810 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.811 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.811 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.812 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.812 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.812 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.813 DEBUG:boto:Data: 2023-10-12T01:09:26.813 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.814 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.814 DEBUG:boto:Params: {} 2023-10-12T01:09:26.815 DEBUG:boto:Token: None 2023-10-12T01:09:26.815 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.816 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ZBdA1Dag5hZYUHRP5G6UtdHtik= 2023-10-12T01:09:26.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ZBdA1Dag5hZYUHRP5G6UtdHtik=', 'Content-Length': '0'} 2023-10-12T01:09:26.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a82b190eaa6e088-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.819 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.819 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.820 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.820 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.821 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.821 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.822 DEBUG:boto:Data: 2023-10-12T01:09:26.822 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.823 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.823 DEBUG:boto:Params: {} 2023-10-12T01:09:26.824 DEBUG:boto:Token: None 2023-10-12T01:09:26.824 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ZBdA1Dag5hZYUHRP5G6UtdHtik= 2023-10-12T01:09:26.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ZBdA1Dag5hZYUHRP5G6UtdHtik=', 'Content-Length': '0'} 2023-10-12T01:09:26.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018874497cf33f6bf-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:09:26.829 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.829 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.830 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.830 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.830 DEBUG:boto:Method: GET 2023-10-12T01:09:26.831 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.831 DEBUG:boto:Data: 2023-10-12T01:09:26.832 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.833 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.833 DEBUG:boto:Params: {} 2023-10-12T01:09:26.833 DEBUG:boto:Token: None 2023-10-12T01:09:26.834 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4= 2023-10-12T01:09:26.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4=', 'Content-Length': '0'} 2023-10-12T01:09:26.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070952ac3e0026461-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.878 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.879 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.879 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.880 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.880 DEBUG:boto:Method: GET 2023-10-12T01:09:26.881 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.881 DEBUG:boto:Data: 2023-10-12T01:09:26.882 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.882 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.883 DEBUG:boto:Params: {} 2023-10-12T01:09:26.883 DEBUG:boto:Token: None 2023-10-12T01:09:26.884 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:09:26.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4= 2023-10-12T01:09:26.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Wc0ea2hcaMQH6Pda94Mqoka1gm4=', 'Content-Length': '0'} 2023-10-12T01:09:26.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cb6ba118e0ac75f-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.888 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.888 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.889 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.889 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.889 DEBUG:boto:Method: GET 2023-10-12T01:09:26.890 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.890 DEBUG:boto:Data: 2023-10-12T01:09:26.891 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.892 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.892 DEBUG:boto:Params: {} 2023-10-12T01:09:26.892 DEBUG:boto:Token: None 2023-10-12T01:09:26.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a7c7d3fbc2a0d0-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.897 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.898 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.898 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.899 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.899 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.900 DEBUG:boto:Method: GET 2023-10-12T01:09:26.900 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.900 DEBUG:boto:Data: 2023-10-12T01:09:26.901 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.902 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.903 DEBUG:boto:Params: {} 2023-10-12T01:09:26.903 DEBUG:boto:Token: None 2023-10-12T01:09:26.903 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7f537f708895276-0065274746-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.909 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.910 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.910 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.911 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.911 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:09:26.911 DEBUG:boto:Data: 2023-10-12T01:09:26.912 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.912 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.913 DEBUG:boto:Params: {} 2023-10-12T01:09:26.913 DEBUG:boto:Token: None 2023-10-12T01:09:26.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51 2023-10-12T01:09:26.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29ldoA1cboXiW9OBvJ6FzJU5d9c= 2023-10-12T01:09:26.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29ldoA1cboXiW9OBvJ6FzJU5d9c=', 'Content-Length': '0'} 2023-10-12T01:09:26.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad80c7008d1186cd-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.917 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.918 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.918 DEBUG:boto:Method: HEAD 2023-10-12T01:09:26.919 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:09:26.919 DEBUG:boto:Data: 2023-10-12T01:09:26.919 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.920 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.920 DEBUG:boto:Params: {} 2023-10-12T01:09:26.921 DEBUG:boto:Token: None 2023-10-12T01:09:26.921 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51 2023-10-12T01:09:26.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29ldoA1cboXiW9OBvJ6FzJU5d9c= 2023-10-12T01:09:26.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29ldoA1cboXiW9OBvJ6FzJU5d9c=', 'Content-Length': '0'} 2023-10-12T01:09:26.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2aa33211d5f5e94-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:09:26.925 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.926 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.926 DEBUG:boto:Method: GET 2023-10-12T01:09:26.927 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:09:26.927 DEBUG:boto:Data: 2023-10-12T01:09:26.927 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.928 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.929 DEBUG:boto:Params: {} 2023-10-12T01:09:26.929 DEBUG:boto:Token: None 2023-10-12T01:09:26.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51 2023-10-12T01:09:26.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f/l+sWewZ/IxO5/YIeMQyY0uZ04= 2023-10-12T01:09:26.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f/l+sWewZ/IxO5/YIeMQyY0uZ04=', 'Content-Length': '0'} 2023-10-12T01:09:26.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000234240cd185049b8-0065274746-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.933 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:09:26.934 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:09:26.934 DEBUG:boto:Method: GET 2023-10-12T01:09:26.935 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:09:26.935 DEBUG:boto:Data: 2023-10-12T01:09:26.936 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.937 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.937 DEBUG:boto:Params: {} 2023-10-12T01:09:26.937 DEBUG:boto:Token: None 2023-10-12T01:09:26.938 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/obj51 2023-10-12T01:09:26.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f/l+sWewZ/IxO5/YIeMQyY0uZ04= 2023-10-12T01:09:26.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f/l+sWewZ/IxO5/YIeMQyY0uZ04=', 'Content-Length': '0'} 2023-10-12T01:09:26.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028dea640bf6fbd5c-0065274746-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.982 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.983 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.983 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.984 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.984 DEBUG:boto:Method: GET 2023-10-12T01:09:26.985 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.985 DEBUG:boto:Data: 2023-10-12T01:09:26.986 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:26.986 DEBUG:boto:Port: 8001 2023-10-12T01:09:26.987 DEBUG:boto:Params: {} 2023-10-12T01:09:26.987 DEBUG:boto:Token: None 2023-10-12T01:09:26.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:26.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:26.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bed61e58bb71b22-0065274746-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:26.992 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:26.993 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:26.993 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:26.994 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.994 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.995 DEBUG:boto:Method: GET 2023-10-12T01:09:26.995 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:09:26.996 DEBUG:boto:Data: 2023-10-12T01:09:26.996 DEBUG:boto:Headers: {} 2023-10-12T01:09:26.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:26.997 DEBUG:boto:Port: 8000 2023-10-12T01:09:26.997 DEBUG:boto:Params: {} 2023-10-12T01:09:26.998 DEBUG:boto:Token: None 2023-10-12T01:09:26.999 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:26 GMT /wafmoa-59/ 2023-10-12T01:09:26.999 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE= 2023-10-12T01:09:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i6xCGRJSSnNKsH2G4zqm+cgG8oE=', 'Content-Length': '0'} 2023-10-12T01:09:27.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8717e9864996bf6-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.004 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:09:27.005 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.006 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.006 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.007 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.007 DEBUG:boto:Method: GET 2023-10-12T01:09:27.007 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.008 DEBUG:boto:Data: 2023-10-12T01:09:27.008 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.009 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.010 DEBUG:boto:Params: {} 2023-10-12T01:09:27.010 DEBUG:boto:Token: None 2023-10-12T01:09:27.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0= 2023-10-12T01:09:27.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0=', 'Content-Length': '0'} 2023-10-12T01:09:27.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17371665e68fe56-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.015 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.015 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.016 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.016 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.016 DEBUG:boto:Method: GET 2023-10-12T01:09:27.017 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.017 DEBUG:boto:Data: 2023-10-12T01:09:27.018 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.019 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.019 DEBUG:boto:Params: {} 2023-10-12T01:09:27.019 DEBUG:boto:Token: None 2023-10-12T01:09:27.020 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0= 2023-10-12T01:09:27.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0=', 'Content-Length': '0'} 2023-10-12T01:09:27.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001887a2f37f980fb2-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.024 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.024 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.024 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.025 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.025 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.026 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.026 DEBUG:boto:Data: 2023-10-12T01:09:27.026 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.027 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.028 DEBUG:boto:Params: {} 2023-10-12T01:09:27.028 DEBUG:boto:Token: None 2023-10-12T01:09:27.029 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hgDHdtF2YH5R2VIfpbMyPcmVqVE= 2023-10-12T01:09:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hgDHdtF2YH5R2VIfpbMyPcmVqVE=', 'Content-Length': '0'} 2023-10-12T01:09:27.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e073642dee20a4-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.032 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.032 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.033 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.033 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.034 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.034 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.034 DEBUG:boto:Data: 2023-10-12T01:09:27.035 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.036 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.036 DEBUG:boto:Params: {} 2023-10-12T01:09:27.036 DEBUG:boto:Token: None 2023-10-12T01:09:27.037 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.037 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hgDHdtF2YH5R2VIfpbMyPcmVqVE= 2023-10-12T01:09:27.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hgDHdtF2YH5R2VIfpbMyPcmVqVE=', 'Content-Length': '0'} 2023-10-12T01:09:27.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009837688f5a880fa8-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:09:27.041 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.041 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.042 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.042 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.043 DEBUG:boto:Method: GET 2023-10-12T01:09:27.043 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.044 DEBUG:boto:Data: 2023-10-12T01:09:27.044 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.045 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.046 DEBUG:boto:Params: {} 2023-10-12T01:09:27.046 DEBUG:boto:Token: None 2023-10-12T01:09:27.047 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0= 2023-10-12T01:09:27.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0=', 'Content-Length': '0'} 2023-10-12T01:09:27.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7084152e1d3c80c-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.050 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.051 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.051 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.051 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.052 DEBUG:boto:Method: GET 2023-10-12T01:09:27.052 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.053 DEBUG:boto:Data: 2023-10-12T01:09:27.053 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.054 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.055 DEBUG:boto:Params: {} 2023-10-12T01:09:27.055 DEBUG:boto:Token: None 2023-10-12T01:09:27.056 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:09:27.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0= 2023-10-12T01:09:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ns3107XBYNkMDUzctnREOVAkoe0=', 'Content-Length': '0'} 2023-10-12T01:09:27.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57602f1b7a22bcf-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.103 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.103 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.104 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.104 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.105 DEBUG:boto:Method: GET 2023-10-12T01:09:27.105 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.106 DEBUG:boto:Data: 2023-10-12T01:09:27.106 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.107 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.107 DEBUG:boto:Params: {} 2023-10-12T01:09:27.108 DEBUG:boto:Token: None 2023-10-12T01:09:27.108 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.109 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043ab78c54e5f03f3-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.113 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.113 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.114 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.114 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.115 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.115 DEBUG:boto:Method: GET 2023-10-12T01:09:27.116 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.116 DEBUG:boto:Data: 2023-10-12T01:09:27.116 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.117 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.118 DEBUG:boto:Params: {} 2023-10-12T01:09:27.118 DEBUG:boto:Token: None 2023-10-12T01:09:27.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a486b8deff8e4369-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.124 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.125 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.125 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.126 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.126 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:09:27.127 DEBUG:boto:Data: 2023-10-12T01:09:27.127 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.128 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.128 DEBUG:boto:Params: {} 2023-10-12T01:09:27.129 DEBUG:boto:Token: None 2023-10-12T01:09:27.129 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55 2023-10-12T01:09:27.130 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFc9AGiqPeLBv7VnEmLsQGS1plA= 2023-10-12T01:09:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFc9AGiqPeLBv7VnEmLsQGS1plA=', 'Content-Length': '0'} 2023-10-12T01:09:27.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002105d39686f5d22c-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.133 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.133 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.134 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.134 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:09:27.134 DEBUG:boto:Data: 2023-10-12T01:09:27.135 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.136 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.136 DEBUG:boto:Params: {} 2023-10-12T01:09:27.137 DEBUG:boto:Token: None 2023-10-12T01:09:27.137 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55 2023-10-12T01:09:27.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CFc9AGiqPeLBv7VnEmLsQGS1plA= 2023-10-12T01:09:27.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CFc9AGiqPeLBv7VnEmLsQGS1plA=', 'Content-Length': '0'} 2023-10-12T01:09:27.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412a0abed9a7e000-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:09:27.141 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.141 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.142 DEBUG:boto:Method: GET 2023-10-12T01:09:27.142 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:09:27.143 DEBUG:boto:Data: 2023-10-12T01:09:27.143 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.144 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.144 DEBUG:boto:Params: {} 2023-10-12T01:09:27.145 DEBUG:boto:Token: None 2023-10-12T01:09:27.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55 2023-10-12T01:09:27.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dcRuG9JdtFRO2lPao8XEUGqyH/Y= 2023-10-12T01:09:27.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dcRuG9JdtFRO2lPao8XEUGqyH/Y=', 'Content-Length': '0'} 2023-10-12T01:09:27.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000251285a331bdb370-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.194 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:09:27.195 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:09:27.195 DEBUG:boto:Method: GET 2023-10-12T01:09:27.196 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:09:27.196 DEBUG:boto:Data: 2023-10-12T01:09:27.197 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.198 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.199 DEBUG:boto:Params: {} 2023-10-12T01:09:27.199 DEBUG:boto:Token: None 2023-10-12T01:09:27.200 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj55 2023-10-12T01:09:27.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dcRuG9JdtFRO2lPao8XEUGqyH/Y= 2023-10-12T01:09:27.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dcRuG9JdtFRO2lPao8XEUGqyH/Y=', 'Content-Length': '0'} 2023-10-12T01:09:27.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004929acd2f97d3fa2-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.205 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.205 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.206 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.206 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.207 DEBUG:boto:Method: GET 2023-10-12T01:09:27.207 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.208 DEBUG:boto:Data: 2023-10-12T01:09:27.208 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.209 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.209 DEBUG:boto:Params: {} 2023-10-12T01:09:27.210 DEBUG:boto:Token: None 2023-10-12T01:09:27.210 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.211 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bcbff040c777360-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.214 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.215 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.216 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.216 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.217 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.217 DEBUG:boto:Method: GET 2023-10-12T01:09:27.217 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:09:27.218 DEBUG:boto:Data: 2023-10-12T01:09:27.218 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.219 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.220 DEBUG:boto:Params: {} 2023-10-12T01:09:27.220 DEBUG:boto:Token: None 2023-10-12T01:09:27.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae0d0e2efe44dad-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.270 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:09:27.272 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.272 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.272 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.273 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.273 DEBUG:boto:Method: GET 2023-10-12T01:09:27.274 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.274 DEBUG:boto:Data: 2023-10-12T01:09:27.274 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.275 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.276 DEBUG:boto:Params: {} 2023-10-12T01:09:27.276 DEBUG:boto:Token: None 2023-10-12T01:09:27.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM= 2023-10-12T01:09:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM=', 'Content-Length': '0'} 2023-10-12T01:09:27.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b79c8ca82f46a56-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.280 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.281 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.281 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.281 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.282 DEBUG:boto:Method: GET 2023-10-12T01:09:27.282 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.283 DEBUG:boto:Data: 2023-10-12T01:09:27.283 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.284 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.284 DEBUG:boto:Params: {} 2023-10-12T01:09:27.285 DEBUG:boto:Token: None 2023-10-12T01:09:27.285 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM= 2023-10-12T01:09:27.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM=', 'Content-Length': '0'} 2023-10-12T01:09:27.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4952b69c859352f-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.289 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.289 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.290 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.290 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.291 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.291 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.291 DEBUG:boto:Data: 2023-10-12T01:09:27.292 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.293 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.293 DEBUG:boto:Params: {} 2023-10-12T01:09:27.293 DEBUG:boto:Token: None 2023-10-12T01:09:27.294 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.294 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ynbOXvx0pVkGRvrfv4dggoo5mzE= 2023-10-12T01:09:27.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ynbOXvx0pVkGRvrfv4dggoo5mzE=', 'Content-Length': '0'} 2023-10-12T01:09:27.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f847a990f08007b8-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.297 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.298 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.298 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.299 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.299 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.299 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.300 DEBUG:boto:Data: 2023-10-12T01:09:27.300 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.301 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.302 DEBUG:boto:Params: {} 2023-10-12T01:09:27.302 DEBUG:boto:Token: None 2023-10-12T01:09:27.303 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ynbOXvx0pVkGRvrfv4dggoo5mzE= 2023-10-12T01:09:27.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ynbOXvx0pVkGRvrfv4dggoo5mzE=', 'Content-Length': '0'} 2023-10-12T01:09:27.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ebb22ff0501f210-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:09:27.307 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.307 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.308 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.308 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.308 DEBUG:boto:Method: GET 2023-10-12T01:09:27.309 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.309 DEBUG:boto:Data: 2023-10-12T01:09:27.310 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.310 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.311 DEBUG:boto:Params: {} 2023-10-12T01:09:27.311 DEBUG:boto:Token: None 2023-10-12T01:09:27.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.312 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM= 2023-10-12T01:09:27.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM=', 'Content-Length': '0'} 2023-10-12T01:09:27.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fecd7e91cf2f23b3-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.315 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.316 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.316 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.317 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.317 DEBUG:boto:Method: GET 2023-10-12T01:09:27.317 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.318 DEBUG:boto:Data: 2023-10-12T01:09:27.318 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.319 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.319 DEBUG:boto:Params: {} 2023-10-12T01:09:27.320 DEBUG:boto:Token: None 2023-10-12T01:09:27.320 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:09:27.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM= 2023-10-12T01:09:27.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d+EglVUXhPueDU32Yl2YkS/oewM=', 'Content-Length': '0'} 2023-10-12T01:09:27.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e1b35986055093-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.324 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.325 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.325 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.326 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.326 DEBUG:boto:Method: GET 2023-10-12T01:09:27.327 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.327 DEBUG:boto:Data: 2023-10-12T01:09:27.327 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.328 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.329 DEBUG:boto:Params: {} 2023-10-12T01:09:27.329 DEBUG:boto:Token: None 2023-10-12T01:09:27.330 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017072225ce89aa41-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.334 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.335 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.335 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.336 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.336 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.337 DEBUG:boto:Method: GET 2023-10-12T01:09:27.337 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.337 DEBUG:boto:Data: 2023-10-12T01:09:27.338 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.339 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.339 DEBUG:boto:Params: {} 2023-10-12T01:09:27.339 DEBUG:boto:Token: None 2023-10-12T01:09:27.340 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148d3059308e1d1a-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.345 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.346 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.346 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.349 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.350 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:09:27.350 DEBUG:boto:Data: 2023-10-12T01:09:27.351 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.351 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.352 DEBUG:boto:Params: {} 2023-10-12T01:09:27.352 DEBUG:boto:Token: None 2023-10-12T01:09:27.353 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57 2023-10-12T01:09:27.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E0l6Y677bZWa6lEbcwseeziLXaI= 2023-10-12T01:09:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E0l6Y677bZWa6lEbcwseeziLXaI=', 'Content-Length': '0'} 2023-10-12T01:09:27.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047321779367f0399-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.356 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.356 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.357 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.357 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:09:27.358 DEBUG:boto:Data: 2023-10-12T01:09:27.358 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.359 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.359 DEBUG:boto:Params: {} 2023-10-12T01:09:27.360 DEBUG:boto:Token: None 2023-10-12T01:09:27.360 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57 2023-10-12T01:09:27.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E0l6Y677bZWa6lEbcwseeziLXaI= 2023-10-12T01:09:27.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E0l6Y677bZWa6lEbcwseeziLXaI=', 'Content-Length': '0'} 2023-10-12T01:09:27.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f37b017055b849-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:09:27.364 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.365 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.365 DEBUG:boto:Method: GET 2023-10-12T01:09:27.366 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:09:27.366 DEBUG:boto:Data: 2023-10-12T01:09:27.367 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.367 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.368 DEBUG:boto:Params: {} 2023-10-12T01:09:27.368 DEBUG:boto:Token: None 2023-10-12T01:09:27.369 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57 2023-10-12T01:09:27.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XbBHvAzZk6pPzU/mmIx78m0MvFc= 2023-10-12T01:09:27.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XbBHvAzZk6pPzU/mmIx78m0MvFc=', 'Content-Length': '0'} 2023-10-12T01:09:27.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe170d63be87e1ac-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.372 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:09:27.373 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:09:27.373 DEBUG:boto:Method: GET 2023-10-12T01:09:27.374 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:09:27.374 DEBUG:boto:Data: 2023-10-12T01:09:27.375 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.376 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.376 DEBUG:boto:Params: {} 2023-10-12T01:09:27.377 DEBUG:boto:Token: None 2023-10-12T01:09:27.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj57 2023-10-12T01:09:27.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XbBHvAzZk6pPzU/mmIx78m0MvFc= 2023-10-12T01:09:27.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XbBHvAzZk6pPzU/mmIx78m0MvFc=', 'Content-Length': '0'} 2023-10-12T01:09:27.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c29111507a9cf9-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.426 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.427 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.427 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.428 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.428 DEBUG:boto:Method: GET 2023-10-12T01:09:27.429 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.429 DEBUG:boto:Data: 2023-10-12T01:09:27.429 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.430 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.431 DEBUG:boto:Params: {} 2023-10-12T01:09:27.431 DEBUG:boto:Token: None 2023-10-12T01:09:27.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b917edd8bfab483-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.436 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.437 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.438 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.438 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.438 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.439 DEBUG:boto:Method: GET 2023-10-12T01:09:27.439 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:09:27.440 DEBUG:boto:Data: 2023-10-12T01:09:27.440 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.441 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.441 DEBUG:boto:Params: {} 2023-10-12T01:09:27.442 DEBUG:boto:Token: None 2023-10-12T01:09:27.442 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224ef568d2681031-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.448 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:09:27.449 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.449 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.450 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.450 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.451 DEBUG:boto:Method: GET 2023-10-12T01:09:27.451 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.451 DEBUG:boto:Data: 2023-10-12T01:09:27.452 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.453 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.453 DEBUG:boto:Params: {} 2023-10-12T01:09:27.454 DEBUG:boto:Token: None 2023-10-12T01:09:27.454 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw= 2023-10-12T01:09:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw=', 'Content-Length': '0'} 2023-10-12T01:09:27.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009395f45be5661985-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.502 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.503 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.503 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.504 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.504 DEBUG:boto:Method: GET 2023-10-12T01:09:27.505 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.505 DEBUG:boto:Data: 2023-10-12T01:09:27.506 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.506 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.507 DEBUG:boto:Params: {} 2023-10-12T01:09:27.507 DEBUG:boto:Token: None 2023-10-12T01:09:27.508 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw= 2023-10-12T01:09:27.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw=', 'Content-Length': '0'} 2023-10-12T01:09:27.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b052d03d5885c1c-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.512 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.512 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.513 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.513 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.514 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.514 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.515 DEBUG:boto:Data: 2023-10-12T01:09:27.515 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.516 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.516 DEBUG:boto:Params: {} 2023-10-12T01:09:27.517 DEBUG:boto:Token: None 2023-10-12T01:09:27.517 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z22lm64VAivV8uujs+K5V0Dpxao= 2023-10-12T01:09:27.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z22lm64VAivV8uujs+K5V0Dpxao=', 'Content-Length': '0'} 2023-10-12T01:09:27.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017990ad6a2c98101-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.521 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.521 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.521 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.522 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.522 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.523 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.523 DEBUG:boto:Data: 2023-10-12T01:09:27.524 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.524 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.525 DEBUG:boto:Params: {} 2023-10-12T01:09:27.525 DEBUG:boto:Token: None 2023-10-12T01:09:27.526 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z22lm64VAivV8uujs+K5V0Dpxao= 2023-10-12T01:09:27.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z22lm64VAivV8uujs+K5V0Dpxao=', 'Content-Length': '0'} 2023-10-12T01:09:27.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dacfa39180c2136-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:09:27.529 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.530 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.530 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.531 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.531 DEBUG:boto:Method: GET 2023-10-12T01:09:27.532 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.532 DEBUG:boto:Data: 2023-10-12T01:09:27.532 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.533 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.534 DEBUG:boto:Params: {} 2023-10-12T01:09:27.534 DEBUG:boto:Token: None 2023-10-12T01:09:27.535 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw= 2023-10-12T01:09:27.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw=', 'Content-Length': '0'} 2023-10-12T01:09:27.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db50adeaae54f861-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.538 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.539 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.539 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.540 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.540 DEBUG:boto:Method: GET 2023-10-12T01:09:27.540 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.541 DEBUG:boto:Data: 2023-10-12T01:09:27.541 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.542 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.543 DEBUG:boto:Params: {} 2023-10-12T01:09:27.543 DEBUG:boto:Token: None 2023-10-12T01:09:27.544 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:09:27.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw= 2023-10-12T01:09:27.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:faUgb35ttJ8PwcDA9tV/BqJBuaw=', 'Content-Length': '0'} 2023-10-12T01:09:27.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2bf320301dfc8ff-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.547 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.548 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.548 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.549 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.549 DEBUG:boto:Method: GET 2023-10-12T01:09:27.550 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.550 DEBUG:boto:Data: 2023-10-12T01:09:27.550 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.551 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.553 DEBUG:boto:Params: {} 2023-10-12T01:09:27.554 DEBUG:boto:Token: None 2023-10-12T01:09:27.554 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.555 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ae6f81c77baaafb-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.558 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.559 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.559 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.560 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.560 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.561 DEBUG:boto:Method: GET 2023-10-12T01:09:27.561 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.561 DEBUG:boto:Data: 2023-10-12T01:09:27.562 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.563 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.563 DEBUG:boto:Params: {} 2023-10-12T01:09:27.564 DEBUG:boto:Token: None 2023-10-12T01:09:27.564 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003270183f70d037e3-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.610 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.611 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.612 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.612 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.613 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:09:27.613 DEBUG:boto:Data: 2023-10-12T01:09:27.613 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.614 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.615 DEBUG:boto:Params: {} 2023-10-12T01:09:27.615 DEBUG:boto:Token: None 2023-10-12T01:09:27.616 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58 2023-10-12T01:09:27.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TetvU9VWuxvoS2AsdaiFIUlU/ko= 2023-10-12T01:09:27.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TetvU9VWuxvoS2AsdaiFIUlU/ko=', 'Content-Length': '0'} 2023-10-12T01:09:27.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010834002efb25c74-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.619 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.619 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.620 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.620 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:09:27.621 DEBUG:boto:Data: 2023-10-12T01:09:27.621 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.622 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.622 DEBUG:boto:Params: {} 2023-10-12T01:09:27.623 DEBUG:boto:Token: None 2023-10-12T01:09:27.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58 2023-10-12T01:09:27.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TetvU9VWuxvoS2AsdaiFIUlU/ko= 2023-10-12T01:09:27.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TetvU9VWuxvoS2AsdaiFIUlU/ko=', 'Content-Length': '0'} 2023-10-12T01:09:27.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143b1842167f1f7a-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:09:27.627 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.628 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.628 DEBUG:boto:Method: GET 2023-10-12T01:09:27.629 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:09:27.629 DEBUG:boto:Data: 2023-10-12T01:09:27.630 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.630 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.631 DEBUG:boto:Params: {} 2023-10-12T01:09:27.631 DEBUG:boto:Token: None 2023-10-12T01:09:27.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58 2023-10-12T01:09:27.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcX5qGw/Zw/VvbrbM7rtrw183TQ= 2023-10-12T01:09:27.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcX5qGw/Zw/VvbrbM7rtrw183TQ=', 'Content-Length': '0'} 2023-10-12T01:09:27.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9ea3d491b16263b-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.635 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:09:27.636 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:09:27.636 DEBUG:boto:Method: GET 2023-10-12T01:09:27.637 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:09:27.637 DEBUG:boto:Data: 2023-10-12T01:09:27.638 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.638 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.639 DEBUG:boto:Params: {} 2023-10-12T01:09:27.639 DEBUG:boto:Token: None 2023-10-12T01:09:27.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj58 2023-10-12T01:09:27.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UcX5qGw/Zw/VvbrbM7rtrw183TQ= 2023-10-12T01:09:27.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UcX5qGw/Zw/VvbrbM7rtrw183TQ=', 'Content-Length': '0'} 2023-10-12T01:09:27.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947e8f642ff84469-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.644 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.644 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.645 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.645 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.645 DEBUG:boto:Method: GET 2023-10-12T01:09:27.646 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.646 DEBUG:boto:Data: 2023-10-12T01:09:27.647 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.648 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.648 DEBUG:boto:Params: {} 2023-10-12T01:09:27.648 DEBUG:boto:Token: None 2023-10-12T01:09:27.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002931835a3e87b390-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.698 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.699 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.700 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.700 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.700 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.701 DEBUG:boto:Method: GET 2023-10-12T01:09:27.701 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:09:27.702 DEBUG:boto:Data: 2023-10-12T01:09:27.702 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.704 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.704 DEBUG:boto:Params: {} 2023-10-12T01:09:27.705 DEBUG:boto:Token: None 2023-10-12T01:09:27.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b9e8c3d1dc605a-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.711 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:09:27.712 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.713 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.713 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.714 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.714 DEBUG:boto:Method: GET 2023-10-12T01:09:27.715 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.715 DEBUG:boto:Data: 2023-10-12T01:09:27.715 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.716 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.717 DEBUG:boto:Params: {} 2023-10-12T01:09:27.717 DEBUG:boto:Token: None 2023-10-12T01:09:27.718 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE= 2023-10-12T01:09:27.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE=', 'Content-Length': '0'} 2023-10-12T01:09:27.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dc6527717bdbab-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.723 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.723 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.723 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.724 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.725 DEBUG:boto:Method: GET 2023-10-12T01:09:27.725 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.726 DEBUG:boto:Data: 2023-10-12T01:09:27.726 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.727 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.727 DEBUG:boto:Params: {} 2023-10-12T01:09:27.728 DEBUG:boto:Token: None 2023-10-12T01:09:27.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE= 2023-10-12T01:09:27.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE=', 'Content-Length': '0'} 2023-10-12T01:09:27.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c62fd7f7c709ff-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.774 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.775 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.775 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.776 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.776 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.777 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.777 DEBUG:boto:Data: 2023-10-12T01:09:27.778 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.778 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.779 DEBUG:boto:Params: {} 2023-10-12T01:09:27.779 DEBUG:boto:Token: None 2023-10-12T01:09:27.780 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EyM7OAg308C0r5B6AkZ3ICu5c8U= 2023-10-12T01:09:27.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EyM7OAg308C0r5B6AkZ3ICu5c8U=', 'Content-Length': '0'} 2023-10-12T01:09:27.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c96eb9bbe32b722-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.783 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.784 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.784 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.785 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.785 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.785 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.786 DEBUG:boto:Data: 2023-10-12T01:09:27.787 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.788 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.788 DEBUG:boto:Params: {} 2023-10-12T01:09:27.789 DEBUG:boto:Token: None 2023-10-12T01:09:27.789 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.790 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EyM7OAg308C0r5B6AkZ3ICu5c8U= 2023-10-12T01:09:27.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EyM7OAg308C0r5B6AkZ3ICu5c8U=', 'Content-Length': '0'} 2023-10-12T01:09:27.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04847fc71bdc67f-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:09:27.793 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.794 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.794 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.795 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.795 DEBUG:boto:Method: GET 2023-10-12T01:09:27.796 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.796 DEBUG:boto:Data: 2023-10-12T01:09:27.797 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.798 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.799 DEBUG:boto:Params: {} 2023-10-12T01:09:27.799 DEBUG:boto:Token: None 2023-10-12T01:09:27.799 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.800 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE= 2023-10-12T01:09:27.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE=', 'Content-Length': '0'} 2023-10-12T01:09:27.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b260a97306aa1bb-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.803 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.803 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.804 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.804 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.805 DEBUG:boto:Method: GET 2023-10-12T01:09:27.805 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.806 DEBUG:boto:Data: 2023-10-12T01:09:27.806 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.807 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.807 DEBUG:boto:Params: {} 2023-10-12T01:09:27.808 DEBUG:boto:Token: None 2023-10-12T01:09:27.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:09:27.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE= 2023-10-12T01:09:27.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35aWLMJVHGLmzc4nla5lhSX30bE=', 'Content-Length': '0'} 2023-10-12T01:09:27.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085fcbfac91270257-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.812 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.812 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.813 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.813 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.813 DEBUG:boto:Method: GET 2023-10-12T01:09:27.814 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.814 DEBUG:boto:Data: 2023-10-12T01:09:27.815 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.816 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.816 DEBUG:boto:Params: {} 2023-10-12T01:09:27.817 DEBUG:boto:Token: None 2023-10-12T01:09:27.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5570ce856c729d8-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.862 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.863 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.863 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.864 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.864 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.865 DEBUG:boto:Method: GET 2023-10-12T01:09:27.865 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.865 DEBUG:boto:Data: 2023-10-12T01:09:27.866 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.867 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.867 DEBUG:boto:Params: {} 2023-10-12T01:09:27.868 DEBUG:boto:Token: None 2023-10-12T01:09:27.868 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.869 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d758ee38c1a07b7-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.874 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.874 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.875 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.875 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.876 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:09:27.876 DEBUG:boto:Data: 2023-10-12T01:09:27.877 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.877 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.878 DEBUG:boto:Params: {} 2023-10-12T01:09:27.878 DEBUG:boto:Token: None 2023-10-12T01:09:27.879 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62 2023-10-12T01:09:27.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dl9TxtJVt1V6nwpcMR37jXHUMPk= 2023-10-12T01:09:27.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dl9TxtJVt1V6nwpcMR37jXHUMPk=', 'Content-Length': '0'} 2023-10-12T01:09:27.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1cb3b8c444b38c-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.882 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.883 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.883 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.883 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:09:27.884 DEBUG:boto:Data: 2023-10-12T01:09:27.884 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.885 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.885 DEBUG:boto:Params: {} 2023-10-12T01:09:27.886 DEBUG:boto:Token: None 2023-10-12T01:09:27.886 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62 2023-10-12T01:09:27.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Dl9TxtJVt1V6nwpcMR37jXHUMPk= 2023-10-12T01:09:27.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Dl9TxtJVt1V6nwpcMR37jXHUMPk=', 'Content-Length': '0'} 2023-10-12T01:09:27.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb98e4f9f777be9-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:09:27.890 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.891 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.891 DEBUG:boto:Method: GET 2023-10-12T01:09:27.892 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:09:27.892 DEBUG:boto:Data: 2023-10-12T01:09:27.893 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.893 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.894 DEBUG:boto:Params: {} 2023-10-12T01:09:27.894 DEBUG:boto:Token: None 2023-10-12T01:09:27.895 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62 2023-10-12T01:09:27.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yBboaVdPjfYp1qnXj0uHcHZEm0s= 2023-10-12T01:09:27.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yBboaVdPjfYp1qnXj0uHcHZEm0s=', 'Content-Length': '0'} 2023-10-12T01:09:27.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63fb658e8bd9c91-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.898 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:09:27.899 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:09:27.899 DEBUG:boto:Method: GET 2023-10-12T01:09:27.900 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:09:27.900 DEBUG:boto:Data: 2023-10-12T01:09:27.900 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.901 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.902 DEBUG:boto:Params: {} 2023-10-12T01:09:27.902 DEBUG:boto:Token: None 2023-10-12T01:09:27.903 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj62 2023-10-12T01:09:27.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yBboaVdPjfYp1qnXj0uHcHZEm0s= 2023-10-12T01:09:27.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yBboaVdPjfYp1qnXj0uHcHZEm0s=', 'Content-Length': '0'} 2023-10-12T01:09:27.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d387718d7459e71-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.906 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.907 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.907 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.908 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.908 DEBUG:boto:Method: GET 2023-10-12T01:09:27.909 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.909 DEBUG:boto:Data: 2023-10-12T01:09:27.909 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.910 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.911 DEBUG:boto:Params: {} 2023-10-12T01:09:27.911 DEBUG:boto:Token: None 2023-10-12T01:09:27.912 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c4ba9bd65edecb-0065274747-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.916 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.916 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:27.917 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:27.917 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.918 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.918 DEBUG:boto:Method: GET 2023-10-12T01:09:27.918 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:09:27.919 DEBUG:boto:Data: 2023-10-12T01:09:27.919 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.920 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.920 DEBUG:boto:Params: {} 2023-10-12T01:09:27.921 DEBUG:boto:Token: None 2023-10-12T01:09:27.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/ 2023-10-12T01:09:27.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q= 2023-10-12T01:09:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FXSU/YtMvYAvfp0LFXc22QOe29Q=', 'Content-Length': '0'} 2023-10-12T01:09:27.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031b8f0b302a1e30a-0065274747-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.927 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:27.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:09:27.928 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:27.928 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:27.928 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.929 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.929 DEBUG:boto:Method: GET 2023-10-12T01:09:27.930 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.930 DEBUG:boto:Data: 2023-10-12T01:09:27.931 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.931 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.932 DEBUG:boto:Params: {} 2023-10-12T01:09:27.932 DEBUG:boto:Token: None 2023-10-12T01:09:27.933 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2W5G69pKUBp850H3xcWHXAIFWJE= 2023-10-12T01:09:27.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2W5G69pKUBp850H3xcWHXAIFWJE=', 'Content-Length': '0'} 2023-10-12T01:09:27.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af12e600472b7e6a-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.937 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:27.937 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:27.937 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.938 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.938 DEBUG:boto:Method: GET 2023-10-12T01:09:27.939 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.939 DEBUG:boto:Data: 2023-10-12T01:09:27.940 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.940 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.941 DEBUG:boto:Params: {} 2023-10-12T01:09:27.941 DEBUG:boto:Token: None 2023-10-12T01:09:27.942 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2W5G69pKUBp850H3xcWHXAIFWJE= 2023-10-12T01:09:27.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2W5G69pKUBp850H3xcWHXAIFWJE=', 'Content-Length': '0'} 2023-10-12T01:09:27.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3e3bc6ae94db90-0065274747-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.986 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:27.987 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:27.987 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.988 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.988 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.988 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.989 DEBUG:boto:Data: 2023-10-12T01:09:27.989 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:27.990 DEBUG:boto:Port: 8001 2023-10-12T01:09:27.991 DEBUG:boto:Params: {} 2023-10-12T01:09:27.991 DEBUG:boto:Token: None 2023-10-12T01:09:27.992 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:27 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.992 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gDHdjGrFVlGSneme1i/6DXWENMc= 2023-10-12T01:09:27.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gDHdjGrFVlGSneme1i/6DXWENMc=', 'Content-Length': '0'} 2023-10-12T01:09:27.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f0bf6471887f47-0065274747-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:27.995 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:27.995 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:27.996 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.996 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.997 DEBUG:boto:Method: HEAD 2023-10-12T01:09:27.997 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:27.997 DEBUG:boto:Data: 2023-10-12T01:09:27.998 DEBUG:boto:Headers: {} 2023-10-12T01:09:27.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:27.999 DEBUG:boto:Port: 8000 2023-10-12T01:09:27.999 DEBUG:boto:Params: {} 2023-10-12T01:09:27.999 DEBUG:boto:Token: None 2023-10-12T01:09:28.000 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J1NvxMssLQng4xxi0pnbEzs7Hto= 2023-10-12T01:09:28.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J1NvxMssLQng4xxi0pnbEzs7Hto=', 'Content-Length': '0'} 2023-10-12T01:09:28.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad2d1f662629463-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:09:28.004 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.004 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.005 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.005 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.006 DEBUG:boto:Method: GET 2023-10-12T01:09:28.006 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.006 DEBUG:boto:Data: 2023-10-12T01:09:28.007 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.008 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.008 DEBUG:boto:Params: {} 2023-10-12T01:09:28.009 DEBUG:boto:Token: None 2023-10-12T01:09:28.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.010 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vJAEhEyq3RaFeVyy2AiwTAgTAlk= 2023-10-12T01:09:28.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vJAEhEyq3RaFeVyy2AiwTAgTAlk=', 'Content-Length': '0'} 2023-10-12T01:09:28.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029f0f0f9eaf64495-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.054 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.055 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.055 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.056 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.056 DEBUG:boto:Method: GET 2023-10-12T01:09:28.057 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.057 DEBUG:boto:Data: 2023-10-12T01:09:28.058 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.058 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.059 DEBUG:boto:Params: {} 2023-10-12T01:09:28.059 DEBUG:boto:Token: None 2023-10-12T01:09:28.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:09:28.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vJAEhEyq3RaFeVyy2AiwTAgTAlk= 2023-10-12T01:09:28.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vJAEhEyq3RaFeVyy2AiwTAgTAlk=', 'Content-Length': '0'} 2023-10-12T01:09:28.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d058470366046a0-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.064 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.064 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.065 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.065 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.066 DEBUG:boto:Method: GET 2023-10-12T01:09:28.066 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.067 DEBUG:boto:Data: 2023-10-12T01:09:28.067 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.068 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.068 DEBUG:boto:Params: {} 2023-10-12T01:09:28.069 DEBUG:boto:Token: None 2023-10-12T01:09:28.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e940762369301fe-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.073 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.074 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.075 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.075 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.076 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.076 DEBUG:boto:Method: GET 2023-10-12T01:09:28.076 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.077 DEBUG:boto:Data: 2023-10-12T01:09:28.077 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.078 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.079 DEBUG:boto:Params: {} 2023-10-12T01:09:28.079 DEBUG:boto:Token: None 2023-10-12T01:09:28.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e69922dd067f84-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.085 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.086 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.086 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.087 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.087 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:09:28.087 DEBUG:boto:Data: 2023-10-12T01:09:28.088 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.089 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.089 DEBUG:boto:Params: {} 2023-10-12T01:09:28.089 DEBUG:boto:Token: None 2023-10-12T01:09:28.090 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68 2023-10-12T01:09:28.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C2CbCN/AcpKGZ5ftjmlPbupK7iE= 2023-10-12T01:09:28.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C2CbCN/AcpKGZ5ftjmlPbupK7iE=', 'Content-Length': '0'} 2023-10-12T01:09:28.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b5b5b3ba851a0a-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.094 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.094 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.095 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.095 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:09:28.095 DEBUG:boto:Data: 2023-10-12T01:09:28.096 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.097 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.097 DEBUG:boto:Params: {} 2023-10-12T01:09:28.097 DEBUG:boto:Token: None 2023-10-12T01:09:28.098 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68 2023-10-12T01:09:28.098 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C2CbCN/AcpKGZ5ftjmlPbupK7iE= 2023-10-12T01:09:28.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C2CbCN/AcpKGZ5ftjmlPbupK7iE=', 'Content-Length': '0'} 2023-10-12T01:09:28.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c50725b99c6e3c2-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:09:28.102 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.102 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.103 DEBUG:boto:Method: GET 2023-10-12T01:09:28.103 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:09:28.104 DEBUG:boto:Data: 2023-10-12T01:09:28.104 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.105 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.105 DEBUG:boto:Params: {} 2023-10-12T01:09:28.106 DEBUG:boto:Token: None 2023-10-12T01:09:28.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68 2023-10-12T01:09:28.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PSiMhxOJC5XoUktcNkpaduWTgZ8= 2023-10-12T01:09:28.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PSiMhxOJC5XoUktcNkpaduWTgZ8=', 'Content-Length': '0'} 2023-10-12T01:09:28.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010de4e4922524f0d-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.110 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:09:28.110 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:09:28.111 DEBUG:boto:Method: GET 2023-10-12T01:09:28.111 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:09:28.112 DEBUG:boto:Data: 2023-10-12T01:09:28.112 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.113 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.113 DEBUG:boto:Params: {} 2023-10-12T01:09:28.114 DEBUG:boto:Token: None 2023-10-12T01:09:28.114 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj68 2023-10-12T01:09:28.115 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PSiMhxOJC5XoUktcNkpaduWTgZ8= 2023-10-12T01:09:28.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PSiMhxOJC5XoUktcNkpaduWTgZ8=', 'Content-Length': '0'} 2023-10-12T01:09:28.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5d8b755d6ad087-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.158 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.159 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.159 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.160 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.160 DEBUG:boto:Method: GET 2023-10-12T01:09:28.161 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.161 DEBUG:boto:Data: 2023-10-12T01:09:28.162 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.163 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.165 DEBUG:boto:Params: {} 2023-10-12T01:09:28.166 DEBUG:boto:Token: None 2023-10-12T01:09:28.166 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc314e844171a017-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.171 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.171 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.172 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.172 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.173 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.173 DEBUG:boto:Method: GET 2023-10-12T01:09:28.174 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:09:28.174 DEBUG:boto:Data: 2023-10-12T01:09:28.174 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.175 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.176 DEBUG:boto:Params: {} 2023-10-12T01:09:28.176 DEBUG:boto:Token: None 2023-10-12T01:09:28.177 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.177 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48820fc783e1891-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.182 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:09:28.184 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.184 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.184 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.185 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.185 DEBUG:boto:Method: GET 2023-10-12T01:09:28.186 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.186 DEBUG:boto:Data: 2023-10-12T01:09:28.187 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.187 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.188 DEBUG:boto:Params: {} 2023-10-12T01:09:28.188 DEBUG:boto:Token: None 2023-10-12T01:09:28.189 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA= 2023-10-12T01:09:28.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA=', 'Content-Length': '0'} 2023-10-12T01:09:28.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020c7f6224cc130bf-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.193 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.193 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.194 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.194 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.195 DEBUG:boto:Method: GET 2023-10-12T01:09:28.195 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.195 DEBUG:boto:Data: 2023-10-12T01:09:28.196 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.197 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.197 DEBUG:boto:Params: {} 2023-10-12T01:09:28.198 DEBUG:boto:Token: None 2023-10-12T01:09:28.198 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.199 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA= 2023-10-12T01:09:28.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA=', 'Content-Length': '0'} 2023-10-12T01:09:28.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8e7ff018ab7719-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.202 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.202 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.203 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.203 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.204 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.204 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.205 DEBUG:boto:Data: 2023-10-12T01:09:28.205 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.206 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.206 DEBUG:boto:Params: {} 2023-10-12T01:09:28.207 DEBUG:boto:Token: None 2023-10-12T01:09:28.207 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.208 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iRBAGTP5lRS6PKX0bH1IPmhq5SM= 2023-10-12T01:09:28.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iRBAGTP5lRS6PKX0bH1IPmhq5SM=', 'Content-Length': '0'} 2023-10-12T01:09:28.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f36b15e82950fab-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.211 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.211 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.212 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.212 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.212 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.213 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.213 DEBUG:boto:Data: 2023-10-12T01:09:28.214 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.214 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.215 DEBUG:boto:Params: {} 2023-10-12T01:09:28.215 DEBUG:boto:Token: None 2023-10-12T01:09:28.216 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.216 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iRBAGTP5lRS6PKX0bH1IPmhq5SM= 2023-10-12T01:09:28.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iRBAGTP5lRS6PKX0bH1IPmhq5SM=', 'Content-Length': '0'} 2023-10-12T01:09:28.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8e4b564fb26fe4-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:09:28.220 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.220 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.221 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.221 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.222 DEBUG:boto:Method: GET 2023-10-12T01:09:28.222 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.222 DEBUG:boto:Data: 2023-10-12T01:09:28.223 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.224 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.224 DEBUG:boto:Params: {} 2023-10-12T01:09:28.225 DEBUG:boto:Token: None 2023-10-12T01:09:28.225 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA= 2023-10-12T01:09:28.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA=', 'Content-Length': '0'} 2023-10-12T01:09:28.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c3c748419220f2-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.229 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.229 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.230 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.230 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.231 DEBUG:boto:Method: GET 2023-10-12T01:09:28.231 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.232 DEBUG:boto:Data: 2023-10-12T01:09:28.232 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.233 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.233 DEBUG:boto:Params: {} 2023-10-12T01:09:28.234 DEBUG:boto:Token: None 2023-10-12T01:09:28.234 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:09:28.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA= 2023-10-12T01:09:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S8l+ImJVQmrkPVatDFuX+AaSOcA=', 'Content-Length': '0'} 2023-10-12T01:09:28.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e06813bd203323f3-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.238 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.239 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.239 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.239 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.240 DEBUG:boto:Method: GET 2023-10-12T01:09:28.240 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.241 DEBUG:boto:Data: 2023-10-12T01:09:28.241 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.242 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.242 DEBUG:boto:Params: {} 2023-10-12T01:09:28.243 DEBUG:boto:Token: None 2023-10-12T01:09:28.243 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f44ce8c3fa3518-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.247 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.248 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.249 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.249 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.250 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.250 DEBUG:boto:Method: GET 2023-10-12T01:09:28.250 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.251 DEBUG:boto:Data: 2023-10-12T01:09:28.251 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.252 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.253 DEBUG:boto:Params: {} 2023-10-12T01:09:28.253 DEBUG:boto:Token: None 2023-10-12T01:09:28.254 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d575a6e2ead867-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.259 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.260 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.260 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.261 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.261 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:09:28.261 DEBUG:boto:Data: 2023-10-12T01:09:28.262 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.263 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.263 DEBUG:boto:Params: {} 2023-10-12T01:09:28.264 DEBUG:boto:Token: None 2023-10-12T01:09:28.264 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69 2023-10-12T01:09:28.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jPAWw7rX5afpnDCWo9lpHoHCqdg= 2023-10-12T01:09:28.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jPAWw7rX5afpnDCWo9lpHoHCqdg=', 'Content-Length': '0'} 2023-10-12T01:09:28.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f663b8b040e3fd-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.268 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.268 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.268 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.269 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:09:28.269 DEBUG:boto:Data: 2023-10-12T01:09:28.270 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.270 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.271 DEBUG:boto:Params: {} 2023-10-12T01:09:28.271 DEBUG:boto:Token: None 2023-10-12T01:09:28.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69 2023-10-12T01:09:28.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jPAWw7rX5afpnDCWo9lpHoHCqdg= 2023-10-12T01:09:28.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jPAWw7rX5afpnDCWo9lpHoHCqdg=', 'Content-Length': '0'} 2023-10-12T01:09:28.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615d72d676c85ff8-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:09:28.276 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.276 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.277 DEBUG:boto:Method: GET 2023-10-12T01:09:28.277 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:09:28.278 DEBUG:boto:Data: 2023-10-12T01:09:28.278 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.279 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.279 DEBUG:boto:Params: {} 2023-10-12T01:09:28.280 DEBUG:boto:Token: None 2023-10-12T01:09:28.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69 2023-10-12T01:09:28.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TnJN615ve+MVGtnK0p3jhueg28Q= 2023-10-12T01:09:28.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TnJN615ve+MVGtnK0p3jhueg28Q=', 'Content-Length': '0'} 2023-10-12T01:09:28.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c36a2753ced27191-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.330 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:09:28.331 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:09:28.331 DEBUG:boto:Method: GET 2023-10-12T01:09:28.332 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:09:28.332 DEBUG:boto:Data: 2023-10-12T01:09:28.333 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.334 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.334 DEBUG:boto:Params: {} 2023-10-12T01:09:28.335 DEBUG:boto:Token: None 2023-10-12T01:09:28.335 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj69 2023-10-12T01:09:28.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TnJN615ve+MVGtnK0p3jhueg28Q= 2023-10-12T01:09:28.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TnJN615ve+MVGtnK0p3jhueg28Q=', 'Content-Length': '0'} 2023-10-12T01:09:28.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351a93c4734cfe11-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.339 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.339 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.340 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.340 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.341 DEBUG:boto:Method: GET 2023-10-12T01:09:28.341 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.341 DEBUG:boto:Data: 2023-10-12T01:09:28.342 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.343 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.343 DEBUG:boto:Params: {} 2023-10-12T01:09:28.344 DEBUG:boto:Token: None 2023-10-12T01:09:28.344 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.345 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab10b3a67ec4de49-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.348 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.349 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.349 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.350 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.350 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.351 DEBUG:boto:Method: GET 2023-10-12T01:09:28.351 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:09:28.352 DEBUG:boto:Data: 2023-10-12T01:09:28.352 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.353 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.353 DEBUG:boto:Params: {} 2023-10-12T01:09:28.354 DEBUG:boto:Token: None 2023-10-12T01:09:28.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a95c3e632a6579-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.360 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:09:28.361 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.361 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.362 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.362 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.363 DEBUG:boto:Method: GET 2023-10-12T01:09:28.363 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.364 DEBUG:boto:Data: 2023-10-12T01:09:28.364 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.365 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.365 DEBUG:boto:Params: {} 2023-10-12T01:09:28.366 DEBUG:boto:Token: None 2023-10-12T01:09:28.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA= 2023-10-12T01:09:28.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA=', 'Content-Length': '0'} 2023-10-12T01:09:28.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060147d66c2195dec-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.372 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.372 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.373 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.373 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.374 DEBUG:boto:Method: GET 2023-10-12T01:09:28.374 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.375 DEBUG:boto:Data: 2023-10-12T01:09:28.375 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.376 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.376 DEBUG:boto:Params: {} 2023-10-12T01:09:28.377 DEBUG:boto:Token: None 2023-10-12T01:09:28.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA= 2023-10-12T01:09:28.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA=', 'Content-Length': '0'} 2023-10-12T01:09:28.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d1c06c301b9871-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.381 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.382 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.382 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.382 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.383 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.383 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.384 DEBUG:boto:Data: 2023-10-12T01:09:28.384 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.385 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.385 DEBUG:boto:Params: {} 2023-10-12T01:09:28.386 DEBUG:boto:Token: None 2023-10-12T01:09:28.386 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2wHsrezFZIzxhGZ/YoU7eH8qAQ= 2023-10-12T01:09:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2wHsrezFZIzxhGZ/YoU7eH8qAQ=', 'Content-Length': '0'} 2023-10-12T01:09:28.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ea2988c0f2d9b1-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.390 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.390 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.391 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.391 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.391 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.392 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.392 DEBUG:boto:Data: 2023-10-12T01:09:28.393 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.393 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.394 DEBUG:boto:Params: {} 2023-10-12T01:09:28.394 DEBUG:boto:Token: None 2023-10-12T01:09:28.395 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.395 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H2wHsrezFZIzxhGZ/YoU7eH8qAQ= 2023-10-12T01:09:28.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H2wHsrezFZIzxhGZ/YoU7eH8qAQ=', 'Content-Length': '0'} 2023-10-12T01:09:28.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106bc77ee63dfa34-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:09:28.399 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.399 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.400 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.400 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.400 DEBUG:boto:Method: GET 2023-10-12T01:09:28.401 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.401 DEBUG:boto:Data: 2023-10-12T01:09:28.402 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.403 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.403 DEBUG:boto:Params: {} 2023-10-12T01:09:28.404 DEBUG:boto:Token: None 2023-10-12T01:09:28.404 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.405 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA= 2023-10-12T01:09:28.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA=', 'Content-Length': '0'} 2023-10-12T01:09:28.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbec855c7aa043e6-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.454 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.455 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.455 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.456 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.456 DEBUG:boto:Method: GET 2023-10-12T01:09:28.457 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.457 DEBUG:boto:Data: 2023-10-12T01:09:28.457 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.458 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.459 DEBUG:boto:Params: {} 2023-10-12T01:09:28.459 DEBUG:boto:Token: None 2023-10-12T01:09:28.460 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:09:28.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA= 2023-10-12T01:09:28.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W7v1pi5Kc9aYibxxOW7HIjtsknA=', 'Content-Length': '0'} 2023-10-12T01:09:28.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f8e2bbddf7b40f0-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.463 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.464 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.464 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.465 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.465 DEBUG:boto:Method: GET 2023-10-12T01:09:28.466 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.466 DEBUG:boto:Data: 2023-10-12T01:09:28.466 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.467 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.468 DEBUG:boto:Params: {} 2023-10-12T01:09:28.468 DEBUG:boto:Token: None 2023-10-12T01:09:28.469 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d459f83b6c1183e-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.473 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.473 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.474 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.474 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.475 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.475 DEBUG:boto:Method: GET 2023-10-12T01:09:28.476 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.476 DEBUG:boto:Data: 2023-10-12T01:09:28.477 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.478 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.478 DEBUG:boto:Params: {} 2023-10-12T01:09:28.479 DEBUG:boto:Token: None 2023-10-12T01:09:28.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db7dd548d6a94d5-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.526 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.527 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.528 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.528 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.529 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:09:28.529 DEBUG:boto:Data: 2023-10-12T01:09:28.529 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.530 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.531 DEBUG:boto:Params: {} 2023-10-12T01:09:28.531 DEBUG:boto:Token: None 2023-10-12T01:09:28.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76 2023-10-12T01:09:28.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rp2S5UBIS8iPsIwcp34qsIySDKU= 2023-10-12T01:09:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rp2S5UBIS8iPsIwcp34qsIySDKU=', 'Content-Length': '0'} 2023-10-12T01:09:28.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057de9f21d1c9c68e-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.535 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.536 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.536 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.537 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:09:28.537 DEBUG:boto:Data: 2023-10-12T01:09:28.538 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.539 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.539 DEBUG:boto:Params: {} 2023-10-12T01:09:28.539 DEBUG:boto:Token: None 2023-10-12T01:09:28.540 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76 2023-10-12T01:09:28.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rp2S5UBIS8iPsIwcp34qsIySDKU= 2023-10-12T01:09:28.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rp2S5UBIS8iPsIwcp34qsIySDKU=', 'Content-Length': '0'} 2023-10-12T01:09:28.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cb4cffdb2b42d0d-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:09:28.544 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.544 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.545 DEBUG:boto:Method: GET 2023-10-12T01:09:28.545 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:09:28.546 DEBUG:boto:Data: 2023-10-12T01:09:28.546 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.547 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.547 DEBUG:boto:Params: {} 2023-10-12T01:09:28.548 DEBUG:boto:Token: None 2023-10-12T01:09:28.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76 2023-10-12T01:09:28.549 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bea96yZ2Rsrqs72Bwv5nMARovik= 2023-10-12T01:09:28.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bea96yZ2Rsrqs72Bwv5nMARovik=', 'Content-Length': '0'} 2023-10-12T01:09:28.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe45372ceb41d0f5-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.552 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:09:28.553 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:09:28.553 DEBUG:boto:Method: GET 2023-10-12T01:09:28.554 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:09:28.554 DEBUG:boto:Data: 2023-10-12T01:09:28.554 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.555 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.556 DEBUG:boto:Params: {} 2023-10-12T01:09:28.556 DEBUG:boto:Token: None 2023-10-12T01:09:28.557 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj76 2023-10-12T01:09:28.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bea96yZ2Rsrqs72Bwv5nMARovik= 2023-10-12T01:09:28.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bea96yZ2Rsrqs72Bwv5nMARovik=', 'Content-Length': '0'} 2023-10-12T01:09:28.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d1123ab8c356120-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.561 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.561 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.561 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.562 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.562 DEBUG:boto:Method: GET 2023-10-12T01:09:28.563 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.563 DEBUG:boto:Data: 2023-10-12T01:09:28.564 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.565 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.565 DEBUG:boto:Params: {} 2023-10-12T01:09:28.565 DEBUG:boto:Token: None 2023-10-12T01:09:28.566 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9626972326b8211-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.570 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.571 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.571 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.572 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.572 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.573 DEBUG:boto:Method: GET 2023-10-12T01:09:28.573 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:09:28.574 DEBUG:boto:Data: 2023-10-12T01:09:28.574 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.575 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.575 DEBUG:boto:Params: {} 2023-10-12T01:09:28.576 DEBUG:boto:Token: None 2023-10-12T01:09:28.576 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0526f85071e18e-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.582 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:09:28.583 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.583 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.584 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.584 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.585 DEBUG:boto:Method: GET 2023-10-12T01:09:28.585 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.586 DEBUG:boto:Data: 2023-10-12T01:09:28.586 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.587 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.587 DEBUG:boto:Params: {} 2023-10-12T01:09:28.588 DEBUG:boto:Token: None 2023-10-12T01:09:28.588 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk= 2023-10-12T01:09:28.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk=', 'Content-Length': '0'} 2023-10-12T01:09:28.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051deb803228c1470-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.592 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.593 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.593 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.594 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.594 DEBUG:boto:Method: GET 2023-10-12T01:09:28.594 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.595 DEBUG:boto:Data: 2023-10-12T01:09:28.595 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.596 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.597 DEBUG:boto:Params: {} 2023-10-12T01:09:28.597 DEBUG:boto:Token: None 2023-10-12T01:09:28.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk= 2023-10-12T01:09:28.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk=', 'Content-Length': '0'} 2023-10-12T01:09:28.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000decba7371d733190-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.646 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.647 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.647 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.648 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.648 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.649 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.649 DEBUG:boto:Data: 2023-10-12T01:09:28.649 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.650 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.651 DEBUG:boto:Params: {} 2023-10-12T01:09:28.651 DEBUG:boto:Token: None 2023-10-12T01:09:28.651 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O3abM32yu6Cph/RdaDfajdYQFDk= 2023-10-12T01:09:28.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O3abM32yu6Cph/RdaDfajdYQFDk=', 'Content-Length': '0'} 2023-10-12T01:09:28.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015bf059389126f17-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.655 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.656 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.656 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.657 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.657 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.657 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.658 DEBUG:boto:Data: 2023-10-12T01:09:28.658 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.659 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.659 DEBUG:boto:Params: {} 2023-10-12T01:09:28.660 DEBUG:boto:Token: None 2023-10-12T01:09:28.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O3abM32yu6Cph/RdaDfajdYQFDk= 2023-10-12T01:09:28.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O3abM32yu6Cph/RdaDfajdYQFDk=', 'Content-Length': '0'} 2023-10-12T01:09:28.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2a3fd32ee21750-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:09:28.664 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.665 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.665 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.665 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.666 DEBUG:boto:Method: GET 2023-10-12T01:09:28.666 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.667 DEBUG:boto:Data: 2023-10-12T01:09:28.667 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.668 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.668 DEBUG:boto:Params: {} 2023-10-12T01:09:28.669 DEBUG:boto:Token: None 2023-10-12T01:09:28.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk= 2023-10-12T01:09:28.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk=', 'Content-Length': '0'} 2023-10-12T01:09:28.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004266b3d3bfb68057-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.714 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.715 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.715 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.716 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.716 DEBUG:boto:Method: GET 2023-10-12T01:09:28.717 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.717 DEBUG:boto:Data: 2023-10-12T01:09:28.717 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.718 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.719 DEBUG:boto:Params: {} 2023-10-12T01:09:28.719 DEBUG:boto:Token: None 2023-10-12T01:09:28.720 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:09:28.720 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk= 2023-10-12T01:09:28.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjANa85oWA4SqXn305Isdn3LjLk=', 'Content-Length': '0'} 2023-10-12T01:09:28.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f93e2beedd3867-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.723 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.724 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.724 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.725 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.725 DEBUG:boto:Method: GET 2023-10-12T01:09:28.726 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.726 DEBUG:boto:Data: 2023-10-12T01:09:28.726 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.727 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.728 DEBUG:boto:Params: {} 2023-10-12T01:09:28.728 DEBUG:boto:Token: None 2023-10-12T01:09:28.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fa8786d5c8c283-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.733 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.733 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.734 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.734 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.735 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.735 DEBUG:boto:Method: GET 2023-10-12T01:09:28.736 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.736 DEBUG:boto:Data: 2023-10-12T01:09:28.736 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.737 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.738 DEBUG:boto:Params: {} 2023-10-12T01:09:28.738 DEBUG:boto:Token: None 2023-10-12T01:09:28.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.739 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002656ce9b00d1b46d-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.744 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.745 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.745 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.746 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.746 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:09:28.746 DEBUG:boto:Data: 2023-10-12T01:09:28.747 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.748 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.748 DEBUG:boto:Params: {} 2023-10-12T01:09:28.749 DEBUG:boto:Token: None 2023-10-12T01:09:28.749 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85 2023-10-12T01:09:28.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3lUtZF3vOwXmyVhtwYERTIaNK2I= 2023-10-12T01:09:28.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3lUtZF3vOwXmyVhtwYERTIaNK2I=', 'Content-Length': '0'} 2023-10-12T01:09:28.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0cecdeb6f2ad1b7-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.753 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.753 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.754 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.754 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:09:28.755 DEBUG:boto:Data: 2023-10-12T01:09:28.755 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.756 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.756 DEBUG:boto:Params: {} 2023-10-12T01:09:28.757 DEBUG:boto:Token: None 2023-10-12T01:09:28.757 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85 2023-10-12T01:09:28.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3lUtZF3vOwXmyVhtwYERTIaNK2I= 2023-10-12T01:09:28.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3lUtZF3vOwXmyVhtwYERTIaNK2I=', 'Content-Length': '0'} 2023-10-12T01:09:28.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb4fd5149f50ee0-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:09:28.761 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.761 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.762 DEBUG:boto:Method: GET 2023-10-12T01:09:28.762 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:09:28.763 DEBUG:boto:Data: 2023-10-12T01:09:28.763 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.764 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.765 DEBUG:boto:Params: {} 2023-10-12T01:09:28.765 DEBUG:boto:Token: None 2023-10-12T01:09:28.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85 2023-10-12T01:09:28.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:69aKMY8+sTPRupWBBMRILrORCF4= 2023-10-12T01:09:28.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:69aKMY8+sTPRupWBBMRILrORCF4=', 'Content-Length': '0'} 2023-10-12T01:09:28.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f818b9fe5e6a4cf-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.769 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:09:28.770 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:09:28.770 DEBUG:boto:Method: GET 2023-10-12T01:09:28.771 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:09:28.771 DEBUG:boto:Data: 2023-10-12T01:09:28.772 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.773 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.773 DEBUG:boto:Params: {} 2023-10-12T01:09:28.773 DEBUG:boto:Token: None 2023-10-12T01:09:28.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj85 2023-10-12T01:09:28.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:69aKMY8+sTPRupWBBMRILrORCF4= 2023-10-12T01:09:28.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:69aKMY8+sTPRupWBBMRILrORCF4=', 'Content-Length': '0'} 2023-10-12T01:09:28.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43039336338fb77-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.818 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.819 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.819 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.820 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.820 DEBUG:boto:Method: GET 2023-10-12T01:09:28.821 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.821 DEBUG:boto:Data: 2023-10-12T01:09:28.822 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.823 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.823 DEBUG:boto:Params: {} 2023-10-12T01:09:28.823 DEBUG:boto:Token: None 2023-10-12T01:09:28.824 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95786f35d5e6830-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.828 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.829 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.829 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.830 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.830 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.830 DEBUG:boto:Method: GET 2023-10-12T01:09:28.831 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:09:28.831 DEBUG:boto:Data: 2023-10-12T01:09:28.832 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.833 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.833 DEBUG:boto:Params: {} 2023-10-12T01:09:28.833 DEBUG:boto:Token: None 2023-10-12T01:09:28.834 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.834 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d39aa2d94c231ee-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.840 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:09:28.841 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.841 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.842 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.842 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.843 DEBUG:boto:Method: GET 2023-10-12T01:09:28.843 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.844 DEBUG:boto:Data: 2023-10-12T01:09:28.844 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.845 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.845 DEBUG:boto:Params: {} 2023-10-12T01:09:28.846 DEBUG:boto:Token: None 2023-10-12T01:09:28.847 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.847 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8= 2023-10-12T01:09:28.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8=', 'Content-Length': '0'} 2023-10-12T01:09:28.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ccc3f2d175f4d50-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.850 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.851 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.851 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.852 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.852 DEBUG:boto:Method: GET 2023-10-12T01:09:28.853 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.853 DEBUG:boto:Data: 2023-10-12T01:09:28.854 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.855 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.855 DEBUG:boto:Params: {} 2023-10-12T01:09:28.856 DEBUG:boto:Token: None 2023-10-12T01:09:28.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8= 2023-10-12T01:09:28.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8=', 'Content-Length': '0'} 2023-10-12T01:09:28.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6446b9632d7e527-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.861 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.861 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.862 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.862 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.863 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.863 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.863 DEBUG:boto:Data: 2023-10-12T01:09:28.864 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.865 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.865 DEBUG:boto:Params: {} 2023-10-12T01:09:28.866 DEBUG:boto:Token: None 2023-10-12T01:09:28.866 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRiIUyhBcchzgu3czDegD+VMBRM= 2023-10-12T01:09:28.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRiIUyhBcchzgu3czDegD+VMBRM=', 'Content-Length': '0'} 2023-10-12T01:09:28.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8b6a6623c1af9b-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.870 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.871 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.871 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.872 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.872 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.873 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.873 DEBUG:boto:Data: 2023-10-12T01:09:28.873 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.874 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.875 DEBUG:boto:Params: {} 2023-10-12T01:09:28.875 DEBUG:boto:Token: None 2023-10-12T01:09:28.876 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YRiIUyhBcchzgu3czDegD+VMBRM= 2023-10-12T01:09:28.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YRiIUyhBcchzgu3czDegD+VMBRM=', 'Content-Length': '0'} 2023-10-12T01:09:28.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c280c67420285d-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:09:28.880 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.880 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.881 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.881 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.882 DEBUG:boto:Method: GET 2023-10-12T01:09:28.882 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.883 DEBUG:boto:Data: 2023-10-12T01:09:28.883 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.884 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.885 DEBUG:boto:Params: {} 2023-10-12T01:09:28.885 DEBUG:boto:Token: None 2023-10-12T01:09:28.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.886 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8= 2023-10-12T01:09:28.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8=', 'Content-Length': '0'} 2023-10-12T01:09:28.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf36ca53ef62dd51-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.890 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.892 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.892 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.893 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.893 DEBUG:boto:Method: GET 2023-10-12T01:09:28.893 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.894 DEBUG:boto:Data: 2023-10-12T01:09:28.894 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.895 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.896 DEBUG:boto:Params: {} 2023-10-12T01:09:28.896 DEBUG:boto:Token: None 2023-10-12T01:09:28.897 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:09:28.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8= 2023-10-12T01:09:28.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SFcSioF56RTv5oL0oT2vOQYRqx8=', 'Content-Length': '0'} 2023-10-12T01:09:28.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b327f4b33eb363-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.946 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.947 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.947 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.948 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.948 DEBUG:boto:Method: GET 2023-10-12T01:09:28.949 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.949 DEBUG:boto:Data: 2023-10-12T01:09:28.950 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.951 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.951 DEBUG:boto:Params: {} 2023-10-12T01:09:28.951 DEBUG:boto:Token: None 2023-10-12T01:09:28.952 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.953 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782ef3c9c45499e9-0065274748-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.956 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.957 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:28.958 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:28.958 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.958 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.959 DEBUG:boto:Method: GET 2023-10-12T01:09:28.959 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:09:28.960 DEBUG:boto:Data: 2023-10-12T01:09:28.960 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.961 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.961 DEBUG:boto:Params: {} 2023-10-12T01:09:28.962 DEBUG:boto:Token: None 2023-10-12T01:09:28.962 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/ 2023-10-12T01:09:28.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI= 2023-10-12T01:09:28.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZ35zY/6GBdU1Y27tYECKkLosoI=', 'Content-Length': '0'} 2023-10-12T01:09:28.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef75e2f40b3a3e4a-0065274748-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.968 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:28.969 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.969 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.969 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.970 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:09:28.970 DEBUG:boto:Data: 2023-10-12T01:09:28.971 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.971 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.972 DEBUG:boto:Params: {} 2023-10-12T01:09:28.972 DEBUG:boto:Token: None 2023-10-12T01:09:28.973 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87 2023-10-12T01:09:28.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2G0TdzC6t1l+jMIyWntJMIoa4tA= 2023-10-12T01:09:28.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2G0TdzC6t1l+jMIyWntJMIoa4tA=', 'Content-Length': '0'} 2023-10-12T01:09:28.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a043531cc7169cb-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.976 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.977 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.977 DEBUG:boto:Method: HEAD 2023-10-12T01:09:28.978 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:09:28.978 DEBUG:boto:Data: 2023-10-12T01:09:28.978 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:28.979 DEBUG:boto:Port: 8000 2023-10-12T01:09:28.980 DEBUG:boto:Params: {} 2023-10-12T01:09:28.980 DEBUG:boto:Token: None 2023-10-12T01:09:28.981 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87 2023-10-12T01:09:28.981 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2G0TdzC6t1l+jMIyWntJMIoa4tA= 2023-10-12T01:09:28.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2G0TdzC6t1l+jMIyWntJMIoa4tA=', 'Content-Length': '0'} 2023-10-12T01:09:28.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a807d62414db0bf3-0065274748-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:28.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:09:28.985 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:28.985 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:28.985 DEBUG:boto:Method: GET 2023-10-12T01:09:28.986 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:09:28.986 DEBUG:boto:Data: 2023-10-12T01:09:28.987 DEBUG:boto:Headers: {} 2023-10-12T01:09:28.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:28.989 DEBUG:boto:Port: 8001 2023-10-12T01:09:28.989 DEBUG:boto:Params: {} 2023-10-12T01:09:28.990 DEBUG:boto:Token: None 2023-10-12T01:09:28.990 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:28 GMT /wafmoa-59/obj87 2023-10-12T01:09:28.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kEh8iDcSa/TkeFaXxKGnG0MIZyg= 2023-10-12T01:09:28.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:28 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kEh8iDcSa/TkeFaXxKGnG0MIZyg=', 'Content-Length': '0'} 2023-10-12T01:09:28.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e31df479fae7e90-0065274748-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.034 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:09:29.035 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:09:29.035 DEBUG:boto:Method: GET 2023-10-12T01:09:29.036 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:09:29.036 DEBUG:boto:Data: 2023-10-12T01:09:29.037 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.038 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.038 DEBUG:boto:Params: {} 2023-10-12T01:09:29.039 DEBUG:boto:Token: None 2023-10-12T01:09:29.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj87 2023-10-12T01:09:29.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TXZWRN9zHjGfInjVjTGjF9UOdkM= 2023-10-12T01:09:29.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TXZWRN9zHjGfInjVjTGjF9UOdkM=', 'Content-Length': '0'} 2023-10-12T01:09:29.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74ece9ad083d305-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.043 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.043 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.044 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.044 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.045 DEBUG:boto:Method: GET 2023-10-12T01:09:29.045 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.046 DEBUG:boto:Data: 2023-10-12T01:09:29.046 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.047 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.047 DEBUG:boto:Params: {} 2023-10-12T01:09:29.048 DEBUG:boto:Token: None 2023-10-12T01:09:29.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027f589d7b283cb13-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.052 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.053 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.054 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.054 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.054 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.055 DEBUG:boto:Method: GET 2023-10-12T01:09:29.055 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:09:29.056 DEBUG:boto:Data: 2023-10-12T01:09:29.056 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.057 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.057 DEBUG:boto:Params: {} 2023-10-12T01:09:29.058 DEBUG:boto:Token: None 2023-10-12T01:09:29.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fe729e7a34e89c-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.106 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:09:29.108 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.108 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.109 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.109 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.109 DEBUG:boto:Method: GET 2023-10-12T01:09:29.110 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.110 DEBUG:boto:Data: 2023-10-12T01:09:29.111 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.112 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.112 DEBUG:boto:Params: {} 2023-10-12T01:09:29.113 DEBUG:boto:Token: None 2023-10-12T01:09:29.113 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8= 2023-10-12T01:09:29.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8=', 'Content-Length': '0'} 2023-10-12T01:09:29.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9c814c27689718-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.117 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.117 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.118 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.118 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.119 DEBUG:boto:Method: GET 2023-10-12T01:09:29.119 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.120 DEBUG:boto:Data: 2023-10-12T01:09:29.120 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.121 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.121 DEBUG:boto:Params: {} 2023-10-12T01:09:29.122 DEBUG:boto:Token: None 2023-10-12T01:09:29.122 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.123 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8= 2023-10-12T01:09:29.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8=', 'Content-Length': '0'} 2023-10-12T01:09:29.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000169914685db1b889-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.126 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.126 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.127 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.127 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.128 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.128 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.128 DEBUG:boto:Data: 2023-10-12T01:09:29.129 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.130 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.130 DEBUG:boto:Params: {} 2023-10-12T01:09:29.131 DEBUG:boto:Token: None 2023-10-12T01:09:29.131 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/jskkys5FYTF6HCNR1kq2mlOZE= 2023-10-12T01:09:29.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/jskkys5FYTF6HCNR1kq2mlOZE=', 'Content-Length': '0'} 2023-10-12T01:09:29.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5627a9f533ade71-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.135 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.135 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.135 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.136 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.136 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.137 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.137 DEBUG:boto:Data: 2023-10-12T01:09:29.138 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.139 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.139 DEBUG:boto:Params: {} 2023-10-12T01:09:29.139 DEBUG:boto:Token: None 2023-10-12T01:09:29.140 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/jskkys5FYTF6HCNR1kq2mlOZE= 2023-10-12T01:09:29.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/jskkys5FYTF6HCNR1kq2mlOZE=', 'Content-Length': '0'} 2023-10-12T01:09:29.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da8759746e0f18d-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:09:29.144 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.145 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.145 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.146 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.146 DEBUG:boto:Method: GET 2023-10-12T01:09:29.146 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.147 DEBUG:boto:Data: 2023-10-12T01:09:29.147 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.148 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.149 DEBUG:boto:Params: {} 2023-10-12T01:09:29.149 DEBUG:boto:Token: None 2023-10-12T01:09:29.150 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8= 2023-10-12T01:09:29.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8=', 'Content-Length': '0'} 2023-10-12T01:09:29.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005415f8940c8ef3a0-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.153 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.154 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.155 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.155 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.155 DEBUG:boto:Method: GET 2023-10-12T01:09:29.156 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.156 DEBUG:boto:Data: 2023-10-12T01:09:29.157 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.158 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.158 DEBUG:boto:Params: {} 2023-10-12T01:09:29.159 DEBUG:boto:Token: None 2023-10-12T01:09:29.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:09:29.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8= 2023-10-12T01:09:29.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MW4pfMV9SmDHlLH0kF0zFD9ME+8=', 'Content-Length': '0'} 2023-10-12T01:09:29.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a790c0f202965505-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.163 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.163 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.163 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.164 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.164 DEBUG:boto:Method: GET 2023-10-12T01:09:29.165 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.165 DEBUG:boto:Data: 2023-10-12T01:09:29.165 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.166 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.167 DEBUG:boto:Params: {} 2023-10-12T01:09:29.167 DEBUG:boto:Token: None 2023-10-12T01:09:29.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002978fa3e6506ff67-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.171 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.172 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.172 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.173 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.173 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.174 DEBUG:boto:Method: GET 2023-10-12T01:09:29.174 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.175 DEBUG:boto:Data: 2023-10-12T01:09:29.175 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.176 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.176 DEBUG:boto:Params: {} 2023-10-12T01:09:29.177 DEBUG:boto:Token: None 2023-10-12T01:09:29.177 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.178 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b302b8cfa35e5f49-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.226 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.227 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.228 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.228 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.228 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:09:29.229 DEBUG:boto:Data: 2023-10-12T01:09:29.229 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.230 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.230 DEBUG:boto:Params: {} 2023-10-12T01:09:29.231 DEBUG:boto:Token: None 2023-10-12T01:09:29.231 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92 2023-10-12T01:09:29.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ND94vilhcgIOEghXUtVxQqbsJcw= 2023-10-12T01:09:29.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ND94vilhcgIOEghXUtVxQqbsJcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5e17598f26f0af-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.235 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.235 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.236 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.236 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:09:29.237 DEBUG:boto:Data: 2023-10-12T01:09:29.237 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.238 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.238 DEBUG:boto:Params: {} 2023-10-12T01:09:29.239 DEBUG:boto:Token: None 2023-10-12T01:09:29.239 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92 2023-10-12T01:09:29.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ND94vilhcgIOEghXUtVxQqbsJcw= 2023-10-12T01:09:29.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ND94vilhcgIOEghXUtVxQqbsJcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba4742904fce9a1-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:09:29.243 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.243 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.244 DEBUG:boto:Method: GET 2023-10-12T01:09:29.244 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:09:29.245 DEBUG:boto:Data: 2023-10-12T01:09:29.245 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.246 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.246 DEBUG:boto:Params: {} 2023-10-12T01:09:29.247 DEBUG:boto:Token: None 2023-10-12T01:09:29.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92 2023-10-12T01:09:29.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ne4TreLNhO22hJa1QJS2gTO70oo= 2023-10-12T01:09:29.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ne4TreLNhO22hJa1QJS2gTO70oo=', 'Content-Length': '0'} 2023-10-12T01:09:29.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d79993e1e914d2f7-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.251 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:09:29.251 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:09:29.252 DEBUG:boto:Method: GET 2023-10-12T01:09:29.252 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:09:29.253 DEBUG:boto:Data: 2023-10-12T01:09:29.253 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.254 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.254 DEBUG:boto:Params: {} 2023-10-12T01:09:29.255 DEBUG:boto:Token: None 2023-10-12T01:09:29.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj92 2023-10-12T01:09:29.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ne4TreLNhO22hJa1QJS2gTO70oo= 2023-10-12T01:09:29.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ne4TreLNhO22hJa1QJS2gTO70oo=', 'Content-Length': '0'} 2023-10-12T01:09:29.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a0d7a614d865597-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.259 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.260 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.260 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.260 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.261 DEBUG:boto:Method: GET 2023-10-12T01:09:29.261 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.261 DEBUG:boto:Data: 2023-10-12T01:09:29.262 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.263 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.263 DEBUG:boto:Params: {} 2023-10-12T01:09:29.264 DEBUG:boto:Token: None 2023-10-12T01:09:29.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004535ee759a9e5c0d-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.310 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.311 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.311 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.312 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.312 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.313 DEBUG:boto:Method: GET 2023-10-12T01:09:29.313 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:09:29.314 DEBUG:boto:Data: 2023-10-12T01:09:29.314 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.315 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.315 DEBUG:boto:Params: {} 2023-10-12T01:09:29.316 DEBUG:boto:Token: None 2023-10-12T01:09:29.316 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a2d1b7a861af397-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.321 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:09:29.322 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.323 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.323 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.324 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.324 DEBUG:boto:Method: GET 2023-10-12T01:09:29.325 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.325 DEBUG:boto:Data: 2023-10-12T01:09:29.325 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.326 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.327 DEBUG:boto:Params: {} 2023-10-12T01:09:29.327 DEBUG:boto:Token: None 2023-10-12T01:09:29.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M= 2023-10-12T01:09:29.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b456bce603bf12a7-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.331 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.332 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.332 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.333 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.333 DEBUG:boto:Method: GET 2023-10-12T01:09:29.334 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.334 DEBUG:boto:Data: 2023-10-12T01:09:29.335 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.335 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.336 DEBUG:boto:Params: {} 2023-10-12T01:09:29.336 DEBUG:boto:Token: None 2023-10-12T01:09:29.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M= 2023-10-12T01:09:29.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dddf3ec85fac3387-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.341 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.341 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.341 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.342 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.342 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.343 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.343 DEBUG:boto:Data: 2023-10-12T01:09:29.343 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.344 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.345 DEBUG:boto:Params: {} 2023-10-12T01:09:29.345 DEBUG:boto:Token: None 2023-10-12T01:09:29.345 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i0VRcOtZacM7Jcz/j99/KS1pO1s= 2023-10-12T01:09:29.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i0VRcOtZacM7Jcz/j99/KS1pO1s=', 'Content-Length': '0'} 2023-10-12T01:09:29.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf015d7ce93e4af-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.349 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.349 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.350 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.350 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.351 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.351 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.351 DEBUG:boto:Data: 2023-10-12T01:09:29.352 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.353 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.353 DEBUG:boto:Params: {} 2023-10-12T01:09:29.353 DEBUG:boto:Token: None 2023-10-12T01:09:29.354 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i0VRcOtZacM7Jcz/j99/KS1pO1s= 2023-10-12T01:09:29.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i0VRcOtZacM7Jcz/j99/KS1pO1s=', 'Content-Length': '0'} 2023-10-12T01:09:29.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c35173b723cda57-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:09:29.358 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.359 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.359 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.360 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.360 DEBUG:boto:Method: GET 2023-10-12T01:09:29.360 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.361 DEBUG:boto:Data: 2023-10-12T01:09:29.361 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.362 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.362 DEBUG:boto:Params: {} 2023-10-12T01:09:29.363 DEBUG:boto:Token: None 2023-10-12T01:09:29.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M= 2023-10-12T01:09:29.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a91ef38db99ae61-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.367 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.367 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.368 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.368 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.369 DEBUG:boto:Method: GET 2023-10-12T01:09:29.369 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.370 DEBUG:boto:Data: 2023-10-12T01:09:29.370 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.371 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.371 DEBUG:boto:Params: {} 2023-10-12T01:09:29.372 DEBUG:boto:Token: None 2023-10-12T01:09:29.372 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:09:29.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M= 2023-10-12T01:09:29.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M9q864dUih+XHrWTasp6mavGD0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550c34d1548fd6e4-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.376 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.377 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.377 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.378 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.378 DEBUG:boto:Method: GET 2023-10-12T01:09:29.379 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.379 DEBUG:boto:Data: 2023-10-12T01:09:29.380 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.380 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.381 DEBUG:boto:Params: {} 2023-10-12T01:09:29.381 DEBUG:boto:Token: None 2023-10-12T01:09:29.382 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.382 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ac6c43573e29ca-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.426 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.427 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.428 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.428 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.429 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.429 DEBUG:boto:Method: GET 2023-10-12T01:09:29.429 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.430 DEBUG:boto:Data: 2023-10-12T01:09:29.430 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.431 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.432 DEBUG:boto:Params: {} 2023-10-12T01:09:29.432 DEBUG:boto:Token: None 2023-10-12T01:09:29.433 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e195c04ad8e278-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.440 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.440 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.441 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.441 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.442 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:09:29.442 DEBUG:boto:Data: 2023-10-12T01:09:29.443 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.443 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.444 DEBUG:boto:Params: {} 2023-10-12T01:09:29.444 DEBUG:boto:Token: None 2023-10-12T01:09:29.445 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93 2023-10-12T01:09:29.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lt+5wwkn68S+lR3G1hqK7fIo3u8= 2023-10-12T01:09:29.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lt+5wwkn68S+lR3G1hqK7fIo3u8=', 'Content-Length': '0'} 2023-10-12T01:09:29.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3ea49844560a90-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.448 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.449 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.449 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.449 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:09:29.450 DEBUG:boto:Data: 2023-10-12T01:09:29.450 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.451 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.452 DEBUG:boto:Params: {} 2023-10-12T01:09:29.452 DEBUG:boto:Token: None 2023-10-12T01:09:29.452 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93 2023-10-12T01:09:29.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lt+5wwkn68S+lR3G1hqK7fIo3u8= 2023-10-12T01:09:29.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lt+5wwkn68S+lR3G1hqK7fIo3u8=', 'Content-Length': '0'} 2023-10-12T01:09:29.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654cb2d407afdd6e-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:09:29.456 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.457 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.457 DEBUG:boto:Method: GET 2023-10-12T01:09:29.457 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:09:29.458 DEBUG:boto:Data: 2023-10-12T01:09:29.458 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.459 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.460 DEBUG:boto:Params: {} 2023-10-12T01:09:29.460 DEBUG:boto:Token: None 2023-10-12T01:09:29.461 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93 2023-10-12T01:09:29.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DXFGpxADKB0JNghQ3jzK7XP8Fq0= 2023-10-12T01:09:29.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DXFGpxADKB0JNghQ3jzK7XP8Fq0=', 'Content-Length': '0'} 2023-10-12T01:09:29.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e2ca0c6af388b5-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.464 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:09:29.465 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:09:29.465 DEBUG:boto:Method: GET 2023-10-12T01:09:29.466 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:09:29.466 DEBUG:boto:Data: 2023-10-12T01:09:29.467 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.467 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.468 DEBUG:boto:Params: {} 2023-10-12T01:09:29.468 DEBUG:boto:Token: None 2023-10-12T01:09:29.469 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj93 2023-10-12T01:09:29.469 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DXFGpxADKB0JNghQ3jzK7XP8Fq0= 2023-10-12T01:09:29.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DXFGpxADKB0JNghQ3jzK7XP8Fq0=', 'Content-Length': '0'} 2023-10-12T01:09:29.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a624a7dac9f110-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.514 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.515 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.516 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.516 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.516 DEBUG:boto:Method: GET 2023-10-12T01:09:29.517 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.517 DEBUG:boto:Data: 2023-10-12T01:09:29.518 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.519 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.519 DEBUG:boto:Params: {} 2023-10-12T01:09:29.520 DEBUG:boto:Token: None 2023-10-12T01:09:29.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d6ba9e6593c6aa-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.524 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.525 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.525 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.526 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.526 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.527 DEBUG:boto:Method: GET 2023-10-12T01:09:29.527 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:09:29.527 DEBUG:boto:Data: 2023-10-12T01:09:29.528 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.529 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.529 DEBUG:boto:Params: {} 2023-10-12T01:09:29.530 DEBUG:boto:Token: None 2023-10-12T01:09:29.530 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb665432e8ae3356-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.535 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:09:29.536 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.537 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.537 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.538 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.538 DEBUG:boto:Method: GET 2023-10-12T01:09:29.539 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.539 DEBUG:boto:Data: 2023-10-12T01:09:29.539 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.540 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.541 DEBUG:boto:Params: {} 2023-10-12T01:09:29.541 DEBUG:boto:Token: None 2023-10-12T01:09:29.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo= 2023-10-12T01:09:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo=', 'Content-Length': '0'} 2023-10-12T01:09:29.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8a846795cb8cfb-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.545 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.546 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.546 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.547 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.547 DEBUG:boto:Method: GET 2023-10-12T01:09:29.548 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.548 DEBUG:boto:Data: 2023-10-12T01:09:29.548 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.549 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.550 DEBUG:boto:Params: {} 2023-10-12T01:09:29.550 DEBUG:boto:Token: None 2023-10-12T01:09:29.551 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo= 2023-10-12T01:09:29.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo=', 'Content-Length': '0'} 2023-10-12T01:09:29.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15b578fc23966dc-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.554 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.555 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.555 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.556 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.556 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.556 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.557 DEBUG:boto:Data: 2023-10-12T01:09:29.557 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.558 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.558 DEBUG:boto:Params: {} 2023-10-12T01:09:29.559 DEBUG:boto:Token: None 2023-10-12T01:09:29.559 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SNUiYiHDq82oFBW2KCEcRYqB19U= 2023-10-12T01:09:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SNUiYiHDq82oFBW2KCEcRYqB19U=', 'Content-Length': '0'} 2023-10-12T01:09:29.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef246b01c34c7a5d-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.563 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.563 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.563 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.564 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.564 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.565 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.565 DEBUG:boto:Data: 2023-10-12T01:09:29.566 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.566 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.567 DEBUG:boto:Params: {} 2023-10-12T01:09:29.567 DEBUG:boto:Token: None 2023-10-12T01:09:29.568 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.568 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SNUiYiHDq82oFBW2KCEcRYqB19U= 2023-10-12T01:09:29.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SNUiYiHDq82oFBW2KCEcRYqB19U=', 'Content-Length': '0'} 2023-10-12T01:09:29.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24ea81f4ac77bb1-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:09:29.572 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.572 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.572 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.573 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.573 DEBUG:boto:Method: GET 2023-10-12T01:09:29.574 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.574 DEBUG:boto:Data: 2023-10-12T01:09:29.575 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.575 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.576 DEBUG:boto:Params: {} 2023-10-12T01:09:29.576 DEBUG:boto:Token: None 2023-10-12T01:09:29.577 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo= 2023-10-12T01:09:29.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo=', 'Content-Length': '0'} 2023-10-12T01:09:29.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000486c9e4a846731eb-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.581 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.581 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.582 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.582 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.582 DEBUG:boto:Method: GET 2023-10-12T01:09:29.583 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.583 DEBUG:boto:Data: 2023-10-12T01:09:29.584 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.585 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.585 DEBUG:boto:Params: {} 2023-10-12T01:09:29.586 DEBUG:boto:Token: None 2023-10-12T01:09:29.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:09:29.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo= 2023-10-12T01:09:29.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6pHt4K06+d5mppGsTk2uJiaEwLo=', 'Content-Length': '0'} 2023-10-12T01:09:29.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7818f9bd7f4271b-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.630 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.631 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.631 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.632 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.632 DEBUG:boto:Method: GET 2023-10-12T01:09:29.633 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.633 DEBUG:boto:Data: 2023-10-12T01:09:29.633 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.634 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.635 DEBUG:boto:Params: {} 2023-10-12T01:09:29.635 DEBUG:boto:Token: None 2023-10-12T01:09:29.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed0bb64d83c941b-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.640 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.641 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.641 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.641 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.642 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.642 DEBUG:boto:Method: GET 2023-10-12T01:09:29.643 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.643 DEBUG:boto:Data: 2023-10-12T01:09:29.644 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.644 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.645 DEBUG:boto:Params: {} 2023-10-12T01:09:29.645 DEBUG:boto:Token: None 2023-10-12T01:09:29.646 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b966fb754e4041-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.651 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.651 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.652 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.652 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.653 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:09:29.653 DEBUG:boto:Data: 2023-10-12T01:09:29.653 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.654 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.655 DEBUG:boto:Params: {} 2023-10-12T01:09:29.655 DEBUG:boto:Token: None 2023-10-12T01:09:29.656 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95 2023-10-12T01:09:29.656 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hP64s1Vy38juTUoBsqWd3vBV/7o= 2023-10-12T01:09:29.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hP64s1Vy38juTUoBsqWd3vBV/7o=', 'Content-Length': '0'} 2023-10-12T01:09:29.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e65fd45ba17d221-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.659 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.660 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.660 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.660 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:09:29.661 DEBUG:boto:Data: 2023-10-12T01:09:29.661 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.662 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.662 DEBUG:boto:Params: {} 2023-10-12T01:09:29.663 DEBUG:boto:Token: None 2023-10-12T01:09:29.663 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95 2023-10-12T01:09:29.664 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hP64s1Vy38juTUoBsqWd3vBV/7o= 2023-10-12T01:09:29.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hP64s1Vy38juTUoBsqWd3vBV/7o=', 'Content-Length': '0'} 2023-10-12T01:09:29.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060817fef29e7f351-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:09:29.667 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.667 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.668 DEBUG:boto:Method: GET 2023-10-12T01:09:29.668 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:09:29.669 DEBUG:boto:Data: 2023-10-12T01:09:29.669 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.670 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.670 DEBUG:boto:Params: {} 2023-10-12T01:09:29.671 DEBUG:boto:Token: None 2023-10-12T01:09:29.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95 2023-10-12T01:09:29.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wtleI79oYHyKW6oLjn5iGwTlk0M= 2023-10-12T01:09:29.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wtleI79oYHyKW6oLjn5iGwTlk0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056128a115e043c71-0065274749-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.718 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:09:29.719 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:09:29.719 DEBUG:boto:Method: GET 2023-10-12T01:09:29.720 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:09:29.720 DEBUG:boto:Data: 2023-10-12T01:09:29.721 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.722 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.722 DEBUG:boto:Params: {} 2023-10-12T01:09:29.722 DEBUG:boto:Token: None 2023-10-12T01:09:29.723 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj95 2023-10-12T01:09:29.723 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wtleI79oYHyKW6oLjn5iGwTlk0M= 2023-10-12T01:09:29.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wtleI79oYHyKW6oLjn5iGwTlk0M=', 'Content-Length': '0'} 2023-10-12T01:09:29.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c1698c8eecb435-0065274749-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.727 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.727 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.728 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.728 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.728 DEBUG:boto:Method: GET 2023-10-12T01:09:29.729 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.729 DEBUG:boto:Data: 2023-10-12T01:09:29.730 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:09:29.731 DEBUG:boto:Port: 8001 2023-10-12T01:09:29.731 DEBUG:boto:Params: {} 2023-10-12T01:09:29.731 DEBUG:boto:Token: None 2023-10-12T01:09:29.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26fc60c66aabb65-0065274749-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.736 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.737 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.737 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.737 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.738 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.738 DEBUG:boto:Method: GET 2023-10-12T01:09:29.739 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:09:29.739 DEBUG:boto:Data: 2023-10-12T01:09:29.740 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.740 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.741 DEBUG:boto:Params: {} 2023-10-12T01:09:29.741 DEBUG:boto:Token: None 2023-10-12T01:09:29.742 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw= 2023-10-12T01:09:29.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/2sHdGWVVrAXesV0ajtprk3gxcw=', 'Content-Length': '0'} 2023-10-12T01:09:29.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b395975976277c-0065274749-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.747 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:09:29.748 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:09:29.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:09:29.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:09:29.749 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:09:29.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:09:29.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:09:29.855 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:09:29.856 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-59 2023-10-12T01:09:29.866 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-10-12T01:09:29.925 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.925 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.926 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.926 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:29.927 DEBUG:boto:Data: 2023-10-12T01:09:29.927 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.928 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.928 DEBUG:boto:Params: {} 2023-10-12T01:09:29.929 DEBUG:boto:Token: None 2023-10-12T01:09:29.929 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.930 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ob+TxmQWo0IgPGhbqWLdc/IAqAY= 2023-10-12T01:09:29.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ob+TxmQWo0IgPGhbqWLdc/IAqAY=', 'Content-Length': '0'} 2023-10-12T01:09:29.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062c4b12c33866845-0065274749-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.935 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:09:29.935 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:09:29.936 DEBUG:boto:Method: PUT 2023-10-12T01:09:29.936 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:09:29.937 DEBUG:boto:Data: 2023-10-12T01:09:29.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:09:29.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.938 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.938 DEBUG:boto:Params: {} 2023-10-12T01:09:29.939 DEBUG:boto:Token: None 2023-10-12T01:09:29.939 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj37 2023-10-12T01:09:29.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YSSCdd4q7Qh/Cf1amRK+LX3K5Nc= 2023-10-12T01:09:29.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YSSCdd4q7Qh/Cf1amRK+LX3K5Nc='} 2023-10-12T01:09:29.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cca2adc465f52ab1-0065274749-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.945 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:29.946 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:29.946 DEBUG:boto:Method: HEAD 2023-10-12T01:09:29.947 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:29.947 DEBUG:boto:Data: 2023-10-12T01:09:29.947 DEBUG:boto:Headers: {} 2023-10-12T01:09:29.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.948 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.949 DEBUG:boto:Params: {} 2023-10-12T01:09:29.949 DEBUG:boto:Token: None 2023-10-12T01:09:29.950 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/ 2023-10-12T01:09:29.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ob+TxmQWo0IgPGhbqWLdc/IAqAY= 2023-10-12T01:09:29.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ob+TxmQWo0IgPGhbqWLdc/IAqAY=', 'Content-Length': '0'} 2023-10-12T01:09:29.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007639f8e8d722efdd-0065274749-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:29.954 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:09:29.955 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:09:29.955 DEBUG:boto:Method: PUT 2023-10-12T01:09:29.955 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:09:29.956 DEBUG:boto:Data: 2023-10-12T01:09:29.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:09:29.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:29.957 DEBUG:boto:Port: 8000 2023-10-12T01:09:29.957 DEBUG:boto:Params: {} 2023-10-12T01:09:29.958 DEBUG:boto:Token: None 2023-10-12T01:09:29.958 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:09:29 GMT /wafmoa-59/obj74 2023-10-12T01:09:29.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WeXuqfmgenSsgHCAFeKxCfe7Y5E= 2023-10-12T01:09:29.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:09:29 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WeXuqfmgenSsgHCAFeKxCfe7Y5E='} 2023-10-12T01:09:30.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007a21215a41a8431-0065274749-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.007 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:30.008 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:30.008 DEBUG:boto:Method: HEAD 2023-10-12T01:09:30.008 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:30.009 DEBUG:boto:Data: 2023-10-12T01:09:30.009 DEBUG:boto:Headers: {} 2023-10-12T01:09:30.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.010 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.010 DEBUG:boto:Params: {} 2023-10-12T01:09:30.011 DEBUG:boto:Token: None 2023-10-12T01:09:30.011 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/ 2023-10-12T01:09:30.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU= 2023-10-12T01:09:30.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU=', 'Content-Length': '0'} 2023-10-12T01:09:30.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b666b57b4891707c-006527474a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.017 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:09:30.017 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:09:30.017 DEBUG:boto:Method: PUT 2023-10-12T01:09:30.018 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:09:30.018 DEBUG:boto:Data: 2023-10-12T01:09:30.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:09:30.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.020 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.020 DEBUG:boto:Params: {} 2023-10-12T01:09:30.020 DEBUG:boto:Token: None 2023-10-12T01:09:30.021 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/obj111 2023-10-12T01:09:30.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zyDr1iU4TO/ZM79jfNzMil1LSCo= 2023-10-12T01:09:30.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zyDr1iU4TO/ZM79jfNzMil1LSCo='} 2023-10-12T01:09:30.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4d3a963f4dfd92a-006527474a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.028 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:30.029 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:30.029 DEBUG:boto:Method: HEAD 2023-10-12T01:09:30.030 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:30.030 DEBUG:boto:Data: 2023-10-12T01:09:30.031 DEBUG:boto:Headers: {} 2023-10-12T01:09:30.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.032 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.032 DEBUG:boto:Params: {} 2023-10-12T01:09:30.033 DEBUG:boto:Token: None 2023-10-12T01:09:30.034 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/ 2023-10-12T01:09:30.034 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU= 2023-10-12T01:09:30.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU=', 'Content-Length': '0'} 2023-10-12T01:09:30.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfecca6ea60c2324-006527474a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.038 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:09:30.039 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:09:30.039 DEBUG:boto:Method: PUT 2023-10-12T01:09:30.040 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:09:30.040 DEBUG:boto:Data: 2023-10-12T01:09:30.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:09:30.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.041 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.042 DEBUG:boto:Params: {} 2023-10-12T01:09:30.042 DEBUG:boto:Token: None 2023-10-12T01:09:30.043 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/obj148 2023-10-12T01:09:30.043 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oP8/uLISHC/WhuiH4Shp299hvWs= 2023-10-12T01:09:30.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oP8/uLISHC/WhuiH4Shp299hvWs='} 2023-10-12T01:09:30.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000305db60fef418588-006527474a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.088 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:09:30.089 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:09:30.089 DEBUG:boto:Method: HEAD 2023-10-12T01:09:30.089 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:09:30.090 DEBUG:boto:Data: 2023-10-12T01:09:30.090 DEBUG:boto:Headers: {} 2023-10-12T01:09:30.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.091 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.092 DEBUG:boto:Params: {} 2023-10-12T01:09:30.093 DEBUG:boto:Token: None 2023-10-12T01:09:30.093 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/ 2023-10-12T01:09:30.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU= 2023-10-12T01:09:30.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zCaKeHV0eKcZKoaApg78Y+n/YkU=', 'Content-Length': '0'} 2023-10-12T01:09:30.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5efc25707c41e14-006527474a-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.098 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:09:30.099 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:09:30.099 DEBUG:boto:Method: PUT 2023-10-12T01:09:30.100 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:09:30.100 DEBUG:boto:Data: 2023-10-12T01:09:30.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:09:30.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:09:30.102 DEBUG:boto:Port: 8000 2023-10-12T01:09:30.102 DEBUG:boto:Params: {} 2023-10-12T01:09:30.103 DEBUG:boto:Token: None 2023-10-12T01:09:30.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:09:30 GMT /wafmoa-59/obj185 2023-10-12T01:09:30.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t+Z67mP7OShcvAOIFbanGlY5qQQ= 2023-10-12T01:09:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:09:30 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t+Z67mP7OShcvAOIFbanGlY5qQQ='} 2023-10-12T01:09:30.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a124a2e3d3f432bd-006527474a-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:09:30.110 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:09:30.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:09:30.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:09:30.112 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:09:30.168 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:09:30.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:09:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:30.202+0000 7fe832cce780 1 waiting to reach latest gen 6, on 5.. 2023-10-12T01:09:35.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:35.203+0000 7fe832cce780 1 waiting to reach latest gen 6, on 5.. 2023-10-12T01:09:40.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:40.204+0000 7fe832cce780 1 waiting to reach latest gen 6, on 5.. 2023-10-12T01:09:45.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:45.207+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:45.210 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:09:45.211 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:09:50.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:50.210+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:50.210 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:09:50.211 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:09:55.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:09:55.212+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:09:55.213 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:09:55.213 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:00.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:00.214+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:00.215 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:00.215 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:05.216+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:05.218 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:05.218 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:10.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:10.219+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:10.220 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:10.220 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:15.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:15.222+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:15.223 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:15.224 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:20.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:20.226+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:20.226 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:20.226 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:25.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:25.229+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:25.229 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.75.6, , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:25.229 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:30.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:30.230+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:30.231 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.75.6, , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:30.232 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:35.233+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:35.234 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.75.6, , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:35.234 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:40.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:40.235+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:40.237 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.75.6, , , , , , , , , , , , , , , , , , , ] 2023-10-12T01:10:40.237 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:45.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:45.238+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:45.239 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:45.239 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:50.240+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:50.242 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:50.242 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:10:55.244+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:10:55.244 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:10:55.245 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:00.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:00.247+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:00.247 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:00.247 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:05.250+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:05.250 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:05.250 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:10.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:10.251+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:10.253 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:10.253 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:15.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:15.254+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:15.255 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:15.255 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:20.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:20.257+0000 7fe832cce780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:20.258 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:20.258 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:25.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:25.259+0000 7fe832cce780 1 bucket sync caught up with source: 2023-10-12T01:11:25.261 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:25.261 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:11:25.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:25.259+0000 7fe832cce780 0 bucket checkpoint complete 2023-10-12T01:11:25.278 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:11:25.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:11:25.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:11:25.280 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:11:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:25.384+0000 7f3bd212a780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:25.385 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000005.243.6] 2023-10-12T01:11:25.386 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000006.245.6] 2023-10-12T01:11:30.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:30.386+0000 7f3bd212a780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:30.387 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000005.243.6] 2023-10-12T01:11:30.387 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000006.245.6] 2023-10-12T01:11:35.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:35.388+0000 7f3bd212a780 1 waiting for incremental sync to catch up: 2023-10-12T01:11:35.388 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000005.243.6] 2023-10-12T01:11:35.388 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000006.245.6] 2023-10-12T01:11:40.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:40.389+0000 7f3bd212a780 1 bucket sync caught up with source: 2023-10-12T01:11:40.389 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000006.245.6] 2023-10-12T01:11:40.390 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.340.6, 00000000002.299.6, 00000000002.193.6, 00000000005.242.6, 00000000004.225.6, 00000000001.140.6, 00000000001.305.6, 00000000004.317.6, 00000000007.256.6, 00000000006.245.6] 2023-10-12T01:11:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:40.389+0000 7f3bd212a780 0 bucket checkpoint complete 2023-10-12T01:11:40.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:11:40.406 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.407 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.407 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.407 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:11:40.408 DEBUG:boto:Data: 2023-10-12T01:11:40.408 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.409 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.409 DEBUG:boto:Params: {} 2023-10-12T01:11:40.410 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:11:40.411 DEBUG:boto:Token: None 2023-10-12T01:11:40.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.412 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mVr98fErgz8xS0kdBiCv5cfGJt0= 2023-10-12T01:11:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mVr98fErgz8xS0kdBiCv5cfGJt0=', 'Content-Length': '0'} 2023-10-12T01:11:40.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026adeee9af8936f7-00652747cc-1bba-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.418 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.418 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.419 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.419 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:11:40.420 DEBUG:boto:Data: 2023-10-12T01:11:40.420 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.421 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.421 DEBUG:boto:Params: {} 2023-10-12T01:11:40.422 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:11:40.423 DEBUG:boto:Token: None 2023-10-12T01:11:40.423 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mVr98fErgz8xS0kdBiCv5cfGJt0= 2023-10-12T01:11:40.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mVr98fErgz8xS0kdBiCv5cfGJt0=', 'Content-Length': '0'} 2023-10-12T01:11:40.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031cbc14628cd669b-00652747cc-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.430 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:11:40.431 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.431 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.432 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.432 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.432 DEBUG:boto:Method: GET 2023-10-12T01:11:40.433 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.433 DEBUG:boto:Data: 2023-10-12T01:11:40.434 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.435 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.435 DEBUG:boto:Params: {} 2023-10-12T01:11:40.436 DEBUG:boto:Token: None 2023-10-12T01:11:40.436 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/?versions 2023-10-12T01:11:40.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI= 2023-10-12T01:11:40.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI=', 'Content-Length': '0'} 2023-10-12T01:11:40.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000271b2b1e34ebff6-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.441 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:11:40.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:11:40.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:11:40.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:11:40.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:11:40.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:11:40.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:11:40.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:11:40.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:11:40.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:11:40.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:11:40.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:11:40.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:11:40.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:11:40.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:11:40.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:11:40.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:11:40.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:11:40.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:11:40.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:11:40.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:11:40.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:11:40.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:11:40.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:11:40.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:11:40.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:11:40.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:11:40.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:11:40.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:11:40.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:11:40.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:11:40.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:11:40.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:11:40.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:11:40.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:11:40.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:11:40.460 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.461 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.461 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.462 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.462 DEBUG:boto:Method: GET 2023-10-12T01:11:40.463 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.463 DEBUG:boto:Data: 2023-10-12T01:11:40.463 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.464 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.465 DEBUG:boto:Params: {} 2023-10-12T01:11:40.465 DEBUG:boto:Token: None 2023-10-12T01:11:40.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/?versions 2023-10-12T01:11:40.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI= 2023-10-12T01:11:40.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI=', 'Content-Length': '0'} 2023-10-12T01:11:40.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda9a84ca42de396-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.472 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:11:40.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:11:40.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:11:40.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:11:40.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:11:40.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:11:40.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:11:40.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:11:40.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:11:40.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:11:40.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:11:40.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:11:40.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:11:40.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:11:40.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:11:40.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:11:40.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:11:40.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:11:40.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:11:40.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:11:40.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:11:40.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:11:40.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:11:40.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:11:40.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:11:40.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:11:40.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:11:40.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:11:40.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:11:40.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:11:40.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:11:40.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:11:40.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:11:40.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:11:40.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:11:40.489 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.489 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.489 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.490 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.490 DEBUG:boto:Method: GET 2023-10-12T01:11:40.491 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.491 DEBUG:boto:Data: 2023-10-12T01:11:40.492 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.493 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.493 DEBUG:boto:Params: {} 2023-10-12T01:11:40.494 DEBUG:boto:Token: None 2023-10-12T01:11:40.494 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/?versions 2023-10-12T01:11:40.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI= 2023-10-12T01:11:40.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI=', 'Content-Length': '0'} 2023-10-12T01:11:40.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513c374a4bb474f5-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.499 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.501 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.501 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.502 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.502 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.503 DEBUG:boto:Method: GET 2023-10-12T01:11:40.503 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:11:40.504 DEBUG:boto:Data: 2023-10-12T01:11:40.504 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.505 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.505 DEBUG:boto:Params: {} 2023-10-12T01:11:40.506 DEBUG:boto:Token: None 2023-10-12T01:11:40.506 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/?versions 2023-10-12T01:11:40.507 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI= 2023-10-12T01:11:40.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LyPcmig3tyD/nVCGSP5HGBDCSGI=', 'Content-Length': '0'} 2023-10-12T01:11:40.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee9b10b4529e76f-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.512 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:11:40.515 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.516 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.516 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.517 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.517 DEBUG:boto:Method: GET 2023-10-12T01:11:40.518 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.518 DEBUG:boto:Data: 2023-10-12T01:11:40.518 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.519 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.520 DEBUG:boto:Params: {} 2023-10-12T01:11:40.520 DEBUG:boto:Token: None 2023-10-12T01:11:40.521 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.521 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc= 2023-10-12T01:11:40.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc=', 'Content-Length': '0'} 2023-10-12T01:11:40.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000001201ed3c31173a-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.525 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.526 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.526 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.527 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.527 DEBUG:boto:Method: GET 2023-10-12T01:11:40.528 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.528 DEBUG:boto:Data: 2023-10-12T01:11:40.529 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.530 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.530 DEBUG:boto:Params: {} 2023-10-12T01:11:40.531 DEBUG:boto:Token: None 2023-10-12T01:11:40.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc= 2023-10-12T01:11:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc=', 'Content-Length': '0'} 2023-10-12T01:11:40.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4da69566b0d31bc-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.535 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.535 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.536 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.536 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.537 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.537 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.538 DEBUG:boto:Data: 2023-10-12T01:11:40.538 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.539 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.540 DEBUG:boto:Params: {} 2023-10-12T01:11:40.540 DEBUG:boto:Token: None 2023-10-12T01:11:40.541 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RaKSrP5QrDpNYqh2vpNoTIFrT3g= 2023-10-12T01:11:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RaKSrP5QrDpNYqh2vpNoTIFrT3g=', 'Content-Length': '0'} 2023-10-12T01:11:40.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083b1eae41ca2e488-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.544 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.545 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.545 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.546 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.546 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.547 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.547 DEBUG:boto:Data: 2023-10-12T01:11:40.547 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.548 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.549 DEBUG:boto:Params: {} 2023-10-12T01:11:40.549 DEBUG:boto:Token: None 2023-10-12T01:11:40.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RaKSrP5QrDpNYqh2vpNoTIFrT3g= 2023-10-12T01:11:40.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RaKSrP5QrDpNYqh2vpNoTIFrT3g=', 'Content-Length': '0'} 2023-10-12T01:11:40.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c39ecde6d2682a-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:11:40.554 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.554 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.555 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.555 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.556 DEBUG:boto:Method: GET 2023-10-12T01:11:40.556 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.557 DEBUG:boto:Data: 2023-10-12T01:11:40.557 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.558 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.558 DEBUG:boto:Params: {} 2023-10-12T01:11:40.559 DEBUG:boto:Token: None 2023-10-12T01:11:40.559 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc= 2023-10-12T01:11:40.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc=', 'Content-Length': '0'} 2023-10-12T01:11:40.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008efaebbabb2af19c-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.564 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.564 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.565 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.565 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.566 DEBUG:boto:Method: GET 2023-10-12T01:11:40.566 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.567 DEBUG:boto:Data: 2023-10-12T01:11:40.567 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.568 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.568 DEBUG:boto:Params: {} 2023-10-12T01:11:40.569 DEBUG:boto:Token: None 2023-10-12T01:11:40.569 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:11:40.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc= 2023-10-12T01:11:40.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kP8utknxk8klqzJy5A86XwTWaCc=', 'Content-Length': '0'} 2023-10-12T01:11:40.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df412acde1c13f09-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.618 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.619 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.619 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.620 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.620 DEBUG:boto:Method: GET 2023-10-12T01:11:40.621 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.621 DEBUG:boto:Data: 2023-10-12T01:11:40.622 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.623 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.623 DEBUG:boto:Params: {} 2023-10-12T01:11:40.624 DEBUG:boto:Token: None 2023-10-12T01:11:40.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1db15410549607-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.629 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.630 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.631 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.631 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.631 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.632 DEBUG:boto:Method: GET 2023-10-12T01:11:40.632 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.633 DEBUG:boto:Data: 2023-10-12T01:11:40.633 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.634 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.634 DEBUG:boto:Params: {} 2023-10-12T01:11:40.635 DEBUG:boto:Token: None 2023-10-12T01:11:40.635 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043659257f540a12a-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.641 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.642 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.642 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.643 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.643 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:11:40.644 DEBUG:boto:Data: 2023-10-12T01:11:40.644 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.645 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.645 DEBUG:boto:Params: {} 2023-10-12T01:11:40.646 DEBUG:boto:Token: None 2023-10-12T01:11:40.646 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11 2023-10-12T01:11:40.647 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7RyPLJtHAvVCRi9TH3ycJoj/VZQ= 2023-10-12T01:11:40.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7RyPLJtHAvVCRi9TH3ycJoj/VZQ=', 'Content-Length': '0'} 2023-10-12T01:11:40.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bcd99e2bf300ff9-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.650 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.650 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.651 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.651 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:11:40.652 DEBUG:boto:Data: 2023-10-12T01:11:40.652 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.653 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.653 DEBUG:boto:Params: {} 2023-10-12T01:11:40.654 DEBUG:boto:Token: None 2023-10-12T01:11:40.654 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11 2023-10-12T01:11:40.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7RyPLJtHAvVCRi9TH3ycJoj/VZQ= 2023-10-12T01:11:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7RyPLJtHAvVCRi9TH3ycJoj/VZQ=', 'Content-Length': '0'} 2023-10-12T01:11:40.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a181215dc3dc54-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:11:40.658 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.659 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.659 DEBUG:boto:Method: GET 2023-10-12T01:11:40.659 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:11:40.660 DEBUG:boto:Data: 2023-10-12T01:11:40.660 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.661 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.662 DEBUG:boto:Params: {} 2023-10-12T01:11:40.662 DEBUG:boto:Token: None 2023-10-12T01:11:40.663 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11 2023-10-12T01:11:40.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aOkDUD2QrkrcvtkMwubIooxlAhY= 2023-10-12T01:11:40.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aOkDUD2QrkrcvtkMwubIooxlAhY=', 'Content-Length': '0'} 2023-10-12T01:11:40.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc586ac9f037aee5-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.668 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:11:40.669 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:11:40.669 DEBUG:boto:Method: GET 2023-10-12T01:11:40.670 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:11:40.670 DEBUG:boto:Data: 2023-10-12T01:11:40.671 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.672 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.672 DEBUG:boto:Params: {} 2023-10-12T01:11:40.673 DEBUG:boto:Token: None 2023-10-12T01:11:40.673 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj11 2023-10-12T01:11:40.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aOkDUD2QrkrcvtkMwubIooxlAhY= 2023-10-12T01:11:40.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aOkDUD2QrkrcvtkMwubIooxlAhY=', 'Content-Length': '0'} 2023-10-12T01:11:40.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e7ad6962b888e5-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.677 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.677 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.678 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.678 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.679 DEBUG:boto:Method: GET 2023-10-12T01:11:40.679 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.680 DEBUG:boto:Data: 2023-10-12T01:11:40.681 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.682 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.682 DEBUG:boto:Params: {} 2023-10-12T01:11:40.683 DEBUG:boto:Token: None 2023-10-12T01:11:40.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d74d3652c1a9e1-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.688 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.689 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.689 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.690 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.691 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.691 DEBUG:boto:Method: GET 2023-10-12T01:11:40.692 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:11:40.692 DEBUG:boto:Data: 2023-10-12T01:11:40.693 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.694 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.694 DEBUG:boto:Params: {} 2023-10-12T01:11:40.694 DEBUG:boto:Token: None 2023-10-12T01:11:40.695 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.696 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006108e8861b917482-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.701 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:11:40.702 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.703 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.703 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.703 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.704 DEBUG:boto:Method: GET 2023-10-12T01:11:40.704 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.705 DEBUG:boto:Data: 2023-10-12T01:11:40.705 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.706 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.706 DEBUG:boto:Params: {} 2023-10-12T01:11:40.707 DEBUG:boto:Token: None 2023-10-12T01:11:40.707 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k= 2023-10-12T01:11:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k=', 'Content-Length': '0'} 2023-10-12T01:11:40.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fe513c1d173557-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.711 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.712 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.712 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.713 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.713 DEBUG:boto:Method: GET 2023-10-12T01:11:40.714 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.714 DEBUG:boto:Data: 2023-10-12T01:11:40.714 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.715 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.716 DEBUG:boto:Params: {} 2023-10-12T01:11:40.716 DEBUG:boto:Token: None 2023-10-12T01:11:40.717 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k= 2023-10-12T01:11:40.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k=', 'Content-Length': '0'} 2023-10-12T01:11:40.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2cb4203b18caab0-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.762 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.763 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.764 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.764 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.764 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.765 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.765 DEBUG:boto:Data: 2023-10-12T01:11:40.766 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.766 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.767 DEBUG:boto:Params: {} 2023-10-12T01:11:40.767 DEBUG:boto:Token: None 2023-10-12T01:11:40.768 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8Vrj62dI+/ctNrVNc2Icvq+SBE= 2023-10-12T01:11:40.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8Vrj62dI+/ctNrVNc2Icvq+SBE=', 'Content-Length': '0'} 2023-10-12T01:11:40.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000258da2425518a0-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.772 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.772 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.772 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.773 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.774 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.774 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.774 DEBUG:boto:Data: 2023-10-12T01:11:40.775 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.776 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.776 DEBUG:boto:Params: {} 2023-10-12T01:11:40.776 DEBUG:boto:Token: None 2023-10-12T01:11:40.777 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.777 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8Vrj62dI+/ctNrVNc2Icvq+SBE= 2023-10-12T01:11:40.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8Vrj62dI+/ctNrVNc2Icvq+SBE=', 'Content-Length': '0'} 2023-10-12T01:11:40.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ccb36a40fdd3f2-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:11:40.781 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.781 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.782 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.782 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.783 DEBUG:boto:Method: GET 2023-10-12T01:11:40.783 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.783 DEBUG:boto:Data: 2023-10-12T01:11:40.784 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.785 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.785 DEBUG:boto:Params: {} 2023-10-12T01:11:40.786 DEBUG:boto:Token: None 2023-10-12T01:11:40.786 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k= 2023-10-12T01:11:40.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k=', 'Content-Length': '0'} 2023-10-12T01:11:40.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb8c6fdd14153c2-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.830 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.831 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.831 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.832 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.832 DEBUG:boto:Method: GET 2023-10-12T01:11:40.833 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.833 DEBUG:boto:Data: 2023-10-12T01:11:40.833 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.834 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.835 DEBUG:boto:Params: {} 2023-10-12T01:11:40.835 DEBUG:boto:Token: None 2023-10-12T01:11:40.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:11:40.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k= 2023-10-12T01:11:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H90AUZiU5tb8L9JWhBVjs8h+95k=', 'Content-Length': '0'} 2023-10-12T01:11:40.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b23d0334be49d056-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.840 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.840 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.840 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.841 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.841 DEBUG:boto:Method: GET 2023-10-12T01:11:40.842 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.842 DEBUG:boto:Data: 2023-10-12T01:11:40.843 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.843 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.844 DEBUG:boto:Params: {} 2023-10-12T01:11:40.844 DEBUG:boto:Token: None 2023-10-12T01:11:40.845 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b153d4bf281bff4a-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.849 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.850 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.850 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.851 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.851 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.852 DEBUG:boto:Method: GET 2023-10-12T01:11:40.852 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.853 DEBUG:boto:Data: 2023-10-12T01:11:40.853 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.854 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.854 DEBUG:boto:Params: {} 2023-10-12T01:11:40.855 DEBUG:boto:Token: None 2023-10-12T01:11:40.855 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.856 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9b58f9776f9ca27-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.861 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.862 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.862 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.862 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.863 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:11:40.863 DEBUG:boto:Data: 2023-10-12T01:11:40.864 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.864 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.865 DEBUG:boto:Params: {} 2023-10-12T01:11:40.865 DEBUG:boto:Token: None 2023-10-12T01:11:40.866 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111 2023-10-12T01:11:40.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KIxT8vy1M3sDC49LecCCTUXdziA= 2023-10-12T01:11:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KIxT8vy1M3sDC49LecCCTUXdziA=', 'Content-Length': '0'} 2023-10-12T01:11:40.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb06d97f914af0fa-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.869 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.869 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.870 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.870 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:11:40.871 DEBUG:boto:Data: 2023-10-12T01:11:40.871 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.872 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.872 DEBUG:boto:Params: {} 2023-10-12T01:11:40.873 DEBUG:boto:Token: None 2023-10-12T01:11:40.873 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111 2023-10-12T01:11:40.873 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KIxT8vy1M3sDC49LecCCTUXdziA= 2023-10-12T01:11:40.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KIxT8vy1M3sDC49LecCCTUXdziA=', 'Content-Length': '0'} 2023-10-12T01:11:40.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a47e9db3d74f264d-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:11:40.877 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.877 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.878 DEBUG:boto:Method: GET 2023-10-12T01:11:40.878 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:11:40.879 DEBUG:boto:Data: 2023-10-12T01:11:40.879 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.880 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.881 DEBUG:boto:Params: {} 2023-10-12T01:11:40.881 DEBUG:boto:Token: None 2023-10-12T01:11:40.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111 2023-10-12T01:11:40.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5/zfQ/+U8A/gCrlswx1U4w+4+MY= 2023-10-12T01:11:40.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5/zfQ/+U8A/gCrlswx1U4w+4+MY=', 'Content-Length': '0'} 2023-10-12T01:11:40.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab1c745b05dcfdc-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.885 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:11:40.886 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:11:40.886 DEBUG:boto:Method: GET 2023-10-12T01:11:40.887 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:11:40.887 DEBUG:boto:Data: 2023-10-12T01:11:40.887 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.888 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.889 DEBUG:boto:Params: {} 2023-10-12T01:11:40.889 DEBUG:boto:Token: None 2023-10-12T01:11:40.890 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj111 2023-10-12T01:11:40.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5/zfQ/+U8A/gCrlswx1U4w+4+MY= 2023-10-12T01:11:40.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5/zfQ/+U8A/gCrlswx1U4w+4+MY=', 'Content-Length': '0'} 2023-10-12T01:11:40.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008110f54e7c70641f-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.934 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.935 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.936 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.936 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.936 DEBUG:boto:Method: GET 2023-10-12T01:11:40.937 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.937 DEBUG:boto:Data: 2023-10-12T01:11:40.938 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.939 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.939 DEBUG:boto:Params: {} 2023-10-12T01:11:40.940 DEBUG:boto:Token: None 2023-10-12T01:11:40.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008450a9cf863897be-00652747cc-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.945 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.946 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:40.946 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:40.947 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.947 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.948 DEBUG:boto:Method: GET 2023-10-12T01:11:40.948 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:11:40.949 DEBUG:boto:Data: 2023-10-12T01:11:40.949 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.950 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.950 DEBUG:boto:Params: {} 2023-10-12T01:11:40.951 DEBUG:boto:Token: None 2023-10-12T01:11:40.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/ 2023-10-12T01:11:40.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s= 2023-10-12T01:11:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4N6nuEgER7V9Ex+pVDoyksxAw2s=', 'Content-Length': '0'} 2023-10-12T01:11:40.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066057363c7a0f626-00652747cc-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.957 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:40.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:11:40.958 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:40.959 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:40.959 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.960 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.960 DEBUG:boto:Method: GET 2023-10-12T01:11:40.961 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.961 DEBUG:boto:Data: 2023-10-12T01:11:40.961 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.962 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.963 DEBUG:boto:Params: {} 2023-10-12T01:11:40.963 DEBUG:boto:Token: None 2023-10-12T01:11:40.964 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDxEKQ+evk84nhAfm3P+9eIOSTA= 2023-10-12T01:11:40.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDxEKQ+evk84nhAfm3P+9eIOSTA=', 'Content-Length': '0'} 2023-10-12T01:11:40.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053356048ddb48c0c-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.968 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:40.968 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:40.968 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.969 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.969 DEBUG:boto:Method: GET 2023-10-12T01:11:40.970 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.970 DEBUG:boto:Data: 2023-10-12T01:11:40.971 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.972 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.972 DEBUG:boto:Params: {} 2023-10-12T01:11:40.972 DEBUG:boto:Token: None 2023-10-12T01:11:40.973 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.973 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BDxEKQ+evk84nhAfm3P+9eIOSTA= 2023-10-12T01:11:40.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BDxEKQ+evk84nhAfm3P+9eIOSTA=', 'Content-Length': '0'} 2023-10-12T01:11:40.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68d56e0f852f811-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.977 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:40.978 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:40.978 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.979 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.979 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.980 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.980 DEBUG:boto:Data: 2023-10-12T01:11:40.981 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:40.981 DEBUG:boto:Port: 8001 2023-10-12T01:11:40.982 DEBUG:boto:Params: {} 2023-10-12T01:11:40.982 DEBUG:boto:Token: None 2023-10-12T01:11:40.983 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.984 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+KoihnabCg491SuAcafpF0v5WQ= 2023-10-12T01:11:40.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+KoihnabCg491SuAcafpF0v5WQ=', 'Content-Length': '0'} 2023-10-12T01:11:40.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089655fda975cdd22-00652747cc-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.987 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:40.987 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:40.988 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.988 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.989 DEBUG:boto:Method: HEAD 2023-10-12T01:11:40.989 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.990 DEBUG:boto:Data: 2023-10-12T01:11:40.990 DEBUG:boto:Headers: {} 2023-10-12T01:11:40.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:40.991 DEBUG:boto:Port: 8000 2023-10-12T01:11:40.991 DEBUG:boto:Params: {} 2023-10-12T01:11:40.992 DEBUG:boto:Token: None 2023-10-12T01:11:40.992 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:40 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I+KoihnabCg491SuAcafpF0v5WQ= 2023-10-12T01:11:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I+KoihnabCg491SuAcafpF0v5WQ=', 'Content-Length': '0'} 2023-10-12T01:11:40.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d68d01da2d8319-00652747cc-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:40.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:11:40.996 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:40.997 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:40.997 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.998 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.998 DEBUG:boto:Method: GET 2023-10-12T01:11:40.999 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:40.999 DEBUG:boto:Data: 2023-10-12T01:11:41.000 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.000 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.001 DEBUG:boto:Params: {} 2023-10-12T01:11:41.002 DEBUG:boto:Token: None 2023-10-12T01:11:41.003 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:41.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pMR6GKoLwd0fTnD/lCg0cr7kFaw= 2023-10-12T01:11:41.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pMR6GKoLwd0fTnD/lCg0cr7kFaw=', 'Content-Length': '0'} 2023-10-12T01:11:41.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba174a21b5d49a7-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.008 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:41.009 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:41.010 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:41.011 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:11:41.011 DEBUG:boto:Method: GET 2023-10-12T01:11:41.012 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:41.013 DEBUG:boto:Data: 2023-10-12T01:11:41.014 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.016 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.016 DEBUG:boto:Params: {} 2023-10-12T01:11:41.017 DEBUG:boto:Token: None 2023-10-12T01:11:41.018 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:11:41.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pMR6GKoLwd0fTnD/lCg0cr7kFaw= 2023-10-12T01:11:41.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pMR6GKoLwd0fTnD/lCg0cr7kFaw=', 'Content-Length': '0'} 2023-10-12T01:11:41.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50960db8149086c-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.066 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.067 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.068 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.068 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.069 DEBUG:boto:Method: GET 2023-10-12T01:11:41.069 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.069 DEBUG:boto:Data: 2023-10-12T01:11:41.070 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.071 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.071 DEBUG:boto:Params: {} 2023-10-12T01:11:41.072 DEBUG:boto:Token: None 2023-10-12T01:11:41.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c69585fdead6d03-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.078 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.079 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.079 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.080 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.080 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.081 DEBUG:boto:Method: GET 2023-10-12T01:11:41.081 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.082 DEBUG:boto:Data: 2023-10-12T01:11:41.083 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.084 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.084 DEBUG:boto:Params: {} 2023-10-12T01:11:41.085 DEBUG:boto:Token: None 2023-10-12T01:11:41.086 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.087 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000480f491426d1ac4a-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.091 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.092 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:41.093 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:41.093 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.094 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:11:41.095 DEBUG:boto:Data: 2023-10-12T01:11:41.095 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.097 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.098 DEBUG:boto:Params: {} 2023-10-12T01:11:41.098 DEBUG:boto:Token: None 2023-10-12T01:11:41.099 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115 2023-10-12T01:11:41.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZtA/HK5DxnoCCjXjxYwiJzR+WyU= 2023-10-12T01:11:41.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZtA/HK5DxnoCCjXjxYwiJzR+WyU=', 'Content-Length': '0'} 2023-10-12T01:11:41.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c2e75e64bc314a0-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.104 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:41.104 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:41.104 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.105 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:11:41.105 DEBUG:boto:Data: 2023-10-12T01:11:41.106 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.106 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.107 DEBUG:boto:Params: {} 2023-10-12T01:11:41.107 DEBUG:boto:Token: None 2023-10-12T01:11:41.108 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115 2023-10-12T01:11:41.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZtA/HK5DxnoCCjXjxYwiJzR+WyU= 2023-10-12T01:11:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZtA/HK5DxnoCCjXjxYwiJzR+WyU=', 'Content-Length': '0'} 2023-10-12T01:11:41.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe506aceaa0e48f-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:11:41.111 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:41.112 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:41.112 DEBUG:boto:Method: GET 2023-10-12T01:11:41.112 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:11:41.113 DEBUG:boto:Data: 2023-10-12T01:11:41.113 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.114 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.115 DEBUG:boto:Params: {} 2023-10-12T01:11:41.115 DEBUG:boto:Token: None 2023-10-12T01:11:41.115 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115 2023-10-12T01:11:41.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EnD94vXfkyGAzOkkijNhwmEHRnQ= 2023-10-12T01:11:41.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EnD94vXfkyGAzOkkijNhwmEHRnQ=', 'Content-Length': '0'} 2023-10-12T01:11:41.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027dfcea68d719bd3-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.166 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:11:41.167 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:11:41.168 DEBUG:boto:Method: GET 2023-10-12T01:11:41.168 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:11:41.168 DEBUG:boto:Data: 2023-10-12T01:11:41.169 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.170 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.170 DEBUG:boto:Params: {} 2023-10-12T01:11:41.171 DEBUG:boto:Token: None 2023-10-12T01:11:41.171 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj115 2023-10-12T01:11:41.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EnD94vXfkyGAzOkkijNhwmEHRnQ= 2023-10-12T01:11:41.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EnD94vXfkyGAzOkkijNhwmEHRnQ=', 'Content-Length': '0'} 2023-10-12T01:11:41.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185feebb22e54f5a-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.175 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.176 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.176 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.176 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.177 DEBUG:boto:Method: GET 2023-10-12T01:11:41.177 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.178 DEBUG:boto:Data: 2023-10-12T01:11:41.178 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.179 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.180 DEBUG:boto:Params: {} 2023-10-12T01:11:41.180 DEBUG:boto:Token: None 2023-10-12T01:11:41.181 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd51978bba25e5f9-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.185 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.186 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.186 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.187 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.187 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.187 DEBUG:boto:Method: GET 2023-10-12T01:11:41.188 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:11:41.188 DEBUG:boto:Data: 2023-10-12T01:11:41.189 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.190 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.190 DEBUG:boto:Params: {} 2023-10-12T01:11:41.191 DEBUG:boto:Token: None 2023-10-12T01:11:41.191 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003580385d93e8ddb3-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.242 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:11:41.244 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.244 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.245 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.245 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.245 DEBUG:boto:Method: GET 2023-10-12T01:11:41.246 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.246 DEBUG:boto:Data: 2023-10-12T01:11:41.247 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.248 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.248 DEBUG:boto:Params: {} 2023-10-12T01:11:41.249 DEBUG:boto:Token: None 2023-10-12T01:11:41.249 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU= 2023-10-12T01:11:41.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU=', 'Content-Length': '0'} 2023-10-12T01:11:41.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919d8f3937d7f261-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.253 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.254 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.254 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.255 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.255 DEBUG:boto:Method: GET 2023-10-12T01:11:41.256 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.257 DEBUG:boto:Data: 2023-10-12T01:11:41.257 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.259 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.260 DEBUG:boto:Params: {} 2023-10-12T01:11:41.260 DEBUG:boto:Token: None 2023-10-12T01:11:41.261 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.262 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU= 2023-10-12T01:11:41.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU=', 'Content-Length': '0'} 2023-10-12T01:11:41.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540ee3abead03089-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.267 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.267 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.268 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.268 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.268 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.269 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.269 DEBUG:boto:Data: 2023-10-12T01:11:41.270 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.271 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.271 DEBUG:boto:Params: {} 2023-10-12T01:11:41.271 DEBUG:boto:Token: None 2023-10-12T01:11:41.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZGsXOLj/st2GDK6EHtRAqFN3Tw= 2023-10-12T01:11:41.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZGsXOLj/st2GDK6EHtRAqFN3Tw=', 'Content-Length': '0'} 2023-10-12T01:11:41.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07720d6784003b3-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.275 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.276 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.277 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.278 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.278 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.278 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.279 DEBUG:boto:Data: 2023-10-12T01:11:41.279 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.280 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.280 DEBUG:boto:Params: {} 2023-10-12T01:11:41.281 DEBUG:boto:Token: None 2023-10-12T01:11:41.281 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tZGsXOLj/st2GDK6EHtRAqFN3Tw= 2023-10-12T01:11:41.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tZGsXOLj/st2GDK6EHtRAqFN3Tw=', 'Content-Length': '0'} 2023-10-12T01:11:41.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f076000f6427bbc4-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:11:41.286 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.286 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.286 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.287 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.287 DEBUG:boto:Method: GET 2023-10-12T01:11:41.288 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.288 DEBUG:boto:Data: 2023-10-12T01:11:41.289 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.289 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.290 DEBUG:boto:Params: {} 2023-10-12T01:11:41.290 DEBUG:boto:Token: None 2023-10-12T01:11:41.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU= 2023-10-12T01:11:41.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU=', 'Content-Length': '0'} 2023-10-12T01:11:41.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9e9de5daaa1c5b-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.296 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.297 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.298 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.298 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.299 DEBUG:boto:Method: GET 2023-10-12T01:11:41.300 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.301 DEBUG:boto:Data: 2023-10-12T01:11:41.301 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.302 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.303 DEBUG:boto:Params: {} 2023-10-12T01:11:41.303 DEBUG:boto:Token: None 2023-10-12T01:11:41.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:11:41.304 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU= 2023-10-12T01:11:41.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jYMmNpuv0URpO353C5bYphzIaWU=', 'Content-Length': '0'} 2023-10-12T01:11:41.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1cb3d48451e41f5-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.307 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.308 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.308 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.308 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.309 DEBUG:boto:Method: GET 2023-10-12T01:11:41.309 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.310 DEBUG:boto:Data: 2023-10-12T01:11:41.310 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.311 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.311 DEBUG:boto:Params: {} 2023-10-12T01:11:41.312 DEBUG:boto:Token: None 2023-10-12T01:11:41.312 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb50ea620d57ba3f-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.358 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.359 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.360 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.361 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.361 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.362 DEBUG:boto:Method: GET 2023-10-12T01:11:41.362 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.363 DEBUG:boto:Data: 2023-10-12T01:11:41.363 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.364 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.365 DEBUG:boto:Params: {} 2023-10-12T01:11:41.365 DEBUG:boto:Token: None 2023-10-12T01:11:41.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44a342b1628e224-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.373 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.373 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.374 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.374 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.375 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:11:41.375 DEBUG:boto:Data: 2023-10-12T01:11:41.376 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.377 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.377 DEBUG:boto:Params: {} 2023-10-12T01:11:41.377 DEBUG:boto:Token: None 2023-10-12T01:11:41.378 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116 2023-10-12T01:11:41.378 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fH30rgcyoHw/IHbMRHdKHUivIzw= 2023-10-12T01:11:41.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fH30rgcyoHw/IHbMRHdKHUivIzw=', 'Content-Length': '0'} 2023-10-12T01:11:41.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000794a833430d500-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.383 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.383 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.384 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.384 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:11:41.385 DEBUG:boto:Data: 2023-10-12T01:11:41.385 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.387 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.387 DEBUG:boto:Params: {} 2023-10-12T01:11:41.388 DEBUG:boto:Token: None 2023-10-12T01:11:41.389 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116 2023-10-12T01:11:41.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fH30rgcyoHw/IHbMRHdKHUivIzw= 2023-10-12T01:11:41.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fH30rgcyoHw/IHbMRHdKHUivIzw=', 'Content-Length': '0'} 2023-10-12T01:11:41.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048444dacd77f0a15-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:11:41.396 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.396 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.397 DEBUG:boto:Method: GET 2023-10-12T01:11:41.397 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:11:41.398 DEBUG:boto:Data: 2023-10-12T01:11:41.398 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.399 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.399 DEBUG:boto:Params: {} 2023-10-12T01:11:41.400 DEBUG:boto:Token: None 2023-10-12T01:11:41.400 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116 2023-10-12T01:11:41.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H+Gs2B0VBXMYhYYHnPT9eh/mp84= 2023-10-12T01:11:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H+Gs2B0VBXMYhYYHnPT9eh/mp84=', 'Content-Length': '0'} 2023-10-12T01:11:41.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd380fa1a254138-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.404 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:11:41.405 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:11:41.405 DEBUG:boto:Method: GET 2023-10-12T01:11:41.406 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:11:41.407 DEBUG:boto:Data: 2023-10-12T01:11:41.407 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.408 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.409 DEBUG:boto:Params: {} 2023-10-12T01:11:41.409 DEBUG:boto:Token: None 2023-10-12T01:11:41.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj116 2023-10-12T01:11:41.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H+Gs2B0VBXMYhYYHnPT9eh/mp84= 2023-10-12T01:11:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H+Gs2B0VBXMYhYYHnPT9eh/mp84=', 'Content-Length': '0'} 2023-10-12T01:11:41.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7aeefcc5d4a6456-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.414 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.415 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.416 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.417 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.417 DEBUG:boto:Method: GET 2023-10-12T01:11:41.418 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.419 DEBUG:boto:Data: 2023-10-12T01:11:41.420 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.421 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.422 DEBUG:boto:Params: {} 2023-10-12T01:11:41.423 DEBUG:boto:Token: None 2023-10-12T01:11:41.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f854821307012359-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.429 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.429 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.430 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.430 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.431 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.431 DEBUG:boto:Method: GET 2023-10-12T01:11:41.432 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:11:41.433 DEBUG:boto:Data: 2023-10-12T01:11:41.433 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.434 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.434 DEBUG:boto:Params: {} 2023-10-12T01:11:41.435 DEBUG:boto:Token: None 2023-10-12T01:11:41.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a7f3407e08b838-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.442 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:11:41.443 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.444 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.444 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.444 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.445 DEBUG:boto:Method: GET 2023-10-12T01:11:41.445 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.446 DEBUG:boto:Data: 2023-10-12T01:11:41.446 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.447 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.448 DEBUG:boto:Params: {} 2023-10-12T01:11:41.448 DEBUG:boto:Token: None 2023-10-12T01:11:41.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.449 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc= 2023-10-12T01:11:41.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc=', 'Content-Length': '0'} 2023-10-12T01:11:41.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3e3eebae3101ce-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.494 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.495 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.495 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.496 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.496 DEBUG:boto:Method: GET 2023-10-12T01:11:41.497 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.497 DEBUG:boto:Data: 2023-10-12T01:11:41.498 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.499 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.499 DEBUG:boto:Params: {} 2023-10-12T01:11:41.500 DEBUG:boto:Token: None 2023-10-12T01:11:41.500 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc= 2023-10-12T01:11:41.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc=', 'Content-Length': '0'} 2023-10-12T01:11:41.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37cb582b2f24cc2-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.504 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.505 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.505 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.506 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.506 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.507 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.507 DEBUG:boto:Data: 2023-10-12T01:11:41.507 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.508 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.509 DEBUG:boto:Params: {} 2023-10-12T01:11:41.509 DEBUG:boto:Token: None 2023-10-12T01:11:41.510 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:teyPCvDVwN1gIWEELEuibL3V9Zc= 2023-10-12T01:11:41.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:teyPCvDVwN1gIWEELEuibL3V9Zc=', 'Content-Length': '0'} 2023-10-12T01:11:41.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7552d7df4e95940-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.513 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.514 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.514 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.515 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.515 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.516 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.516 DEBUG:boto:Data: 2023-10-12T01:11:41.516 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.517 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.518 DEBUG:boto:Params: {} 2023-10-12T01:11:41.518 DEBUG:boto:Token: None 2023-10-12T01:11:41.519 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.519 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:teyPCvDVwN1gIWEELEuibL3V9Zc= 2023-10-12T01:11:41.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:teyPCvDVwN1gIWEELEuibL3V9Zc=', 'Content-Length': '0'} 2023-10-12T01:11:41.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b479df3e888d858-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:11:41.523 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.523 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.524 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.526 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.527 DEBUG:boto:Method: GET 2023-10-12T01:11:41.527 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.528 DEBUG:boto:Data: 2023-10-12T01:11:41.528 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.530 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.530 DEBUG:boto:Params: {} 2023-10-12T01:11:41.531 DEBUG:boto:Token: None 2023-10-12T01:11:41.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.533 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc= 2023-10-12T01:11:41.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc=', 'Content-Length': '0'} 2023-10-12T01:11:41.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c939aee72f3022aa-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.538 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.539 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.539 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.540 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.540 DEBUG:boto:Method: GET 2023-10-12T01:11:41.541 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.541 DEBUG:boto:Data: 2023-10-12T01:11:41.542 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.543 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.543 DEBUG:boto:Params: {} 2023-10-12T01:11:41.544 DEBUG:boto:Token: None 2023-10-12T01:11:41.544 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:11:41.545 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc= 2023-10-12T01:11:41.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RJiRuRjqA1f7zYmMzKh9BwwnVJc=', 'Content-Length': '0'} 2023-10-12T01:11:41.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b90aaacff3f0da-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.548 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.548 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.549 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.549 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.550 DEBUG:boto:Method: GET 2023-10-12T01:11:41.550 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.551 DEBUG:boto:Data: 2023-10-12T01:11:41.551 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.552 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.553 DEBUG:boto:Params: {} 2023-10-12T01:11:41.553 DEBUG:boto:Token: None 2023-10-12T01:11:41.554 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.554 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f403f60490311e0-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.558 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.559 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.559 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.559 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.560 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.561 DEBUG:boto:Method: GET 2023-10-12T01:11:41.561 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.562 DEBUG:boto:Data: 2023-10-12T01:11:41.562 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.563 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.564 DEBUG:boto:Params: {} 2023-10-12T01:11:41.564 DEBUG:boto:Token: None 2023-10-12T01:11:41.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc033c6d80de4e31-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.610 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.611 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.612 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.612 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.613 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:11:41.613 DEBUG:boto:Data: 2023-10-12T01:11:41.613 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.614 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.615 DEBUG:boto:Params: {} 2023-10-12T01:11:41.615 DEBUG:boto:Token: None 2023-10-12T01:11:41.616 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124 2023-10-12T01:11:41.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IhyaAJD05jDJUbHgvOSIRO/luLw= 2023-10-12T01:11:41.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IhyaAJD05jDJUbHgvOSIRO/luLw=', 'Content-Length': '0'} 2023-10-12T01:11:41.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9299596ed94e17a-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.619 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.620 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.620 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.621 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:11:41.621 DEBUG:boto:Data: 2023-10-12T01:11:41.622 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.623 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.623 DEBUG:boto:Params: {} 2023-10-12T01:11:41.624 DEBUG:boto:Token: None 2023-10-12T01:11:41.624 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124 2023-10-12T01:11:41.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IhyaAJD05jDJUbHgvOSIRO/luLw= 2023-10-12T01:11:41.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IhyaAJD05jDJUbHgvOSIRO/luLw=', 'Content-Length': '0'} 2023-10-12T01:11:41.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513a28503d7232af-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:11:41.628 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.629 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.629 DEBUG:boto:Method: GET 2023-10-12T01:11:41.629 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:11:41.630 DEBUG:boto:Data: 2023-10-12T01:11:41.630 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.631 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.632 DEBUG:boto:Params: {} 2023-10-12T01:11:41.632 DEBUG:boto:Token: None 2023-10-12T01:11:41.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124 2023-10-12T01:11:41.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UGBwILXwNMDfi1BZEcAWaWxWQVE= 2023-10-12T01:11:41.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UGBwILXwNMDfi1BZEcAWaWxWQVE=', 'Content-Length': '0'} 2023-10-12T01:11:41.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af486dbd3b1a309d-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.636 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:11:41.637 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:11:41.637 DEBUG:boto:Method: GET 2023-10-12T01:11:41.638 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:11:41.638 DEBUG:boto:Data: 2023-10-12T01:11:41.639 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.639 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.640 DEBUG:boto:Params: {} 2023-10-12T01:11:41.640 DEBUG:boto:Token: None 2023-10-12T01:11:41.641 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj124 2023-10-12T01:11:41.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UGBwILXwNMDfi1BZEcAWaWxWQVE= 2023-10-12T01:11:41.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UGBwILXwNMDfi1BZEcAWaWxWQVE=', 'Content-Length': '0'} 2023-10-12T01:11:41.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe923cfbeb7e6c1a-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.644 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.645 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.645 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.646 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.646 DEBUG:boto:Method: GET 2023-10-12T01:11:41.647 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.647 DEBUG:boto:Data: 2023-10-12T01:11:41.647 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.648 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.649 DEBUG:boto:Params: {} 2023-10-12T01:11:41.649 DEBUG:boto:Token: None 2023-10-12T01:11:41.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007662feb8e0f80e27-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.654 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.655 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.655 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.656 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.656 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.656 DEBUG:boto:Method: GET 2023-10-12T01:11:41.657 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:11:41.657 DEBUG:boto:Data: 2023-10-12T01:11:41.658 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.659 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.659 DEBUG:boto:Params: {} 2023-10-12T01:11:41.660 DEBUG:boto:Token: None 2023-10-12T01:11:41.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c91fa97756e553f-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.666 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:11:41.667 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.667 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.668 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.668 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.669 DEBUG:boto:Method: GET 2023-10-12T01:11:41.669 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.670 DEBUG:boto:Data: 2023-10-12T01:11:41.670 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.671 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.671 DEBUG:boto:Params: {} 2023-10-12T01:11:41.672 DEBUG:boto:Token: None 2023-10-12T01:11:41.672 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.673 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs= 2023-10-12T01:11:41.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs=', 'Content-Length': '0'} 2023-10-12T01:11:41.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b4845da7ec6d23-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.676 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.677 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.677 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.677 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.678 DEBUG:boto:Method: GET 2023-10-12T01:11:41.678 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.679 DEBUG:boto:Data: 2023-10-12T01:11:41.679 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.680 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.680 DEBUG:boto:Params: {} 2023-10-12T01:11:41.681 DEBUG:boto:Token: None 2023-10-12T01:11:41.681 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs= 2023-10-12T01:11:41.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs=', 'Content-Length': '0'} 2023-10-12T01:11:41.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb18134fe6db58d1-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.734 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.735 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.736 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.736 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.737 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.737 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.738 DEBUG:boto:Data: 2023-10-12T01:11:41.739 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.741 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.741 DEBUG:boto:Params: {} 2023-10-12T01:11:41.742 DEBUG:boto:Token: None 2023-10-12T01:11:41.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.743 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GDb7A8SCEuLrPzSDRN7BiZVJzyY= 2023-10-12T01:11:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GDb7A8SCEuLrPzSDRN7BiZVJzyY=', 'Content-Length': '0'} 2023-10-12T01:11:41.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87abc6de1ea3056-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.747 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.748 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.748 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.748 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.749 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.750 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.751 DEBUG:boto:Data: 2023-10-12T01:11:41.751 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.753 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.754 DEBUG:boto:Params: {} 2023-10-12T01:11:41.755 DEBUG:boto:Token: None 2023-10-12T01:11:41.755 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GDb7A8SCEuLrPzSDRN7BiZVJzyY= 2023-10-12T01:11:41.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GDb7A8SCEuLrPzSDRN7BiZVJzyY=', 'Content-Length': '0'} 2023-10-12T01:11:41.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be72720b9d94bf0-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:11:41.761 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.761 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.762 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.762 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.763 DEBUG:boto:Method: GET 2023-10-12T01:11:41.763 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.763 DEBUG:boto:Data: 2023-10-12T01:11:41.764 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.765 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.765 DEBUG:boto:Params: {} 2023-10-12T01:11:41.766 DEBUG:boto:Token: None 2023-10-12T01:11:41.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs= 2023-10-12T01:11:41.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs=', 'Content-Length': '0'} 2023-10-12T01:11:41.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf50de52af278e6c-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.810 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.811 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.812 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.813 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.814 DEBUG:boto:Method: GET 2023-10-12T01:11:41.814 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.815 DEBUG:boto:Data: 2023-10-12T01:11:41.816 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.817 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.818 DEBUG:boto:Params: {} 2023-10-12T01:11:41.819 DEBUG:boto:Token: None 2023-10-12T01:11:41.820 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:11:41.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs= 2023-10-12T01:11:41.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1ja+JZXjMYCFe+bX0t7+QKFozSs=', 'Content-Length': '0'} 2023-10-12T01:11:41.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f48327d3cb69b94-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.825 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.826 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.827 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.828 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.829 DEBUG:boto:Method: GET 2023-10-12T01:11:41.829 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.830 DEBUG:boto:Data: 2023-10-12T01:11:41.831 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.833 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.833 DEBUG:boto:Params: {} 2023-10-12T01:11:41.835 DEBUG:boto:Token: None 2023-10-12T01:11:41.836 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec475ef4d24fd1f-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.842 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.843 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.844 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.845 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.846 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.847 DEBUG:boto:Method: GET 2023-10-12T01:11:41.847 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.848 DEBUG:boto:Data: 2023-10-12T01:11:41.849 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.850 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.851 DEBUG:boto:Params: {} 2023-10-12T01:11:41.852 DEBUG:boto:Token: None 2023-10-12T01:11:41.853 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.854 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baae54fcabe4d4cf-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.860 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.861 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.861 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.862 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.862 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:11:41.863 DEBUG:boto:Data: 2023-10-12T01:11:41.863 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.864 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.864 DEBUG:boto:Params: {} 2023-10-12T01:11:41.865 DEBUG:boto:Token: None 2023-10-12T01:11:41.865 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145 2023-10-12T01:11:41.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wFcFcCt2aQIPs9Hk3VxEgbiewGA= 2023-10-12T01:11:41.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wFcFcCt2aQIPs9Hk3VxEgbiewGA=', 'Content-Length': '0'} 2023-10-12T01:11:41.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf660933aa37689c-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.871 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.871 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.872 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.873 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:11:41.874 DEBUG:boto:Data: 2023-10-12T01:11:41.874 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.875 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.875 DEBUG:boto:Params: {} 2023-10-12T01:11:41.876 DEBUG:boto:Token: None 2023-10-12T01:11:41.876 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145 2023-10-12T01:11:41.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wFcFcCt2aQIPs9Hk3VxEgbiewGA= 2023-10-12T01:11:41.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wFcFcCt2aQIPs9Hk3VxEgbiewGA=', 'Content-Length': '0'} 2023-10-12T01:11:41.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a296886607525372-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:11:41.881 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.881 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.882 DEBUG:boto:Method: GET 2023-10-12T01:11:41.882 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:11:41.882 DEBUG:boto:Data: 2023-10-12T01:11:41.883 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.884 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.884 DEBUG:boto:Params: {} 2023-10-12T01:11:41.885 DEBUG:boto:Token: None 2023-10-12T01:11:41.885 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145 2023-10-12T01:11:41.886 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F7NmH2BzSJriT3Hw2byAj96Zy2I= 2023-10-12T01:11:41.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F7NmH2BzSJriT3Hw2byAj96Zy2I=', 'Content-Length': '0'} 2023-10-12T01:11:41.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0117a69b6e6d6c-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.890 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:11:41.890 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:11:41.891 DEBUG:boto:Method: GET 2023-10-12T01:11:41.891 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:11:41.892 DEBUG:boto:Data: 2023-10-12T01:11:41.892 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.893 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.894 DEBUG:boto:Params: {} 2023-10-12T01:11:41.894 DEBUG:boto:Token: None 2023-10-12T01:11:41.895 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj145 2023-10-12T01:11:41.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F7NmH2BzSJriT3Hw2byAj96Zy2I= 2023-10-12T01:11:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F7NmH2BzSJriT3Hw2byAj96Zy2I=', 'Content-Length': '0'} 2023-10-12T01:11:41.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff76bd571b3a59f8-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.898 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.899 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.899 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.900 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.900 DEBUG:boto:Method: GET 2023-10-12T01:11:41.900 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.901 DEBUG:boto:Data: 2023-10-12T01:11:41.901 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.902 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.903 DEBUG:boto:Params: {} 2023-10-12T01:11:41.903 DEBUG:boto:Token: None 2023-10-12T01:11:41.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2d58d0f99263ce-00652747cd-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.908 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.909 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:41.909 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:41.910 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.910 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.911 DEBUG:boto:Method: GET 2023-10-12T01:11:41.911 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:11:41.911 DEBUG:boto:Data: 2023-10-12T01:11:41.912 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.913 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.913 DEBUG:boto:Params: {} 2023-10-12T01:11:41.914 DEBUG:boto:Token: None 2023-10-12T01:11:41.914 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/ 2023-10-12T01:11:41.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM= 2023-10-12T01:11:41.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q6WkCswMdjRHedQslN4paIZL3iM=', 'Content-Length': '0'} 2023-10-12T01:11:41.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee9430e0cbe9185-00652747cd-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.920 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:41.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:11:41.921 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:41.921 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:41.922 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.922 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.923 DEBUG:boto:Method: GET 2023-10-12T01:11:41.923 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.924 DEBUG:boto:Data: 2023-10-12T01:11:41.924 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.925 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.925 DEBUG:boto:Params: {} 2023-10-12T01:11:41.926 DEBUG:boto:Token: None 2023-10-12T01:11:41.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+mjlw6mv1PxRDrvogWTkWj5vAk8= 2023-10-12T01:11:41.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+mjlw6mv1PxRDrvogWTkWj5vAk8=', 'Content-Length': '0'} 2023-10-12T01:11:41.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5bae3724a6c7042-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.930 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:41.930 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:41.931 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.931 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.932 DEBUG:boto:Method: GET 2023-10-12T01:11:41.932 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.932 DEBUG:boto:Data: 2023-10-12T01:11:41.933 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.934 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.934 DEBUG:boto:Params: {} 2023-10-12T01:11:41.935 DEBUG:boto:Token: None 2023-10-12T01:11:41.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+mjlw6mv1PxRDrvogWTkWj5vAk8= 2023-10-12T01:11:41.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+mjlw6mv1PxRDrvogWTkWj5vAk8=', 'Content-Length': '0'} 2023-10-12T01:11:41.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a202939228f13f-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.982 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:41.983 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:41.983 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.984 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.984 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.985 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.985 DEBUG:boto:Data: 2023-10-12T01:11:41.985 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:41.986 DEBUG:boto:Port: 8001 2023-10-12T01:11:41.987 DEBUG:boto:Params: {} 2023-10-12T01:11:41.987 DEBUG:boto:Token: None 2023-10-12T01:11:41.988 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZrcoqLSaMT/sSZzO63eIiK/g9MQ= 2023-10-12T01:11:41.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZrcoqLSaMT/sSZzO63eIiK/g9MQ=', 'Content-Length': '0'} 2023-10-12T01:11:41.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa29297c40546a1-00652747cd-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:41.991 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:41.992 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:41.992 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.993 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.993 DEBUG:boto:Method: HEAD 2023-10-12T01:11:41.993 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.994 DEBUG:boto:Data: 2023-10-12T01:11:41.994 DEBUG:boto:Headers: {} 2023-10-12T01:11:41.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:41.995 DEBUG:boto:Port: 8000 2023-10-12T01:11:41.996 DEBUG:boto:Params: {} 2023-10-12T01:11:41.996 DEBUG:boto:Token: None 2023-10-12T01:11:41.996 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:41 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:41.997 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZrcoqLSaMT/sSZzO63eIiK/g9MQ= 2023-10-12T01:11:41.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZrcoqLSaMT/sSZzO63eIiK/g9MQ=', 'Content-Length': '0'} 2023-10-12T01:11:42.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004770c0e8d9500c94-00652747cd-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:11:42.001 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.001 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.001 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.002 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.002 DEBUG:boto:Method: GET 2023-10-12T01:11:42.003 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.003 DEBUG:boto:Data: 2023-10-12T01:11:42.004 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.004 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.005 DEBUG:boto:Params: {} 2023-10-12T01:11:42.005 DEBUG:boto:Token: None 2023-10-12T01:11:42.006 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jZ0/AzelCcAjFlrou+3y/agx4BY= 2023-10-12T01:11:42.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jZ0/AzelCcAjFlrou+3y/agx4BY=', 'Content-Length': '0'} 2023-10-12T01:11:42.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b31854e73897acc-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.054 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.055 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.055 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.056 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.056 DEBUG:boto:Method: GET 2023-10-12T01:11:42.057 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.057 DEBUG:boto:Data: 2023-10-12T01:11:42.058 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.059 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.059 DEBUG:boto:Params: {} 2023-10-12T01:11:42.059 DEBUG:boto:Token: None 2023-10-12T01:11:42.060 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:11:42.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jZ0/AzelCcAjFlrou+3y/agx4BY= 2023-10-12T01:11:42.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jZ0/AzelCcAjFlrou+3y/agx4BY=', 'Content-Length': '0'} 2023-10-12T01:11:42.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000171f354ba74d70e-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.064 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.064 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.065 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.065 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.065 DEBUG:boto:Method: GET 2023-10-12T01:11:42.066 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.066 DEBUG:boto:Data: 2023-10-12T01:11:42.067 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.068 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.068 DEBUG:boto:Params: {} 2023-10-12T01:11:42.069 DEBUG:boto:Token: None 2023-10-12T01:11:42.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3462a8bda11079c-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.073 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.074 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.075 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.075 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.075 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.076 DEBUG:boto:Method: GET 2023-10-12T01:11:42.076 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.077 DEBUG:boto:Data: 2023-10-12T01:11:42.077 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.078 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.078 DEBUG:boto:Params: {} 2023-10-12T01:11:42.079 DEBUG:boto:Token: None 2023-10-12T01:11:42.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d3240b1a7dda77-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.085 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.085 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.086 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.086 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.087 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:11:42.087 DEBUG:boto:Data: 2023-10-12T01:11:42.088 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.089 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.089 DEBUG:boto:Params: {} 2023-10-12T01:11:42.089 DEBUG:boto:Token: None 2023-10-12T01:11:42.090 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148 2023-10-12T01:11:42.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cs03CDwJzgfRifl5sJQsah4Epck= 2023-10-12T01:11:42.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cs03CDwJzgfRifl5sJQsah4Epck=', 'Content-Length': '0'} 2023-10-12T01:11:42.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda23614b293e54f-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.093 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.094 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.094 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.095 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:11:42.095 DEBUG:boto:Data: 2023-10-12T01:11:42.096 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.096 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.097 DEBUG:boto:Params: {} 2023-10-12T01:11:42.097 DEBUG:boto:Token: None 2023-10-12T01:11:42.098 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148 2023-10-12T01:11:42.098 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cs03CDwJzgfRifl5sJQsah4Epck= 2023-10-12T01:11:42.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cs03CDwJzgfRifl5sJQsah4Epck=', 'Content-Length': '0'} 2023-10-12T01:11:42.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719419711dd5ea68-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:11:42.102 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.102 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.103 DEBUG:boto:Method: GET 2023-10-12T01:11:42.103 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:11:42.103 DEBUG:boto:Data: 2023-10-12T01:11:42.104 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.105 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.105 DEBUG:boto:Params: {} 2023-10-12T01:11:42.106 DEBUG:boto:Token: None 2023-10-12T01:11:42.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148 2023-10-12T01:11:42.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aZhBZYU56auAXViMmuYcgP6TAII= 2023-10-12T01:11:42.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aZhBZYU56auAXViMmuYcgP6TAII=', 'Content-Length': '0'} 2023-10-12T01:11:42.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7a52bf67b657f4-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.110 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:11:42.110 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:11:42.111 DEBUG:boto:Method: GET 2023-10-12T01:11:42.111 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:11:42.112 DEBUG:boto:Data: 2023-10-12T01:11:42.112 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.113 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.113 DEBUG:boto:Params: {} 2023-10-12T01:11:42.114 DEBUG:boto:Token: None 2023-10-12T01:11:42.114 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj148 2023-10-12T01:11:42.115 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aZhBZYU56auAXViMmuYcgP6TAII= 2023-10-12T01:11:42.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aZhBZYU56auAXViMmuYcgP6TAII=', 'Content-Length': '0'} 2023-10-12T01:11:42.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000018b6e784bd599eb-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.158 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.159 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.159 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.160 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.160 DEBUG:boto:Method: GET 2023-10-12T01:11:42.161 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.161 DEBUG:boto:Data: 2023-10-12T01:11:42.161 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.162 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.163 DEBUG:boto:Params: {} 2023-10-12T01:11:42.163 DEBUG:boto:Token: None 2023-10-12T01:11:42.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e893f290f125cc-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.168 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.169 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.169 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.170 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.170 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.171 DEBUG:boto:Method: GET 2023-10-12T01:11:42.171 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:11:42.172 DEBUG:boto:Data: 2023-10-12T01:11:42.172 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.173 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.173 DEBUG:boto:Params: {} 2023-10-12T01:11:42.174 DEBUG:boto:Token: None 2023-10-12T01:11:42.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e508f185c843d82-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.180 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:11:42.181 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.182 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.182 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.182 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.183 DEBUG:boto:Method: GET 2023-10-12T01:11:42.183 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.184 DEBUG:boto:Data: 2023-10-12T01:11:42.184 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.185 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.185 DEBUG:boto:Params: {} 2023-10-12T01:11:42.186 DEBUG:boto:Token: None 2023-10-12T01:11:42.186 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8= 2023-10-12T01:11:42.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8=', 'Content-Length': '0'} 2023-10-12T01:11:42.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000079075599bcf01c7-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.190 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.191 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.191 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.191 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.192 DEBUG:boto:Method: GET 2023-10-12T01:11:42.192 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.193 DEBUG:boto:Data: 2023-10-12T01:11:42.193 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.194 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.194 DEBUG:boto:Params: {} 2023-10-12T01:11:42.195 DEBUG:boto:Token: None 2023-10-12T01:11:42.195 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8= 2023-10-12T01:11:42.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8=', 'Content-Length': '0'} 2023-10-12T01:11:42.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30c2e1d1d2183f0-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.199 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.200 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.200 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.201 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.201 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.201 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.202 DEBUG:boto:Data: 2023-10-12T01:11:42.202 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.203 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.203 DEBUG:boto:Params: {} 2023-10-12T01:11:42.204 DEBUG:boto:Token: None 2023-10-12T01:11:42.204 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.205 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4D2+MjI347hvfGh6525yJBcFJo= 2023-10-12T01:11:42.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4D2+MjI347hvfGh6525yJBcFJo=', 'Content-Length': '0'} 2023-10-12T01:11:42.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023bdf41708d0fb9b-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.208 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.208 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.209 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.209 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.209 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.210 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.210 DEBUG:boto:Data: 2023-10-12T01:11:42.211 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.211 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.212 DEBUG:boto:Params: {} 2023-10-12T01:11:42.212 DEBUG:boto:Token: None 2023-10-12T01:11:42.213 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y4D2+MjI347hvfGh6525yJBcFJo= 2023-10-12T01:11:42.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y4D2+MjI347hvfGh6525yJBcFJo=', 'Content-Length': '0'} 2023-10-12T01:11:42.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000774e97b7345c842a-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:11:42.216 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.217 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.217 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.218 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.218 DEBUG:boto:Method: GET 2023-10-12T01:11:42.219 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.219 DEBUG:boto:Data: 2023-10-12T01:11:42.219 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.220 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.221 DEBUG:boto:Params: {} 2023-10-12T01:11:42.221 DEBUG:boto:Token: None 2023-10-12T01:11:42.222 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8= 2023-10-12T01:11:42.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8=', 'Content-Length': '0'} 2023-10-12T01:11:42.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079403f4eecf97e73-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.266 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.267 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.267 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.268 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.268 DEBUG:boto:Method: GET 2023-10-12T01:11:42.269 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.269 DEBUG:boto:Data: 2023-10-12T01:11:42.269 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.270 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.271 DEBUG:boto:Params: {} 2023-10-12T01:11:42.271 DEBUG:boto:Token: None 2023-10-12T01:11:42.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:11:42.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8= 2023-10-12T01:11:42.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qj04RfIFfZMv+fEaIa1YdnUWlC8=', 'Content-Length': '0'} 2023-10-12T01:11:42.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be889302f6ceecd-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.276 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.276 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.277 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.277 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.278 DEBUG:boto:Method: GET 2023-10-12T01:11:42.278 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.278 DEBUG:boto:Data: 2023-10-12T01:11:42.279 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.280 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.280 DEBUG:boto:Params: {} 2023-10-12T01:11:42.281 DEBUG:boto:Token: None 2023-10-12T01:11:42.281 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.282 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008afac51dff5daec5-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.285 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.286 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.287 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.287 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.287 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.288 DEBUG:boto:Method: GET 2023-10-12T01:11:42.288 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.289 DEBUG:boto:Data: 2023-10-12T01:11:42.289 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.290 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.290 DEBUG:boto:Params: {} 2023-10-12T01:11:42.291 DEBUG:boto:Token: None 2023-10-12T01:11:42.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097b10b58f1fe39dd-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.297 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.298 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.301 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.301 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.302 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:11:42.302 DEBUG:boto:Data: 2023-10-12T01:11:42.302 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.303 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.304 DEBUG:boto:Params: {} 2023-10-12T01:11:42.304 DEBUG:boto:Token: None 2023-10-12T01:11:42.305 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155 2023-10-12T01:11:42.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2MGgVX9atGjt0QQM75T9SKi1d9Y= 2023-10-12T01:11:42.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MGgVX9atGjt0QQM75T9SKi1d9Y=', 'Content-Length': '0'} 2023-10-12T01:11:42.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f5f817ed0e5616-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.308 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.309 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.309 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.310 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:11:42.310 DEBUG:boto:Data: 2023-10-12T01:11:42.310 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.311 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.312 DEBUG:boto:Params: {} 2023-10-12T01:11:42.312 DEBUG:boto:Token: None 2023-10-12T01:11:42.312 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155 2023-10-12T01:11:42.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2MGgVX9atGjt0QQM75T9SKi1d9Y= 2023-10-12T01:11:42.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MGgVX9atGjt0QQM75T9SKi1d9Y=', 'Content-Length': '0'} 2023-10-12T01:11:42.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36e61c559f318d4-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:11:42.317 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.317 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.318 DEBUG:boto:Method: GET 2023-10-12T01:11:42.318 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:11:42.319 DEBUG:boto:Data: 2023-10-12T01:11:42.319 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.320 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.320 DEBUG:boto:Params: {} 2023-10-12T01:11:42.321 DEBUG:boto:Token: None 2023-10-12T01:11:42.321 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155 2023-10-12T01:11:42.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u5DfwL5/t3s8us0QfhTsq+JY5zI= 2023-10-12T01:11:42.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u5DfwL5/t3s8us0QfhTsq+JY5zI=', 'Content-Length': '0'} 2023-10-12T01:11:42.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af58e7612e34610-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.325 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:11:42.326 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:11:42.326 DEBUG:boto:Method: GET 2023-10-12T01:11:42.327 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:11:42.327 DEBUG:boto:Data: 2023-10-12T01:11:42.327 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.328 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.329 DEBUG:boto:Params: {} 2023-10-12T01:11:42.329 DEBUG:boto:Token: None 2023-10-12T01:11:42.330 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj155 2023-10-12T01:11:42.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u5DfwL5/t3s8us0QfhTsq+JY5zI= 2023-10-12T01:11:42.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u5DfwL5/t3s8us0QfhTsq+JY5zI=', 'Content-Length': '0'} 2023-10-12T01:11:42.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b6336fa0fdebd8-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.333 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.334 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.334 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.335 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.335 DEBUG:boto:Method: GET 2023-10-12T01:11:42.335 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.336 DEBUG:boto:Data: 2023-10-12T01:11:42.336 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.337 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.338 DEBUG:boto:Params: {} 2023-10-12T01:11:42.338 DEBUG:boto:Token: None 2023-10-12T01:11:42.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618ea0392f3927c9-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.343 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.344 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.344 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.345 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.345 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.345 DEBUG:boto:Method: GET 2023-10-12T01:11:42.346 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:11:42.346 DEBUG:boto:Data: 2023-10-12T01:11:42.347 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.347 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.348 DEBUG:boto:Params: {} 2023-10-12T01:11:42.348 DEBUG:boto:Token: None 2023-10-12T01:11:42.349 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005711f28eda50aa3e-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.355 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:11:42.356 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.356 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.357 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.357 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.357 DEBUG:boto:Method: GET 2023-10-12T01:11:42.358 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.358 DEBUG:boto:Data: 2023-10-12T01:11:42.359 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.360 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.360 DEBUG:boto:Params: {} 2023-10-12T01:11:42.361 DEBUG:boto:Token: None 2023-10-12T01:11:42.361 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84= 2023-10-12T01:11:42.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84=', 'Content-Length': '0'} 2023-10-12T01:11:42.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ac5e2f1a08af2b-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.365 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.365 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.366 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.366 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.367 DEBUG:boto:Method: GET 2023-10-12T01:11:42.367 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.367 DEBUG:boto:Data: 2023-10-12T01:11:42.368 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.369 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.369 DEBUG:boto:Params: {} 2023-10-12T01:11:42.370 DEBUG:boto:Token: None 2023-10-12T01:11:42.370 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84= 2023-10-12T01:11:42.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84=', 'Content-Length': '0'} 2023-10-12T01:11:42.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21e751892bd2d11-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.418 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.419 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.419 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.420 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.420 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.421 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.421 DEBUG:boto:Data: 2023-10-12T01:11:42.421 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.422 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.423 DEBUG:boto:Params: {} 2023-10-12T01:11:42.423 DEBUG:boto:Token: None 2023-10-12T01:11:42.424 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tdTPOzdCl2uIQuLjSDZABzcJvOM= 2023-10-12T01:11:42.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tdTPOzdCl2uIQuLjSDZABzcJvOM=', 'Content-Length': '0'} 2023-10-12T01:11:42.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a994d3722d51c12-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.427 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.428 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.428 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.429 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.429 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.429 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.430 DEBUG:boto:Data: 2023-10-12T01:11:42.430 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.431 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.431 DEBUG:boto:Params: {} 2023-10-12T01:11:42.432 DEBUG:boto:Token: None 2023-10-12T01:11:42.432 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.433 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tdTPOzdCl2uIQuLjSDZABzcJvOM= 2023-10-12T01:11:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tdTPOzdCl2uIQuLjSDZABzcJvOM=', 'Content-Length': '0'} 2023-10-12T01:11:42.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbf10984ab933b1-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:11:42.436 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.437 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.437 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.437 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.438 DEBUG:boto:Method: GET 2023-10-12T01:11:42.438 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.439 DEBUG:boto:Data: 2023-10-12T01:11:42.439 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.440 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.440 DEBUG:boto:Params: {} 2023-10-12T01:11:42.441 DEBUG:boto:Token: None 2023-10-12T01:11:42.441 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.442 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84= 2023-10-12T01:11:42.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84=', 'Content-Length': '0'} 2023-10-12T01:11:42.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042bc8ec7c792a737-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.490 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.491 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.492 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.492 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.492 DEBUG:boto:Method: GET 2023-10-12T01:11:42.493 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.493 DEBUG:boto:Data: 2023-10-12T01:11:42.494 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.494 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.495 DEBUG:boto:Params: {} 2023-10-12T01:11:42.495 DEBUG:boto:Token: None 2023-10-12T01:11:42.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:11:42.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84= 2023-10-12T01:11:42.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NJsNGiWJHKr6sPuM8vrjTvj3v84=', 'Content-Length': '0'} 2023-10-12T01:11:42.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58cfd47e0607a59-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.500 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.500 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.500 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.501 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.501 DEBUG:boto:Method: GET 2023-10-12T01:11:42.502 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.502 DEBUG:boto:Data: 2023-10-12T01:11:42.503 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.504 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.504 DEBUG:boto:Params: {} 2023-10-12T01:11:42.505 DEBUG:boto:Token: None 2023-10-12T01:11:42.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090978e2dd9930327-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.510 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.511 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.512 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.512 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.512 DEBUG:boto:Method: GET 2023-10-12T01:11:42.513 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.513 DEBUG:boto:Data: 2023-10-12T01:11:42.514 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.515 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.515 DEBUG:boto:Params: {} 2023-10-12T01:11:42.515 DEBUG:boto:Token: None 2023-10-12T01:11:42.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3c03ef155f1c81-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.521 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.522 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.523 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.523 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.524 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:11:42.524 DEBUG:boto:Data: 2023-10-12T01:11:42.524 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.525 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.525 DEBUG:boto:Params: {} 2023-10-12T01:11:42.526 DEBUG:boto:Token: None 2023-10-12T01:11:42.526 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17 2023-10-12T01:11:42.527 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyUqkKhl9xz7deOnJR4pDK0LHpQ= 2023-10-12T01:11:42.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyUqkKhl9xz7deOnJR4pDK0LHpQ=', 'Content-Length': '0'} 2023-10-12T01:11:42.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7dd32d73711da4-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.530 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.530 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.530 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.531 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:11:42.531 DEBUG:boto:Data: 2023-10-12T01:11:42.532 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.532 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.533 DEBUG:boto:Params: {} 2023-10-12T01:11:42.533 DEBUG:boto:Token: None 2023-10-12T01:11:42.534 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17 2023-10-12T01:11:42.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lyUqkKhl9xz7deOnJR4pDK0LHpQ= 2023-10-12T01:11:42.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lyUqkKhl9xz7deOnJR4pDK0LHpQ=', 'Content-Length': '0'} 2023-10-12T01:11:42.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67112df195b6061-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:11:42.538 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.538 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.539 DEBUG:boto:Method: GET 2023-10-12T01:11:42.539 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:11:42.539 DEBUG:boto:Data: 2023-10-12T01:11:42.540 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.541 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.541 DEBUG:boto:Params: {} 2023-10-12T01:11:42.542 DEBUG:boto:Token: None 2023-10-12T01:11:42.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17 2023-10-12T01:11:42.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/DsJK0kidRyNWlTdiCYcX9YEKzs= 2023-10-12T01:11:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/DsJK0kidRyNWlTdiCYcX9YEKzs=', 'Content-Length': '0'} 2023-10-12T01:11:42.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a427c2404be79c11-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.546 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:11:42.546 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:11:42.547 DEBUG:boto:Method: GET 2023-10-12T01:11:42.547 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:11:42.548 DEBUG:boto:Data: 2023-10-12T01:11:42.548 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.549 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.549 DEBUG:boto:Params: {} 2023-10-12T01:11:42.550 DEBUG:boto:Token: None 2023-10-12T01:11:42.550 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj17 2023-10-12T01:11:42.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/DsJK0kidRyNWlTdiCYcX9YEKzs= 2023-10-12T01:11:42.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/DsJK0kidRyNWlTdiCYcX9YEKzs=', 'Content-Length': '0'} 2023-10-12T01:11:42.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fbc843f242d283f-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.598 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.599 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.599 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.600 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.600 DEBUG:boto:Method: GET 2023-10-12T01:11:42.601 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.601 DEBUG:boto:Data: 2023-10-12T01:11:42.602 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.602 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.603 DEBUG:boto:Params: {} 2023-10-12T01:11:42.603 DEBUG:boto:Token: None 2023-10-12T01:11:42.604 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044942b3c24f58048-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.608 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.609 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.610 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.610 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.611 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.611 DEBUG:boto:Method: GET 2023-10-12T01:11:42.612 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:11:42.612 DEBUG:boto:Data: 2023-10-12T01:11:42.613 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.614 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.614 DEBUG:boto:Params: {} 2023-10-12T01:11:42.615 DEBUG:boto:Token: None 2023-10-12T01:11:42.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4505c77deb2e00-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.620 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:11:42.622 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.622 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.622 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.623 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.623 DEBUG:boto:Method: GET 2023-10-12T01:11:42.624 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.624 DEBUG:boto:Data: 2023-10-12T01:11:42.625 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.625 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.626 DEBUG:boto:Params: {} 2023-10-12T01:11:42.626 DEBUG:boto:Token: None 2023-10-12T01:11:42.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI= 2023-10-12T01:11:42.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI=', 'Content-Length': '0'} 2023-10-12T01:11:42.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c12593900f8611b-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.631 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.632 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.632 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.632 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.633 DEBUG:boto:Method: GET 2023-10-12T01:11:42.633 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.634 DEBUG:boto:Data: 2023-10-12T01:11:42.634 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.635 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.635 DEBUG:boto:Params: {} 2023-10-12T01:11:42.636 DEBUG:boto:Token: None 2023-10-12T01:11:42.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI= 2023-10-12T01:11:42.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI=', 'Content-Length': '0'} 2023-10-12T01:11:42.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b44d91f5d03a84a-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.640 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.640 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.641 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.641 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.642 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.642 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.643 DEBUG:boto:Data: 2023-10-12T01:11:42.643 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.644 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.644 DEBUG:boto:Params: {} 2023-10-12T01:11:42.645 DEBUG:boto:Token: None 2023-10-12T01:11:42.645 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pO8pag2Z2NH2oXz8Angixrw0Tl0= 2023-10-12T01:11:42.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pO8pag2Z2NH2oXz8Angixrw0Tl0=', 'Content-Length': '0'} 2023-10-12T01:11:42.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008691e2708d0d5b98-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.649 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.649 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.650 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.650 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.650 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.651 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.651 DEBUG:boto:Data: 2023-10-12T01:11:42.652 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.652 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.653 DEBUG:boto:Params: {} 2023-10-12T01:11:42.653 DEBUG:boto:Token: None 2023-10-12T01:11:42.654 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pO8pag2Z2NH2oXz8Angixrw0Tl0= 2023-10-12T01:11:42.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pO8pag2Z2NH2oXz8Angixrw0Tl0=', 'Content-Length': '0'} 2023-10-12T01:11:42.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbe1a65167904c1-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:11:42.658 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.658 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.658 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.659 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.659 DEBUG:boto:Method: GET 2023-10-12T01:11:42.660 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.660 DEBUG:boto:Data: 2023-10-12T01:11:42.661 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.661 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.662 DEBUG:boto:Params: {} 2023-10-12T01:11:42.662 DEBUG:boto:Token: None 2023-10-12T01:11:42.663 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI= 2023-10-12T01:11:42.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI=', 'Content-Length': '0'} 2023-10-12T01:11:42.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008173814d8690593a-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.706 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.707 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.707 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.708 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.708 DEBUG:boto:Method: GET 2023-10-12T01:11:42.709 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.709 DEBUG:boto:Data: 2023-10-12T01:11:42.710 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.711 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.711 DEBUG:boto:Params: {} 2023-10-12T01:11:42.711 DEBUG:boto:Token: None 2023-10-12T01:11:42.712 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:11:42.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI= 2023-10-12T01:11:42.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n53BxdwPrfUx4mk1f7B6dzJ28VI=', 'Content-Length': '0'} 2023-10-12T01:11:42.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e343704b340398c-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.718 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.719 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.719 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.719 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.720 DEBUG:boto:Method: GET 2023-10-12T01:11:42.720 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.721 DEBUG:boto:Data: 2023-10-12T01:11:42.721 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.722 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.723 DEBUG:boto:Params: {} 2023-10-12T01:11:42.723 DEBUG:boto:Token: None 2023-10-12T01:11:42.724 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003645a4625340a6f0-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.728 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.729 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.729 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.729 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.730 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.730 DEBUG:boto:Method: GET 2023-10-12T01:11:42.731 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.731 DEBUG:boto:Data: 2023-10-12T01:11:42.732 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.732 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.733 DEBUG:boto:Params: {} 2023-10-12T01:11:42.734 DEBUG:boto:Token: None 2023-10-12T01:11:42.734 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5d4a9b8427916e-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.740 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.741 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.741 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.742 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.742 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:11:42.743 DEBUG:boto:Data: 2023-10-12T01:11:42.743 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.744 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.744 DEBUG:boto:Params: {} 2023-10-12T01:11:42.745 DEBUG:boto:Token: None 2023-10-12T01:11:42.746 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185 2023-10-12T01:11:42.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fC6Vsne/iGyrdQt4H6//kimMvI= 2023-10-12T01:11:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fC6Vsne/iGyrdQt4H6//kimMvI=', 'Content-Length': '0'} 2023-10-12T01:11:42.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c57e28dba5637d03-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.749 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.749 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.750 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.750 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:11:42.751 DEBUG:boto:Data: 2023-10-12T01:11:42.751 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.752 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.752 DEBUG:boto:Params: {} 2023-10-12T01:11:42.753 DEBUG:boto:Token: None 2023-10-12T01:11:42.753 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185 2023-10-12T01:11:42.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+fC6Vsne/iGyrdQt4H6//kimMvI= 2023-10-12T01:11:42.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+fC6Vsne/iGyrdQt4H6//kimMvI=', 'Content-Length': '0'} 2023-10-12T01:11:42.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde19f1377a259ce-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:11:42.757 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.758 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.758 DEBUG:boto:Method: GET 2023-10-12T01:11:42.759 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:11:42.759 DEBUG:boto:Data: 2023-10-12T01:11:42.760 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.761 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.761 DEBUG:boto:Params: {} 2023-10-12T01:11:42.761 DEBUG:boto:Token: None 2023-10-12T01:11:42.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185 2023-10-12T01:11:42.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6vHsi2uF13/yzUpB9oKKVtjqnU= 2023-10-12T01:11:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6vHsi2uF13/yzUpB9oKKVtjqnU=', 'Content-Length': '0'} 2023-10-12T01:11:42.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4a18dab627184d-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.766 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:11:42.766 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:11:42.766 DEBUG:boto:Method: GET 2023-10-12T01:11:42.767 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:11:42.767 DEBUG:boto:Data: 2023-10-12T01:11:42.768 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.769 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.769 DEBUG:boto:Params: {} 2023-10-12T01:11:42.769 DEBUG:boto:Token: None 2023-10-12T01:11:42.770 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj185 2023-10-12T01:11:42.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6vHsi2uF13/yzUpB9oKKVtjqnU= 2023-10-12T01:11:42.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6vHsi2uF13/yzUpB9oKKVtjqnU=', 'Content-Length': '0'} 2023-10-12T01:11:42.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b513e46c85a653ca-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.774 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.774 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.774 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.775 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.775 DEBUG:boto:Method: GET 2023-10-12T01:11:42.776 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.776 DEBUG:boto:Data: 2023-10-12T01:11:42.777 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.777 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.778 DEBUG:boto:Params: {} 2023-10-12T01:11:42.778 DEBUG:boto:Token: None 2023-10-12T01:11:42.779 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ce90c8cbd021ed-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.783 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.784 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.784 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.785 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.785 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.786 DEBUG:boto:Method: GET 2023-10-12T01:11:42.786 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:11:42.787 DEBUG:boto:Data: 2023-10-12T01:11:42.787 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.788 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.788 DEBUG:boto:Params: {} 2023-10-12T01:11:42.789 DEBUG:boto:Token: None 2023-10-12T01:11:42.789 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077e50114d7c213f-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.795 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:11:42.796 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.796 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.797 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.797 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.798 DEBUG:boto:Method: GET 2023-10-12T01:11:42.798 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.798 DEBUG:boto:Data: 2023-10-12T01:11:42.799 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.800 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.800 DEBUG:boto:Params: {} 2023-10-12T01:11:42.801 DEBUG:boto:Token: None 2023-10-12T01:11:42.801 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.802 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo= 2023-10-12T01:11:42.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo=', 'Content-Length': '0'} 2023-10-12T01:11:42.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d236af9a3fae3a46-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.846 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.847 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.848 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.848 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.848 DEBUG:boto:Method: GET 2023-10-12T01:11:42.849 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.849 DEBUG:boto:Data: 2023-10-12T01:11:42.850 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.851 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.851 DEBUG:boto:Params: {} 2023-10-12T01:11:42.852 DEBUG:boto:Token: None 2023-10-12T01:11:42.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.853 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo= 2023-10-12T01:11:42.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo=', 'Content-Length': '0'} 2023-10-12T01:11:42.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001196f3ffd47edbb1-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.856 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.857 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.857 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.858 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.858 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.858 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.859 DEBUG:boto:Data: 2023-10-12T01:11:42.859 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.860 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.860 DEBUG:boto:Params: {} 2023-10-12T01:11:42.861 DEBUG:boto:Token: None 2023-10-12T01:11:42.861 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bT67uIvJUls5yqSAJ5G/zp9KnD4= 2023-10-12T01:11:42.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bT67uIvJUls5yqSAJ5G/zp9KnD4=', 'Content-Length': '0'} 2023-10-12T01:11:42.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c98f3c9e8f93226-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.865 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.865 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.866 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.866 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.867 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.867 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.867 DEBUG:boto:Data: 2023-10-12T01:11:42.868 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.869 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.869 DEBUG:boto:Params: {} 2023-10-12T01:11:42.869 DEBUG:boto:Token: None 2023-10-12T01:11:42.870 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bT67uIvJUls5yqSAJ5G/zp9KnD4= 2023-10-12T01:11:42.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bT67uIvJUls5yqSAJ5G/zp9KnD4=', 'Content-Length': '0'} 2023-10-12T01:11:42.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bb711b490fc94e-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:11:42.874 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.874 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.874 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.875 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.875 DEBUG:boto:Method: GET 2023-10-12T01:11:42.876 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.876 DEBUG:boto:Data: 2023-10-12T01:11:42.877 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.878 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.878 DEBUG:boto:Params: {} 2023-10-12T01:11:42.879 DEBUG:boto:Token: None 2023-10-12T01:11:42.879 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo= 2023-10-12T01:11:42.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo=', 'Content-Length': '0'} 2023-10-12T01:11:42.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd993a963749c944-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.882 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.883 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.883 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.884 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.884 DEBUG:boto:Method: GET 2023-10-12T01:11:42.885 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.885 DEBUG:boto:Data: 2023-10-12T01:11:42.885 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.886 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.887 DEBUG:boto:Params: {} 2023-10-12T01:11:42.887 DEBUG:boto:Token: None 2023-10-12T01:11:42.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:11:42.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo= 2023-10-12T01:11:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3k5NFyZuY+0XGVxHueptp7B4/oo=', 'Content-Length': '0'} 2023-10-12T01:11:42.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd7f4e5cebf3d2a-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.934 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.935 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.935 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.936 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.936 DEBUG:boto:Method: GET 2023-10-12T01:11:42.937 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.937 DEBUG:boto:Data: 2023-10-12T01:11:42.937 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.938 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.939 DEBUG:boto:Params: {} 2023-10-12T01:11:42.939 DEBUG:boto:Token: None 2023-10-12T01:11:42.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597d55e1012e586d-00652747ce-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.944 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.945 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.945 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.946 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.946 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.947 DEBUG:boto:Method: GET 2023-10-12T01:11:42.947 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.947 DEBUG:boto:Data: 2023-10-12T01:11:42.948 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.949 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.949 DEBUG:boto:Params: {} 2023-10-12T01:11:42.950 DEBUG:boto:Token: None 2023-10-12T01:11:42.950 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:42.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d37401257d66d7-00652747ce-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.956 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:42.957 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.957 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.958 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.958 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:11:42.959 DEBUG:boto:Data: 2023-10-12T01:11:42.959 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.960 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.960 DEBUG:boto:Params: {} 2023-10-12T01:11:42.961 DEBUG:boto:Token: None 2023-10-12T01:11:42.961 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19 2023-10-12T01:11:42.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cpnyLdgriKifgtdnTTusmX/Z3Ic= 2023-10-12T01:11:42.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cpnyLdgriKifgtdnTTusmX/Z3Ic=', 'Content-Length': '0'} 2023-10-12T01:11:42.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d808c4db6f48a808-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.965 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.965 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.966 DEBUG:boto:Method: HEAD 2023-10-12T01:11:42.966 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:11:42.967 DEBUG:boto:Data: 2023-10-12T01:11:42.967 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.968 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.968 DEBUG:boto:Params: {} 2023-10-12T01:11:42.969 DEBUG:boto:Token: None 2023-10-12T01:11:42.969 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19 2023-10-12T01:11:42.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cpnyLdgriKifgtdnTTusmX/Z3Ic= 2023-10-12T01:11:42.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cpnyLdgriKifgtdnTTusmX/Z3Ic=', 'Content-Length': '0'} 2023-10-12T01:11:42.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c5a225011d3b82-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:11:42.974 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.975 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.975 DEBUG:boto:Method: GET 2023-10-12T01:11:42.976 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:11:42.976 DEBUG:boto:Data: 2023-10-12T01:11:42.976 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.977 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.978 DEBUG:boto:Params: {} 2023-10-12T01:11:42.978 DEBUG:boto:Token: None 2023-10-12T01:11:42.979 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19 2023-10-12T01:11:42.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWdn2OCjRzXXH9Y9zWxeEO8pw5g= 2023-10-12T01:11:42.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWdn2OCjRzXXH9Y9zWxeEO8pw5g=', 'Content-Length': '0'} 2023-10-12T01:11:42.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f5e87d9b0d99de8-00652747ce-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.983 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:11:42.983 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:11:42.984 DEBUG:boto:Method: GET 2023-10-12T01:11:42.984 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:11:42.984 DEBUG:boto:Data: 2023-10-12T01:11:42.985 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:42.986 DEBUG:boto:Port: 8000 2023-10-12T01:11:42.987 DEBUG:boto:Params: {} 2023-10-12T01:11:42.987 DEBUG:boto:Token: None 2023-10-12T01:11:42.987 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/obj19 2023-10-12T01:11:42.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWdn2OCjRzXXH9Y9zWxeEO8pw5g= 2023-10-12T01:11:42.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWdn2OCjRzXXH9Y9zWxeEO8pw5g=', 'Content-Length': '0'} 2023-10-12T01:11:42.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068972d7797f8fe91-00652747ce-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:42.991 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:42.992 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:42.992 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.993 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.993 DEBUG:boto:Method: GET 2023-10-12T01:11:42.994 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:11:42.994 DEBUG:boto:Data: 2023-10-12T01:11:42.995 DEBUG:boto:Headers: {} 2023-10-12T01:11:42.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:42.996 DEBUG:boto:Port: 8001 2023-10-12T01:11:42.996 DEBUG:boto:Params: {} 2023-10-12T01:11:42.997 DEBUG:boto:Token: None 2023-10-12T01:11:42.997 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:42 GMT /wafmoa-59/ 2023-10-12T01:11:42.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI= 2023-10-12T01:11:42.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akK4wc+mIgxTFo4fkwv5Vd2J2yI=', 'Content-Length': '0'} 2023-10-12T01:11:43.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09c1de6d0704a0f-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.001 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.002 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.003 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.003 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:43.004 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:11:43.004 DEBUG:boto:Method: GET 2023-10-12T01:11:43.004 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:11:43.005 DEBUG:boto:Data: 2023-10-12T01:11:43.005 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.006 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.007 DEBUG:boto:Params: {} 2023-10-12T01:11:43.007 DEBUG:boto:Token: None 2023-10-12T01:11:43.008 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b20738b7284337c-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.058 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:11:43.060 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.060 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.061 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.061 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.061 DEBUG:boto:Method: GET 2023-10-12T01:11:43.062 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.062 DEBUG:boto:Data: 2023-10-12T01:11:43.063 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.064 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.064 DEBUG:boto:Params: {} 2023-10-12T01:11:43.065 DEBUG:boto:Token: None 2023-10-12T01:11:43.065 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U= 2023-10-12T01:11:43.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U=', 'Content-Length': '0'} 2023-10-12T01:11:43.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b635e4524036d0-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.069 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.070 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.070 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.071 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.071 DEBUG:boto:Method: GET 2023-10-12T01:11:43.072 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.072 DEBUG:boto:Data: 2023-10-12T01:11:43.073 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.073 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.074 DEBUG:boto:Params: {} 2023-10-12T01:11:43.074 DEBUG:boto:Token: None 2023-10-12T01:11:43.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U= 2023-10-12T01:11:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U=', 'Content-Length': '0'} 2023-10-12T01:11:43.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4a191e70460bbf4-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.079 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.079 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.080 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.080 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.081 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.081 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.081 DEBUG:boto:Data: 2023-10-12T01:11:43.082 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.083 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.083 DEBUG:boto:Params: {} 2023-10-12T01:11:43.084 DEBUG:boto:Token: None 2023-10-12T01:11:43.084 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ayAcf5VzOoIRc7y9lzb3juB6Kss= 2023-10-12T01:11:43.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ayAcf5VzOoIRc7y9lzb3juB6Kss=', 'Content-Length': '0'} 2023-10-12T01:11:43.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aef309c670442785-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.087 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.088 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.088 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.089 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.089 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.089 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.090 DEBUG:boto:Data: 2023-10-12T01:11:43.090 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.091 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.092 DEBUG:boto:Params: {} 2023-10-12T01:11:43.092 DEBUG:boto:Token: None 2023-10-12T01:11:43.092 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ayAcf5VzOoIRc7y9lzb3juB6Kss= 2023-10-12T01:11:43.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ayAcf5VzOoIRc7y9lzb3juB6Kss=', 'Content-Length': '0'} 2023-10-12T01:11:43.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc56ba6972854e8-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:11:43.096 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.097 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.097 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.097 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.098 DEBUG:boto:Method: GET 2023-10-12T01:11:43.098 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.099 DEBUG:boto:Data: 2023-10-12T01:11:43.099 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.100 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.100 DEBUG:boto:Params: {} 2023-10-12T01:11:43.101 DEBUG:boto:Token: None 2023-10-12T01:11:43.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U= 2023-10-12T01:11:43.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U=', 'Content-Length': '0'} 2023-10-12T01:11:43.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f16baf6de76987-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.146 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.147 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.147 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.148 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.148 DEBUG:boto:Method: GET 2023-10-12T01:11:43.149 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.149 DEBUG:boto:Data: 2023-10-12T01:11:43.149 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.150 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.151 DEBUG:boto:Params: {} 2023-10-12T01:11:43.151 DEBUG:boto:Token: None 2023-10-12T01:11:43.152 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:11:43.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U= 2023-10-12T01:11:43.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SWiTNU86GDu61VZW0KWNDfP056U=', 'Content-Length': '0'} 2023-10-12T01:11:43.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002feeb08b7f571f49-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.156 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.156 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.156 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.157 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.157 DEBUG:boto:Method: GET 2023-10-12T01:11:43.158 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.158 DEBUG:boto:Data: 2023-10-12T01:11:43.158 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.159 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.160 DEBUG:boto:Params: {} 2023-10-12T01:11:43.160 DEBUG:boto:Token: None 2023-10-12T01:11:43.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e573f5d6ad2ac6-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.165 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.166 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.167 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.167 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.167 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.168 DEBUG:boto:Method: GET 2023-10-12T01:11:43.168 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.169 DEBUG:boto:Data: 2023-10-12T01:11:43.169 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.170 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.171 DEBUG:boto:Params: {} 2023-10-12T01:11:43.171 DEBUG:boto:Token: None 2023-10-12T01:11:43.172 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f09200b4411cc7-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.218 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.219 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.220 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.220 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.221 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:11:43.221 DEBUG:boto:Data: 2023-10-12T01:11:43.221 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.222 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.223 DEBUG:boto:Params: {} 2023-10-12T01:11:43.224 DEBUG:boto:Token: None 2023-10-12T01:11:43.224 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22 2023-10-12T01:11:43.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B7dS/zirFZaSQWxazRiOEFxsRIA= 2023-10-12T01:11:43.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B7dS/zirFZaSQWxazRiOEFxsRIA=', 'Content-Length': '0'} 2023-10-12T01:11:43.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061af1a7557293142-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.228 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.228 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.229 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.229 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:11:43.230 DEBUG:boto:Data: 2023-10-12T01:11:43.230 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.231 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.231 DEBUG:boto:Params: {} 2023-10-12T01:11:43.232 DEBUG:boto:Token: None 2023-10-12T01:11:43.232 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22 2023-10-12T01:11:43.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B7dS/zirFZaSQWxazRiOEFxsRIA= 2023-10-12T01:11:43.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B7dS/zirFZaSQWxazRiOEFxsRIA=', 'Content-Length': '0'} 2023-10-12T01:11:43.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6880e8d42ca1a5b-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:11:43.237 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.237 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.238 DEBUG:boto:Method: GET 2023-10-12T01:11:43.238 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:11:43.239 DEBUG:boto:Data: 2023-10-12T01:11:43.239 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.240 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.240 DEBUG:boto:Params: {} 2023-10-12T01:11:43.241 DEBUG:boto:Token: None 2023-10-12T01:11:43.241 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22 2023-10-12T01:11:43.242 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mhs6k1yNrEW11dBzbdZ4ptiBtsQ= 2023-10-12T01:11:43.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mhs6k1yNrEW11dBzbdZ4ptiBtsQ=', 'Content-Length': '0'} 2023-10-12T01:11:43.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077fd2f8f3e87f14d-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.245 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:11:43.245 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:11:43.246 DEBUG:boto:Method: GET 2023-10-12T01:11:43.246 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:11:43.247 DEBUG:boto:Data: 2023-10-12T01:11:43.247 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.248 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.248 DEBUG:boto:Params: {} 2023-10-12T01:11:43.249 DEBUG:boto:Token: None 2023-10-12T01:11:43.249 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj22 2023-10-12T01:11:43.250 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mhs6k1yNrEW11dBzbdZ4ptiBtsQ= 2023-10-12T01:11:43.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mhs6k1yNrEW11dBzbdZ4ptiBtsQ=', 'Content-Length': '0'} 2023-10-12T01:11:43.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4966111c7275db1-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.253 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.254 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.254 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.255 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.255 DEBUG:boto:Method: GET 2023-10-12T01:11:43.256 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.256 DEBUG:boto:Data: 2023-10-12T01:11:43.257 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.257 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.258 DEBUG:boto:Params: {} 2023-10-12T01:11:43.258 DEBUG:boto:Token: None 2023-10-12T01:11:43.259 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa28f6d42649640-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.263 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.264 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.264 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.265 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.265 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.266 DEBUG:boto:Method: GET 2023-10-12T01:11:43.266 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:11:43.267 DEBUG:boto:Data: 2023-10-12T01:11:43.267 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.268 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.268 DEBUG:boto:Params: {} 2023-10-12T01:11:43.269 DEBUG:boto:Token: None 2023-10-12T01:11:43.269 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081cd60a69dd57450-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.275 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:11:43.277 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.277 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.278 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.278 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.278 DEBUG:boto:Method: GET 2023-10-12T01:11:43.279 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.279 DEBUG:boto:Data: 2023-10-12T01:11:43.280 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.281 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.281 DEBUG:boto:Params: {} 2023-10-12T01:11:43.282 DEBUG:boto:Token: None 2023-10-12T01:11:43.282 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0= 2023-10-12T01:11:43.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0=', 'Content-Length': '0'} 2023-10-12T01:11:43.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c876d16abf8bcf9-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.330 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.331 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.332 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.332 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.333 DEBUG:boto:Method: GET 2023-10-12T01:11:43.333 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.334 DEBUG:boto:Data: 2023-10-12T01:11:43.334 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.335 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.335 DEBUG:boto:Params: {} 2023-10-12T01:11:43.336 DEBUG:boto:Token: None 2023-10-12T01:11:43.337 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0= 2023-10-12T01:11:43.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0=', 'Content-Length': '0'} 2023-10-12T01:11:43.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093118153f83fc0ad-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.341 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.341 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.341 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.342 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.342 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.343 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.343 DEBUG:boto:Data: 2023-10-12T01:11:43.344 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.344 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.345 DEBUG:boto:Params: {} 2023-10-12T01:11:43.345 DEBUG:boto:Token: None 2023-10-12T01:11:43.346 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A2QBESAao10yCc8fI5a41kR1+FM= 2023-10-12T01:11:43.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A2QBESAao10yCc8fI5a41kR1+FM=', 'Content-Length': '0'} 2023-10-12T01:11:43.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f714cf7b124a098-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.349 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.350 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.350 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.351 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.351 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.352 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.352 DEBUG:boto:Data: 2023-10-12T01:11:43.352 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.353 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.354 DEBUG:boto:Params: {} 2023-10-12T01:11:43.354 DEBUG:boto:Token: None 2023-10-12T01:11:43.355 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.355 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A2QBESAao10yCc8fI5a41kR1+FM= 2023-10-12T01:11:43.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A2QBESAao10yCc8fI5a41kR1+FM=', 'Content-Length': '0'} 2023-10-12T01:11:43.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741fd8e9dd3630ff-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:11:43.360 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.360 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.360 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.361 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.361 DEBUG:boto:Method: GET 2023-10-12T01:11:43.362 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.362 DEBUG:boto:Data: 2023-10-12T01:11:43.363 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.363 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.364 DEBUG:boto:Params: {} 2023-10-12T01:11:43.364 DEBUG:boto:Token: None 2023-10-12T01:11:43.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0= 2023-10-12T01:11:43.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0=', 'Content-Length': '0'} 2023-10-12T01:11:43.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b1d336b27e40cd-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.369 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.369 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.370 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.370 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.370 DEBUG:boto:Method: GET 2023-10-12T01:11:43.371 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.371 DEBUG:boto:Data: 2023-10-12T01:11:43.372 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.373 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.373 DEBUG:boto:Params: {} 2023-10-12T01:11:43.374 DEBUG:boto:Token: None 2023-10-12T01:11:43.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:11:43.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0= 2023-10-12T01:11:43.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cbr3NidOqvexpFK5nNJ7yN0d8c0=', 'Content-Length': '0'} 2023-10-12T01:11:43.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e66020ebc8c0f3b-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.418 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.419 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.420 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.420 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.421 DEBUG:boto:Method: GET 2023-10-12T01:11:43.421 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.422 DEBUG:boto:Data: 2023-10-12T01:11:43.422 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.423 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.424 DEBUG:boto:Params: {} 2023-10-12T01:11:43.424 DEBUG:boto:Token: None 2023-10-12T01:11:43.425 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025b7a51fe0995c5-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.429 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.430 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.431 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.431 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.431 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.432 DEBUG:boto:Method: GET 2023-10-12T01:11:43.432 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.433 DEBUG:boto:Data: 2023-10-12T01:11:43.433 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.434 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.435 DEBUG:boto:Params: {} 2023-10-12T01:11:43.435 DEBUG:boto:Token: None 2023-10-12T01:11:43.436 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059dd62a2c3794ee4-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.442 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.443 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.443 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.444 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.444 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:11:43.444 DEBUG:boto:Data: 2023-10-12T01:11:43.445 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.446 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.446 DEBUG:boto:Params: {} 2023-10-12T01:11:43.447 DEBUG:boto:Token: None 2023-10-12T01:11:43.447 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23 2023-10-12T01:11:43.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tvRC4tahpNrkD/2nCwc7tAqgwbQ= 2023-10-12T01:11:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tvRC4tahpNrkD/2nCwc7tAqgwbQ=', 'Content-Length': '0'} 2023-10-12T01:11:43.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ff552e8467d057-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.451 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.451 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.452 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.452 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:11:43.453 DEBUG:boto:Data: 2023-10-12T01:11:43.453 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.454 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.454 DEBUG:boto:Params: {} 2023-10-12T01:11:43.455 DEBUG:boto:Token: None 2023-10-12T01:11:43.455 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23 2023-10-12T01:11:43.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tvRC4tahpNrkD/2nCwc7tAqgwbQ= 2023-10-12T01:11:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tvRC4tahpNrkD/2nCwc7tAqgwbQ=', 'Content-Length': '0'} 2023-10-12T01:11:43.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b4c20c398e9235-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:11:43.459 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.460 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.460 DEBUG:boto:Method: GET 2023-10-12T01:11:43.461 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:11:43.461 DEBUG:boto:Data: 2023-10-12T01:11:43.461 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.462 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.463 DEBUG:boto:Params: {} 2023-10-12T01:11:43.463 DEBUG:boto:Token: None 2023-10-12T01:11:43.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23 2023-10-12T01:11:43.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xO8NTWSsY4/nRmPiF2O3YaYo4wY= 2023-10-12T01:11:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xO8NTWSsY4/nRmPiF2O3YaYo4wY=', 'Content-Length': '0'} 2023-10-12T01:11:43.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b726f119deabf6b-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.468 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:11:43.468 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:11:43.469 DEBUG:boto:Method: GET 2023-10-12T01:11:43.469 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:11:43.470 DEBUG:boto:Data: 2023-10-12T01:11:43.470 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.471 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.471 DEBUG:boto:Params: {} 2023-10-12T01:11:43.472 DEBUG:boto:Token: None 2023-10-12T01:11:43.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj23 2023-10-12T01:11:43.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xO8NTWSsY4/nRmPiF2O3YaYo4wY= 2023-10-12T01:11:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xO8NTWSsY4/nRmPiF2O3YaYo4wY=', 'Content-Length': '0'} 2023-10-12T01:11:43.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4690e08088f8044-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.518 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.519 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.519 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.520 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.520 DEBUG:boto:Method: GET 2023-10-12T01:11:43.521 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.521 DEBUG:boto:Data: 2023-10-12T01:11:43.522 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.523 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.523 DEBUG:boto:Params: {} 2023-10-12T01:11:43.524 DEBUG:boto:Token: None 2023-10-12T01:11:43.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc7f389096bc620c-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.528 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.529 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.529 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.530 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.530 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.531 DEBUG:boto:Method: GET 2023-10-12T01:11:43.531 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:11:43.532 DEBUG:boto:Data: 2023-10-12T01:11:43.532 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.533 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.533 DEBUG:boto:Params: {} 2023-10-12T01:11:43.534 DEBUG:boto:Token: None 2023-10-12T01:11:43.534 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077294ce8944cf72d-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.540 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:11:43.541 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.542 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.542 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.543 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.543 DEBUG:boto:Method: GET 2023-10-12T01:11:43.543 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.544 DEBUG:boto:Data: 2023-10-12T01:11:43.544 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.545 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.545 DEBUG:boto:Params: {} 2023-10-12T01:11:43.546 DEBUG:boto:Token: None 2023-10-12T01:11:43.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU= 2023-10-12T01:11:43.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU=', 'Content-Length': '0'} 2023-10-12T01:11:43.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2d98218e09cd9f-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.550 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.551 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.551 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.552 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.552 DEBUG:boto:Method: GET 2023-10-12T01:11:43.553 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.553 DEBUG:boto:Data: 2023-10-12T01:11:43.553 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.554 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.555 DEBUG:boto:Params: {} 2023-10-12T01:11:43.555 DEBUG:boto:Token: None 2023-10-12T01:11:43.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU= 2023-10-12T01:11:43.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU=', 'Content-Length': '0'} 2023-10-12T01:11:43.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51fb479b487b24e-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.559 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.560 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.560 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.560 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.561 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.561 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.562 DEBUG:boto:Data: 2023-10-12T01:11:43.562 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.564 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.564 DEBUG:boto:Params: {} 2023-10-12T01:11:43.565 DEBUG:boto:Token: None 2023-10-12T01:11:43.565 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4KucIyV7QdmFQ43NvHwJMZhVmu4= 2023-10-12T01:11:43.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4KucIyV7QdmFQ43NvHwJMZhVmu4=', 'Content-Length': '0'} 2023-10-12T01:11:43.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918a381accac69a2-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.569 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.569 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.569 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.570 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.570 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.571 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.571 DEBUG:boto:Data: 2023-10-12T01:11:43.572 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.573 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.573 DEBUG:boto:Params: {} 2023-10-12T01:11:43.574 DEBUG:boto:Token: None 2023-10-12T01:11:43.574 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.575 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4KucIyV7QdmFQ43NvHwJMZhVmu4= 2023-10-12T01:11:43.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4KucIyV7QdmFQ43NvHwJMZhVmu4=', 'Content-Length': '0'} 2023-10-12T01:11:43.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e3e911de280bee-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:11:43.579 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.580 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.581 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.581 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.582 DEBUG:boto:Method: GET 2023-10-12T01:11:43.583 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.583 DEBUG:boto:Data: 2023-10-12T01:11:43.584 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.585 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.585 DEBUG:boto:Params: {} 2023-10-12T01:11:43.586 DEBUG:boto:Token: None 2023-10-12T01:11:43.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU= 2023-10-12T01:11:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU=', 'Content-Length': '0'} 2023-10-12T01:11:43.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f8d93e3d1d41e5-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.590 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.591 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.591 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.592 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.592 DEBUG:boto:Method: GET 2023-10-12T01:11:43.592 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.593 DEBUG:boto:Data: 2023-10-12T01:11:43.593 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.594 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.595 DEBUG:boto:Params: {} 2023-10-12T01:11:43.595 DEBUG:boto:Token: None 2023-10-12T01:11:43.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:11:43.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU= 2023-10-12T01:11:43.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/jyaENc+7s45MUCOGZwVZXSAefU=', 'Content-Length': '0'} 2023-10-12T01:11:43.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a72c10addf4caf5-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.642 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.643 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.643 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.644 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.644 DEBUG:boto:Method: GET 2023-10-12T01:11:43.645 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.645 DEBUG:boto:Data: 2023-10-12T01:11:43.646 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.647 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.647 DEBUG:boto:Params: {} 2023-10-12T01:11:43.648 DEBUG:boto:Token: None 2023-10-12T01:11:43.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.649 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad2cd8d40214e2a-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.652 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.653 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.654 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.654 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.655 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.655 DEBUG:boto:Method: GET 2023-10-12T01:11:43.656 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.656 DEBUG:boto:Data: 2023-10-12T01:11:43.656 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.657 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.658 DEBUG:boto:Params: {} 2023-10-12T01:11:43.658 DEBUG:boto:Token: None 2023-10-12T01:11:43.659 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.659 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c674950f095ccf-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.665 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.666 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.666 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.666 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.667 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:11:43.667 DEBUG:boto:Data: 2023-10-12T01:11:43.668 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.668 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.669 DEBUG:boto:Params: {} 2023-10-12T01:11:43.669 DEBUG:boto:Token: None 2023-10-12T01:11:43.670 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29 2023-10-12T01:11:43.670 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pMH5Bjb7b6sV9mC6+81l8JYjlc= 2023-10-12T01:11:43.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pMH5Bjb7b6sV9mC6+81l8JYjlc=', 'Content-Length': '0'} 2023-10-12T01:11:43.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03053fae8f51d97-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.673 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.674 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.674 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.675 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:11:43.675 DEBUG:boto:Data: 2023-10-12T01:11:43.675 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.676 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.677 DEBUG:boto:Params: {} 2023-10-12T01:11:43.677 DEBUG:boto:Token: None 2023-10-12T01:11:43.677 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29 2023-10-12T01:11:43.678 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5pMH5Bjb7b6sV9mC6+81l8JYjlc= 2023-10-12T01:11:43.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5pMH5Bjb7b6sV9mC6+81l8JYjlc=', 'Content-Length': '0'} 2023-10-12T01:11:43.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122e03654d0a4023-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:11:43.682 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.682 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.682 DEBUG:boto:Method: GET 2023-10-12T01:11:43.683 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:11:43.683 DEBUG:boto:Data: 2023-10-12T01:11:43.684 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.685 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.685 DEBUG:boto:Params: {} 2023-10-12T01:11:43.686 DEBUG:boto:Token: None 2023-10-12T01:11:43.686 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29 2023-10-12T01:11:43.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TA0chjhut2iz1pLCIuBRXjgZgxM= 2023-10-12T01:11:43.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TA0chjhut2iz1pLCIuBRXjgZgxM=', 'Content-Length': '0'} 2023-10-12T01:11:43.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3aa6feb268c6068-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.734 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:11:43.735 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:11:43.735 DEBUG:boto:Method: GET 2023-10-12T01:11:43.736 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:11:43.736 DEBUG:boto:Data: 2023-10-12T01:11:43.737 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.738 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.738 DEBUG:boto:Params: {} 2023-10-12T01:11:43.739 DEBUG:boto:Token: None 2023-10-12T01:11:43.741 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj29 2023-10-12T01:11:43.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TA0chjhut2iz1pLCIuBRXjgZgxM= 2023-10-12T01:11:43.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TA0chjhut2iz1pLCIuBRXjgZgxM=', 'Content-Length': '0'} 2023-10-12T01:11:43.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed65972d1bbd7fa6-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.745 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.746 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.746 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.747 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.747 DEBUG:boto:Method: GET 2023-10-12T01:11:43.747 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.748 DEBUG:boto:Data: 2023-10-12T01:11:43.748 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.749 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.750 DEBUG:boto:Params: {} 2023-10-12T01:11:43.750 DEBUG:boto:Token: None 2023-10-12T01:11:43.750 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.751 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5a93ac1fc26356-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.755 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.755 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.756 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.756 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.757 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.757 DEBUG:boto:Method: GET 2023-10-12T01:11:43.757 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:11:43.758 DEBUG:boto:Data: 2023-10-12T01:11:43.758 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.759 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.759 DEBUG:boto:Params: {} 2023-10-12T01:11:43.760 DEBUG:boto:Token: None 2023-10-12T01:11:43.760 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09c71f2fb8bc01f-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.766 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:11:43.767 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.767 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.768 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.768 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.768 DEBUG:boto:Method: GET 2023-10-12T01:11:43.769 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.769 DEBUG:boto:Data: 2023-10-12T01:11:43.770 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.771 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.771 DEBUG:boto:Params: {} 2023-10-12T01:11:43.772 DEBUG:boto:Token: None 2023-10-12T01:11:43.772 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.773 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc= 2023-10-12T01:11:43.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc=', 'Content-Length': '0'} 2023-10-12T01:11:43.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c32870241b20f44-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.776 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.776 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.777 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.777 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.778 DEBUG:boto:Method: GET 2023-10-12T01:11:43.778 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.779 DEBUG:boto:Data: 2023-10-12T01:11:43.779 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.780 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.780 DEBUG:boto:Params: {} 2023-10-12T01:11:43.781 DEBUG:boto:Token: None 2023-10-12T01:11:43.781 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc= 2023-10-12T01:11:43.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc=', 'Content-Length': '0'} 2023-10-12T01:11:43.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318e43969fb64321-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.785 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.785 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.786 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.786 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.787 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.787 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.787 DEBUG:boto:Data: 2023-10-12T01:11:43.788 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.789 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.789 DEBUG:boto:Params: {} 2023-10-12T01:11:43.789 DEBUG:boto:Token: None 2023-10-12T01:11:43.790 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.790 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5xG1wOH4l2F+4Jw5b705MNGreT0= 2023-10-12T01:11:43.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5xG1wOH4l2F+4Jw5b705MNGreT0=', 'Content-Length': '0'} 2023-10-12T01:11:43.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9fb83b0d7327aa-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.793 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.794 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.794 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.795 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.795 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.796 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.796 DEBUG:boto:Data: 2023-10-12T01:11:43.796 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.797 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.798 DEBUG:boto:Params: {} 2023-10-12T01:11:43.798 DEBUG:boto:Token: None 2023-10-12T01:11:43.799 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5xG1wOH4l2F+4Jw5b705MNGreT0= 2023-10-12T01:11:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5xG1wOH4l2F+4Jw5b705MNGreT0=', 'Content-Length': '0'} 2023-10-12T01:11:43.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014db28f92d51dbaa-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:11:43.803 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.803 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.803 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.804 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.804 DEBUG:boto:Method: GET 2023-10-12T01:11:43.805 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.805 DEBUG:boto:Data: 2023-10-12T01:11:43.806 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.806 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.807 DEBUG:boto:Params: {} 2023-10-12T01:11:43.807 DEBUG:boto:Token: None 2023-10-12T01:11:43.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc= 2023-10-12T01:11:43.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc=', 'Content-Length': '0'} 2023-10-12T01:11:43.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f58d0b9e28a731-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.854 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.855 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.856 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.856 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.857 DEBUG:boto:Method: GET 2023-10-12T01:11:43.857 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.858 DEBUG:boto:Data: 2023-10-12T01:11:43.858 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.859 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.859 DEBUG:boto:Params: {} 2023-10-12T01:11:43.860 DEBUG:boto:Token: None 2023-10-12T01:11:43.861 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:11:43.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc= 2023-10-12T01:11:43.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:U7G+BsX4unXODyOrBXsjSM5Zykc=', 'Content-Length': '0'} 2023-10-12T01:11:43.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009084a0bb249886d5-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.865 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.865 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.866 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.866 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.867 DEBUG:boto:Method: GET 2023-10-12T01:11:43.867 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.867 DEBUG:boto:Data: 2023-10-12T01:11:43.868 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.869 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.869 DEBUG:boto:Params: {} 2023-10-12T01:11:43.870 DEBUG:boto:Token: None 2023-10-12T01:11:43.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000125a670910dbb813-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.874 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.875 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.876 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.876 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.877 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.877 DEBUG:boto:Method: GET 2023-10-12T01:11:43.878 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.878 DEBUG:boto:Data: 2023-10-12T01:11:43.878 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.879 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.880 DEBUG:boto:Params: {} 2023-10-12T01:11:43.880 DEBUG:boto:Token: None 2023-10-12T01:11:43.881 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.881 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cc2449c5945a28a-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.886 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.887 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.888 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.888 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.888 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:11:43.889 DEBUG:boto:Data: 2023-10-12T01:11:43.889 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.890 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.891 DEBUG:boto:Params: {} 2023-10-12T01:11:43.891 DEBUG:boto:Token: None 2023-10-12T01:11:43.891 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31 2023-10-12T01:11:43.892 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:efo6MRIBdsf48DV4jWHSWgVudaE= 2023-10-12T01:11:43.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:efo6MRIBdsf48DV4jWHSWgVudaE=', 'Content-Length': '0'} 2023-10-12T01:11:43.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030025bc195a33ef9-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.895 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.895 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.896 DEBUG:boto:Method: HEAD 2023-10-12T01:11:43.896 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:11:43.897 DEBUG:boto:Data: 2023-10-12T01:11:43.897 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.898 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.898 DEBUG:boto:Params: {} 2023-10-12T01:11:43.899 DEBUG:boto:Token: None 2023-10-12T01:11:43.899 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31 2023-10-12T01:11:43.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:efo6MRIBdsf48DV4jWHSWgVudaE= 2023-10-12T01:11:43.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:efo6MRIBdsf48DV4jWHSWgVudaE=', 'Content-Length': '0'} 2023-10-12T01:11:43.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058dc0dc3caa8a866-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:11:43.903 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.904 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.904 DEBUG:boto:Method: GET 2023-10-12T01:11:43.904 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:11:43.905 DEBUG:boto:Data: 2023-10-12T01:11:43.905 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.906 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.907 DEBUG:boto:Params: {} 2023-10-12T01:11:43.907 DEBUG:boto:Token: None 2023-10-12T01:11:43.908 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31 2023-10-12T01:11:43.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BPt9dRK8os2Fv9Kp0K3G6fv0kT8= 2023-10-12T01:11:43.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BPt9dRK8os2Fv9Kp0K3G6fv0kT8=', 'Content-Length': '0'} 2023-10-12T01:11:43.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d708a8d816eac37e-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.911 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:11:43.912 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:11:43.912 DEBUG:boto:Method: GET 2023-10-12T01:11:43.913 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:11:43.913 DEBUG:boto:Data: 2023-10-12T01:11:43.914 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.915 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.915 DEBUG:boto:Params: {} 2023-10-12T01:11:43.916 DEBUG:boto:Token: None 2023-10-12T01:11:43.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj31 2023-10-12T01:11:43.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BPt9dRK8os2Fv9Kp0K3G6fv0kT8= 2023-10-12T01:11:43.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BPt9dRK8os2Fv9Kp0K3G6fv0kT8=', 'Content-Length': '0'} 2023-10-12T01:11:43.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a1b07dd2114e90-00652747cf-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.920 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.920 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.921 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.921 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.922 DEBUG:boto:Method: GET 2023-10-12T01:11:43.922 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.922 DEBUG:boto:Data: 2023-10-12T01:11:43.923 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.924 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.924 DEBUG:boto:Params: {} 2023-10-12T01:11:43.925 DEBUG:boto:Token: None 2023-10-12T01:11:43.925 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.926 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094bc8437762902a6-00652747cf-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.970 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.971 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:43.972 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:43.972 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.972 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.973 DEBUG:boto:Method: GET 2023-10-12T01:11:43.973 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:11:43.974 DEBUG:boto:Data: 2023-10-12T01:11:43.975 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.976 DEBUG:boto:Port: 8000 2023-10-12T01:11:43.976 DEBUG:boto:Params: {} 2023-10-12T01:11:43.977 DEBUG:boto:Token: None 2023-10-12T01:11:43.977 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/ 2023-10-12T01:11:43.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g= 2023-10-12T01:11:43.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JNret1GIvrEIu/5DDFQZBl9qK/g=', 'Content-Length': '0'} 2023-10-12T01:11:43.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bfd41212f89568b-00652747cf-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.983 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:43.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:11:43.985 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:43.985 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:43.985 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.986 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.986 DEBUG:boto:Method: GET 2023-10-12T01:11:43.987 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.987 DEBUG:boto:Data: 2023-10-12T01:11:43.988 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:43.988 DEBUG:boto:Port: 8001 2023-10-12T01:11:43.990 DEBUG:boto:Params: {} 2023-10-12T01:11:43.991 DEBUG:boto:Token: None 2023-10-12T01:11:43.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:43 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.992 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8+WfNN1DfqnApExJ8NQ6/jTUPB0= 2023-10-12T01:11:43.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8+WfNN1DfqnApExJ8NQ6/jTUPB0=', 'Content-Length': '0'} 2023-10-12T01:11:43.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c93930386e40af1-00652747cf-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:43.995 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:43.996 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:43.996 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.997 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.997 DEBUG:boto:Method: GET 2023-10-12T01:11:43.998 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:43.998 DEBUG:boto:Data: 2023-10-12T01:11:43.999 DEBUG:boto:Headers: {} 2023-10-12T01:11:43.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:43.999 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.000 DEBUG:boto:Params: {} 2023-10-12T01:11:44.001 DEBUG:boto:Token: None 2023-10-12T01:11:44.001 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.002 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0= 2023-10-12T01:11:44.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0=', 'Content-Length': '0'} 2023-10-12T01:11:44.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232edfac2802531a-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.050 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.051 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.051 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.052 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.052 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.053 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.053 DEBUG:boto:Data: 2023-10-12T01:11:44.054 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.054 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.055 DEBUG:boto:Params: {} 2023-10-12T01:11:44.056 DEBUG:boto:Token: None 2023-10-12T01:11:44.056 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/PWmv00ymJewv4IGcuPMXupq05Q= 2023-10-12T01:11:44.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/PWmv00ymJewv4IGcuPMXupq05Q=', 'Content-Length': '0'} 2023-10-12T01:11:44.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddae9028e83e1aba-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.060 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.061 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.061 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.062 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.062 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.062 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.063 DEBUG:boto:Data: 2023-10-12T01:11:44.063 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.064 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.065 DEBUG:boto:Params: {} 2023-10-12T01:11:44.065 DEBUG:boto:Token: None 2023-10-12T01:11:44.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/PWmv00ymJewv4IGcuPMXupq05Q= 2023-10-12T01:11:44.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/PWmv00ymJewv4IGcuPMXupq05Q=', 'Content-Length': '0'} 2023-10-12T01:11:44.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b8968f2a66f4b9-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:11:44.070 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.070 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.071 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.071 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.072 DEBUG:boto:Method: GET 2023-10-12T01:11:44.072 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.073 DEBUG:boto:Data: 2023-10-12T01:11:44.073 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.074 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.074 DEBUG:boto:Params: {} 2023-10-12T01:11:44.075 DEBUG:boto:Token: None 2023-10-12T01:11:44.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0= 2023-10-12T01:11:44.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0=', 'Content-Length': '0'} 2023-10-12T01:11:44.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe4ef81debe2a89-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.080 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.080 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.081 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.081 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.082 DEBUG:boto:Method: GET 2023-10-12T01:11:44.082 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.082 DEBUG:boto:Data: 2023-10-12T01:11:44.083 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.084 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.084 DEBUG:boto:Params: {} 2023-10-12T01:11:44.085 DEBUG:boto:Token: None 2023-10-12T01:11:44.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:11:44.086 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0= 2023-10-12T01:11:44.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sX0JEpQRE7G/qyb6MH+zwU3+kE0=', 'Content-Length': '0'} 2023-10-12T01:11:44.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d26d381cbaf80f-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.090 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.091 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.091 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.092 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.092 DEBUG:boto:Method: GET 2023-10-12T01:11:44.092 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.093 DEBUG:boto:Data: 2023-10-12T01:11:44.093 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.094 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.095 DEBUG:boto:Params: {} 2023-10-12T01:11:44.095 DEBUG:boto:Token: None 2023-10-12T01:11:44.096 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f73455813f4aaf-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.100 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.101 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.101 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.101 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.102 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.102 DEBUG:boto:Method: GET 2023-10-12T01:11:44.103 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.103 DEBUG:boto:Data: 2023-10-12T01:11:44.104 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.105 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.105 DEBUG:boto:Params: {} 2023-10-12T01:11:44.105 DEBUG:boto:Token: None 2023-10-12T01:11:44.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004981340adcfbfe2c-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.112 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.113 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.113 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.114 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.114 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:11:44.115 DEBUG:boto:Data: 2023-10-12T01:11:44.115 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.116 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.116 DEBUG:boto:Params: {} 2023-10-12T01:11:44.117 DEBUG:boto:Token: None 2023-10-12T01:11:44.117 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33 2023-10-12T01:11:44.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ry4/EP1dTH6x7KW6j5Tkir8CUbI= 2023-10-12T01:11:44.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ry4/EP1dTH6x7KW6j5Tkir8CUbI=', 'Content-Length': '0'} 2023-10-12T01:11:44.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d0f15ce3e96978-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.121 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.121 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.122 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.122 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:11:44.123 DEBUG:boto:Data: 2023-10-12T01:11:44.123 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.124 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.124 DEBUG:boto:Params: {} 2023-10-12T01:11:44.125 DEBUG:boto:Token: None 2023-10-12T01:11:44.125 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33 2023-10-12T01:11:44.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ry4/EP1dTH6x7KW6j5Tkir8CUbI= 2023-10-12T01:11:44.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ry4/EP1dTH6x7KW6j5Tkir8CUbI=', 'Content-Length': '0'} 2023-10-12T01:11:44.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c3c3844fa3a5ac-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:11:44.129 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.129 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.130 DEBUG:boto:Method: GET 2023-10-12T01:11:44.130 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:11:44.131 DEBUG:boto:Data: 2023-10-12T01:11:44.131 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.132 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.132 DEBUG:boto:Params: {} 2023-10-12T01:11:44.133 DEBUG:boto:Token: None 2023-10-12T01:11:44.133 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33 2023-10-12T01:11:44.134 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ThP003IFb5vqjezEOiqAPa3jDNQ= 2023-10-12T01:11:44.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ThP003IFb5vqjezEOiqAPa3jDNQ=', 'Content-Length': '0'} 2023-10-12T01:11:44.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb2ad13ba37a2de-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.178 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:11:44.179 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:11:44.179 DEBUG:boto:Method: GET 2023-10-12T01:11:44.180 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:11:44.180 DEBUG:boto:Data: 2023-10-12T01:11:44.181 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.182 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.182 DEBUG:boto:Params: {} 2023-10-12T01:11:44.183 DEBUG:boto:Token: None 2023-10-12T01:11:44.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj33 2023-10-12T01:11:44.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ThP003IFb5vqjezEOiqAPa3jDNQ= 2023-10-12T01:11:44.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ThP003IFb5vqjezEOiqAPa3jDNQ=', 'Content-Length': '0'} 2023-10-12T01:11:44.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c7c6294429a149-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.187 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.188 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.188 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.191 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.192 DEBUG:boto:Method: GET 2023-10-12T01:11:44.192 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.193 DEBUG:boto:Data: 2023-10-12T01:11:44.193 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.194 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.194 DEBUG:boto:Params: {} 2023-10-12T01:11:44.195 DEBUG:boto:Token: None 2023-10-12T01:11:44.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d8b97abf91a5d8-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.200 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.201 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.201 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.202 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.202 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.203 DEBUG:boto:Method: GET 2023-10-12T01:11:44.203 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:11:44.203 DEBUG:boto:Data: 2023-10-12T01:11:44.204 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.205 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.205 DEBUG:boto:Params: {} 2023-10-12T01:11:44.206 DEBUG:boto:Token: None 2023-10-12T01:11:44.206 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.207 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace7f0252af1183d-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.258 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:11:44.260 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.260 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.261 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.261 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.262 DEBUG:boto:Method: GET 2023-10-12T01:11:44.262 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.263 DEBUG:boto:Data: 2023-10-12T01:11:44.263 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.264 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.264 DEBUG:boto:Params: {} 2023-10-12T01:11:44.265 DEBUG:boto:Token: None 2023-10-12T01:11:44.266 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M= 2023-10-12T01:11:44.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M=', 'Content-Length': '0'} 2023-10-12T01:11:44.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7308678c9031e1-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.270 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.271 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.271 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.272 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.272 DEBUG:boto:Method: GET 2023-10-12T01:11:44.273 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.273 DEBUG:boto:Data: 2023-10-12T01:11:44.273 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.274 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.275 DEBUG:boto:Params: {} 2023-10-12T01:11:44.275 DEBUG:boto:Token: None 2023-10-12T01:11:44.276 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M= 2023-10-12T01:11:44.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M=', 'Content-Length': '0'} 2023-10-12T01:11:44.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714db79c056da38e-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.280 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.280 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.280 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.281 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.281 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.282 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.282 DEBUG:boto:Data: 2023-10-12T01:11:44.283 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.284 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.284 DEBUG:boto:Params: {} 2023-10-12T01:11:44.285 DEBUG:boto:Token: None 2023-10-12T01:11:44.285 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aJiqauCGfvddygxdCc63Xnxo6Tc= 2023-10-12T01:11:44.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aJiqauCGfvddygxdCc63Xnxo6Tc=', 'Content-Length': '0'} 2023-10-12T01:11:44.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46690f161d4e4aa-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.289 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.289 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.289 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.290 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.290 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.291 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.291 DEBUG:boto:Data: 2023-10-12T01:11:44.292 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.293 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.293 DEBUG:boto:Params: {} 2023-10-12T01:11:44.294 DEBUG:boto:Token: None 2023-10-12T01:11:44.294 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.295 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aJiqauCGfvddygxdCc63Xnxo6Tc= 2023-10-12T01:11:44.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aJiqauCGfvddygxdCc63Xnxo6Tc=', 'Content-Length': '0'} 2023-10-12T01:11:44.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b56f704478ccd9f-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:11:44.299 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.299 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.299 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.300 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.300 DEBUG:boto:Method: GET 2023-10-12T01:11:44.301 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.301 DEBUG:boto:Data: 2023-10-12T01:11:44.302 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.303 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.303 DEBUG:boto:Params: {} 2023-10-12T01:11:44.304 DEBUG:boto:Token: None 2023-10-12T01:11:44.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M= 2023-10-12T01:11:44.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M=', 'Content-Length': '0'} 2023-10-12T01:11:44.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793e3148df345eb5-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.308 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.309 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.310 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.310 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.311 DEBUG:boto:Method: GET 2023-10-12T01:11:44.311 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.311 DEBUG:boto:Data: 2023-10-12T01:11:44.312 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.313 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.313 DEBUG:boto:Params: {} 2023-10-12T01:11:44.313 DEBUG:boto:Token: None 2023-10-12T01:11:44.314 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:11:44.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M= 2023-10-12T01:11:44.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8IH4HOjqOXX/ZOivNNYt+H8GY6M=', 'Content-Length': '0'} 2023-10-12T01:11:44.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25525ecff07f135-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.318 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.318 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.318 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.319 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.319 DEBUG:boto:Method: GET 2023-10-12T01:11:44.320 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.320 DEBUG:boto:Data: 2023-10-12T01:11:44.321 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.322 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.322 DEBUG:boto:Params: {} 2023-10-12T01:11:44.323 DEBUG:boto:Token: None 2023-10-12T01:11:44.323 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb69bf8929b4b32-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.327 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.328 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.329 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.329 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.329 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.330 DEBUG:boto:Method: GET 2023-10-12T01:11:44.330 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.331 DEBUG:boto:Data: 2023-10-12T01:11:44.331 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.332 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.332 DEBUG:boto:Params: {} 2023-10-12T01:11:44.333 DEBUG:boto:Token: None 2023-10-12T01:11:44.333 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f100a50d005cd6-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.339 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.340 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.340 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.340 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.341 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:11:44.341 DEBUG:boto:Data: 2023-10-12T01:11:44.342 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.342 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.343 DEBUG:boto:Params: {} 2023-10-12T01:11:44.343 DEBUG:boto:Token: None 2023-10-12T01:11:44.344 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34 2023-10-12T01:11:44.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5NcZwVaW9gTLsVMGUw5pRIr7k00= 2023-10-12T01:11:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5NcZwVaW9gTLsVMGUw5pRIr7k00=', 'Content-Length': '0'} 2023-10-12T01:11:44.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed868ee2840c1bb-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.347 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.348 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.348 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.349 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:11:44.350 DEBUG:boto:Data: 2023-10-12T01:11:44.350 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.351 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.352 DEBUG:boto:Params: {} 2023-10-12T01:11:44.352 DEBUG:boto:Token: None 2023-10-12T01:11:44.353 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34 2023-10-12T01:11:44.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5NcZwVaW9gTLsVMGUw5pRIr7k00= 2023-10-12T01:11:44.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5NcZwVaW9gTLsVMGUw5pRIr7k00=', 'Content-Length': '0'} 2023-10-12T01:11:44.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6d66f6d351dfdc-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:11:44.359 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.359 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.360 DEBUG:boto:Method: GET 2023-10-12T01:11:44.360 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:11:44.361 DEBUG:boto:Data: 2023-10-12T01:11:44.361 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.362 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.362 DEBUG:boto:Params: {} 2023-10-12T01:11:44.363 DEBUG:boto:Token: None 2023-10-12T01:11:44.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34 2023-10-12T01:11:44.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cqWIQtSIUFF1uwlVZEeF0MDuXFo= 2023-10-12T01:11:44.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cqWIQtSIUFF1uwlVZEeF0MDuXFo=', 'Content-Length': '0'} 2023-10-12T01:11:44.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f0671a764074b0b-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.414 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:11:44.415 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:11:44.416 DEBUG:boto:Method: GET 2023-10-12T01:11:44.416 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:11:44.416 DEBUG:boto:Data: 2023-10-12T01:11:44.417 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.418 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.418 DEBUG:boto:Params: {} 2023-10-12T01:11:44.419 DEBUG:boto:Token: None 2023-10-12T01:11:44.419 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj34 2023-10-12T01:11:44.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cqWIQtSIUFF1uwlVZEeF0MDuXFo= 2023-10-12T01:11:44.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cqWIQtSIUFF1uwlVZEeF0MDuXFo=', 'Content-Length': '0'} 2023-10-12T01:11:44.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3c999d6a2157e6-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.423 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.424 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.424 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.424 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.425 DEBUG:boto:Method: GET 2023-10-12T01:11:44.425 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.426 DEBUG:boto:Data: 2023-10-12T01:11:44.426 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.427 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.427 DEBUG:boto:Params: {} 2023-10-12T01:11:44.428 DEBUG:boto:Token: None 2023-10-12T01:11:44.428 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f5d2f1c603323d-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.432 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.433 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.434 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.434 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.434 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.435 DEBUG:boto:Method: GET 2023-10-12T01:11:44.435 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:11:44.436 DEBUG:boto:Data: 2023-10-12T01:11:44.436 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.437 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.437 DEBUG:boto:Params: {} 2023-10-12T01:11:44.438 DEBUG:boto:Token: None 2023-10-12T01:11:44.439 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edc629e322bb0aa5-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.445 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:11:44.446 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.446 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.447 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.447 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.447 DEBUG:boto:Method: GET 2023-10-12T01:11:44.448 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.448 DEBUG:boto:Data: 2023-10-12T01:11:44.449 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.450 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.450 DEBUG:boto:Params: {} 2023-10-12T01:11:44.451 DEBUG:boto:Token: None 2023-10-12T01:11:44.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0= 2023-10-12T01:11:44.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0=', 'Content-Length': '0'} 2023-10-12T01:11:44.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536dcc36cd3e8912-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.455 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.456 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.456 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.457 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.457 DEBUG:boto:Method: GET 2023-10-12T01:11:44.458 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.458 DEBUG:boto:Data: 2023-10-12T01:11:44.458 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.459 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.460 DEBUG:boto:Params: {} 2023-10-12T01:11:44.460 DEBUG:boto:Token: None 2023-10-12T01:11:44.461 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0= 2023-10-12T01:11:44.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0=', 'Content-Length': '0'} 2023-10-12T01:11:44.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744c767fbebf8089-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.465 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.465 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.466 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.466 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.466 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.467 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.467 DEBUG:boto:Data: 2023-10-12T01:11:44.468 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.468 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.469 DEBUG:boto:Params: {} 2023-10-12T01:11:44.469 DEBUG:boto:Token: None 2023-10-12T01:11:44.470 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lpRMU6MP/l+eBiAQ+NU33r2p2iU= 2023-10-12T01:11:44.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lpRMU6MP/l+eBiAQ+NU33r2p2iU=', 'Content-Length': '0'} 2023-10-12T01:11:44.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064517eb833687854-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.474 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.474 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.474 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.475 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.475 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.476 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.476 DEBUG:boto:Data: 2023-10-12T01:11:44.477 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.478 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.478 DEBUG:boto:Params: {} 2023-10-12T01:11:44.478 DEBUG:boto:Token: None 2023-10-12T01:11:44.479 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lpRMU6MP/l+eBiAQ+NU33r2p2iU= 2023-10-12T01:11:44.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lpRMU6MP/l+eBiAQ+NU33r2p2iU=', 'Content-Length': '0'} 2023-10-12T01:11:44.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb4576692db6b5b-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:11:44.483 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.484 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.484 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.484 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.485 DEBUG:boto:Method: GET 2023-10-12T01:11:44.485 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.486 DEBUG:boto:Data: 2023-10-12T01:11:44.486 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.487 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.488 DEBUG:boto:Params: {} 2023-10-12T01:11:44.488 DEBUG:boto:Token: None 2023-10-12T01:11:44.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0= 2023-10-12T01:11:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0=', 'Content-Length': '0'} 2023-10-12T01:11:44.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958064ffeace96e4-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.492 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.493 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.493 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.494 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.494 DEBUG:boto:Method: GET 2023-10-12T01:11:44.495 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.495 DEBUG:boto:Data: 2023-10-12T01:11:44.495 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.496 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.497 DEBUG:boto:Params: {} 2023-10-12T01:11:44.497 DEBUG:boto:Token: None 2023-10-12T01:11:44.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:11:44.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0= 2023-10-12T01:11:44.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tMo3yJciwajinq6BfqFgn35tiO0=', 'Content-Length': '0'} 2023-10-12T01:11:44.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051460c1f5674f991-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.501 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.502 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.502 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.503 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.503 DEBUG:boto:Method: GET 2023-10-12T01:11:44.503 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.504 DEBUG:boto:Data: 2023-10-12T01:11:44.504 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.505 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.506 DEBUG:boto:Params: {} 2023-10-12T01:11:44.506 DEBUG:boto:Token: None 2023-10-12T01:11:44.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.507 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c381365fa2484bb-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.511 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.512 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.512 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.513 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.513 DEBUG:boto:Method: GET 2023-10-12T01:11:44.514 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.514 DEBUG:boto:Data: 2023-10-12T01:11:44.514 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.515 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.516 DEBUG:boto:Params: {} 2023-10-12T01:11:44.516 DEBUG:boto:Token: None 2023-10-12T01:11:44.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017be3f33b7563b09-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.522 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.523 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.523 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.524 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.524 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:11:44.525 DEBUG:boto:Data: 2023-10-12T01:11:44.525 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.526 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.526 DEBUG:boto:Params: {} 2023-10-12T01:11:44.527 DEBUG:boto:Token: None 2023-10-12T01:11:44.527 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37 2023-10-12T01:11:44.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UobmfTQ46ISZvdsvV7NfSx/fww4= 2023-10-12T01:11:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UobmfTQ46ISZvdsvV7NfSx/fww4=', 'Content-Length': '0'} 2023-10-12T01:11:44.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1b0f76c2bcbce7-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.531 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.531 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.532 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.532 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:11:44.532 DEBUG:boto:Data: 2023-10-12T01:11:44.533 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.534 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.534 DEBUG:boto:Params: {} 2023-10-12T01:11:44.535 DEBUG:boto:Token: None 2023-10-12T01:11:44.535 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37 2023-10-12T01:11:44.536 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UobmfTQ46ISZvdsvV7NfSx/fww4= 2023-10-12T01:11:44.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UobmfTQ46ISZvdsvV7NfSx/fww4=', 'Content-Length': '0'} 2023-10-12T01:11:44.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1bfe9c1165868d-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:11:44.539 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.540 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.540 DEBUG:boto:Method: GET 2023-10-12T01:11:44.540 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:11:44.541 DEBUG:boto:Data: 2023-10-12T01:11:44.541 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.542 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.543 DEBUG:boto:Params: {} 2023-10-12T01:11:44.543 DEBUG:boto:Token: None 2023-10-12T01:11:44.544 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37 2023-10-12T01:11:44.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JaWrFdRVH+vFOyUlHjrCCKs906s= 2023-10-12T01:11:44.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JaWrFdRVH+vFOyUlHjrCCKs906s=', 'Content-Length': '0'} 2023-10-12T01:11:44.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e53923587fb4cf6-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.590 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:11:44.591 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:11:44.592 DEBUG:boto:Method: GET 2023-10-12T01:11:44.592 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:11:44.592 DEBUG:boto:Data: 2023-10-12T01:11:44.593 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.594 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.594 DEBUG:boto:Params: {} 2023-10-12T01:11:44.595 DEBUG:boto:Token: None 2023-10-12T01:11:44.595 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj37 2023-10-12T01:11:44.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JaWrFdRVH+vFOyUlHjrCCKs906s= 2023-10-12T01:11:44.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JaWrFdRVH+vFOyUlHjrCCKs906s=', 'Content-Length': '0'} 2023-10-12T01:11:44.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b530725eaa8a31db-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.600 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.600 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.600 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.601 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.601 DEBUG:boto:Method: GET 2023-10-12T01:11:44.602 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.602 DEBUG:boto:Data: 2023-10-12T01:11:44.603 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.603 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.604 DEBUG:boto:Params: {} 2023-10-12T01:11:44.604 DEBUG:boto:Token: None 2023-10-12T01:11:44.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc4dc537fa6a1ca6-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.609 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.610 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.610 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.610 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.611 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.611 DEBUG:boto:Method: GET 2023-10-12T01:11:44.612 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:11:44.612 DEBUG:boto:Data: 2023-10-12T01:11:44.613 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.613 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.614 DEBUG:boto:Params: {} 2023-10-12T01:11:44.615 DEBUG:boto:Token: None 2023-10-12T01:11:44.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c41564eda3f5d7-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.662 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:11:44.664 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.664 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.665 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.665 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.666 DEBUG:boto:Method: GET 2023-10-12T01:11:44.666 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.667 DEBUG:boto:Data: 2023-10-12T01:11:44.667 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.668 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.668 DEBUG:boto:Params: {} 2023-10-12T01:11:44.669 DEBUG:boto:Token: None 2023-10-12T01:11:44.669 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.670 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY= 2023-10-12T01:11:44.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY=', 'Content-Length': '0'} 2023-10-12T01:11:44.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846aa3e258b258b1-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.674 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.674 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.675 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.675 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.675 DEBUG:boto:Method: GET 2023-10-12T01:11:44.676 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.676 DEBUG:boto:Data: 2023-10-12T01:11:44.677 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.678 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.678 DEBUG:boto:Params: {} 2023-10-12T01:11:44.679 DEBUG:boto:Token: None 2023-10-12T01:11:44.679 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY= 2023-10-12T01:11:44.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY=', 'Content-Length': '0'} 2023-10-12T01:11:44.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000695b29a01aab6eb-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.683 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.684 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.684 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.685 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.685 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.686 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.686 DEBUG:boto:Data: 2023-10-12T01:11:44.686 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.687 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.688 DEBUG:boto:Params: {} 2023-10-12T01:11:44.688 DEBUG:boto:Token: None 2023-10-12T01:11:44.689 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dWmz/LTNH444V/ZyZvZQDmrAX8M= 2023-10-12T01:11:44.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dWmz/LTNH444V/ZyZvZQDmrAX8M=', 'Content-Length': '0'} 2023-10-12T01:11:44.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082887707105d4e75-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.693 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.694 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.694 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.694 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.695 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.695 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.696 DEBUG:boto:Data: 2023-10-12T01:11:44.696 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.697 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.697 DEBUG:boto:Params: {} 2023-10-12T01:11:44.698 DEBUG:boto:Token: None 2023-10-12T01:11:44.698 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dWmz/LTNH444V/ZyZvZQDmrAX8M= 2023-10-12T01:11:44.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dWmz/LTNH444V/ZyZvZQDmrAX8M=', 'Content-Length': '0'} 2023-10-12T01:11:44.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003521d675ac8719e8-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:11:44.703 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.703 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.703 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.704 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.704 DEBUG:boto:Method: GET 2023-10-12T01:11:44.705 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.705 DEBUG:boto:Data: 2023-10-12T01:11:44.706 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.707 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.707 DEBUG:boto:Params: {} 2023-10-12T01:11:44.708 DEBUG:boto:Token: None 2023-10-12T01:11:44.708 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY= 2023-10-12T01:11:44.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY=', 'Content-Length': '0'} 2023-10-12T01:11:44.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001498d190a769dc0b-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.712 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.713 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.713 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.713 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.714 DEBUG:boto:Method: GET 2023-10-12T01:11:44.714 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.715 DEBUG:boto:Data: 2023-10-12T01:11:44.715 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.716 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.717 DEBUG:boto:Params: {} 2023-10-12T01:11:44.717 DEBUG:boto:Token: None 2023-10-12T01:11:44.718 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:11:44.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY= 2023-10-12T01:11:44.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NOKjw7Hl0yx5Z5jxJxVqQ5PR8vY=', 'Content-Length': '0'} 2023-10-12T01:11:44.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c55cec3399d44bb-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.722 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.722 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.723 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.723 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.723 DEBUG:boto:Method: GET 2023-10-12T01:11:44.724 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.724 DEBUG:boto:Data: 2023-10-12T01:11:44.725 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.726 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.726 DEBUG:boto:Params: {} 2023-10-12T01:11:44.727 DEBUG:boto:Token: None 2023-10-12T01:11:44.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f05aa86fde69c90d-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.731 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.732 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.733 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.733 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.734 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.734 DEBUG:boto:Method: GET 2023-10-12T01:11:44.735 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.739 DEBUG:boto:Data: 2023-10-12T01:11:44.739 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.740 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.741 DEBUG:boto:Params: {} 2023-10-12T01:11:44.741 DEBUG:boto:Token: None 2023-10-12T01:11:44.742 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.742 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000739f34a3e3f7a73-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.748 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.748 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.749 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.749 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.750 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:11:44.750 DEBUG:boto:Data: 2023-10-12T01:11:44.751 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.752 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.752 DEBUG:boto:Params: {} 2023-10-12T01:11:44.752 DEBUG:boto:Token: None 2023-10-12T01:11:44.753 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38 2023-10-12T01:11:44.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hgSYHWi+gPDsMsBt0oYmb2Gs4RM= 2023-10-12T01:11:44.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hgSYHWi+gPDsMsBt0oYmb2Gs4RM=', 'Content-Length': '0'} 2023-10-12T01:11:44.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d5a002a2a1a3bf4-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.757 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.758 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.758 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.758 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:11:44.759 DEBUG:boto:Data: 2023-10-12T01:11:44.759 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.760 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.760 DEBUG:boto:Params: {} 2023-10-12T01:11:44.761 DEBUG:boto:Token: None 2023-10-12T01:11:44.761 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38 2023-10-12T01:11:44.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hgSYHWi+gPDsMsBt0oYmb2Gs4RM= 2023-10-12T01:11:44.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hgSYHWi+gPDsMsBt0oYmb2Gs4RM=', 'Content-Length': '0'} 2023-10-12T01:11:44.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc7334996d62d22-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:11:44.766 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.767 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.767 DEBUG:boto:Method: GET 2023-10-12T01:11:44.767 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:11:44.768 DEBUG:boto:Data: 2023-10-12T01:11:44.768 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.769 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.770 DEBUG:boto:Params: {} 2023-10-12T01:11:44.770 DEBUG:boto:Token: None 2023-10-12T01:11:44.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38 2023-10-12T01:11:44.771 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jyeO29YHsOnJbuh24E4nuGZ3odM= 2023-10-12T01:11:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jyeO29YHsOnJbuh24E4nuGZ3odM=', 'Content-Length': '0'} 2023-10-12T01:11:44.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaef6e3deea35b0a-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.775 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:11:44.775 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:11:44.776 DEBUG:boto:Method: GET 2023-10-12T01:11:44.776 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:11:44.776 DEBUG:boto:Data: 2023-10-12T01:11:44.777 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.778 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.778 DEBUG:boto:Params: {} 2023-10-12T01:11:44.779 DEBUG:boto:Token: None 2023-10-12T01:11:44.779 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj38 2023-10-12T01:11:44.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jyeO29YHsOnJbuh24E4nuGZ3odM= 2023-10-12T01:11:44.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jyeO29YHsOnJbuh24E4nuGZ3odM=', 'Content-Length': '0'} 2023-10-12T01:11:44.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e531459c782d4a-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.827 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.828 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.829 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.829 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.830 DEBUG:boto:Method: GET 2023-10-12T01:11:44.831 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.832 DEBUG:boto:Data: 2023-10-12T01:11:44.833 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.834 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.835 DEBUG:boto:Params: {} 2023-10-12T01:11:44.836 DEBUG:boto:Token: None 2023-10-12T01:11:44.837 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a889d5e54069ebf-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.842 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.843 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.843 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.844 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.844 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.845 DEBUG:boto:Method: GET 2023-10-12T01:11:44.845 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:11:44.846 DEBUG:boto:Data: 2023-10-12T01:11:44.846 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.847 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.848 DEBUG:boto:Params: {} 2023-10-12T01:11:44.848 DEBUG:boto:Token: None 2023-10-12T01:11:44.849 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b760e47216eb283d-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.855 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:11:44.856 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.856 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.857 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.857 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.857 DEBUG:boto:Method: GET 2023-10-12T01:11:44.858 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.858 DEBUG:boto:Data: 2023-10-12T01:11:44.859 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.859 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.860 DEBUG:boto:Params: {} 2023-10-12T01:11:44.860 DEBUG:boto:Token: None 2023-10-12T01:11:44.861 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8= 2023-10-12T01:11:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8=', 'Content-Length': '0'} 2023-10-12T01:11:44.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80be96a8be66661-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.864 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.865 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.865 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.866 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.866 DEBUG:boto:Method: GET 2023-10-12T01:11:44.867 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.867 DEBUG:boto:Data: 2023-10-12T01:11:44.867 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.868 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.869 DEBUG:boto:Params: {} 2023-10-12T01:11:44.869 DEBUG:boto:Token: None 2023-10-12T01:11:44.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.870 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8= 2023-10-12T01:11:44.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8=', 'Content-Length': '0'} 2023-10-12T01:11:44.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a67cc555fcec9f-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.874 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.874 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.875 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.875 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.875 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.876 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.876 DEBUG:boto:Data: 2023-10-12T01:11:44.877 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.877 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.878 DEBUG:boto:Params: {} 2023-10-12T01:11:44.878 DEBUG:boto:Token: None 2023-10-12T01:11:44.879 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X9xm6jQDVdLYy8lAjQwvF9SEeHE= 2023-10-12T01:11:44.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X9xm6jQDVdLYy8lAjQwvF9SEeHE=', 'Content-Length': '0'} 2023-10-12T01:11:44.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaaa4d1f7ab425da-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.882 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.882 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.883 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.883 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.884 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.884 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.884 DEBUG:boto:Data: 2023-10-12T01:11:44.885 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.885 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.886 DEBUG:boto:Params: {} 2023-10-12T01:11:44.886 DEBUG:boto:Token: None 2023-10-12T01:11:44.887 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X9xm6jQDVdLYy8lAjQwvF9SEeHE= 2023-10-12T01:11:44.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X9xm6jQDVdLYy8lAjQwvF9SEeHE=', 'Content-Length': '0'} 2023-10-12T01:11:44.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42bf57a3355bcf9-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:11:44.890 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.891 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.891 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.892 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.892 DEBUG:boto:Method: GET 2023-10-12T01:11:44.893 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.893 DEBUG:boto:Data: 2023-10-12T01:11:44.893 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.894 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.895 DEBUG:boto:Params: {} 2023-10-12T01:11:44.895 DEBUG:boto:Token: None 2023-10-12T01:11:44.896 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8= 2023-10-12T01:11:44.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8=', 'Content-Length': '0'} 2023-10-12T01:11:44.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35cf163c8666604-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.899 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.900 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.900 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.901 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.901 DEBUG:boto:Method: GET 2023-10-12T01:11:44.902 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.902 DEBUG:boto:Data: 2023-10-12T01:11:44.903 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.903 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.904 DEBUG:boto:Params: {} 2023-10-12T01:11:44.904 DEBUG:boto:Token: None 2023-10-12T01:11:44.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:11:44.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8= 2023-10-12T01:11:44.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJJxxJwDHIGHH2m3TkNgHMDvjw8=', 'Content-Length': '0'} 2023-10-12T01:11:44.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5e62ebbde87294-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.908 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.909 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.909 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.910 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.910 DEBUG:boto:Method: GET 2023-10-12T01:11:44.910 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.911 DEBUG:boto:Data: 2023-10-12T01:11:44.911 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.912 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.912 DEBUG:boto:Params: {} 2023-10-12T01:11:44.913 DEBUG:boto:Token: None 2023-10-12T01:11:44.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e576153340fef73e-00652747d0-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.958 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.959 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:44.960 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:44.960 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.961 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.961 DEBUG:boto:Method: GET 2023-10-12T01:11:44.961 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:11:44.962 DEBUG:boto:Data: 2023-10-12T01:11:44.962 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.963 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.964 DEBUG:boto:Params: {} 2023-10-12T01:11:44.964 DEBUG:boto:Token: None 2023-10-12T01:11:44.964 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/ 2023-10-12T01:11:44.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc= 2023-10-12T01:11:44.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RfAF7z1U+NgBSxL6Sr55W8r2lgc=', 'Content-Length': '0'} 2023-10-12T01:11:44.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df8f6f4f056f5149-00652747d0-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.970 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:44.974 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.974 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.974 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.975 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:11:44.975 DEBUG:boto:Data: 2023-10-12T01:11:44.976 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.977 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.977 DEBUG:boto:Params: {} 2023-10-12T01:11:44.977 DEBUG:boto:Token: None 2023-10-12T01:11:44.978 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44 2023-10-12T01:11:44.978 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z/Ywf8hUqDEKByt3oQSVtO3dFAU= 2023-10-12T01:11:44.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z/Ywf8hUqDEKByt3oQSVtO3dFAU=', 'Content-Length': '0'} 2023-10-12T01:11:44.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ca98bdeebd2e26-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.981 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.982 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.982 DEBUG:boto:Method: HEAD 2023-10-12T01:11:44.983 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:11:44.983 DEBUG:boto:Data: 2023-10-12T01:11:44.984 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:44.984 DEBUG:boto:Port: 8000 2023-10-12T01:11:44.985 DEBUG:boto:Params: {} 2023-10-12T01:11:44.985 DEBUG:boto:Token: None 2023-10-12T01:11:44.986 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44 2023-10-12T01:11:44.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z/Ywf8hUqDEKByt3oQSVtO3dFAU= 2023-10-12T01:11:44.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z/Ywf8hUqDEKByt3oQSVtO3dFAU=', 'Content-Length': '0'} 2023-10-12T01:11:44.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000583f844e8311cb4e-00652747d0-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:11:44.990 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.990 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.991 DEBUG:boto:Method: GET 2023-10-12T01:11:44.991 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:11:44.992 DEBUG:boto:Data: 2023-10-12T01:11:44.992 DEBUG:boto:Headers: {} 2023-10-12T01:11:44.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:44.993 DEBUG:boto:Port: 8001 2023-10-12T01:11:44.993 DEBUG:boto:Params: {} 2023-10-12T01:11:44.994 DEBUG:boto:Token: None 2023-10-12T01:11:44.994 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:44 GMT /wafmoa-59/obj44 2023-10-12T01:11:44.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CRgoLGvC4ZCqF5KZzjvDs5ubp0w= 2023-10-12T01:11:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CRgoLGvC4ZCqF5KZzjvDs5ubp0w=', 'Content-Length': '0'} 2023-10-12T01:11:44.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006064b22ae448eae6-00652747d0-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:44.998 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:11:44.999 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:11:44.999 DEBUG:boto:Method: GET 2023-10-12T01:11:44.999 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:11:45.000 DEBUG:boto:Data: 2023-10-12T01:11:45.000 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.001 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.002 DEBUG:boto:Params: {} 2023-10-12T01:11:45.002 DEBUG:boto:Token: None 2023-10-12T01:11:45.003 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj44 2023-10-12T01:11:45.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9lY7O4MLsYZikizkbEh/ZAorJNQ= 2023-10-12T01:11:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9lY7O4MLsYZikizkbEh/ZAorJNQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8c82af117d92b83-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.007 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.007 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.007 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.008 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.008 DEBUG:boto:Method: GET 2023-10-12T01:11:45.009 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.009 DEBUG:boto:Data: 2023-10-12T01:11:45.010 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.011 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.011 DEBUG:boto:Params: {} 2023-10-12T01:11:45.012 DEBUG:boto:Token: None 2023-10-12T01:11:45.012 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161765f4dbf03021-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.016 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.017 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.017 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.018 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.018 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.019 DEBUG:boto:Method: GET 2023-10-12T01:11:45.019 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:11:45.020 DEBUG:boto:Data: 2023-10-12T01:11:45.020 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.021 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.022 DEBUG:boto:Params: {} 2023-10-12T01:11:45.022 DEBUG:boto:Token: None 2023-10-12T01:11:45.023 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7498d987ebbb6f5-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.028 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:11:45.029 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.030 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.030 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.031 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.031 DEBUG:boto:Method: GET 2023-10-12T01:11:45.032 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.032 DEBUG:boto:Data: 2023-10-12T01:11:45.033 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.034 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.034 DEBUG:boto:Params: {} 2023-10-12T01:11:45.035 DEBUG:boto:Token: None 2023-10-12T01:11:45.035 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q= 2023-10-12T01:11:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b95c6217daeb7b7-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.039 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.040 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.040 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.041 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.041 DEBUG:boto:Method: GET 2023-10-12T01:11:45.042 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.042 DEBUG:boto:Data: 2023-10-12T01:11:45.043 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.044 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.044 DEBUG:boto:Params: {} 2023-10-12T01:11:45.045 DEBUG:boto:Token: None 2023-10-12T01:11:45.045 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q= 2023-10-12T01:11:45.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4964b24fbfda68-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.049 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.049 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.050 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.050 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.050 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.051 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.051 DEBUG:boto:Data: 2023-10-12T01:11:45.052 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.052 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.053 DEBUG:boto:Params: {} 2023-10-12T01:11:45.053 DEBUG:boto:Token: None 2023-10-12T01:11:45.054 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.054 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M6HVQpfwKdReRaeKVWTupssdE5A= 2023-10-12T01:11:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M6HVQpfwKdReRaeKVWTupssdE5A=', 'Content-Length': '0'} 2023-10-12T01:11:45.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab6313a73db8775-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.058 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.058 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.059 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.059 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.060 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.060 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.060 DEBUG:boto:Data: 2023-10-12T01:11:45.061 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.062 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.062 DEBUG:boto:Params: {} 2023-10-12T01:11:45.063 DEBUG:boto:Token: None 2023-10-12T01:11:45.063 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M6HVQpfwKdReRaeKVWTupssdE5A= 2023-10-12T01:11:45.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M6HVQpfwKdReRaeKVWTupssdE5A=', 'Content-Length': '0'} 2023-10-12T01:11:45.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaaada6043260297-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:11:45.067 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.068 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.068 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.069 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.069 DEBUG:boto:Method: GET 2023-10-12T01:11:45.070 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.070 DEBUG:boto:Data: 2023-10-12T01:11:45.070 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.071 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.072 DEBUG:boto:Params: {} 2023-10-12T01:11:45.072 DEBUG:boto:Token: None 2023-10-12T01:11:45.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q= 2023-10-12T01:11:45.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011deba022393f0b-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.118 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.119 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.120 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.120 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.121 DEBUG:boto:Method: GET 2023-10-12T01:11:45.121 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.122 DEBUG:boto:Data: 2023-10-12T01:11:45.122 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.123 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.124 DEBUG:boto:Params: {} 2023-10-12T01:11:45.124 DEBUG:boto:Token: None 2023-10-12T01:11:45.125 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:11:45.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q= 2023-10-12T01:11:45.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VhvVB/oTRiqUqg3+UMbsX8pG59Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f75f24a53498fc-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.129 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.130 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.130 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.130 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.131 DEBUG:boto:Method: GET 2023-10-12T01:11:45.131 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.132 DEBUG:boto:Data: 2023-10-12T01:11:45.132 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.133 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.134 DEBUG:boto:Params: {} 2023-10-12T01:11:45.134 DEBUG:boto:Token: None 2023-10-12T01:11:45.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a4f2261f982a47-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.139 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.140 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.140 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.141 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.141 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.142 DEBUG:boto:Method: GET 2023-10-12T01:11:45.142 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.143 DEBUG:boto:Data: 2023-10-12T01:11:45.143 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.144 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.144 DEBUG:boto:Params: {} 2023-10-12T01:11:45.145 DEBUG:boto:Token: None 2023-10-12T01:11:45.145 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049417cdf2fd58732-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.152 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.152 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.153 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.153 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.154 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:11:45.154 DEBUG:boto:Data: 2023-10-12T01:11:45.155 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.156 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.156 DEBUG:boto:Params: {} 2023-10-12T01:11:45.157 DEBUG:boto:Token: None 2023-10-12T01:11:45.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46 2023-10-12T01:11:45.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CUWN1qjLbW7N/0zvfM5VcH2gyc4= 2023-10-12T01:11:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CUWN1qjLbW7N/0zvfM5VcH2gyc4=', 'Content-Length': '0'} 2023-10-12T01:11:45.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf590847df261c9-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.161 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.162 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.162 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.163 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:11:45.163 DEBUG:boto:Data: 2023-10-12T01:11:45.164 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.165 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.165 DEBUG:boto:Params: {} 2023-10-12T01:11:45.166 DEBUG:boto:Token: None 2023-10-12T01:11:45.166 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46 2023-10-12T01:11:45.167 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CUWN1qjLbW7N/0zvfM5VcH2gyc4= 2023-10-12T01:11:45.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CUWN1qjLbW7N/0zvfM5VcH2gyc4=', 'Content-Length': '0'} 2023-10-12T01:11:45.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dd9377768681e2-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:11:45.170 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.171 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.171 DEBUG:boto:Method: GET 2023-10-12T01:11:45.172 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:11:45.172 DEBUG:boto:Data: 2023-10-12T01:11:45.173 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.173 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.174 DEBUG:boto:Params: {} 2023-10-12T01:11:45.174 DEBUG:boto:Token: None 2023-10-12T01:11:45.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46 2023-10-12T01:11:45.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kG+JYBtBqGCKHhIAAjSd29MOLwY= 2023-10-12T01:11:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kG+JYBtBqGCKHhIAAjSd29MOLwY=', 'Content-Length': '0'} 2023-10-12T01:11:45.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a28c4f9ac959d38-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.179 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:11:45.179 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:11:45.180 DEBUG:boto:Method: GET 2023-10-12T01:11:45.180 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:11:45.180 DEBUG:boto:Data: 2023-10-12T01:11:45.181 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.182 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.182 DEBUG:boto:Params: {} 2023-10-12T01:11:45.183 DEBUG:boto:Token: None 2023-10-12T01:11:45.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj46 2023-10-12T01:11:45.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kG+JYBtBqGCKHhIAAjSd29MOLwY= 2023-10-12T01:11:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kG+JYBtBqGCKHhIAAjSd29MOLwY=', 'Content-Length': '0'} 2023-10-12T01:11:45.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a520fe8d9e0a74-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.230 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.231 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.231 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.232 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.232 DEBUG:boto:Method: GET 2023-10-12T01:11:45.232 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.233 DEBUG:boto:Data: 2023-10-12T01:11:45.233 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.234 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.235 DEBUG:boto:Params: {} 2023-10-12T01:11:45.235 DEBUG:boto:Token: None 2023-10-12T01:11:45.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d58231df572b3499-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.240 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.241 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.241 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.242 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.242 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.243 DEBUG:boto:Method: GET 2023-10-12T01:11:45.243 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:11:45.243 DEBUG:boto:Data: 2023-10-12T01:11:45.244 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.245 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.245 DEBUG:boto:Params: {} 2023-10-12T01:11:45.246 DEBUG:boto:Token: None 2023-10-12T01:11:45.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deeb5f12e1b3a9a3-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.251 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:11:45.253 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.253 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.254 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.254 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.254 DEBUG:boto:Method: GET 2023-10-12T01:11:45.255 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.255 DEBUG:boto:Data: 2023-10-12T01:11:45.256 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.257 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.257 DEBUG:boto:Params: {} 2023-10-12T01:11:45.258 DEBUG:boto:Token: None 2023-10-12T01:11:45.258 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A= 2023-10-12T01:11:45.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A=', 'Content-Length': '0'} 2023-10-12T01:11:45.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013338b8ded505664-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.306 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.307 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.308 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.309 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.310 DEBUG:boto:Method: GET 2023-10-12T01:11:45.311 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.312 DEBUG:boto:Data: 2023-10-12T01:11:45.313 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.314 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.314 DEBUG:boto:Params: {} 2023-10-12T01:11:45.315 DEBUG:boto:Token: None 2023-10-12T01:11:45.315 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A= 2023-10-12T01:11:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A=', 'Content-Length': '0'} 2023-10-12T01:11:45.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe4abc73c043450-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.320 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.320 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.321 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.321 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.321 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.322 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.322 DEBUG:boto:Data: 2023-10-12T01:11:45.323 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.323 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.324 DEBUG:boto:Params: {} 2023-10-12T01:11:45.324 DEBUG:boto:Token: None 2023-10-12T01:11:45.325 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pAh1fYerLCQs1OIPuxYuulUNd4Q= 2023-10-12T01:11:45.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pAh1fYerLCQs1OIPuxYuulUNd4Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789b01862e980855-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.328 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.329 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.329 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.330 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.330 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.330 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.331 DEBUG:boto:Data: 2023-10-12T01:11:45.331 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.332 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.332 DEBUG:boto:Params: {} 2023-10-12T01:11:45.333 DEBUG:boto:Token: None 2023-10-12T01:11:45.333 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pAh1fYerLCQs1OIPuxYuulUNd4Q= 2023-10-12T01:11:45.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pAh1fYerLCQs1OIPuxYuulUNd4Q=', 'Content-Length': '0'} 2023-10-12T01:11:45.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952134ca199f04e6-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:11:45.337 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.338 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.338 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.339 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.339 DEBUG:boto:Method: GET 2023-10-12T01:11:45.340 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.340 DEBUG:boto:Data: 2023-10-12T01:11:45.340 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.341 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.342 DEBUG:boto:Params: {} 2023-10-12T01:11:45.342 DEBUG:boto:Token: None 2023-10-12T01:11:45.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A= 2023-10-12T01:11:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A=', 'Content-Length': '0'} 2023-10-12T01:11:45.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000012bc386bbb49b3a-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.346 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.347 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.347 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.348 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.348 DEBUG:boto:Method: GET 2023-10-12T01:11:45.349 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.349 DEBUG:boto:Data: 2023-10-12T01:11:45.349 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.350 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.351 DEBUG:boto:Params: {} 2023-10-12T01:11:45.351 DEBUG:boto:Token: None 2023-10-12T01:11:45.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:11:45.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A= 2023-10-12T01:11:45.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v/+kLzlwLWpFaEByjABuMFqtI+A=', 'Content-Length': '0'} 2023-10-12T01:11:45.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c167a9cff75bc2-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.356 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.356 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.357 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.357 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.357 DEBUG:boto:Method: GET 2023-10-12T01:11:45.358 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.358 DEBUG:boto:Data: 2023-10-12T01:11:45.359 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.360 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.360 DEBUG:boto:Params: {} 2023-10-12T01:11:45.361 DEBUG:boto:Token: None 2023-10-12T01:11:45.361 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.362 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b8b673b4c5ebd1-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.365 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.366 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.367 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.367 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.367 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.368 DEBUG:boto:Method: GET 2023-10-12T01:11:45.368 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.369 DEBUG:boto:Data: 2023-10-12T01:11:45.369 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.370 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.371 DEBUG:boto:Params: {} 2023-10-12T01:11:45.371 DEBUG:boto:Token: None 2023-10-12T01:11:45.372 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.372 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ef00257a293c4c3-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.422 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.423 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.424 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.424 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.424 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:11:45.425 DEBUG:boto:Data: 2023-10-12T01:11:45.425 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.426 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.427 DEBUG:boto:Params: {} 2023-10-12T01:11:45.427 DEBUG:boto:Token: None 2023-10-12T01:11:45.428 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51 2023-10-12T01:11:45.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w9eZ4RzrcejBqMhqqfoucvp7FHc= 2023-10-12T01:11:45.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w9eZ4RzrcejBqMhqqfoucvp7FHc=', 'Content-Length': '0'} 2023-10-12T01:11:45.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6e97952d8bc0cb-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.431 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.432 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.432 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.432 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:11:45.433 DEBUG:boto:Data: 2023-10-12T01:11:45.433 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.434 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.435 DEBUG:boto:Params: {} 2023-10-12T01:11:45.435 DEBUG:boto:Token: None 2023-10-12T01:11:45.436 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51 2023-10-12T01:11:45.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w9eZ4RzrcejBqMhqqfoucvp7FHc= 2023-10-12T01:11:45.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w9eZ4RzrcejBqMhqqfoucvp7FHc=', 'Content-Length': '0'} 2023-10-12T01:11:45.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349b42eda8219a68-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:11:45.439 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.440 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.440 DEBUG:boto:Method: GET 2023-10-12T01:11:45.441 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:11:45.441 DEBUG:boto:Data: 2023-10-12T01:11:45.442 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.443 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.443 DEBUG:boto:Params: {} 2023-10-12T01:11:45.444 DEBUG:boto:Token: None 2023-10-12T01:11:45.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51 2023-10-12T01:11:45.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w/D6nFXh8Jpg27zvL3FDHkI6G08= 2023-10-12T01:11:45.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w/D6nFXh8Jpg27zvL3FDHkI6G08=', 'Content-Length': '0'} 2023-10-12T01:11:45.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6a6d6eb8c5e96e-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.447 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:11:45.448 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:11:45.448 DEBUG:boto:Method: GET 2023-10-12T01:11:45.449 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:11:45.449 DEBUG:boto:Data: 2023-10-12T01:11:45.450 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.451 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.451 DEBUG:boto:Params: {} 2023-10-12T01:11:45.452 DEBUG:boto:Token: None 2023-10-12T01:11:45.452 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj51 2023-10-12T01:11:45.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w/D6nFXh8Jpg27zvL3FDHkI6G08= 2023-10-12T01:11:45.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w/D6nFXh8Jpg27zvL3FDHkI6G08=', 'Content-Length': '0'} 2023-10-12T01:11:45.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008698d5741d6a6522-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.456 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.456 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.457 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.457 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.458 DEBUG:boto:Method: GET 2023-10-12T01:11:45.458 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.458 DEBUG:boto:Data: 2023-10-12T01:11:45.459 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.460 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.460 DEBUG:boto:Params: {} 2023-10-12T01:11:45.461 DEBUG:boto:Token: None 2023-10-12T01:11:45.461 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.462 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b91e6ad0138891e-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.510 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.512 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.512 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.513 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.513 DEBUG:boto:Method: GET 2023-10-12T01:11:45.514 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:11:45.514 DEBUG:boto:Data: 2023-10-12T01:11:45.515 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.515 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.516 DEBUG:boto:Params: {} 2023-10-12T01:11:45.516 DEBUG:boto:Token: None 2023-10-12T01:11:45.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bd38db9ccd2715-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.522 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:11:45.524 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.524 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.525 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.525 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.525 DEBUG:boto:Method: GET 2023-10-12T01:11:45.526 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.526 DEBUG:boto:Data: 2023-10-12T01:11:45.527 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.528 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.528 DEBUG:boto:Params: {} 2023-10-12T01:11:45.529 DEBUG:boto:Token: None 2023-10-12T01:11:45.529 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY= 2023-10-12T01:11:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY=', 'Content-Length': '0'} 2023-10-12T01:11:45.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93f4dbb71173c7f-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.533 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.533 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.533 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.534 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.534 DEBUG:boto:Method: GET 2023-10-12T01:11:45.535 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.535 DEBUG:boto:Data: 2023-10-12T01:11:45.536 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.537 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.537 DEBUG:boto:Params: {} 2023-10-12T01:11:45.538 DEBUG:boto:Token: None 2023-10-12T01:11:45.538 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY= 2023-10-12T01:11:45.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY=', 'Content-Length': '0'} 2023-10-12T01:11:45.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f9f9801aef4839-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.542 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.542 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.543 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.543 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.543 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.544 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.544 DEBUG:boto:Data: 2023-10-12T01:11:45.545 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.545 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.546 DEBUG:boto:Params: {} 2023-10-12T01:11:45.546 DEBUG:boto:Token: None 2023-10-12T01:11:45.547 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5L6FCcBX0vAUbrJdytcEMSxxAyk= 2023-10-12T01:11:45.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5L6FCcBX0vAUbrJdytcEMSxxAyk=', 'Content-Length': '0'} 2023-10-12T01:11:45.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4516893c834b0e-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.550 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.551 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.551 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.551 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.552 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.552 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.553 DEBUG:boto:Data: 2023-10-12T01:11:45.553 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.554 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.554 DEBUG:boto:Params: {} 2023-10-12T01:11:45.555 DEBUG:boto:Token: None 2023-10-12T01:11:45.555 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5L6FCcBX0vAUbrJdytcEMSxxAyk= 2023-10-12T01:11:45.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5L6FCcBX0vAUbrJdytcEMSxxAyk=', 'Content-Length': '0'} 2023-10-12T01:11:45.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008522367cfdc80b3c-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:11:45.559 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.560 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.560 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.561 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.561 DEBUG:boto:Method: GET 2023-10-12T01:11:45.562 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.562 DEBUG:boto:Data: 2023-10-12T01:11:45.563 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.563 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.564 DEBUG:boto:Params: {} 2023-10-12T01:11:45.564 DEBUG:boto:Token: None 2023-10-12T01:11:45.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.565 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY= 2023-10-12T01:11:45.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY=', 'Content-Length': '0'} 2023-10-12T01:11:45.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f4509c29e06d64-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.568 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.569 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.569 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.570 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.570 DEBUG:boto:Method: GET 2023-10-12T01:11:45.571 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.571 DEBUG:boto:Data: 2023-10-12T01:11:45.572 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.572 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.573 DEBUG:boto:Params: {} 2023-10-12T01:11:45.573 DEBUG:boto:Token: None 2023-10-12T01:11:45.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:11:45.574 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY= 2023-10-12T01:11:45.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r+EXs8JTCgDKyjv26YbKRoTW8dY=', 'Content-Length': '0'} 2023-10-12T01:11:45.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6547f2cf18498d0-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.578 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.578 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.579 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.579 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.579 DEBUG:boto:Method: GET 2023-10-12T01:11:45.580 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.580 DEBUG:boto:Data: 2023-10-12T01:11:45.581 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.582 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.582 DEBUG:boto:Params: {} 2023-10-12T01:11:45.585 DEBUG:boto:Token: None 2023-10-12T01:11:45.586 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002185492664f63ca9-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.590 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.591 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.591 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.592 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.592 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.592 DEBUG:boto:Method: GET 2023-10-12T01:11:45.593 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.593 DEBUG:boto:Data: 2023-10-12T01:11:45.594 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.595 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.595 DEBUG:boto:Params: {} 2023-10-12T01:11:45.596 DEBUG:boto:Token: None 2023-10-12T01:11:45.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c95f7f4d13c2924d-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.601 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.602 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.603 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.603 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.603 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:11:45.604 DEBUG:boto:Data: 2023-10-12T01:11:45.604 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.605 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.605 DEBUG:boto:Params: {} 2023-10-12T01:11:45.606 DEBUG:boto:Token: None 2023-10-12T01:11:45.606 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55 2023-10-12T01:11:45.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I68B2EgQCpYeS6RPWI+mE03TOIE= 2023-10-12T01:11:45.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I68B2EgQCpYeS6RPWI+mE03TOIE=', 'Content-Length': '0'} 2023-10-12T01:11:45.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db79917526969e14-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.610 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.610 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.611 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.611 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:11:45.611 DEBUG:boto:Data: 2023-10-12T01:11:45.612 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.613 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.613 DEBUG:boto:Params: {} 2023-10-12T01:11:45.614 DEBUG:boto:Token: None 2023-10-12T01:11:45.614 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55 2023-10-12T01:11:45.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I68B2EgQCpYeS6RPWI+mE03TOIE= 2023-10-12T01:11:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I68B2EgQCpYeS6RPWI+mE03TOIE=', 'Content-Length': '0'} 2023-10-12T01:11:45.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fc4279a8ef8260a-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:11:45.618 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.619 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.619 DEBUG:boto:Method: GET 2023-10-12T01:11:45.620 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:11:45.620 DEBUG:boto:Data: 2023-10-12T01:11:45.621 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.621 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.622 DEBUG:boto:Params: {} 2023-10-12T01:11:45.622 DEBUG:boto:Token: None 2023-10-12T01:11:45.623 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55 2023-10-12T01:11:45.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IH+rclsqlCqoO5+mSXNmtTyhsbc= 2023-10-12T01:11:45.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IH+rclsqlCqoO5+mSXNmtTyhsbc=', 'Content-Length': '0'} 2023-10-12T01:11:45.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762bd96d98f1a39b-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.627 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:11:45.627 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:11:45.628 DEBUG:boto:Method: GET 2023-10-12T01:11:45.628 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:11:45.629 DEBUG:boto:Data: 2023-10-12T01:11:45.629 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.630 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.631 DEBUG:boto:Params: {} 2023-10-12T01:11:45.631 DEBUG:boto:Token: None 2023-10-12T01:11:45.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj55 2023-10-12T01:11:45.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IH+rclsqlCqoO5+mSXNmtTyhsbc= 2023-10-12T01:11:45.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IH+rclsqlCqoO5+mSXNmtTyhsbc=', 'Content-Length': '0'} 2023-10-12T01:11:45.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008266b7fac1d8af35-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.678 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.679 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.679 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.680 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.680 DEBUG:boto:Method: GET 2023-10-12T01:11:45.681 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.681 DEBUG:boto:Data: 2023-10-12T01:11:45.682 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.683 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.683 DEBUG:boto:Params: {} 2023-10-12T01:11:45.685 DEBUG:boto:Token: None 2023-10-12T01:11:45.685 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bfeef118c22195-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.689 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.690 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.691 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.691 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.692 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.692 DEBUG:boto:Method: GET 2023-10-12T01:11:45.693 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:11:45.693 DEBUG:boto:Data: 2023-10-12T01:11:45.694 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.695 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.695 DEBUG:boto:Params: {} 2023-10-12T01:11:45.696 DEBUG:boto:Token: None 2023-10-12T01:11:45.696 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001997e2669ec282ad-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.701 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:11:45.703 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.703 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.704 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.704 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.705 DEBUG:boto:Method: GET 2023-10-12T01:11:45.705 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.705 DEBUG:boto:Data: 2023-10-12T01:11:45.706 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.707 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.707 DEBUG:boto:Params: {} 2023-10-12T01:11:45.708 DEBUG:boto:Token: None 2023-10-12T01:11:45.708 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc= 2023-10-12T01:11:45.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc=', 'Content-Length': '0'} 2023-10-12T01:11:45.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd772799fdd4e97-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.712 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.712 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.713 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.713 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.714 DEBUG:boto:Method: GET 2023-10-12T01:11:45.714 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.715 DEBUG:boto:Data: 2023-10-12T01:11:45.715 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.716 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.716 DEBUG:boto:Params: {} 2023-10-12T01:11:45.717 DEBUG:boto:Token: None 2023-10-12T01:11:45.717 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc= 2023-10-12T01:11:45.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc=', 'Content-Length': '0'} 2023-10-12T01:11:45.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487b4019056ea32b-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.721 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.722 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.722 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.722 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.723 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.723 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.724 DEBUG:boto:Data: 2023-10-12T01:11:45.724 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.725 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.725 DEBUG:boto:Params: {} 2023-10-12T01:11:45.726 DEBUG:boto:Token: None 2023-10-12T01:11:45.726 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfOIm4pLNYtnBmueoaA6Ax+M5Zg= 2023-10-12T01:11:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfOIm4pLNYtnBmueoaA6Ax+M5Zg=', 'Content-Length': '0'} 2023-10-12T01:11:45.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009624b2c5ae1964c9-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.729 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.730 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.730 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.731 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.731 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.731 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.732 DEBUG:boto:Data: 2023-10-12T01:11:45.732 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.733 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.734 DEBUG:boto:Params: {} 2023-10-12T01:11:45.734 DEBUG:boto:Token: None 2023-10-12T01:11:45.735 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.735 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AfOIm4pLNYtnBmueoaA6Ax+M5Zg= 2023-10-12T01:11:45.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AfOIm4pLNYtnBmueoaA6Ax+M5Zg=', 'Content-Length': '0'} 2023-10-12T01:11:45.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a07972728b19ab-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:11:45.739 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.739 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.739 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.740 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.740 DEBUG:boto:Method: GET 2023-10-12T01:11:45.741 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.741 DEBUG:boto:Data: 2023-10-12T01:11:45.741 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.742 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.743 DEBUG:boto:Params: {} 2023-10-12T01:11:45.743 DEBUG:boto:Token: None 2023-10-12T01:11:45.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc= 2023-10-12T01:11:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc=', 'Content-Length': '0'} 2023-10-12T01:11:45.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2b9e3708921b8a8-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.747 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.748 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.748 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.749 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.749 DEBUG:boto:Method: GET 2023-10-12T01:11:45.750 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.750 DEBUG:boto:Data: 2023-10-12T01:11:45.751 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.752 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.752 DEBUG:boto:Params: {} 2023-10-12T01:11:45.752 DEBUG:boto:Token: None 2023-10-12T01:11:45.753 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:11:45.753 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc= 2023-10-12T01:11:45.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFiJUIbH8dARkg4i1wu0Am7Xgwc=', 'Content-Length': '0'} 2023-10-12T01:11:45.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b0e35b317c569c-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.756 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.757 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.757 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.758 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.758 DEBUG:boto:Method: GET 2023-10-12T01:11:45.759 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.759 DEBUG:boto:Data: 2023-10-12T01:11:45.759 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.760 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.761 DEBUG:boto:Params: {} 2023-10-12T01:11:45.761 DEBUG:boto:Token: None 2023-10-12T01:11:45.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb74e0dd7ede2c97-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.806 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.807 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.808 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.808 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.809 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.809 DEBUG:boto:Method: GET 2023-10-12T01:11:45.810 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.810 DEBUG:boto:Data: 2023-10-12T01:11:45.810 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.811 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.812 DEBUG:boto:Params: {} 2023-10-12T01:11:45.812 DEBUG:boto:Token: None 2023-10-12T01:11:45.813 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056dc6bac66d1c50e-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.818 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.819 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.819 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.820 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.820 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:11:45.821 DEBUG:boto:Data: 2023-10-12T01:11:45.821 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.822 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.823 DEBUG:boto:Params: {} 2023-10-12T01:11:45.823 DEBUG:boto:Token: None 2023-10-12T01:11:45.823 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57 2023-10-12T01:11:45.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W9r3XPxyFIOn4+XYWdcD2suvtUo= 2023-10-12T01:11:45.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W9r3XPxyFIOn4+XYWdcD2suvtUo=', 'Content-Length': '0'} 2023-10-12T01:11:45.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e28287e8481904-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.827 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.827 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.827 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.828 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:11:45.828 DEBUG:boto:Data: 2023-10-12T01:11:45.829 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.829 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.830 DEBUG:boto:Params: {} 2023-10-12T01:11:45.830 DEBUG:boto:Token: None 2023-10-12T01:11:45.831 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57 2023-10-12T01:11:45.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W9r3XPxyFIOn4+XYWdcD2suvtUo= 2023-10-12T01:11:45.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W9r3XPxyFIOn4+XYWdcD2suvtUo=', 'Content-Length': '0'} 2023-10-12T01:11:45.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0e437d64a8d5814-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:11:45.835 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.835 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.836 DEBUG:boto:Method: GET 2023-10-12T01:11:45.836 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:11:45.837 DEBUG:boto:Data: 2023-10-12T01:11:45.837 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.838 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.839 DEBUG:boto:Params: {} 2023-10-12T01:11:45.839 DEBUG:boto:Token: None 2023-10-12T01:11:45.839 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57 2023-10-12T01:11:45.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PrXI498ZrXmVR90qiykhRKtOyw8= 2023-10-12T01:11:45.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PrXI498ZrXmVR90qiykhRKtOyw8=', 'Content-Length': '0'} 2023-10-12T01:11:45.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613bcd25b5452951-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.843 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:11:45.843 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:11:45.844 DEBUG:boto:Method: GET 2023-10-12T01:11:45.844 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:11:45.845 DEBUG:boto:Data: 2023-10-12T01:11:45.845 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.846 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.847 DEBUG:boto:Params: {} 2023-10-12T01:11:45.847 DEBUG:boto:Token: None 2023-10-12T01:11:45.848 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj57 2023-10-12T01:11:45.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PrXI498ZrXmVR90qiykhRKtOyw8= 2023-10-12T01:11:45.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PrXI498ZrXmVR90qiykhRKtOyw8=', 'Content-Length': '0'} 2023-10-12T01:11:45.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e11d397c86e10c88-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.851 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.851 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.852 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.852 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.853 DEBUG:boto:Method: GET 2023-10-12T01:11:45.853 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.854 DEBUG:boto:Data: 2023-10-12T01:11:45.854 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.855 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.855 DEBUG:boto:Params: {} 2023-10-12T01:11:45.856 DEBUG:boto:Token: None 2023-10-12T01:11:45.856 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007acd744821e67caa-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.861 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.861 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.862 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.862 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.863 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.863 DEBUG:boto:Method: GET 2023-10-12T01:11:45.864 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:11:45.864 DEBUG:boto:Data: 2023-10-12T01:11:45.864 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.865 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.866 DEBUG:boto:Params: {} 2023-10-12T01:11:45.866 DEBUG:boto:Token: None 2023-10-12T01:11:45.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc8f1158f767029-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.914 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:11:45.916 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.916 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.917 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.917 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.917 DEBUG:boto:Method: GET 2023-10-12T01:11:45.918 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.918 DEBUG:boto:Data: 2023-10-12T01:11:45.919 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.919 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.920 DEBUG:boto:Params: {} 2023-10-12T01:11:45.920 DEBUG:boto:Token: None 2023-10-12T01:11:45.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw= 2023-10-12T01:11:45.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw=', 'Content-Length': '0'} 2023-10-12T01:11:45.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfce26f10f87fa4e-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.925 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.925 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.926 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.926 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.927 DEBUG:boto:Method: GET 2023-10-12T01:11:45.927 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.928 DEBUG:boto:Data: 2023-10-12T01:11:45.928 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.929 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.929 DEBUG:boto:Params: {} 2023-10-12T01:11:45.930 DEBUG:boto:Token: None 2023-10-12T01:11:45.930 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.931 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw= 2023-10-12T01:11:45.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw=', 'Content-Length': '0'} 2023-10-12T01:11:45.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99baafd6f828510-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.934 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.934 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.935 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.935 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.936 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.936 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.936 DEBUG:boto:Data: 2023-10-12T01:11:45.937 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.938 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.938 DEBUG:boto:Params: {} 2023-10-12T01:11:45.938 DEBUG:boto:Token: None 2023-10-12T01:11:45.939 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VZwPfY4ywluynPuTHpk1D+NlpK0= 2023-10-12T01:11:45.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VZwPfY4ywluynPuTHpk1D+NlpK0=', 'Content-Length': '0'} 2023-10-12T01:11:45.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007586d8be79e5e35c-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.942 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.943 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.943 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.944 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.944 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.944 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.945 DEBUG:boto:Data: 2023-10-12T01:11:45.945 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.946 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.947 DEBUG:boto:Params: {} 2023-10-12T01:11:45.947 DEBUG:boto:Token: None 2023-10-12T01:11:45.947 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.948 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VZwPfY4ywluynPuTHpk1D+NlpK0= 2023-10-12T01:11:45.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VZwPfY4ywluynPuTHpk1D+NlpK0=', 'Content-Length': '0'} 2023-10-12T01:11:45.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5ccf1ca6d07592-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:11:45.951 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.952 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.952 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.952 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.953 DEBUG:boto:Method: GET 2023-10-12T01:11:45.953 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.954 DEBUG:boto:Data: 2023-10-12T01:11:45.954 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.955 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.956 DEBUG:boto:Params: {} 2023-10-12T01:11:45.956 DEBUG:boto:Token: None 2023-10-12T01:11:45.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw= 2023-10-12T01:11:45.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw=', 'Content-Length': '0'} 2023-10-12T01:11:45.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077639aa67aefaa55-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.961 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.961 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.961 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.962 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.962 DEBUG:boto:Method: GET 2023-10-12T01:11:45.963 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.963 DEBUG:boto:Data: 2023-10-12T01:11:45.963 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.964 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.965 DEBUG:boto:Params: {} 2023-10-12T01:11:45.965 DEBUG:boto:Token: None 2023-10-12T01:11:45.965 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:11:45.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw= 2023-10-12T01:11:45.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Yq/oSM/LTLaVx/6mGboPNTliXcw=', 'Content-Length': '0'} 2023-10-12T01:11:45.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016dc9f6a8159ed62-00652747d1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.969 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.970 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.970 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.971 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.971 DEBUG:boto:Method: GET 2023-10-12T01:11:45.972 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.972 DEBUG:boto:Data: 2023-10-12T01:11:45.973 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.973 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.974 DEBUG:boto:Params: {} 2023-10-12T01:11:45.974 DEBUG:boto:Token: None 2023-10-12T01:11:45.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000321537de55b9b87c-00652747d1-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.979 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.980 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:45.980 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:45.981 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.981 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.982 DEBUG:boto:Method: GET 2023-10-12T01:11:45.982 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:11:45.983 DEBUG:boto:Data: 2023-10-12T01:11:45.983 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:45.984 DEBUG:boto:Port: 8000 2023-10-12T01:11:45.984 DEBUG:boto:Params: {} 2023-10-12T01:11:45.985 DEBUG:boto:Token: None 2023-10-12T01:11:45.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/ 2023-10-12T01:11:45.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ= 2023-10-12T01:11:45.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CgRoUDRpshhbOmirr/94SCFctgQ=', 'Content-Length': '0'} 2023-10-12T01:11:45.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb587cce4f754115-00652747d1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.991 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:45.992 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.992 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:45.993 DEBUG:boto:Method: HEAD 2023-10-12T01:11:45.993 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:11:45.993 DEBUG:boto:Data: 2023-10-12T01:11:45.994 DEBUG:boto:Headers: {} 2023-10-12T01:11:45.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:45.995 DEBUG:boto:Port: 8001 2023-10-12T01:11:45.995 DEBUG:boto:Params: {} 2023-10-12T01:11:45.995 DEBUG:boto:Token: None 2023-10-12T01:11:45.996 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:45 GMT /wafmoa-59/obj58 2023-10-12T01:11:45.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IT6OiUYvu8+FwnbFkuErs8QNg5g= 2023-10-12T01:11:45.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IT6OiUYvu8+FwnbFkuErs8QNg5g=', 'Content-Length': '0'} 2023-10-12T01:11:45.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003686aeb82e57a60f-00652747d1-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:45.999 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:45.999 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:46.000 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.000 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:11:46.001 DEBUG:boto:Data: 2023-10-12T01:11:46.001 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.002 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.002 DEBUG:boto:Params: {} 2023-10-12T01:11:46.003 DEBUG:boto:Token: None 2023-10-12T01:11:46.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj58 2023-10-12T01:11:46.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lMP1/q9lmFpxsl3ZIIAZbSYrHuc= 2023-10-12T01:11:46.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lMP1/q9lmFpxsl3ZIIAZbSYrHuc=', 'Content-Length': '0'} 2023-10-12T01:11:46.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e3c9f6ae33261f-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:11:46.007 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:46.007 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:46.008 DEBUG:boto:Method: GET 2023-10-12T01:11:46.008 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:11:46.008 DEBUG:boto:Data: 2023-10-12T01:11:46.009 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.010 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.010 DEBUG:boto:Params: {} 2023-10-12T01:11:46.011 DEBUG:boto:Token: None 2023-10-12T01:11:46.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj58 2023-10-12T01:11:46.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sf1TY0f9g0dOBPX1pYCXmOD8KuQ= 2023-10-12T01:11:46.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sf1TY0f9g0dOBPX1pYCXmOD8KuQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f62e3874d3aeb6-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.054 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:11:46.055 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:11:46.055 DEBUG:boto:Method: GET 2023-10-12T01:11:46.056 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:11:46.056 DEBUG:boto:Data: 2023-10-12T01:11:46.057 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.057 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.058 DEBUG:boto:Params: {} 2023-10-12T01:11:46.059 DEBUG:boto:Token: None 2023-10-12T01:11:46.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj58 2023-10-12T01:11:46.061 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sf1TY0f9g0dOBPX1pYCXmOD8KuQ= 2023-10-12T01:11:46.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sf1TY0f9g0dOBPX1pYCXmOD8KuQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d4c0a48216a6cd-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.064 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.064 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.065 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.065 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.066 DEBUG:boto:Method: GET 2023-10-12T01:11:46.066 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.067 DEBUG:boto:Data: 2023-10-12T01:11:46.067 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.068 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.068 DEBUG:boto:Params: {} 2023-10-12T01:11:46.069 DEBUG:boto:Token: None 2023-10-12T01:11:46.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ddba042c47f5f1d-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.074 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.074 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.075 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.075 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.076 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.076 DEBUG:boto:Method: GET 2023-10-12T01:11:46.077 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:11:46.077 DEBUG:boto:Data: 2023-10-12T01:11:46.077 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.078 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.079 DEBUG:boto:Params: {} 2023-10-12T01:11:46.079 DEBUG:boto:Token: None 2023-10-12T01:11:46.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac1b1fc1a119f028-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.086 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:11:46.087 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.087 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.088 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.088 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.089 DEBUG:boto:Method: GET 2023-10-12T01:11:46.089 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.089 DEBUG:boto:Data: 2023-10-12T01:11:46.090 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.091 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.091 DEBUG:boto:Params: {} 2023-10-12T01:11:46.092 DEBUG:boto:Token: None 2023-10-12T01:11:46.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA= 2023-10-12T01:11:46.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA=', 'Content-Length': '0'} 2023-10-12T01:11:46.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6933708bbee297-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.096 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.096 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.097 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.097 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.097 DEBUG:boto:Method: GET 2023-10-12T01:11:46.098 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.098 DEBUG:boto:Data: 2023-10-12T01:11:46.099 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.100 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.100 DEBUG:boto:Params: {} 2023-10-12T01:11:46.100 DEBUG:boto:Token: None 2023-10-12T01:11:46.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.101 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA= 2023-10-12T01:11:46.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA=', 'Content-Length': '0'} 2023-10-12T01:11:46.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ef832cdf5d5d0f-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.105 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.105 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.106 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.106 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.106 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.107 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.107 DEBUG:boto:Data: 2023-10-12T01:11:46.108 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.109 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.109 DEBUG:boto:Params: {} 2023-10-12T01:11:46.109 DEBUG:boto:Token: None 2023-10-12T01:11:46.110 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z6EUf4yBWhCifLbOtmTtbnKoTsk= 2023-10-12T01:11:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z6EUf4yBWhCifLbOtmTtbnKoTsk=', 'Content-Length': '0'} 2023-10-12T01:11:46.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866f523d53455211-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.113 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.114 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.114 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.115 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.115 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.115 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.116 DEBUG:boto:Data: 2023-10-12T01:11:46.116 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.117 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.118 DEBUG:boto:Params: {} 2023-10-12T01:11:46.118 DEBUG:boto:Token: None 2023-10-12T01:11:46.119 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.119 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z6EUf4yBWhCifLbOtmTtbnKoTsk= 2023-10-12T01:11:46.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z6EUf4yBWhCifLbOtmTtbnKoTsk=', 'Content-Length': '0'} 2023-10-12T01:11:46.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089dfb94fd6819871-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:11:46.123 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.123 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.124 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.124 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.125 DEBUG:boto:Method: GET 2023-10-12T01:11:46.125 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.126 DEBUG:boto:Data: 2023-10-12T01:11:46.126 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.127 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.127 DEBUG:boto:Params: {} 2023-10-12T01:11:46.128 DEBUG:boto:Token: None 2023-10-12T01:11:46.128 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA= 2023-10-12T01:11:46.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA=', 'Content-Length': '0'} 2023-10-12T01:11:46.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ea92a66cd9b54a6-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.132 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.133 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.133 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.133 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.134 DEBUG:boto:Method: GET 2023-10-12T01:11:46.134 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.135 DEBUG:boto:Data: 2023-10-12T01:11:46.135 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.136 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.137 DEBUG:boto:Params: {} 2023-10-12T01:11:46.137 DEBUG:boto:Token: None 2023-10-12T01:11:46.138 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:11:46.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA= 2023-10-12T01:11:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kD7SBo2ZnOKkjregBiJcfFbXjwA=', 'Content-Length': '0'} 2023-10-12T01:11:46.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246444f4d445b4cc-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.141 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.142 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.142 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.143 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.143 DEBUG:boto:Method: GET 2023-10-12T01:11:46.143 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.144 DEBUG:boto:Data: 2023-10-12T01:11:46.144 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.145 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.146 DEBUG:boto:Params: {} 2023-10-12T01:11:46.146 DEBUG:boto:Token: None 2023-10-12T01:11:46.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa61ef975df6013-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.151 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.152 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.152 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.153 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.153 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.153 DEBUG:boto:Method: GET 2023-10-12T01:11:46.154 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.154 DEBUG:boto:Data: 2023-10-12T01:11:46.155 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.156 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.156 DEBUG:boto:Params: {} 2023-10-12T01:11:46.157 DEBUG:boto:Token: None 2023-10-12T01:11:46.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc4a40afac943553-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.163 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.163 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.164 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.164 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.165 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:11:46.165 DEBUG:boto:Data: 2023-10-12T01:11:46.166 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.166 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.167 DEBUG:boto:Params: {} 2023-10-12T01:11:46.167 DEBUG:boto:Token: None 2023-10-12T01:11:46.168 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62 2023-10-12T01:11:46.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kj0OwvSmivdL8H9FfYtY8ZbHBHc= 2023-10-12T01:11:46.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kj0OwvSmivdL8H9FfYtY8ZbHBHc=', 'Content-Length': '0'} 2023-10-12T01:11:46.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b845ce30fcb5014-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.171 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.172 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.172 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.173 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:11:46.173 DEBUG:boto:Data: 2023-10-12T01:11:46.173 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.174 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.175 DEBUG:boto:Params: {} 2023-10-12T01:11:46.175 DEBUG:boto:Token: None 2023-10-12T01:11:46.175 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62 2023-10-12T01:11:46.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Kj0OwvSmivdL8H9FfYtY8ZbHBHc= 2023-10-12T01:11:46.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Kj0OwvSmivdL8H9FfYtY8ZbHBHc=', 'Content-Length': '0'} 2023-10-12T01:11:46.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bbb6ee0cdeb3c13-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:11:46.179 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.180 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.180 DEBUG:boto:Method: GET 2023-10-12T01:11:46.181 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:11:46.181 DEBUG:boto:Data: 2023-10-12T01:11:46.181 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.182 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.183 DEBUG:boto:Params: {} 2023-10-12T01:11:46.183 DEBUG:boto:Token: None 2023-10-12T01:11:46.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62 2023-10-12T01:11:46.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GgylWP33NssoEGvxKix0dIuzbpU= 2023-10-12T01:11:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GgylWP33NssoEGvxKix0dIuzbpU=', 'Content-Length': '0'} 2023-10-12T01:11:46.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd3d1408f513b95-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.230 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:11:46.231 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:11:46.232 DEBUG:boto:Method: GET 2023-10-12T01:11:46.232 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:11:46.232 DEBUG:boto:Data: 2023-10-12T01:11:46.233 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.234 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.234 DEBUG:boto:Params: {} 2023-10-12T01:11:46.235 DEBUG:boto:Token: None 2023-10-12T01:11:46.235 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj62 2023-10-12T01:11:46.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GgylWP33NssoEGvxKix0dIuzbpU= 2023-10-12T01:11:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GgylWP33NssoEGvxKix0dIuzbpU=', 'Content-Length': '0'} 2023-10-12T01:11:46.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007773e51c55bf9282-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.239 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.240 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.240 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.241 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.241 DEBUG:boto:Method: GET 2023-10-12T01:11:46.241 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.242 DEBUG:boto:Data: 2023-10-12T01:11:46.242 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.243 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.244 DEBUG:boto:Params: {} 2023-10-12T01:11:46.244 DEBUG:boto:Token: None 2023-10-12T01:11:46.245 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a82a07941e14eef-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.250 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.250 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.251 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.251 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.252 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.252 DEBUG:boto:Method: GET 2023-10-12T01:11:46.253 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:11:46.253 DEBUG:boto:Data: 2023-10-12T01:11:46.253 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.254 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.255 DEBUG:boto:Params: {} 2023-10-12T01:11:46.255 DEBUG:boto:Token: None 2023-10-12T01:11:46.256 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dead0ef2ecb02023-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.306 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:11:46.308 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.308 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.309 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.309 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.309 DEBUG:boto:Method: GET 2023-10-12T01:11:46.310 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.310 DEBUG:boto:Data: 2023-10-12T01:11:46.311 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.312 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.312 DEBUG:boto:Params: {} 2023-10-12T01:11:46.313 DEBUG:boto:Token: None 2023-10-12T01:11:46.313 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU= 2023-10-12T01:11:46.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU=', 'Content-Length': '0'} 2023-10-12T01:11:46.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f3e2380743ca57-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.317 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.317 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.318 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.318 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.319 DEBUG:boto:Method: GET 2023-10-12T01:11:46.319 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.320 DEBUG:boto:Data: 2023-10-12T01:11:46.320 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.321 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.321 DEBUG:boto:Params: {} 2023-10-12T01:11:46.322 DEBUG:boto:Token: None 2023-10-12T01:11:46.322 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.323 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU= 2023-10-12T01:11:46.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU=', 'Content-Length': '0'} 2023-10-12T01:11:46.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf10636c9da055ae-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.326 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.327 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.327 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.328 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.328 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.328 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.329 DEBUG:boto:Data: 2023-10-12T01:11:46.329 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.330 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.330 DEBUG:boto:Params: {} 2023-10-12T01:11:46.331 DEBUG:boto:Token: None 2023-10-12T01:11:46.331 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.332 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GrPfsAcyqlnv2S41K1bQKQnFbWA= 2023-10-12T01:11:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GrPfsAcyqlnv2S41K1bQKQnFbWA=', 'Content-Length': '0'} 2023-10-12T01:11:46.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf0bb1ec35858dd-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.335 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.335 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.336 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.336 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.336 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.337 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.337 DEBUG:boto:Data: 2023-10-12T01:11:46.338 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.339 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.339 DEBUG:boto:Params: {} 2023-10-12T01:11:46.339 DEBUG:boto:Token: None 2023-10-12T01:11:46.340 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GrPfsAcyqlnv2S41K1bQKQnFbWA= 2023-10-12T01:11:46.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GrPfsAcyqlnv2S41K1bQKQnFbWA=', 'Content-Length': '0'} 2023-10-12T01:11:46.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258abf865a1d9889-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:11:46.344 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.344 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.345 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.345 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.346 DEBUG:boto:Method: GET 2023-10-12T01:11:46.346 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.346 DEBUG:boto:Data: 2023-10-12T01:11:46.347 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.348 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.348 DEBUG:boto:Params: {} 2023-10-12T01:11:46.349 DEBUG:boto:Token: None 2023-10-12T01:11:46.349 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU= 2023-10-12T01:11:46.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU=', 'Content-Length': '0'} 2023-10-12T01:11:46.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004449153722fc9389-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.353 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.354 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.354 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.354 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.355 DEBUG:boto:Method: GET 2023-10-12T01:11:46.355 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.356 DEBUG:boto:Data: 2023-10-12T01:11:46.356 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.357 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.357 DEBUG:boto:Params: {} 2023-10-12T01:11:46.358 DEBUG:boto:Token: None 2023-10-12T01:11:46.358 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:11:46.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU= 2023-10-12T01:11:46.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8H3VdS8p5qvCoVgU4rk1t55agUU=', 'Content-Length': '0'} 2023-10-12T01:11:46.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ffd0e001d505e0c-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.363 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.363 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.363 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.364 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.364 DEBUG:boto:Method: GET 2023-10-12T01:11:46.365 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.365 DEBUG:boto:Data: 2023-10-12T01:11:46.366 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.366 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.367 DEBUG:boto:Params: {} 2023-10-12T01:11:46.367 DEBUG:boto:Token: None 2023-10-12T01:11:46.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb106871783418f8-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.372 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.373 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.373 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.374 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.374 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.375 DEBUG:boto:Method: GET 2023-10-12T01:11:46.375 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.376 DEBUG:boto:Data: 2023-10-12T01:11:46.376 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.377 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.377 DEBUG:boto:Params: {} 2023-10-12T01:11:46.378 DEBUG:boto:Token: None 2023-10-12T01:11:46.378 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.379 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b4230069fb58293-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.384 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.384 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.385 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.385 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.386 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:11:46.386 DEBUG:boto:Data: 2023-10-12T01:11:46.387 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.388 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.388 DEBUG:boto:Params: {} 2023-10-12T01:11:46.388 DEBUG:boto:Token: None 2023-10-12T01:11:46.389 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68 2023-10-12T01:11:46.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6sPjoDIQs/xoW4gfrCVyrlfcRqY= 2023-10-12T01:11:46.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6sPjoDIQs/xoW4gfrCVyrlfcRqY=', 'Content-Length': '0'} 2023-10-12T01:11:46.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0ba48a04928b22-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.392 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.393 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.393 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.394 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:11:46.394 DEBUG:boto:Data: 2023-10-12T01:11:46.394 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.395 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.396 DEBUG:boto:Params: {} 2023-10-12T01:11:46.396 DEBUG:boto:Token: None 2023-10-12T01:11:46.397 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68 2023-10-12T01:11:46.397 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6sPjoDIQs/xoW4gfrCVyrlfcRqY= 2023-10-12T01:11:46.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6sPjoDIQs/xoW4gfrCVyrlfcRqY=', 'Content-Length': '0'} 2023-10-12T01:11:46.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf55b5d54ad63560-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:11:46.401 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.401 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.402 DEBUG:boto:Method: GET 2023-10-12T01:11:46.402 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:11:46.403 DEBUG:boto:Data: 2023-10-12T01:11:46.405 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.406 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.407 DEBUG:boto:Params: {} 2023-10-12T01:11:46.407 DEBUG:boto:Token: None 2023-10-12T01:11:46.407 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68 2023-10-12T01:11:46.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:67ekwM5JS39DL4hh9I2Wr1jZyXE= 2023-10-12T01:11:46.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:67ekwM5JS39DL4hh9I2Wr1jZyXE=', 'Content-Length': '0'} 2023-10-12T01:11:46.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd840b86fa14cd51-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.458 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:11:46.459 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:11:46.459 DEBUG:boto:Method: GET 2023-10-12T01:11:46.460 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:11:46.460 DEBUG:boto:Data: 2023-10-12T01:11:46.461 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.462 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.462 DEBUG:boto:Params: {} 2023-10-12T01:11:46.463 DEBUG:boto:Token: None 2023-10-12T01:11:46.463 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj68 2023-10-12T01:11:46.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:67ekwM5JS39DL4hh9I2Wr1jZyXE= 2023-10-12T01:11:46.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:67ekwM5JS39DL4hh9I2Wr1jZyXE=', 'Content-Length': '0'} 2023-10-12T01:11:46.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a187a4d71aa5580-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.467 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.468 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.468 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.470 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.470 DEBUG:boto:Method: GET 2023-10-12T01:11:46.471 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.471 DEBUG:boto:Data: 2023-10-12T01:11:46.472 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.473 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.473 DEBUG:boto:Params: {} 2023-10-12T01:11:46.473 DEBUG:boto:Token: None 2023-10-12T01:11:46.474 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000163c1996ae78031b-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.478 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.479 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.479 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.480 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.480 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.481 DEBUG:boto:Method: GET 2023-10-12T01:11:46.481 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:11:46.481 DEBUG:boto:Data: 2023-10-12T01:11:46.482 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.483 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.483 DEBUG:boto:Params: {} 2023-10-12T01:11:46.484 DEBUG:boto:Token: None 2023-10-12T01:11:46.484 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ff933410368c61-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.490 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:11:46.491 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.491 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.492 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.492 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.493 DEBUG:boto:Method: GET 2023-10-12T01:11:46.493 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.494 DEBUG:boto:Data: 2023-10-12T01:11:46.494 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.495 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.495 DEBUG:boto:Params: {} 2023-10-12T01:11:46.496 DEBUG:boto:Token: None 2023-10-12T01:11:46.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI= 2023-10-12T01:11:46.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI=', 'Content-Length': '0'} 2023-10-12T01:11:46.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6ca61f1c6d4215-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.500 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.500 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.501 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.501 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.502 DEBUG:boto:Method: GET 2023-10-12T01:11:46.502 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.503 DEBUG:boto:Data: 2023-10-12T01:11:46.503 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.504 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.504 DEBUG:boto:Params: {} 2023-10-12T01:11:46.505 DEBUG:boto:Token: None 2023-10-12T01:11:46.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI= 2023-10-12T01:11:46.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI=', 'Content-Length': '0'} 2023-10-12T01:11:46.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03cf93205285699-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.509 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.509 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.510 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.510 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.511 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.511 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.511 DEBUG:boto:Data: 2023-10-12T01:11:46.512 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.513 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.513 DEBUG:boto:Params: {} 2023-10-12T01:11:46.513 DEBUG:boto:Token: None 2023-10-12T01:11:46.514 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rYWLzGtH1ZGhM7nFmxAa2thBiq0= 2023-10-12T01:11:46.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rYWLzGtH1ZGhM7nFmxAa2thBiq0=', 'Content-Length': '0'} 2023-10-12T01:11:46.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b667db43df5cc8f-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.517 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.517 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.518 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.518 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.519 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.519 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.520 DEBUG:boto:Data: 2023-10-12T01:11:46.520 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.521 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.521 DEBUG:boto:Params: {} 2023-10-12T01:11:46.522 DEBUG:boto:Token: None 2023-10-12T01:11:46.522 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rYWLzGtH1ZGhM7nFmxAa2thBiq0= 2023-10-12T01:11:46.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rYWLzGtH1ZGhM7nFmxAa2thBiq0=', 'Content-Length': '0'} 2023-10-12T01:11:46.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ecf1531bfb4c5a5-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:11:46.526 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.526 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.527 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.527 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.528 DEBUG:boto:Method: GET 2023-10-12T01:11:46.528 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.528 DEBUG:boto:Data: 2023-10-12T01:11:46.529 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.530 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.530 DEBUG:boto:Params: {} 2023-10-12T01:11:46.531 DEBUG:boto:Token: None 2023-10-12T01:11:46.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI= 2023-10-12T01:11:46.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI=', 'Content-Length': '0'} 2023-10-12T01:11:46.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001483b938d5dc0c2b-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.535 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.535 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.536 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.536 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.537 DEBUG:boto:Method: GET 2023-10-12T01:11:46.537 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.537 DEBUG:boto:Data: 2023-10-12T01:11:46.538 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.539 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.539 DEBUG:boto:Params: {} 2023-10-12T01:11:46.540 DEBUG:boto:Token: None 2023-10-12T01:11:46.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:11:46.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI= 2023-10-12T01:11:46.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7a+3F0HUqLu3M5EbvyxSwujMALI=', 'Content-Length': '0'} 2023-10-12T01:11:46.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030709e2d217f74fd-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.544 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.545 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.545 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.546 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.546 DEBUG:boto:Method: GET 2023-10-12T01:11:46.547 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.547 DEBUG:boto:Data: 2023-10-12T01:11:46.547 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.548 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.549 DEBUG:boto:Params: {} 2023-10-12T01:11:46.549 DEBUG:boto:Token: None 2023-10-12T01:11:46.550 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018cac24b6c092767-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.594 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.595 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.595 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.596 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.596 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.597 DEBUG:boto:Method: GET 2023-10-12T01:11:46.597 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.598 DEBUG:boto:Data: 2023-10-12T01:11:46.598 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.599 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.599 DEBUG:boto:Params: {} 2023-10-12T01:11:46.600 DEBUG:boto:Token: None 2023-10-12T01:11:46.600 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5da000f9f438e4-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.606 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.607 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.607 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.608 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.608 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:11:46.608 DEBUG:boto:Data: 2023-10-12T01:11:46.609 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.610 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.610 DEBUG:boto:Params: {} 2023-10-12T01:11:46.611 DEBUG:boto:Token: None 2023-10-12T01:11:46.611 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69 2023-10-12T01:11:46.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dWNwplVwx/M47IrZkpY/rfBFiFs= 2023-10-12T01:11:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dWNwplVwx/M47IrZkpY/rfBFiFs=', 'Content-Length': '0'} 2023-10-12T01:11:46.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae9b5891f7495f9-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.615 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.615 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.615 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.616 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:11:46.616 DEBUG:boto:Data: 2023-10-12T01:11:46.617 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.618 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.618 DEBUG:boto:Params: {} 2023-10-12T01:11:46.619 DEBUG:boto:Token: None 2023-10-12T01:11:46.619 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69 2023-10-12T01:11:46.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dWNwplVwx/M47IrZkpY/rfBFiFs= 2023-10-12T01:11:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dWNwplVwx/M47IrZkpY/rfBFiFs=', 'Content-Length': '0'} 2023-10-12T01:11:46.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6d18d7773fd18c-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:11:46.623 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.624 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.624 DEBUG:boto:Method: GET 2023-10-12T01:11:46.624 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:11:46.625 DEBUG:boto:Data: 2023-10-12T01:11:46.625 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.626 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.626 DEBUG:boto:Params: {} 2023-10-12T01:11:46.627 DEBUG:boto:Token: None 2023-10-12T01:11:46.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69 2023-10-12T01:11:46.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sQu639l3vD2riH3QpMD+9IVD3Is= 2023-10-12T01:11:46.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sQu639l3vD2riH3QpMD+9IVD3Is=', 'Content-Length': '0'} 2023-10-12T01:11:46.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c696870a6ec6fa5-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.631 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:11:46.631 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:11:46.631 DEBUG:boto:Method: GET 2023-10-12T01:11:46.632 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:11:46.632 DEBUG:boto:Data: 2023-10-12T01:11:46.632 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.633 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.634 DEBUG:boto:Params: {} 2023-10-12T01:11:46.634 DEBUG:boto:Token: None 2023-10-12T01:11:46.635 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj69 2023-10-12T01:11:46.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sQu639l3vD2riH3QpMD+9IVD3Is= 2023-10-12T01:11:46.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sQu639l3vD2riH3QpMD+9IVD3Is=', 'Content-Length': '0'} 2023-10-12T01:11:46.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054159f4fdec759f8-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.638 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.639 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.639 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.639 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.640 DEBUG:boto:Method: GET 2023-10-12T01:11:46.640 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.641 DEBUG:boto:Data: 2023-10-12T01:11:46.641 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.642 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.642 DEBUG:boto:Params: {} 2023-10-12T01:11:46.643 DEBUG:boto:Token: None 2023-10-12T01:11:46.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f7bb308a661269c-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.647 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.648 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.648 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.649 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.649 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.649 DEBUG:boto:Method: GET 2023-10-12T01:11:46.650 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:11:46.650 DEBUG:boto:Data: 2023-10-12T01:11:46.651 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.652 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.652 DEBUG:boto:Params: {} 2023-10-12T01:11:46.652 DEBUG:boto:Token: None 2023-10-12T01:11:46.653 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.653 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d9d37c26313939-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.658 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:11:46.659 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.660 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.660 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.661 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.661 DEBUG:boto:Method: GET 2023-10-12T01:11:46.662 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.662 DEBUG:boto:Data: 2023-10-12T01:11:46.662 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.663 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.664 DEBUG:boto:Params: {} 2023-10-12T01:11:46.664 DEBUG:boto:Token: None 2023-10-12T01:11:46.665 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I= 2023-10-12T01:11:46.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I=', 'Content-Length': '0'} 2023-10-12T01:11:46.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7da8522778b78d-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.668 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.669 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.669 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.670 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.670 DEBUG:boto:Method: GET 2023-10-12T01:11:46.671 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.671 DEBUG:boto:Data: 2023-10-12T01:11:46.672 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.673 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.673 DEBUG:boto:Params: {} 2023-10-12T01:11:46.673 DEBUG:boto:Token: None 2023-10-12T01:11:46.674 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I= 2023-10-12T01:11:46.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I=', 'Content-Length': '0'} 2023-10-12T01:11:46.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3eaa32513515a2-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.678 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.678 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.679 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.679 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.679 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.680 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.680 DEBUG:boto:Data: 2023-10-12T01:11:46.680 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.681 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.682 DEBUG:boto:Params: {} 2023-10-12T01:11:46.682 DEBUG:boto:Token: None 2023-10-12T01:11:46.683 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n54tjCLhpn2nrnZpFxg1gwoe+jI= 2023-10-12T01:11:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n54tjCLhpn2nrnZpFxg1gwoe+jI=', 'Content-Length': '0'} 2023-10-12T01:11:46.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46a89a50b3dee88-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.686 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.687 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.687 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.687 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.688 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.688 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.689 DEBUG:boto:Data: 2023-10-12T01:11:46.689 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.690 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.690 DEBUG:boto:Params: {} 2023-10-12T01:11:46.691 DEBUG:boto:Token: None 2023-10-12T01:11:46.691 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.691 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n54tjCLhpn2nrnZpFxg1gwoe+jI= 2023-10-12T01:11:46.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n54tjCLhpn2nrnZpFxg1gwoe+jI=', 'Content-Length': '0'} 2023-10-12T01:11:46.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f74bf3035d1ec0-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:11:46.695 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.695 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.696 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.696 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.697 DEBUG:boto:Method: GET 2023-10-12T01:11:46.697 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.697 DEBUG:boto:Data: 2023-10-12T01:11:46.698 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.699 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.699 DEBUG:boto:Params: {} 2023-10-12T01:11:46.700 DEBUG:boto:Token: None 2023-10-12T01:11:46.700 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I= 2023-10-12T01:11:46.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I=', 'Content-Length': '0'} 2023-10-12T01:11:46.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93ca95f1638643f-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.704 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.704 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.704 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.705 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.705 DEBUG:boto:Method: GET 2023-10-12T01:11:46.706 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.706 DEBUG:boto:Data: 2023-10-12T01:11:46.706 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.707 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.708 DEBUG:boto:Params: {} 2023-10-12T01:11:46.708 DEBUG:boto:Token: None 2023-10-12T01:11:46.709 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:11:46.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I= 2023-10-12T01:11:46.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EfCP0z4pgF0I6Bt/ZtjvaY18m3I=', 'Content-Length': '0'} 2023-10-12T01:11:46.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122f5a847d024c1a-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.713 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.713 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.714 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.714 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.714 DEBUG:boto:Method: GET 2023-10-12T01:11:46.715 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.715 DEBUG:boto:Data: 2023-10-12T01:11:46.716 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.716 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.717 DEBUG:boto:Params: {} 2023-10-12T01:11:46.717 DEBUG:boto:Token: None 2023-10-12T01:11:46.718 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001887e865233578a7-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.766 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.767 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.768 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.768 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.769 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.769 DEBUG:boto:Method: GET 2023-10-12T01:11:46.770 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.770 DEBUG:boto:Data: 2023-10-12T01:11:46.771 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.772 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.772 DEBUG:boto:Params: {} 2023-10-12T01:11:46.773 DEBUG:boto:Token: None 2023-10-12T01:11:46.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df41a7d0a1ddeae8-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.779 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.780 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.782 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.782 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.783 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:11:46.783 DEBUG:boto:Data: 2023-10-12T01:11:46.783 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.784 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.785 DEBUG:boto:Params: {} 2023-10-12T01:11:46.785 DEBUG:boto:Token: None 2023-10-12T01:11:46.786 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74 2023-10-12T01:11:46.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mTFHyYyNWYSIUO8uuVRusR7MQsw= 2023-10-12T01:11:46.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mTFHyYyNWYSIUO8uuVRusR7MQsw=', 'Content-Length': '0'} 2023-10-12T01:11:46.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007231f67d93953627-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.789 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.790 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.790 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.791 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:11:46.791 DEBUG:boto:Data: 2023-10-12T01:11:46.791 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.792 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.793 DEBUG:boto:Params: {} 2023-10-12T01:11:46.793 DEBUG:boto:Token: None 2023-10-12T01:11:46.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74 2023-10-12T01:11:46.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mTFHyYyNWYSIUO8uuVRusR7MQsw= 2023-10-12T01:11:46.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mTFHyYyNWYSIUO8uuVRusR7MQsw=', 'Content-Length': '0'} 2023-10-12T01:11:46.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c894d9ea2693c4b-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:11:46.798 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.798 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.799 DEBUG:boto:Method: GET 2023-10-12T01:11:46.799 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:11:46.799 DEBUG:boto:Data: 2023-10-12T01:11:46.800 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.801 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.801 DEBUG:boto:Params: {} 2023-10-12T01:11:46.802 DEBUG:boto:Token: None 2023-10-12T01:11:46.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74 2023-10-12T01:11:46.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:43DbpxYEX7gTMFEQ8KfNjeNMwKA= 2023-10-12T01:11:46.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:43DbpxYEX7gTMFEQ8KfNjeNMwKA=', 'Content-Length': '0'} 2023-10-12T01:11:46.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e25601fb6fe37c3-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.806 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:11:46.806 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:11:46.807 DEBUG:boto:Method: GET 2023-10-12T01:11:46.807 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:11:46.808 DEBUG:boto:Data: 2023-10-12T01:11:46.808 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.809 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.809 DEBUG:boto:Params: {} 2023-10-12T01:11:46.810 DEBUG:boto:Token: None 2023-10-12T01:11:46.810 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj74 2023-10-12T01:11:46.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:43DbpxYEX7gTMFEQ8KfNjeNMwKA= 2023-10-12T01:11:46.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:43DbpxYEX7gTMFEQ8KfNjeNMwKA=', 'Content-Length': '0'} 2023-10-12T01:11:46.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ecf8da57dadb99-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.854 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.855 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.855 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.856 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.856 DEBUG:boto:Method: GET 2023-10-12T01:11:46.857 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.857 DEBUG:boto:Data: 2023-10-12T01:11:46.858 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.859 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.859 DEBUG:boto:Params: {} 2023-10-12T01:11:46.860 DEBUG:boto:Token: None 2023-10-12T01:11:46.860 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda2414dd8549a40-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.864 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.865 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.865 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.866 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.866 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.867 DEBUG:boto:Method: GET 2023-10-12T01:11:46.867 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:11:46.868 DEBUG:boto:Data: 2023-10-12T01:11:46.868 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.869 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.869 DEBUG:boto:Params: {} 2023-10-12T01:11:46.870 DEBUG:boto:Token: None 2023-10-12T01:11:46.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f9d2c1c5108c52-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.876 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:11:46.877 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.878 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.878 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.879 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.879 DEBUG:boto:Method: GET 2023-10-12T01:11:46.880 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.880 DEBUG:boto:Data: 2023-10-12T01:11:46.881 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.882 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.882 DEBUG:boto:Params: {} 2023-10-12T01:11:46.883 DEBUG:boto:Token: None 2023-10-12T01:11:46.883 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y= 2023-10-12T01:11:46.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y=', 'Content-Length': '0'} 2023-10-12T01:11:46.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd70a055e49e30f5-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.887 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.887 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.888 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.888 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.889 DEBUG:boto:Method: GET 2023-10-12T01:11:46.889 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.890 DEBUG:boto:Data: 2023-10-12T01:11:46.890 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.891 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.891 DEBUG:boto:Params: {} 2023-10-12T01:11:46.892 DEBUG:boto:Token: None 2023-10-12T01:11:46.892 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y= 2023-10-12T01:11:46.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y=', 'Content-Length': '0'} 2023-10-12T01:11:46.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bc1e6c4bf61f0f-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.896 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.897 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.897 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.897 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.898 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.898 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.899 DEBUG:boto:Data: 2023-10-12T01:11:46.899 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.900 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.900 DEBUG:boto:Params: {} 2023-10-12T01:11:46.901 DEBUG:boto:Token: None 2023-10-12T01:11:46.901 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mb2KUIVkqwwU9Pbyypu+MXeQ5+A= 2023-10-12T01:11:46.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mb2KUIVkqwwU9Pbyypu+MXeQ5+A=', 'Content-Length': '0'} 2023-10-12T01:11:46.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb937c719153d115-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.905 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.905 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.906 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.906 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.906 DEBUG:boto:Method: HEAD 2023-10-12T01:11:46.907 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.907 DEBUG:boto:Data: 2023-10-12T01:11:46.908 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.908 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.909 DEBUG:boto:Params: {} 2023-10-12T01:11:46.909 DEBUG:boto:Token: None 2023-10-12T01:11:46.910 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.910 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mb2KUIVkqwwU9Pbyypu+MXeQ5+A= 2023-10-12T01:11:46.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mb2KUIVkqwwU9Pbyypu+MXeQ5+A=', 'Content-Length': '0'} 2023-10-12T01:11:46.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000faa80f24745af51-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:11:46.914 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.914 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.915 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.915 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.916 DEBUG:boto:Method: GET 2023-10-12T01:11:46.916 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.916 DEBUG:boto:Data: 2023-10-12T01:11:46.917 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.918 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.918 DEBUG:boto:Params: {} 2023-10-12T01:11:46.919 DEBUG:boto:Token: None 2023-10-12T01:11:46.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.920 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y= 2023-10-12T01:11:46.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y=', 'Content-Length': '0'} 2023-10-12T01:11:46.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14f9e4cff07ec52-00652747d2-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.970 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:46.971 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:46.971 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.972 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.972 DEBUG:boto:Method: GET 2023-10-12T01:11:46.973 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.973 DEBUG:boto:Data: 2023-10-12T01:11:46.974 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.975 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.975 DEBUG:boto:Params: {} 2023-10-12T01:11:46.975 DEBUG:boto:Token: None 2023-10-12T01:11:46.976 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:11:46.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y= 2023-10-12T01:11:46.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m6OqsM/+aP1xi1xlg6NqM/XX+/Y=', 'Content-Length': '0'} 2023-10-12T01:11:46.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e54bee4824c2b1-00652747d2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.980 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.980 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.981 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.981 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.981 DEBUG:boto:Method: GET 2023-10-12T01:11:46.982 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.982 DEBUG:boto:Data: 2023-10-12T01:11:46.983 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:46.984 DEBUG:boto:Port: 8001 2023-10-12T01:11:46.984 DEBUG:boto:Params: {} 2023-10-12T01:11:46.985 DEBUG:boto:Token: None 2023-10-12T01:11:46.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.986 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:46.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021a3014022c6596-00652747d2-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:46.989 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:46.990 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:46.991 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:46.991 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.991 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.992 DEBUG:boto:Method: GET 2023-10-12T01:11:46.992 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:11:46.993 DEBUG:boto:Data: 2023-10-12T01:11:46.993 DEBUG:boto:Headers: {} 2023-10-12T01:11:46.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:46.994 DEBUG:boto:Port: 8000 2023-10-12T01:11:46.994 DEBUG:boto:Params: {} 2023-10-12T01:11:46.995 DEBUG:boto:Token: None 2023-10-12T01:11:46.995 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:46 GMT /wafmoa-59/ 2023-10-12T01:11:46.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ= 2023-10-12T01:11:46.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B9KMnkqyIYkfWpbnS5dCj1y/QsQ=', 'Content-Length': '0'} 2023-10-12T01:11:47.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ad918761a678a1-00652747d2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.001 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.002 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:47.002 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:47.003 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.003 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:11:47.003 DEBUG:boto:Data: 2023-10-12T01:11:47.004 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.005 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.005 DEBUG:boto:Params: {} 2023-10-12T01:11:47.006 DEBUG:boto:Token: None 2023-10-12T01:11:47.006 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj76 2023-10-12T01:11:47.007 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EXL4+mXs4MqKVpuEVIwn/XhgdEU= 2023-10-12T01:11:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EXL4+mXs4MqKVpuEVIwn/XhgdEU=', 'Content-Length': '0'} 2023-10-12T01:11:47.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf2bde4a1598e16-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.010 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:47.010 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:47.011 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.011 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:11:47.012 DEBUG:boto:Data: 2023-10-12T01:11:47.012 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.013 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.013 DEBUG:boto:Params: {} 2023-10-12T01:11:47.014 DEBUG:boto:Token: None 2023-10-12T01:11:47.014 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj76 2023-10-12T01:11:47.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EXL4+mXs4MqKVpuEVIwn/XhgdEU= 2023-10-12T01:11:47.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EXL4+mXs4MqKVpuEVIwn/XhgdEU=', 'Content-Length': '0'} 2023-10-12T01:11:47.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd60b0ff388b06a-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:11:47.018 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:47.019 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:47.019 DEBUG:boto:Method: GET 2023-10-12T01:11:47.020 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:11:47.020 DEBUG:boto:Data: 2023-10-12T01:11:47.021 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.021 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.022 DEBUG:boto:Params: {} 2023-10-12T01:11:47.022 DEBUG:boto:Token: None 2023-10-12T01:11:47.023 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj76 2023-10-12T01:11:47.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W3tJ7mg9KPtevyAR0sd5m5hoWqU= 2023-10-12T01:11:47.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W3tJ7mg9KPtevyAR0sd5m5hoWqU=', 'Content-Length': '0'} 2023-10-12T01:11:47.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a89235238b87ec0-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.027 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:11:47.027 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:11:47.028 DEBUG:boto:Method: GET 2023-10-12T01:11:47.028 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:11:47.029 DEBUG:boto:Data: 2023-10-12T01:11:47.029 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.030 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.030 DEBUG:boto:Params: {} 2023-10-12T01:11:47.031 DEBUG:boto:Token: None 2023-10-12T01:11:47.031 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj76 2023-10-12T01:11:47.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W3tJ7mg9KPtevyAR0sd5m5hoWqU= 2023-10-12T01:11:47.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W3tJ7mg9KPtevyAR0sd5m5hoWqU=', 'Content-Length': '0'} 2023-10-12T01:11:47.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a561454f94986e4e-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.035 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.036 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.036 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.036 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.037 DEBUG:boto:Method: GET 2023-10-12T01:11:47.037 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.038 DEBUG:boto:Data: 2023-10-12T01:11:47.038 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.039 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.040 DEBUG:boto:Params: {} 2023-10-12T01:11:47.040 DEBUG:boto:Token: None 2023-10-12T01:11:47.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2463274f41f38df-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.045 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.046 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.046 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.047 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.047 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.047 DEBUG:boto:Method: GET 2023-10-12T01:11:47.048 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:11:47.048 DEBUG:boto:Data: 2023-10-12T01:11:47.049 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.050 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.050 DEBUG:boto:Params: {} 2023-10-12T01:11:47.050 DEBUG:boto:Token: None 2023-10-12T01:11:47.051 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb4b552b105f62ea-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.098 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:11:47.100 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.100 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.101 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.101 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.102 DEBUG:boto:Method: GET 2023-10-12T01:11:47.102 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.103 DEBUG:boto:Data: 2023-10-12T01:11:47.103 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.104 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.104 DEBUG:boto:Params: {} 2023-10-12T01:11:47.105 DEBUG:boto:Token: None 2023-10-12T01:11:47.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s= 2023-10-12T01:11:47.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s=', 'Content-Length': '0'} 2023-10-12T01:11:47.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000336366b2b84c0df-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.110 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.110 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.111 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.111 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.112 DEBUG:boto:Method: GET 2023-10-12T01:11:47.112 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.113 DEBUG:boto:Data: 2023-10-12T01:11:47.113 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.114 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.115 DEBUG:boto:Params: {} 2023-10-12T01:11:47.115 DEBUG:boto:Token: None 2023-10-12T01:11:47.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s= 2023-10-12T01:11:47.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s=', 'Content-Length': '0'} 2023-10-12T01:11:47.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914706bb56fc4542-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.120 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.120 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.121 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.121 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.121 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.122 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.122 DEBUG:boto:Data: 2023-10-12T01:11:47.123 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.124 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.124 DEBUG:boto:Params: {} 2023-10-12T01:11:47.124 DEBUG:boto:Token: None 2023-10-12T01:11:47.125 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4mibsNpfl4pOXDEisVczQaRUWnM= 2023-10-12T01:11:47.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4mibsNpfl4pOXDEisVczQaRUWnM=', 'Content-Length': '0'} 2023-10-12T01:11:47.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eed1b644590c71c-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.129 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.129 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.130 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.130 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.131 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.131 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.131 DEBUG:boto:Data: 2023-10-12T01:11:47.132 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.133 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.133 DEBUG:boto:Params: {} 2023-10-12T01:11:47.134 DEBUG:boto:Token: None 2023-10-12T01:11:47.134 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.135 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4mibsNpfl4pOXDEisVczQaRUWnM= 2023-10-12T01:11:47.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4mibsNpfl4pOXDEisVczQaRUWnM=', 'Content-Length': '0'} 2023-10-12T01:11:47.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc886a4b8dec17d-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:11:47.138 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.139 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.139 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.140 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.140 DEBUG:boto:Method: GET 2023-10-12T01:11:47.141 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.141 DEBUG:boto:Data: 2023-10-12T01:11:47.142 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.143 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.143 DEBUG:boto:Params: {} 2023-10-12T01:11:47.144 DEBUG:boto:Token: None 2023-10-12T01:11:47.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s= 2023-10-12T01:11:47.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s=', 'Content-Length': '0'} 2023-10-12T01:11:47.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925f8e6bb5828f41-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.148 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.149 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.149 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.150 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.150 DEBUG:boto:Method: GET 2023-10-12T01:11:47.150 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.151 DEBUG:boto:Data: 2023-10-12T01:11:47.151 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.152 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.153 DEBUG:boto:Params: {} 2023-10-12T01:11:47.153 DEBUG:boto:Token: None 2023-10-12T01:11:47.154 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:11:47.154 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s= 2023-10-12T01:11:47.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rofek7YS0IWtYqVsAfYfbEC8/6s=', 'Content-Length': '0'} 2023-10-12T01:11:47.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7cc32357ebba3dd-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.157 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.158 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.158 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.159 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.159 DEBUG:boto:Method: GET 2023-10-12T01:11:47.160 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.161 DEBUG:boto:Data: 2023-10-12T01:11:47.161 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.162 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.162 DEBUG:boto:Params: {} 2023-10-12T01:11:47.163 DEBUG:boto:Token: None 2023-10-12T01:11:47.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef8b4488825770cd-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.168 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.168 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.169 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.169 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.170 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.170 DEBUG:boto:Method: GET 2023-10-12T01:11:47.171 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.171 DEBUG:boto:Data: 2023-10-12T01:11:47.172 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.172 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.173 DEBUG:boto:Params: {} 2023-10-12T01:11:47.173 DEBUG:boto:Token: None 2023-10-12T01:11:47.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f629e60498ef44d-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.180 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.180 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.181 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.181 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.182 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:11:47.182 DEBUG:boto:Data: 2023-10-12T01:11:47.183 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.183 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.184 DEBUG:boto:Params: {} 2023-10-12T01:11:47.185 DEBUG:boto:Token: None 2023-10-12T01:11:47.185 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85 2023-10-12T01:11:47.186 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBoF85OfQcnLrcvGqKSQtpLNrps= 2023-10-12T01:11:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBoF85OfQcnLrcvGqKSQtpLNrps=', 'Content-Length': '0'} 2023-10-12T01:11:47.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d6ab4821b50769-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.189 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.189 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.190 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.190 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:11:47.191 DEBUG:boto:Data: 2023-10-12T01:11:47.191 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.192 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.192 DEBUG:boto:Params: {} 2023-10-12T01:11:47.193 DEBUG:boto:Token: None 2023-10-12T01:11:47.193 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85 2023-10-12T01:11:47.193 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBoF85OfQcnLrcvGqKSQtpLNrps= 2023-10-12T01:11:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBoF85OfQcnLrcvGqKSQtpLNrps=', 'Content-Length': '0'} 2023-10-12T01:11:47.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008187203f1691df9d-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:11:47.197 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.198 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.198 DEBUG:boto:Method: GET 2023-10-12T01:11:47.199 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:11:47.199 DEBUG:boto:Data: 2023-10-12T01:11:47.200 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.200 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.201 DEBUG:boto:Params: {} 2023-10-12T01:11:47.201 DEBUG:boto:Token: None 2023-10-12T01:11:47.202 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85 2023-10-12T01:11:47.202 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5Yes1Q0NNeL78BWlvtupFn8n0k= 2023-10-12T01:11:47.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5Yes1Q0NNeL78BWlvtupFn8n0k=', 'Content-Length': '0'} 2023-10-12T01:11:47.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc22ce4bf1a3f842-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.251 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:11:47.251 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:11:47.252 DEBUG:boto:Method: GET 2023-10-12T01:11:47.252 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:11:47.253 DEBUG:boto:Data: 2023-10-12T01:11:47.253 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.254 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.254 DEBUG:boto:Params: {} 2023-10-12T01:11:47.255 DEBUG:boto:Token: None 2023-10-12T01:11:47.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj85 2023-10-12T01:11:47.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d5Yes1Q0NNeL78BWlvtupFn8n0k= 2023-10-12T01:11:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d5Yes1Q0NNeL78BWlvtupFn8n0k=', 'Content-Length': '0'} 2023-10-12T01:11:47.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb9ab582e5b39de-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.259 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.260 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.262 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.262 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.263 DEBUG:boto:Method: GET 2023-10-12T01:11:47.263 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.263 DEBUG:boto:Data: 2023-10-12T01:11:47.264 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.265 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.265 DEBUG:boto:Params: {} 2023-10-12T01:11:47.266 DEBUG:boto:Token: None 2023-10-12T01:11:47.267 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.267 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d70220623da54e-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.271 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.272 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.272 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.273 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.273 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.274 DEBUG:boto:Method: GET 2023-10-12T01:11:47.274 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:11:47.275 DEBUG:boto:Data: 2023-10-12T01:11:47.275 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.276 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.276 DEBUG:boto:Params: {} 2023-10-12T01:11:47.277 DEBUG:boto:Token: None 2023-10-12T01:11:47.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a431e12e7ca8988d-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.283 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:11:47.284 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.285 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.285 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.286 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.286 DEBUG:boto:Method: GET 2023-10-12T01:11:47.286 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.287 DEBUG:boto:Data: 2023-10-12T01:11:47.287 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.288 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.289 DEBUG:boto:Params: {} 2023-10-12T01:11:47.289 DEBUG:boto:Token: None 2023-10-12T01:11:47.290 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo= 2023-10-12T01:11:47.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo=', 'Content-Length': '0'} 2023-10-12T01:11:47.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f595b6669b686a-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.294 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.294 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.295 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.295 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.296 DEBUG:boto:Method: GET 2023-10-12T01:11:47.296 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.297 DEBUG:boto:Data: 2023-10-12T01:11:47.297 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.298 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.298 DEBUG:boto:Params: {} 2023-10-12T01:11:47.299 DEBUG:boto:Token: None 2023-10-12T01:11:47.299 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo= 2023-10-12T01:11:47.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo=', 'Content-Length': '0'} 2023-10-12T01:11:47.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede2735d2481b606-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.303 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.304 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.304 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.304 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.305 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.305 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.306 DEBUG:boto:Data: 2023-10-12T01:11:47.306 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.307 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.307 DEBUG:boto:Params: {} 2023-10-12T01:11:47.308 DEBUG:boto:Token: None 2023-10-12T01:11:47.308 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hPDeex1pACiXSOCftelBgvxfK1I= 2023-10-12T01:11:47.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hPDeex1pACiXSOCftelBgvxfK1I=', 'Content-Length': '0'} 2023-10-12T01:11:47.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000070ff68fb5c8fcdf-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.312 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.312 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.313 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.313 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.313 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.314 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.314 DEBUG:boto:Data: 2023-10-12T01:11:47.315 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.315 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.316 DEBUG:boto:Params: {} 2023-10-12T01:11:47.316 DEBUG:boto:Token: None 2023-10-12T01:11:47.317 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.317 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hPDeex1pACiXSOCftelBgvxfK1I= 2023-10-12T01:11:47.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hPDeex1pACiXSOCftelBgvxfK1I=', 'Content-Length': '0'} 2023-10-12T01:11:47.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003298171c73cafaee-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:11:47.321 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.321 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.322 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.322 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.323 DEBUG:boto:Method: GET 2023-10-12T01:11:47.323 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.324 DEBUG:boto:Data: 2023-10-12T01:11:47.324 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.325 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.325 DEBUG:boto:Params: {} 2023-10-12T01:11:47.326 DEBUG:boto:Token: None 2023-10-12T01:11:47.326 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo= 2023-10-12T01:11:47.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo=', 'Content-Length': '0'} 2023-10-12T01:11:47.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a43b52d9d48df504-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.370 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.371 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.371 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.372 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.372 DEBUG:boto:Method: GET 2023-10-12T01:11:47.373 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.373 DEBUG:boto:Data: 2023-10-12T01:11:47.374 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.375 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.375 DEBUG:boto:Params: {} 2023-10-12T01:11:47.375 DEBUG:boto:Token: None 2023-10-12T01:11:47.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:11:47.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo= 2023-10-12T01:11:47.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P8/qC9jWIuWYnIdhegijskz4pQo=', 'Content-Length': '0'} 2023-10-12T01:11:47.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670da1b1c3b043c1-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.380 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.380 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.381 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.381 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.382 DEBUG:boto:Method: GET 2023-10-12T01:11:47.382 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.383 DEBUG:boto:Data: 2023-10-12T01:11:47.383 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.384 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.384 DEBUG:boto:Params: {} 2023-10-12T01:11:47.385 DEBUG:boto:Token: None 2023-10-12T01:11:47.385 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.386 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006596bca1801adba1-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.391 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.391 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.392 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.392 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.393 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.393 DEBUG:boto:Method: GET 2023-10-12T01:11:47.394 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.394 DEBUG:boto:Data: 2023-10-12T01:11:47.395 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.395 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.396 DEBUG:boto:Params: {} 2023-10-12T01:11:47.397 DEBUG:boto:Token: None 2023-10-12T01:11:47.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f8fe9458430272-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.403 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.404 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.404 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.405 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.405 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:11:47.406 DEBUG:boto:Data: 2023-10-12T01:11:47.406 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.407 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.407 DEBUG:boto:Params: {} 2023-10-12T01:11:47.408 DEBUG:boto:Token: None 2023-10-12T01:11:47.408 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87 2023-10-12T01:11:47.409 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:imlhf+yG3vzUgHZoB7RBMTFFDdQ= 2023-10-12T01:11:47.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:imlhf+yG3vzUgHZoB7RBMTFFDdQ=', 'Content-Length': '0'} 2023-10-12T01:11:47.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce319127f79ed18d-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.412 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.413 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.413 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.413 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:11:47.414 DEBUG:boto:Data: 2023-10-12T01:11:47.414 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.415 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.415 DEBUG:boto:Params: {} 2023-10-12T01:11:47.416 DEBUG:boto:Token: None 2023-10-12T01:11:47.417 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87 2023-10-12T01:11:47.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:imlhf+yG3vzUgHZoB7RBMTFFDdQ= 2023-10-12T01:11:47.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:imlhf+yG3vzUgHZoB7RBMTFFDdQ=', 'Content-Length': '0'} 2023-10-12T01:11:47.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a8e16f6e300041-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:11:47.421 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.421 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.422 DEBUG:boto:Method: GET 2023-10-12T01:11:47.422 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:11:47.423 DEBUG:boto:Data: 2023-10-12T01:11:47.423 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.424 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.425 DEBUG:boto:Params: {} 2023-10-12T01:11:47.425 DEBUG:boto:Token: None 2023-10-12T01:11:47.426 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87 2023-10-12T01:11:47.426 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jtw7sQv+6MnjVV8RZcZmHuN3YNU= 2023-10-12T01:11:47.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jtw7sQv+6MnjVV8RZcZmHuN3YNU=', 'Content-Length': '0'} 2023-10-12T01:11:47.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9feb02117c29882-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.430 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:11:47.430 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:11:47.431 DEBUG:boto:Method: GET 2023-10-12T01:11:47.431 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:11:47.431 DEBUG:boto:Data: 2023-10-12T01:11:47.432 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.433 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.433 DEBUG:boto:Params: {} 2023-10-12T01:11:47.434 DEBUG:boto:Token: None 2023-10-12T01:11:47.434 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj87 2023-10-12T01:11:47.435 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jtw7sQv+6MnjVV8RZcZmHuN3YNU= 2023-10-12T01:11:47.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jtw7sQv+6MnjVV8RZcZmHuN3YNU=', 'Content-Length': '0'} 2023-10-12T01:11:47.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce1f952dbe9e70b-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.438 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.438 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.439 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.439 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.439 DEBUG:boto:Method: GET 2023-10-12T01:11:47.440 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.440 DEBUG:boto:Data: 2023-10-12T01:11:47.441 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.442 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.442 DEBUG:boto:Params: {} 2023-10-12T01:11:47.442 DEBUG:boto:Token: None 2023-10-12T01:11:47.443 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000125c57258f49b5d2-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.447 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.448 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.448 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.449 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.449 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.449 DEBUG:boto:Method: GET 2023-10-12T01:11:47.450 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:11:47.450 DEBUG:boto:Data: 2023-10-12T01:11:47.451 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.451 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.452 DEBUG:boto:Params: {} 2023-10-12T01:11:47.452 DEBUG:boto:Token: None 2023-10-12T01:11:47.453 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.453 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b3c8aed7a996764-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.459 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:11:47.460 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.460 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.460 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.461 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.461 DEBUG:boto:Method: GET 2023-10-12T01:11:47.462 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.462 DEBUG:boto:Data: 2023-10-12T01:11:47.463 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.463 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.464 DEBUG:boto:Params: {} 2023-10-12T01:11:47.464 DEBUG:boto:Token: None 2023-10-12T01:11:47.465 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.465 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24= 2023-10-12T01:11:47.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24=', 'Content-Length': '0'} 2023-10-12T01:11:47.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb417a24829cd46-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.510 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.511 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.511 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.512 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.512 DEBUG:boto:Method: GET 2023-10-12T01:11:47.513 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.513 DEBUG:boto:Data: 2023-10-12T01:11:47.513 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.514 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.515 DEBUG:boto:Params: {} 2023-10-12T01:11:47.515 DEBUG:boto:Token: None 2023-10-12T01:11:47.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24= 2023-10-12T01:11:47.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24=', 'Content-Length': '0'} 2023-10-12T01:11:47.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006873f184061c0f8a-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.520 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.520 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.520 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.521 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.521 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.522 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.522 DEBUG:boto:Data: 2023-10-12T01:11:47.523 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.523 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.524 DEBUG:boto:Params: {} 2023-10-12T01:11:47.524 DEBUG:boto:Token: None 2023-10-12T01:11:47.525 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SSC7mer6jlGIMP2W2BxEUjaPx4= 2023-10-12T01:11:47.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SSC7mer6jlGIMP2W2BxEUjaPx4=', 'Content-Length': '0'} 2023-10-12T01:11:47.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020e11b28b1f8383a-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.529 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.529 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.529 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.530 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.530 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.531 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.531 DEBUG:boto:Data: 2023-10-12T01:11:47.531 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.532 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.533 DEBUG:boto:Params: {} 2023-10-12T01:11:47.533 DEBUG:boto:Token: None 2023-10-12T01:11:47.534 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SSC7mer6jlGIMP2W2BxEUjaPx4= 2023-10-12T01:11:47.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SSC7mer6jlGIMP2W2BxEUjaPx4=', 'Content-Length': '0'} 2023-10-12T01:11:47.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c2f1e33a23c6b4-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:11:47.538 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.538 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.539 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.539 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.540 DEBUG:boto:Method: GET 2023-10-12T01:11:47.540 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.541 DEBUG:boto:Data: 2023-10-12T01:11:47.541 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.542 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.542 DEBUG:boto:Params: {} 2023-10-12T01:11:47.543 DEBUG:boto:Token: None 2023-10-12T01:11:47.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24= 2023-10-12T01:11:47.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24=', 'Content-Length': '0'} 2023-10-12T01:11:47.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000445690340cd29e5-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.547 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.547 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.548 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.548 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.548 DEBUG:boto:Method: GET 2023-10-12T01:11:47.549 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.549 DEBUG:boto:Data: 2023-10-12T01:11:47.550 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.551 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.551 DEBUG:boto:Params: {} 2023-10-12T01:11:47.551 DEBUG:boto:Token: None 2023-10-12T01:11:47.552 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:11:47.552 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24= 2023-10-12T01:11:47.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fGe5fRAcXhLqRKkvdbX4Z2igj24=', 'Content-Length': '0'} 2023-10-12T01:11:47.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028162263e9ffc952-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.598 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.599 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.599 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.600 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.600 DEBUG:boto:Method: GET 2023-10-12T01:11:47.601 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.601 DEBUG:boto:Data: 2023-10-12T01:11:47.602 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.603 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.603 DEBUG:boto:Params: {} 2023-10-12T01:11:47.604 DEBUG:boto:Token: None 2023-10-12T01:11:47.604 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fba42f69c42aa4-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.609 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.609 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.610 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.610 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.611 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.611 DEBUG:boto:Method: GET 2023-10-12T01:11:47.611 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.612 DEBUG:boto:Data: 2023-10-12T01:11:47.612 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.613 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.614 DEBUG:boto:Params: {} 2023-10-12T01:11:47.614 DEBUG:boto:Token: None 2023-10-12T01:11:47.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c002431ea90cd4b-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.620 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.621 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.621 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.622 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.622 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:11:47.623 DEBUG:boto:Data: 2023-10-12T01:11:47.623 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.624 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.624 DEBUG:boto:Params: {} 2023-10-12T01:11:47.625 DEBUG:boto:Token: None 2023-10-12T01:11:47.625 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92 2023-10-12T01:11:47.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpDnRqBh/PBPsBBs8xqNaA2q+U= 2023-10-12T01:11:47.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpDnRqBh/PBPsBBs8xqNaA2q+U=', 'Content-Length': '0'} 2023-10-12T01:11:47.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f783f1fe90fcff16-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.629 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.629 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.630 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.630 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:11:47.631 DEBUG:boto:Data: 2023-10-12T01:11:47.631 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.632 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.632 DEBUG:boto:Params: {} 2023-10-12T01:11:47.633 DEBUG:boto:Token: None 2023-10-12T01:11:47.633 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92 2023-10-12T01:11:47.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpDnRqBh/PBPsBBs8xqNaA2q+U= 2023-10-12T01:11:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpDnRqBh/PBPsBBs8xqNaA2q+U=', 'Content-Length': '0'} 2023-10-12T01:11:47.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008106274971e405fc-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:11:47.637 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.638 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.638 DEBUG:boto:Method: GET 2023-10-12T01:11:47.639 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:11:47.639 DEBUG:boto:Data: 2023-10-12T01:11:47.640 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.640 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.641 DEBUG:boto:Params: {} 2023-10-12T01:11:47.641 DEBUG:boto:Token: None 2023-10-12T01:11:47.642 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92 2023-10-12T01:11:47.642 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YWFFPIRzV6eE35DBIYue6U/3Rik= 2023-10-12T01:11:47.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YWFFPIRzV6eE35DBIYue6U/3Rik=', 'Content-Length': '0'} 2023-10-12T01:11:47.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aacbbec1a184e4d0-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.646 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:11:47.646 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:11:47.647 DEBUG:boto:Method: GET 2023-10-12T01:11:47.647 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:11:47.648 DEBUG:boto:Data: 2023-10-12T01:11:47.648 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.649 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.649 DEBUG:boto:Params: {} 2023-10-12T01:11:47.650 DEBUG:boto:Token: None 2023-10-12T01:11:47.650 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj92 2023-10-12T01:11:47.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YWFFPIRzV6eE35DBIYue6U/3Rik= 2023-10-12T01:11:47.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YWFFPIRzV6eE35DBIYue6U/3Rik=', 'Content-Length': '0'} 2023-10-12T01:11:47.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed2b65b7c3d6077-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.654 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.655 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.655 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.656 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.656 DEBUG:boto:Method: GET 2023-10-12T01:11:47.657 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.657 DEBUG:boto:Data: 2023-10-12T01:11:47.658 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.659 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.659 DEBUG:boto:Params: {} 2023-10-12T01:11:47.660 DEBUG:boto:Token: None 2023-10-12T01:11:47.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ec8f4283b11aeb-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.664 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.665 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.665 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.666 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.666 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.667 DEBUG:boto:Method: GET 2023-10-12T01:11:47.667 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:11:47.668 DEBUG:boto:Data: 2023-10-12T01:11:47.668 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.669 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.670 DEBUG:boto:Params: {} 2023-10-12T01:11:47.670 DEBUG:boto:Token: None 2023-10-12T01:11:47.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d78d6f6b2ee9a52a-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.677 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:11:47.678 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.678 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.679 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.679 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.680 DEBUG:boto:Method: GET 2023-10-12T01:11:47.680 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.681 DEBUG:boto:Data: 2023-10-12T01:11:47.681 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.682 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.683 DEBUG:boto:Params: {} 2023-10-12T01:11:47.683 DEBUG:boto:Token: None 2023-10-12T01:11:47.684 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk= 2023-10-12T01:11:47.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk=', 'Content-Length': '0'} 2023-10-12T01:11:47.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5d78976dba96fb4-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.734 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.735 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.735 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.736 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.736 DEBUG:boto:Method: GET 2023-10-12T01:11:47.737 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.737 DEBUG:boto:Data: 2023-10-12T01:11:47.738 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.739 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.739 DEBUG:boto:Params: {} 2023-10-12T01:11:47.740 DEBUG:boto:Token: None 2023-10-12T01:11:47.740 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk= 2023-10-12T01:11:47.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk=', 'Content-Length': '0'} 2023-10-12T01:11:47.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e97876ae22c40f-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.744 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.745 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.745 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.746 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.746 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.747 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.747 DEBUG:boto:Data: 2023-10-12T01:11:47.747 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.748 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.749 DEBUG:boto:Params: {} 2023-10-12T01:11:47.749 DEBUG:boto:Token: None 2023-10-12T01:11:47.750 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zCPlF/2TWHkqJApoNKfloevG4A= 2023-10-12T01:11:47.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zCPlF/2TWHkqJApoNKfloevG4A=', 'Content-Length': '0'} 2023-10-12T01:11:47.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013823ee949db157-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.754 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.754 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.755 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.755 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.755 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.756 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.756 DEBUG:boto:Data: 2023-10-12T01:11:47.757 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.758 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.758 DEBUG:boto:Params: {} 2023-10-12T01:11:47.759 DEBUG:boto:Token: None 2023-10-12T01:11:47.759 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0zCPlF/2TWHkqJApoNKfloevG4A= 2023-10-12T01:11:47.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0zCPlF/2TWHkqJApoNKfloevG4A=', 'Content-Length': '0'} 2023-10-12T01:11:47.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac5c180ef2f4f66-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:11:47.763 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.763 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.764 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.764 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.765 DEBUG:boto:Method: GET 2023-10-12T01:11:47.765 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.766 DEBUG:boto:Data: 2023-10-12T01:11:47.766 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.767 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.767 DEBUG:boto:Params: {} 2023-10-12T01:11:47.768 DEBUG:boto:Token: None 2023-10-12T01:11:47.769 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.769 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk= 2023-10-12T01:11:47.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk=', 'Content-Length': '0'} 2023-10-12T01:11:47.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4230199fbf30a7a-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.772 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.773 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.773 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.774 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.774 DEBUG:boto:Method: GET 2023-10-12T01:11:47.775 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.775 DEBUG:boto:Data: 2023-10-12T01:11:47.776 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.777 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.777 DEBUG:boto:Params: {} 2023-10-12T01:11:47.778 DEBUG:boto:Token: None 2023-10-12T01:11:47.778 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:11:47.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk= 2023-10-12T01:11:47.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sUGdwPojc9EX/6rLb0JFvdOvTCk=', 'Content-Length': '0'} 2023-10-12T01:11:47.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ff12010bf40d4d-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.782 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.782 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.783 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.783 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.784 DEBUG:boto:Method: GET 2023-10-12T01:11:47.784 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.785 DEBUG:boto:Data: 2023-10-12T01:11:47.785 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.786 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.787 DEBUG:boto:Params: {} 2023-10-12T01:11:47.787 DEBUG:boto:Token: None 2023-10-12T01:11:47.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.788 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000df637e107c7ef7-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.792 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.793 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.793 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.794 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.795 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.795 DEBUG:boto:Method: GET 2023-10-12T01:11:47.796 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.796 DEBUG:boto:Data: 2023-10-12T01:11:47.796 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.797 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.798 DEBUG:boto:Params: {} 2023-10-12T01:11:47.798 DEBUG:boto:Token: None 2023-10-12T01:11:47.799 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073542656e5f358b0-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.805 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.805 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.806 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.806 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.807 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:11:47.807 DEBUG:boto:Data: 2023-10-12T01:11:47.808 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.809 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.809 DEBUG:boto:Params: {} 2023-10-12T01:11:47.810 DEBUG:boto:Token: None 2023-10-12T01:11:47.810 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93 2023-10-12T01:11:47.811 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GXCI/+ZRC8AAlwStsqJhucuRZQ4= 2023-10-12T01:11:47.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GXCI/+ZRC8AAlwStsqJhucuRZQ4=', 'Content-Length': '0'} 2023-10-12T01:11:47.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d11af9753f9a489-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.814 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.814 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.814 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.815 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:11:47.815 DEBUG:boto:Data: 2023-10-12T01:11:47.816 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.817 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.817 DEBUG:boto:Params: {} 2023-10-12T01:11:47.818 DEBUG:boto:Token: None 2023-10-12T01:11:47.818 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93 2023-10-12T01:11:47.818 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GXCI/+ZRC8AAlwStsqJhucuRZQ4= 2023-10-12T01:11:47.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GXCI/+ZRC8AAlwStsqJhucuRZQ4=', 'Content-Length': '0'} 2023-10-12T01:11:47.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000034996d69b75a286-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:11:47.822 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.822 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.823 DEBUG:boto:Method: GET 2023-10-12T01:11:47.823 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:11:47.824 DEBUG:boto:Data: 2023-10-12T01:11:47.824 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.825 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.826 DEBUG:boto:Params: {} 2023-10-12T01:11:47.826 DEBUG:boto:Token: None 2023-10-12T01:11:47.827 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93 2023-10-12T01:11:47.827 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7eZEQLkDQ0IOlFbPkWnQNbg/rqE= 2023-10-12T01:11:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7eZEQLkDQ0IOlFbPkWnQNbg/rqE=', 'Content-Length': '0'} 2023-10-12T01:11:47.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a763b388e7b3b97d-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.830 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:11:47.830 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:11:47.831 DEBUG:boto:Method: GET 2023-10-12T01:11:47.831 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:11:47.832 DEBUG:boto:Data: 2023-10-12T01:11:47.832 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.833 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.833 DEBUG:boto:Params: {} 2023-10-12T01:11:47.834 DEBUG:boto:Token: None 2023-10-12T01:11:47.834 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj93 2023-10-12T01:11:47.838 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7eZEQLkDQ0IOlFbPkWnQNbg/rqE= 2023-10-12T01:11:47.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7eZEQLkDQ0IOlFbPkWnQNbg/rqE=', 'Content-Length': '0'} 2023-10-12T01:11:47.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4aed38510c8e98e-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.882 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.883 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.883 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.884 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.884 DEBUG:boto:Method: GET 2023-10-12T01:11:47.885 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.885 DEBUG:boto:Data: 2023-10-12T01:11:47.886 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.887 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.887 DEBUG:boto:Params: {} 2023-10-12T01:11:47.887 DEBUG:boto:Token: None 2023-10-12T01:11:47.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000598c9f95d85145cf-00652747d3-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.893 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.894 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:47.894 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:47.895 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.895 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.895 DEBUG:boto:Method: GET 2023-10-12T01:11:47.896 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:11:47.896 DEBUG:boto:Data: 2023-10-12T01:11:47.897 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.898 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.898 DEBUG:boto:Params: {} 2023-10-12T01:11:47.899 DEBUG:boto:Token: None 2023-10-12T01:11:47.899 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/ 2023-10-12T01:11:47.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68= 2023-10-12T01:11:47.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nJcn10uDAyiPymK2WQPD3ZzWD68=', 'Content-Length': '0'} 2023-10-12T01:11:47.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d5bd562faf6a55-00652747d3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.905 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:47.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:11:47.906 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.906 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.906 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.907 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.907 DEBUG:boto:Method: GET 2023-10-12T01:11:47.908 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.908 DEBUG:boto:Data: 2023-10-12T01:11:47.909 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.910 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.910 DEBUG:boto:Params: {} 2023-10-12T01:11:47.911 DEBUG:boto:Token: None 2023-10-12T01:11:47.911 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.912 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws= 2023-10-12T01:11:47.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws=', 'Content-Length': '0'} 2023-10-12T01:11:47.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c7fae7633362be-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.958 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.959 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.959 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.960 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.960 DEBUG:boto:Method: GET 2023-10-12T01:11:47.961 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.961 DEBUG:boto:Data: 2023-10-12T01:11:47.962 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.963 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.963 DEBUG:boto:Params: {} 2023-10-12T01:11:47.964 DEBUG:boto:Token: None 2023-10-12T01:11:47.964 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws= 2023-10-12T01:11:47.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws=', 'Content-Length': '0'} 2023-10-12T01:11:47.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13b7d47ec5175eb-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.968 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.969 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.969 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.970 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.970 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.970 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.971 DEBUG:boto:Data: 2023-10-12T01:11:47.971 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.972 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.972 DEBUG:boto:Params: {} 2023-10-12T01:11:47.973 DEBUG:boto:Token: None 2023-10-12T01:11:47.973 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.974 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Evz2nlLFvMY7cr5QGn+FL4DwJ2g= 2023-10-12T01:11:47.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Evz2nlLFvMY7cr5QGn+FL4DwJ2g=', 'Content-Length': '0'} 2023-10-12T01:11:47.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691d0001b96478fe-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.977 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.977 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.978 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.978 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.978 DEBUG:boto:Method: HEAD 2023-10-12T01:11:47.979 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.979 DEBUG:boto:Data: 2023-10-12T01:11:47.980 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.980 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.981 DEBUG:boto:Params: {} 2023-10-12T01:11:47.981 DEBUG:boto:Token: None 2023-10-12T01:11:47.981 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Evz2nlLFvMY7cr5QGn+FL4DwJ2g= 2023-10-12T01:11:47.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Evz2nlLFvMY7cr5QGn+FL4DwJ2g=', 'Content-Length': '0'} 2023-10-12T01:11:47.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001471b01c15fd5e4-00652747d3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:11:47.986 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.986 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.986 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.987 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.987 DEBUG:boto:Method: GET 2023-10-12T01:11:47.988 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.988 DEBUG:boto:Data: 2023-10-12T01:11:47.989 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:47.989 DEBUG:boto:Port: 8001 2023-10-12T01:11:47.990 DEBUG:boto:Params: {} 2023-10-12T01:11:47.990 DEBUG:boto:Token: None 2023-10-12T01:11:47.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:47 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws= 2023-10-12T01:11:47.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QXLR33F+DJ6vFVbjqVPEmHak+ws=', 'Content-Length': '0'} 2023-10-12T01:11:47.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9387b34b17dd70-00652747d3-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:47.995 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:47.995 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:47.995 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.996 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.996 DEBUG:boto:Method: GET 2023-10-12T01:11:47.997 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:47.997 DEBUG:boto:Data: 2023-10-12T01:11:47.998 DEBUG:boto:Headers: {} 2023-10-12T01:11:47.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:47.999 DEBUG:boto:Port: 8000 2023-10-12T01:11:47.999 DEBUG:boto:Params: {} 2023-10-12T01:11:47.999 DEBUG:boto:Token: None 2023-10-12T01:11:48.000 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:11:48.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Czlg45myqUcmxAvrp8ZXvIRyIUQ= 2023-10-12T01:11:48.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Czlg45myqUcmxAvrp8ZXvIRyIUQ=', 'Content-Length': '0'} 2023-10-12T01:11:48.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d405b7f536cd376f-00652747d4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.003 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:48.004 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:48.004 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.005 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.005 DEBUG:boto:Method: GET 2023-10-12T01:11:48.006 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.006 DEBUG:boto:Data: 2023-10-12T01:11:48.006 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:48.007 DEBUG:boto:Port: 8001 2023-10-12T01:11:48.008 DEBUG:boto:Params: {} 2023-10-12T01:11:48.008 DEBUG:boto:Token: None 2023-10-12T01:11:48.009 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/ 2023-10-12T01:11:48.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE= 2023-10-12T01:11:48.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE=', 'Content-Length': '0'} 2023-10-12T01:11:48.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021bcf826739bd2e2-00652747d4-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.013 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:48.014 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:48.014 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:48.015 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.015 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.016 DEBUG:boto:Method: GET 2023-10-12T01:11:48.016 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.016 DEBUG:boto:Data: 2023-10-12T01:11:48.017 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:48.018 DEBUG:boto:Port: 8000 2023-10-12T01:11:48.018 DEBUG:boto:Params: {} 2023-10-12T01:11:48.019 DEBUG:boto:Token: None 2023-10-12T01:11:48.019 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/ 2023-10-12T01:11:48.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE= 2023-10-12T01:11:48.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE=', 'Content-Length': '0'} 2023-10-12T01:11:48.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f344b129f8877fce-00652747d4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.066 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:48.067 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:48.068 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:48.068 DEBUG:boto:Method: HEAD 2023-10-12T01:11:48.068 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:11:48.069 DEBUG:boto:Data: 2023-10-12T01:11:48.069 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:48.070 DEBUG:boto:Port: 8001 2023-10-12T01:11:48.070 DEBUG:boto:Params: {} 2023-10-12T01:11:48.071 DEBUG:boto:Token: None 2023-10-12T01:11:48.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/obj95 2023-10-12T01:11:48.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YJgtzXu/Ip4IK/KtdcJl6iIAwNI= 2023-10-12T01:11:48.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YJgtzXu/Ip4IK/KtdcJl6iIAwNI=', 'Content-Length': '0'} 2023-10-12T01:11:48.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010aed239c81cc587-00652747d4-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.075 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:48.076 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:48.076 DEBUG:boto:Method: HEAD 2023-10-12T01:11:48.077 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:11:48.077 DEBUG:boto:Data: 2023-10-12T01:11:48.077 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:48.078 DEBUG:boto:Port: 8000 2023-10-12T01:11:48.079 DEBUG:boto:Params: {} 2023-10-12T01:11:48.079 DEBUG:boto:Token: None 2023-10-12T01:11:48.080 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/obj95 2023-10-12T01:11:48.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YJgtzXu/Ip4IK/KtdcJl6iIAwNI= 2023-10-12T01:11:48.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YJgtzXu/Ip4IK/KtdcJl6iIAwNI=', 'Content-Length': '0'} 2023-10-12T01:11:48.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e32887f1689d98-00652747d4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:11:48.084 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:48.084 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:48.085 DEBUG:boto:Method: GET 2023-10-12T01:11:48.085 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:11:48.085 DEBUG:boto:Data: 2023-10-12T01:11:48.086 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:48.087 DEBUG:boto:Port: 8001 2023-10-12T01:11:48.087 DEBUG:boto:Params: {} 2023-10-12T01:11:48.088 DEBUG:boto:Token: None 2023-10-12T01:11:48.088 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/obj95 2023-10-12T01:11:48.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LKctN+9npZ0LnaL8SyJHjpaeuFY= 2023-10-12T01:11:48.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LKctN+9npZ0LnaL8SyJHjpaeuFY=', 'Content-Length': '0'} 2023-10-12T01:11:48.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1e7b5985efc6fe-00652747d4-1bba-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.092 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:11:48.092 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:11:48.093 DEBUG:boto:Method: GET 2023-10-12T01:11:48.093 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:11:48.093 DEBUG:boto:Data: 2023-10-12T01:11:48.094 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:48.095 DEBUG:boto:Port: 8000 2023-10-12T01:11:48.095 DEBUG:boto:Params: {} 2023-10-12T01:11:48.096 DEBUG:boto:Token: None 2023-10-12T01:11:48.096 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/obj95 2023-10-12T01:11:48.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LKctN+9npZ0LnaL8SyJHjpaeuFY= 2023-10-12T01:11:48.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LKctN+9npZ0LnaL8SyJHjpaeuFY=', 'Content-Length': '0'} 2023-10-12T01:11:48.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd6c67213a576ff-00652747d4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.100 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:48.100 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:48.101 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.101 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.101 DEBUG:boto:Method: GET 2023-10-12T01:11:48.102 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.102 DEBUG:boto:Data: 2023-10-12T01:11:48.103 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:11:48.104 DEBUG:boto:Port: 8001 2023-10-12T01:11:48.104 DEBUG:boto:Params: {} 2023-10-12T01:11:48.104 DEBUG:boto:Token: None 2023-10-12T01:11:48.105 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/ 2023-10-12T01:11:48.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE= 2023-10-12T01:11:48.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE=', 'Content-Length': '0'} 2023-10-12T01:11:48.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003ea0ab1651fefd3-00652747d4-1bba-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.154 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:48.155 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:11:48.156 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:11:48.156 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.157 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.157 DEBUG:boto:Method: GET 2023-10-12T01:11:48.157 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:11:48.158 DEBUG:boto:Data: 2023-10-12T01:11:48.158 DEBUG:boto:Headers: {} 2023-10-12T01:11:48.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:11:48.159 DEBUG:boto:Port: 8000 2023-10-12T01:11:48.160 DEBUG:boto:Params: {} 2023-10-12T01:11:48.160 DEBUG:boto:Token: None 2023-10-12T01:11:48.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:11:48 GMT /wafmoa-59/ 2023-10-12T01:11:48.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE= 2023-10-12T01:11:48.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:11:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ggrEJCnFslN7Vxy88SOrWMO3VHE=', 'Content-Length': '0'} 2023-10-12T01:11:48.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cfc80a8f6f8234-00652747d4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:11:48.166 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:11:48.167 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:11:48.168 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:11:48.168 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:11:48.169 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:11:48.168+0000 7f4e59db0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 161885) UID: 0 2023-10-12T01:11:48.170 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:11:48.168+0000 7f4e5e0197c0 -1 shutting down 2023-10-12T01:11:53.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:11:54.267 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:11:54.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:11:54.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:11:54.269 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:11:54.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:54.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:54.364 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:11:54.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.364 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-10-12T01:11:54.365 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.365 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.365 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.366 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.366 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.366 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.366 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.367 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.157780Z", 2023-10-12T01:11:54.367 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.367 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.367 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.368 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.368 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.368 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.368 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.369 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.369 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_MNrlq7IghrCQIJSy0bDM1J8K-BuZrOh", 2023-10-12T01:11:54.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.370 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.370 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-10-12T01:11:54.371 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.371 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.371 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.371 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.372 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.372 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.372 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.372 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.372 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.373 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.024750Z", 2023-10-12T01:11:54.373 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.373 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.373 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.373 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.374 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.374 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.374 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.374 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.375 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_p3-c8YlNVOOaMXuLyGsSpfixgb3NBGN", 2023-10-12T01:11:54.375 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.375 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.376 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-10-12T01:11:54.376 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.376 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.377 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.377 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.377 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.378 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.378 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.378 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.378 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.378 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:06.039944Z", 2023-10-12T01:11:54.379 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.379 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.379 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.379 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.379 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.380 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.380 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.380 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.380 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_F1beHHTYOzUkHz6rFBAct-rrvs6Rd29", 2023-10-12T01:11:54.381 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.381 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.381 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.382 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-10-12T01:11:54.382 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.382 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.383 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:11:54.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.383 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.383 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.383 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.384 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.384 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.384 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.399566Z", 2023-10-12T01:11:54.384 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.384 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.385 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.385 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.385 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.385 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.385 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.386 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.386 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_HPEuk9a0r0SIkMPwD3YMu9EDebAYcai", 2023-10-12T01:11:54.386 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-10-12T01:11:54.388 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.388 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.388 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.388 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:11:54.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.408556Z", 2023-10-12T01:11:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_NnFiOIXtZuUQLw57JCGbaARc461W0kp", 2023-10-12T01:11:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-10-12T01:11:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.418345Z", 2023-10-12T01:11:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_oUrV3zVhYAFI0gRhB1YsLCeWGbQqERH", 2023-10-12T01:11:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-10-12T01:11:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.401 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.401 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.401 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.046351Z", 2023-10-12T01:11:54.401 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.401 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.402 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.403 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.406 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_NxpjeJXRw5RBGy0tXBoaiLhxwRDQD_f", 2023-10-12T01:11:54.406 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.406 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.406 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.407 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-10-12T01:11:54.407 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.407 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.408 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.409 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.409 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.409 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.409 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.468156Z", 2023-10-12T01:11:54.409 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.410 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.410 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.410 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.410 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.410 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.411 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.411 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.411 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_eC0DMV7Ah9dlf-R1OjcHaKlv77Slkm5", 2023-10-12T01:11:54.411 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.412 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.413 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-10-12T01:11:54.413 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.413 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.413 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.413 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.414 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.414 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.414 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.414 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.415 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.415 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.917926Z", 2023-10-12T01:11:54.415 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.415 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.415 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.416 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.416 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.416 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.416 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_s1u2zKZjoeIZjd9uommJ0PxqtTSrjsw", 2023-10-12T01:11:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.418 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.418 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-10-12T01:11:54.418 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.419 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.419 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:11:54.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.419 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.420 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.420 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.420 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.420 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.420 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.107246Z", 2023-10-12T01:11:54.421 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.421 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.421 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.421 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.422 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.422 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.422 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.422 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.423 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_MSx5LN2ENVpeOXGYmGOpt5AlVJalewh", 2023-10-12T01:11:54.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.423 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.424 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-10-12T01:11:54.424 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.424 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.425 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.425 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.425 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.426 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.426 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.426 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.426 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.855778Z", 2023-10-12T01:11:54.426 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.427 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.427 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.427 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.428 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.428 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.428 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.429 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_FXzx8lXj7QaS1aRU5AmIyjLDjVNcFLv", 2023-10-12T01:11:54.429 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.429 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.429 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.430 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-10-12T01:11:54.430 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.430 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.431 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.431 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.431 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.431 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.432 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.432 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.432 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.432 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.178423Z", 2023-10-12T01:11:54.432 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.433 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.433 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.433 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.433 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.433 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.434 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.434 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.434 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_uDAW56sddF8t3pJgfOVUOWa9Kvb2MU6", 2023-10-12T01:11:54.434 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.435 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.435 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.435 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-10-12T01:11:54.436 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.436 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.436 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:11:54.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.437 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.437 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.437 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.437 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.438 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.438 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.882871Z", 2023-10-12T01:11:54.438 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.438 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.438 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.439 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.439 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.439 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.439 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.439 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.440 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_qny08PCvE0d06jnLWIk3ETzEpUQjxod", 2023-10-12T01:11:54.440 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.440 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.441 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.441 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-10-12T01:11:54.441 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.442 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.442 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.442 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.442 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.443 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.443 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.443 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.443 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.443 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.261203Z", 2023-10-12T01:11:54.444 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.444 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.444 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.444 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.444 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.445 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.445 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.445 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.446 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_LkEnDiq94tI6T78tcSAbG2Eqs1u9nKi", 2023-10-12T01:11:54.446 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.446 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.446 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.447 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-10-12T01:11:54.447 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.447 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.447 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.448 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.448 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.448 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.448 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.449 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.449 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.449 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.310094Z", 2023-10-12T01:11:54.449 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.449 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.450 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.450 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.450 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.450 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.451 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.451 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.451 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_FYC8byJhYAhuwHNssnbo0g1qMu3MjUE", 2023-10-12T01:11:54.451 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.452 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.452 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-10-12T01:11:54.453 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.453 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.453 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.454 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.454 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.454 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.454 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.454 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.455 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.241163Z", 2023-10-12T01:11:54.455 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.455 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.455 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.456 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.456 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.456 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.456 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.456 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.457 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_h3Pp9YLjCZZkJFNdmjgj52-iUFNBhVH", 2023-10-12T01:11:54.457 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.457 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.458 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-10-12T01:11:54.458 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.458 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.459 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:11:54.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.459 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.459 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.460 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.460 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.460 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.460 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.976899Z", 2023-10-12T01:11:54.460 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.461 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.461 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.461 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.461 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.461 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.462 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.462 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.462 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_QOB7b7j-q83_MRFJ_ofOD6v6P1NxPEd", 2023-10-12T01:11:54.463 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.463 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.464 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-10-12T01:11:54.464 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.464 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.464 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.465 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.465 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.465 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.465 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.942974Z", 2023-10-12T01:11:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_xHNEtERWhgNUOOxluFhLqotMiOdXlD9", 2023-10-12T01:11:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-10-12T01:11:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:11:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.911454Z", 2023-10-12T01:11:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.474 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_-bAlPDoibbpILKX84IboqaQbL4Y1yry", 2023-10-12T01:11:54.474 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.474 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.475 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-10-12T01:11:54.475 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.475 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.476 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.476 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.476 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.476 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.477 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.477 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.477 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.262594Z", 2023-10-12T01:11:54.477 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.478 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.478 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.478 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.478 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.478 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.479 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.479 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.479 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_oPWmQG0HN2_UisGrNa-iqEeO4jzRE_4", 2023-10-12T01:11:54.479 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.480 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.480 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-10-12T01:11:54.481 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.481 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.481 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.482 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.482 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.482 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.482 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.482 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.483 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.902370Z", 2023-10-12T01:11:54.483 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.483 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.483 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.484 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.484 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.484 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.484 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.484 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_HF_JLcK5eZItZkmMqlG6Z3RHnDx1xYF", 2023-10-12T01:11:54.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.485 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.485 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-10-12T01:11:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.486 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.487 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.487 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.487 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.487 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.488 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.488 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.488 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.488 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.996512Z", 2023-10-12T01:11:54.488 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.489 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.489 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.489 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.489 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.490 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.490 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.490 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.490 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CLJ-8rS0uay2xdy_XeB4l8r_e4CxvIZ", 2023-10-12T01:11:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-10-12T01:11:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.493 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.493 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.493 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.493 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.494 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.494 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.281973Z", 2023-10-12T01:11:54.494 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.494 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.494 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.495 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.495 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.495 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.495 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.496 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.496 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_c23HSsKVGNieqtfaaowtDcGeerD6eJ5", 2023-10-12T01:11:54.496 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.497 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.497 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-10-12T01:11:54.497 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.498 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.498 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:11:54.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.498 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.499 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.499 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.499 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.499 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.500 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.929197Z", 2023-10-12T01:11:54.500 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.500 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.500 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.500 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.501 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.501 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.501 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.501 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.502 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_alParVFkhUuLkT-g5CVJALZ4ugRsKY7", 2023-10-12T01:11:54.502 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.502 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.503 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-10-12T01:11:54.503 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.503 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.504 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.504 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.504 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.504 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.323573Z", 2023-10-12T01:11:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.506 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.506 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.506 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.506 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.506 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.507 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.507 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.507 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_w3rQSjW6YVp4rAmESfFP8pqIwp38Urh", 2023-10-12T01:11:54.507 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.508 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.509 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-10-12T01:11:54.509 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.509 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.509 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.510 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.510 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.510 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.510 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.511 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.511 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.329916Z", 2023-10-12T01:11:54.511 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.511 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.511 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.512 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.512 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.512 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.512 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.512 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.513 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_XoolRt3yFytd5hzdE0vOjzGfQJvOdg4", 2023-10-12T01:11:54.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.513 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.514 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-10-12T01:11:54.514 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.514 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.515 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.515 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:11:54.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.515 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.515 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.516 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.516 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.516 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.516 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.058773Z", 2023-10-12T01:11:54.516 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.517 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.517 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.517 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.517 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.517 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.518 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.518 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.518 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_AHV9Pa6uofPcXkYl8FZhXFOBedbaYU6", 2023-10-12T01:11:54.518 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.519 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.519 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.520 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-10-12T01:11:54.520 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.520 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.520 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.520 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.521 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.521 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.521 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.521 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.522 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.522 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.959193Z", 2023-10-12T01:11:54.522 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.522 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.522 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_9-FuLFrNHz8kqMHELtCoJhR2ZzwA1ro", 2023-10-12T01:11:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-10-12T01:11:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.962302Z", 2023-10-12T01:11:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ZhdUpzYnj5zDoV0wBMrjVb9bK0aUX1J", 2023-10-12T01:11:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-10-12T01:11:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:11:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:11:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.992575Z", 2023-10-12T01:11:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ATXiGYzugvrejqWfEs-MBSCZWZPwJJL", 2023-10-12T01:11:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.536 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.536 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-10-12T01:11:54.536 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.537 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.537 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:11:54.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.537 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.538 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.538 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.538 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.538 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.539 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.078457Z", 2023-10-12T01:11:54.539 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.539 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.539 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.539 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.540 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.540 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.540 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.540 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_wued8YrePQLvH9vHaLZrCkdO2I5gxyj", 2023-10-12T01:11:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.542 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-10-12T01:11:54.542 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.542 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.543 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:11:54.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.543 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.543 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.545 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.545 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.545 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:11:54.545 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.341888Z", 2023-10-12T01:11:54.546 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.546 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.546 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.546 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.547 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.547 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.547 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.547 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.548 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_F0vQ1XDr_T0mrXNXaQl1fiIv9bGpHWJ", 2023-10-12T01:11:54.548 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.548 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.549 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-10-12T01:11:54.549 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.549 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.550 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:11:54.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.550 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.550 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.550 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.551 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.551 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:11:54.551 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.979824Z", 2023-10-12T01:11:54.551 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.552 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.552 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.552 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.552 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.552 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.553 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.553 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.553 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_L4VsPeZdOa6gknJxE8LdtXFBlbNLm0u", 2023-10-12T01:11:54.553 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.554 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.554 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-10-12T01:11:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:11:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.388674Z", 2023-10-12T01:11:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_PpFnz5nur41LxbxfSrfZnaY6Wa6k7zA", 2023-10-12T01:11:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:11:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-10-12T01:11:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:11:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:11:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:11:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:11:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:11:54.561 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:11:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:11:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:11:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:11:54.562 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:07.016570Z", 2023-10-12T01:11:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:11:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:11:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:11:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:11:54.563 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:11:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:11:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:11:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:11:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:11:54.564 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_96Fapwnda5LefQIgE5mmYF-bF-k73U1", 2023-10-12T01:11:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:11:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:11:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:11:54.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:11:54.565 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:11:54.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.567 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:54.622 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:54.623 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:54.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.710 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:54.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:54.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:54.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.846 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:54.898 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:54.899 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:54.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:54.983 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.036 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.036 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.123 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.176 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.260 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.399 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.536 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.588 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.673 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.724 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.725 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.811 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:55.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:55.946 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:55.996 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:55.997 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.082 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.221 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.361 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.498 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.551 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.551 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.637 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.688 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.688 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.773 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:56.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:56.908 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:56.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:56.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.047 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.098 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.098 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.185 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.236 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.322 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.376 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.464 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.603 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.739 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:57.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:57.876 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:57.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:57.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.013 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.150 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.288 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.339 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.425 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.563 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.700 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.753 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.753 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.840 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:58.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:11:58.892 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:58.892 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:58.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:58.979 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.030 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.030 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.117 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.168 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.254 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.305 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.393 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.442 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.493 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:11:59.592 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:11:59.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:11:59.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:11:59.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.712+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-10-12T01:11:59.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.712+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:11:59.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.718+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: init 2023-10-12T01:11:59.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.730+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:11:59.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:11:59.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:11:59.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:11:59.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:11:59.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:11:59.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-12T01:11:59.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:11:59.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:11:59.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:11:59.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:11:59.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:11:59.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:11:59.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj116[0] 2023-10-12T01:11:59.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:11:59.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:11:59.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj124[0] 2023-10-12T01:11:59.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:11:59.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:11:59.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj145[0] 2023-10-12T01:11:59.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:11:59.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:11:59.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj148[0] 2023-10-12T01:11:59.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:11:59.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:11:59.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-12T01:11:59.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:11:59.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:11:59.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:11:59.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj17[0] 2023-10-12T01:11:59.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:11:59.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:11:59.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:11:59.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:11:59.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:11:59.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:11:59.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:11:59.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:11:59.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-10-12T01:11:59.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:11:59.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:11:59.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:11:59.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:11:59.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:11:59.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:11:59.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-12T01:11:59.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:11:59.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.777+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:11:59.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:11:59.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:11:59.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:11:59.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:11:59.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:11:59.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:11:59.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:11:59.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-10-12T01:11:59.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:11:59.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:11:59.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:11:59.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:11:59.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:11:59.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:11:59.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:11:59.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:11:59.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:11:59.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:11:59.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:11:59.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:11:59.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.778+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:11:59.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:11:59.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:11:59.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:11:59.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:11:59.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:11:59.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj55[0] 2023-10-12T01:11:59.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:11:59.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:11:59.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj57[0] 2023-10-12T01:11:59.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:11:59.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:11:59.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:11:59.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:11:59.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:11:59.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.781+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj62[0] 2023-10-12T01:11:59.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: success 2023-10-12T01:11:59.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:11:59.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:11:59.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj68[0] 2023-10-12T01:11:59.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:11:59.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:11:59.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:11:59.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:11:59.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:11:59.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:11:59.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:11:59.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:11:59.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:11:59.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:11:59.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:11:59.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:11:59.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj76[0] 2023-10-12T01:11:59.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:11:59.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:11:59.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.802+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj85[0] 2023-10-12T01:11:59.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:11:59.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:11:59.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:11:59.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:11:59.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj87[0] 2023-10-12T01:11:59.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:11:59.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:11:59.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.803+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:11:59.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.804+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: success 2023-10-12T01:11:59.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.804+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:11:59.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.804+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:11:59.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.804+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj93[0] 2023-10-12T01:11:59.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: success 2023-10-12T01:11:59.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:11:59.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: success 2023-10-12T01:11:59.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: success 2023-10-12T01:11:59.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:11:59.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:11:59.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:11:59.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.806+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:11:59.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.807+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:11:59.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.807+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj11 2023-10-12T01:11:59.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.807+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:11:59.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.807+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:11:59.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.833+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: success 2023-10-12T01:11:59.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.833+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:11:59.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.834+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: success 2023-10-12T01:11:59.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.835+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: success 2023-10-12T01:11:59.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.836+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:11:59.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.847+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: success 2023-10-12T01:11:59.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.849+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: success 2023-10-12T01:11:59.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.849+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:11:59.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.850+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: success 2023-10-12T01:11:59.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.852+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:11:59.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.853+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: success 2023-10-12T01:11:59.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.853+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:11:59.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.854+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:11:59.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.855+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: success 2023-10-12T01:11:59.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.855+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: backing out with sync_status=-5 2023-10-12T01:11:59.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.855+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: ERROR: full sync failed. error: -5 2023-10-12T01:11:59.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.857+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-12T01:11:59.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.857+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:11:59.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.861+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: full 2023-10-12T01:11:59.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.873+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:11:59.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:11:59.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:11:59.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:11:59.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:11:59.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:11:59.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj111[0] 2023-10-12T01:11:59.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:11:59.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:11:59.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:11:59.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:11:59.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:11:59.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj116[0] 2023-10-12T01:11:59.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:11:59.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:11:59.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj124[0] 2023-10-12T01:11:59.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:11:59.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:11:59.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj145[0] 2023-10-12T01:11:59.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:11:59.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:11:59.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj148[0] 2023-10-12T01:11:59.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:11:59.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:11:59.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj155[0] 2023-10-12T01:11:59.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:11:59.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:11:59.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj17[0] 2023-10-12T01:11:59.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:11:59.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:11:59.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:11:59.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:11:59.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:11:59.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:11:59.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:11:59.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:11:59.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj22[0] 2023-10-12T01:11:59.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:11:59.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:11:59.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:11:59.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:11:59.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:11:59.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj29[0] 2023-10-12T01:11:59.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:11:59.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:11:59.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:11:59.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:11:59.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:11:59.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:11:59.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:11:59.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:11:59.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj34[0] 2023-10-12T01:11:59.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:11:59.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:11:59.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:11:59.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:11:59.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:11:59.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.918+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:11:59.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:11:59.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:11:59.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:11:59.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:11:59.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:11:59.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.919+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:11:59.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: success 2023-10-12T01:11:59.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:11:59.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:11:59.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:11:59.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:11:59.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:11:59.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:11:59.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:11:59.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:11:59.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj55[0] 2023-10-12T01:11:59.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:11:59.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:11:59.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj57[0] 2023-10-12T01:11:59.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.922+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:11:59.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:11:59.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:11:59.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: success 2023-10-12T01:11:59.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:11:59.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:11:59.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj62[0] 2023-10-12T01:11:59.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: success 2023-10-12T01:11:59.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:11:59.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:11:59.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-10-12T01:11:59.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:11:59.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:11:59.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:11:59.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:11:59.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.923+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:11:59.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: success 2023-10-12T01:11:59.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:11:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:11:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:11:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:11:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:11:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:11:59.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-10-12T01:11:59.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:11:59.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:11:59.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:11:59.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:11:59.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.925+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj85[0] 2023-10-12T01:11:59.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:11:59.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:11:59.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:11:59.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-12T01:11:59.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: success 2023-10-12T01:11:59.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:11:59.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:11:59.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:11:59.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:11:59.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:11:59.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:11:59.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:11:59.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:11:59.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:11:59.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj93[0] 2023-10-12T01:11:59.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:11:59.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:11:59.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:11:59.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.926+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:11:59.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.927+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.928+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:11:59.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.928+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:11:59.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.928+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:11:59.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.928+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: success 2023-10-12T01:11:59.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.928+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:11:59.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.929+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: success 2023-10-12T01:11:59.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.929+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.930+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: success 2023-10-12T01:11:59.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.930+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: success 2023-10-12T01:11:59.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.930+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:11:59.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.931+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:11:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.931+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:11:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.932+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: success 2023-10-12T01:11:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.932+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: success 2023-10-12T01:11:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.933+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:11:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.933+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: success 2023-10-12T01:11:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.933+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj148 2023-10-12T01:11:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.933+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: success 2023-10-12T01:11:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.934+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: success 2023-10-12T01:11:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.934+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: success 2023-10-12T01:11:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.935+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj62 2023-10-12T01:11:59.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.936+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: backing out with sync_status=-5 2023-10-12T01:11:59.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.936+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: ERROR: full sync failed. error: -5 2023-10-12T01:11:59.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.937+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-12T01:11:59.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.937+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:11:59.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.982+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: full 2023-10-12T01:11:59.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:11:59.994+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:12:00.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:12:00.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:12:00.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:12:00.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:12:00.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:12:00.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj111[0] 2023-10-12T01:12:00.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:12:00.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:12:00.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:12:00.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:12:00.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:12:00.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-10-12T01:12:00.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:12:00.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:12:00.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:12:00.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj124[0] 2023-10-12T01:12:00.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:12:00.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:12:00.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj145[0] 2023-10-12T01:12:00.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:12:00.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:12:00.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj148[0] 2023-10-12T01:12:00.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:12:00.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:12:00.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-12T01:12:00.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:12:00.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:12:00.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:12:00.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.039+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj17[0] 2023-10-12T01:12:00.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:12:00.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:12:00.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:12:00.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:12:00.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:12:00.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:12:00.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:12:00.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:12:00.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj22[0] 2023-10-12T01:12:00.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:12:00.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:12:00.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:12:00.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:12:00.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:12:00.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj29[0] 2023-10-12T01:12:00.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:12:00.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:12:00.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:12:00.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:12:00.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:12:00.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:12:00.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:12:00.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:12:00.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj34[0] 2023-10-12T01:12:00.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:12:00.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:12:00.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:12:00.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:12:00.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:12:00.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:12:00.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:12:00.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:12:00.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:12:00.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:12:00.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:12:00.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.040+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:12:00.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:12:00.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: success 2023-10-12T01:12:00.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: success 2023-10-12T01:12:00.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:12:00.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:12:00.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:12:00.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:12:00.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:12:00.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj55[0] 2023-10-12T01:12:00.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:12:00.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:12:00.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj57[0] 2023-10-12T01:12:00.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: success 2023-10-12T01:12:00.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:12:00.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:12:00.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.044+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:12:00.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:12:00.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: success 2023-10-12T01:12:00.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:12:00.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:12:00.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:12:00.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:12:00.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-10-12T01:12:00.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:12:00.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:12:00.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:12:00.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj68[0] 2023-10-12T01:12:00.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:12:00.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:12:00.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:12:00.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:12:00.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:12:00.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:12:00.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:12:00.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:12:00.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: success 2023-10-12T01:12:00.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: success 2023-10-12T01:12:00.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: success 2023-10-12T01:12:00.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:12:00.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:12:00.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:12:00.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj76[0] 2023-10-12T01:12:00.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:12:00.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-12T01:12:00.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:12:00.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:12:00.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-12T01:12:00.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:12:00.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:12:00.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:12:00.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:12:00.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:12:00.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:12:00.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-10-12T01:12:00.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:12:00.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.045+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:12:00.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.046+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:12:00.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.046+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:12:00.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:12:00.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: success 2023-10-12T01:12:00.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:12:00.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:12:00.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.047+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:12:00.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.048+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: success 2023-10-12T01:12:00.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:12:00.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:12:00.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.049+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: success 2023-10-12T01:12:00.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:12:00.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:12:00.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: success 2023-10-12T01:12:00.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: success 2023-10-12T01:12:00.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:12:00.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:12:00.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.050+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj115 2023-10-12T01:12:00.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.052+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: backing out with sync_status=-5 2023-10-12T01:12:00.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.052+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: ERROR: full sync failed. error: -5 2023-10-12T01:12:00.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.053+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.053+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:12:00.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.098+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: full 2023-10-12T01:12:00.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.109+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:12:00.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:12:00.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:12:00.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:12:00.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:12:00.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:12:00.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-12T01:12:00.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:12:00.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:12:00.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:12:00.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:12:00.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:12:00.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:12:00.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj116[0] 2023-10-12T01:12:00.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:12:00.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:12:00.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-10-12T01:12:00.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:12:00.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:12:00.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:12:00.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj145[0] 2023-10-12T01:12:00.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:12:00.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:12:00.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-10-12T01:12:00.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:12:00.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:12:00.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:12:00.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj155[0] 2023-10-12T01:12:00.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:12:00.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.155+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:12:00.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-10-12T01:12:00.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:12:00.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:12:00.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:12:00.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:12:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:12:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:12:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:12:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:12:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:12:00.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj22[0] 2023-10-12T01:12:00.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:12:00.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:12:00.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:12:00.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:12:00.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:12:00.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj29[0] 2023-10-12T01:12:00.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:12:00.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:12:00.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:12:00.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:12:00.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:12:00.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:12:00.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:12:00.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:12:00.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj34[0] 2023-10-12T01:12:00.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:12:00.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:12:00.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:12:00.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:12:00.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:12:00.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:12:00.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:12:00.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:12:00.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:12:00.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:12:00.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:12:00.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.156+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:12:00.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: success 2023-10-12T01:12:00.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:12:00.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:12:00.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:12:00.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:12:00.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:12:00.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-10-12T01:12:00.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: success 2023-10-12T01:12:00.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:12:00.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:12:00.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:12:00.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-10-12T01:12:00.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:12:00.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:12:00.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:12:00.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.160+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:12:00.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: success 2023-10-12T01:12:00.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:12:00.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:12:00.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: success 2023-10-12T01:12:00.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:12:00.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:12:00.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:12:00.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj62[0] 2023-10-12T01:12:00.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:12:00.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:12:00.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: success 2023-10-12T01:12:00.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj68[0] 2023-10-12T01:12:00.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:12:00.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:12:00.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:12:00.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:12:00.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:12:00.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:12:00.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:12:00.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:12:00.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj76[0] 2023-10-12T01:12:00.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:12:00.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj85[0] 2023-10-12T01:12:00.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:12:00.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:12:00.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:12:00.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj87[0] 2023-10-12T01:12:00.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:12:00.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:12:00.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:12:00.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:12:00.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:12:00.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj93[0] 2023-10-12T01:12:00.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:12:00.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:12:00.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:12:00.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.161+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:12:00.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:12:00.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:12:00.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:12:00.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:12:00.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: success 2023-10-12T01:12:00.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.163+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:12:00.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.164+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.164+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.165+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:12:00.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.165+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:12:00.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.165+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: success 2023-10-12T01:12:00.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: success 2023-10-12T01:12:00.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: success 2023-10-12T01:12:00.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: success 2023-10-12T01:12:00.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:12:00.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: success 2023-10-12T01:12:00.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:12:00.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.166+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:12:00.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.167+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:12:00.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.167+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: success 2023-10-12T01:12:00.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.167+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:12:00.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.167+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj11 2023-10-12T01:12:00.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.168+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: backing out with sync_status=-5 2023-10-12T01:12:00.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.168+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: ERROR: full sync failed. error: -5 2023-10-12T01:12:00.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.169+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.169+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:12:00.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.213+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: full 2023-10-12T01:12:00.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.224+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:12:00.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:12:00.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:12:00.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:12:00.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:12:00.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:12:00.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj111[0] 2023-10-12T01:12:00.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:12:00.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:12:00.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:12:00.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:12:00.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:12:00.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj116[0] 2023-10-12T01:12:00.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:12:00.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:12:00.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj124[0] 2023-10-12T01:12:00.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:12:00.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:12:00.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-10-12T01:12:00.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:12:00.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:12:00.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:12:00.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj148[0] 2023-10-12T01:12:00.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:12:00.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:12:00.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj155[0] 2023-10-12T01:12:00.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:12:00.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:12:00.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj17[0] 2023-10-12T01:12:00.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:12:00.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:12:00.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:12:00.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:12:00.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:12:00.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:12:00.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:12:00.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:12:00.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj22[0] 2023-10-12T01:12:00.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:12:00.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:12:00.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:12:00.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.270+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:12:00.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:12:00.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-12T01:12:00.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:12:00.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:12:00.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:12:00.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:12:00.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:12:00.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:12:00.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:12:00.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:12:00.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:12:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj34[0] 2023-10-12T01:12:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:12:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:12:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:12:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:12:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:12:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:12:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:12:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:12:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:12:00.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:12:00.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:12:00.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.271+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:12:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:12:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:12:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:12:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:12:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:12:00.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:12:00.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj55[0] 2023-10-12T01:12:00.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:12:00.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: success 2023-10-12T01:12:00.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: success 2023-10-12T01:12:00.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: success 2023-10-12T01:12:00.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.275+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:12:00.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:12:00.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:12:00.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: success 2023-10-12T01:12:00.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: success 2023-10-12T01:12:00.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj57[0] 2023-10-12T01:12:00.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:12:00.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:12:00.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:12:00.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:12:00.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:12:00.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:12:00.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj62[0] 2023-10-12T01:12:00.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:12:00.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:12:00.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj68[0] 2023-10-12T01:12:00.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:12:00.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:12:00.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:12:00.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:12:00.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:12:00.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:12:00.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:12:00.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:12:00.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj76[0] 2023-10-12T01:12:00.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:12:00.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-12T01:12:00.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-12T01:12:00.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 0 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: ERROR: failed to sync object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:12:00.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:12:00.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.276+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj87[0] 2023-10-12T01:12:00.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.277+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: success 2023-10-12T01:12:00.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.277+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:12:00.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.277+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:12:00.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.277+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:12:00.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:12:00.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: success 2023-10-12T01:12:00.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj124 2023-10-12T01:12:00.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:12:00.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:12:00.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: success 2023-10-12T01:12:00.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:12:00.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:12:00.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:12:00.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: success 2023-10-12T01:12:00.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:12:00.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:12:00.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:12:00.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj93[0] 2023-10-12T01:12:00.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:12:00.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:12:00.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:12:00.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.279+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: a sync operation returned error 2023-10-12T01:12:00.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.280+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:12:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.280+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:12:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.280+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: success 2023-10-12T01:12:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.281+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:12:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.281+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: success 2023-10-12T01:12:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.281+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:12:00.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:12:00.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:12:00.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: success 2023-10-12T01:12:00.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: success 2023-10-12T01:12:00.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: success 2023-10-12T01:12:00.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: success 2023-10-12T01:12:00.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: backing out with sync_status=-5 2023-10-12T01:12:00.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.282+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: ERROR: full sync failed. error: -5 2023-10-12T01:12:00.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.284+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.284+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: waiting on bucket lease 2023-10-12T01:12:00.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.329+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: status after acquiring the lock is: full 2023-10-12T01:12:00.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.341+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bucket for full sync 2023-10-12T01:12:00.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj11 2023-10-12T01:12:00.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj11 op=0 op_state=1 2023-10-12T01:12:00.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj11[0] 2023-10-12T01:12:00.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj111 2023-10-12T01:12:00.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj111 op=0 op_state=1 2023-10-12T01:12:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj111[0] 2023-10-12T01:12:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj115 2023-10-12T01:12:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj115 op=0 op_state=1 2023-10-12T01:12:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj115[0] 2023-10-12T01:12:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj116 2023-10-12T01:12:00.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj116 op=0 op_state=1 2023-10-12T01:12:00.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj116[0] 2023-10-12T01:12:00.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj124 2023-10-12T01:12:00.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj124 op=0 op_state=1 2023-10-12T01:12:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj124[0] 2023-10-12T01:12:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj145 2023-10-12T01:12:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj145 op=0 op_state=1 2023-10-12T01:12:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj145[0] 2023-10-12T01:12:00.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj148 2023-10-12T01:12:00.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj148 op=0 op_state=1 2023-10-12T01:12:00.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj148[0] 2023-10-12T01:12:00.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj155 2023-10-12T01:12:00.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj155 op=0 op_state=1 2023-10-12T01:12:00.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj155[0] 2023-10-12T01:12:00.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj17 2023-10-12T01:12:00.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj17 op=0 op_state=1 2023-10-12T01:12:00.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj17[0] 2023-10-12T01:12:00.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj185 2023-10-12T01:12:00.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj185 op=0 op_state=1 2023-10-12T01:12:00.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.387+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj185[0] 2023-10-12T01:12:00.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj19 2023-10-12T01:12:00.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj19 op=0 op_state=1 2023-10-12T01:12:00.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj19[0] 2023-10-12T01:12:00.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj22 2023-10-12T01:12:00.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj22 op=0 op_state=1 2023-10-12T01:12:00.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj22[0] 2023-10-12T01:12:00.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj23 2023-10-12T01:12:00.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj23 op=0 op_state=1 2023-10-12T01:12:00.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj23[0] 2023-10-12T01:12:00.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj29 2023-10-12T01:12:00.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj29 op=0 op_state=1 2023-10-12T01:12:00.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj29[0] 2023-10-12T01:12:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj31 2023-10-12T01:12:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj31 op=0 op_state=1 2023-10-12T01:12:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj31[0] 2023-10-12T01:12:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj33 2023-10-12T01:12:00.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj33 op=0 op_state=1 2023-10-12T01:12:00.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj33[0] 2023-10-12T01:12:00.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj34 2023-10-12T01:12:00.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj34 op=0 op_state=1 2023-10-12T01:12:00.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj34[0] 2023-10-12T01:12:00.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj37 2023-10-12T01:12:00.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj37 op=0 op_state=1 2023-10-12T01:12:00.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj37[0] 2023-10-12T01:12:00.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj38 2023-10-12T01:12:00.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj38 op=0 op_state=1 2023-10-12T01:12:00.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj38[0] 2023-10-12T01:12:00.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj44 2023-10-12T01:12:00.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj44 op=0 op_state=1 2023-10-12T01:12:00.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj44[0] 2023-10-12T01:12:00.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj46 2023-10-12T01:12:00.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj46 op=0 op_state=1 2023-10-12T01:12:00.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.388+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj46[0] 2023-10-12T01:12:00.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj155]: success 2023-10-12T01:12:00.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj51 2023-10-12T01:12:00.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj51 op=0 op_state=1 2023-10-12T01:12:00.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj51[0] 2023-10-12T01:12:00.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj11]: success 2023-10-12T01:12:00.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj55 2023-10-12T01:12:00.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj55 op=0 op_state=1 2023-10-12T01:12:00.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj55[0] 2023-10-12T01:12:00.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj19]: success 2023-10-12T01:12:00.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj116]: success 2023-10-12T01:12:00.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj57 2023-10-12T01:12:00.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj57 op=0 op_state=1 2023-10-12T01:12:00.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj57[0] 2023-10-12T01:12:00.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj111]: success 2023-10-12T01:12:00.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj58 2023-10-12T01:12:00.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj58 op=0 op_state=1 2023-10-12T01:12:00.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj58[0] 2023-10-12T01:12:00.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj148]: success 2023-10-12T01:12:00.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj62 2023-10-12T01:12:00.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj62 op=0 op_state=1 2023-10-12T01:12:00.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj62[0] 2023-10-12T01:12:00.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj68 2023-10-12T01:12:00.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj68 op=0 op_state=1 2023-10-12T01:12:00.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.391+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj68[0] 2023-10-12T01:12:00.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj31]: success 2023-10-12T01:12:00.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj69 2023-10-12T01:12:00.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj69 op=0 op_state=1 2023-10-12T01:12:00.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj69[0] 2023-10-12T01:12:00.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj185]: success 2023-10-12T01:12:00.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj145]: success 2023-10-12T01:12:00.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj34]: success 2023-10-12T01:12:00.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj74 2023-10-12T01:12:00.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj74 op=0 op_state=1 2023-10-12T01:12:00.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj23]: success 2023-10-12T01:12:00.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj74[0] 2023-10-12T01:12:00.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj115]: success 2023-10-12T01:12:00.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.393+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj116 2023-10-12T01:12:00.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj76 2023-10-12T01:12:00.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj76 op=0 op_state=1 2023-10-12T01:12:00.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj76[0] 2023-10-12T01:12:00.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj85 2023-10-12T01:12:00.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj85 op=0 op_state=1 2023-10-12T01:12:00.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj85[0] 2023-10-12T01:12:00.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj87 2023-10-12T01:12:00.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj87 op=0 op_state=1 2023-10-12T01:12:00.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj87[0] 2023-10-12T01:12:00.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj46]: success 2023-10-12T01:12:00.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj92 2023-10-12T01:12:00.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj92 op=0 op_state=1 2023-10-12T01:12:00.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj92[0] 2023-10-12T01:12:00.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj17]: success 2023-10-12T01:12:00.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj124]: success 2023-10-12T01:12:00.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj93 2023-10-12T01:12:00.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj93 op=0 op_state=1 2023-10-12T01:12:00.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.394+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj93[0] 2023-10-12T01:12:00.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.395+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: [full sync] syncing object: wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0/obj95 2023-10-12T01:12:00.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.395+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b=bucket sync single entry (source_zone=677f5a1d-3ab4-4b5f-af15-9890c2c79648) b= log_entry=obj95 op=0 op_state=1 2023-10-12T01:12:00.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.395+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: bucket sync: sync obj: 677f5a1d-3ab4-4b5f-af15-9890c2c79648/:wafmoa-59[677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24])/obj95[0] 2023-10-12T01:12:00.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj44]: success 2023-10-12T01:12:00.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj58]: success 2023-10-12T01:12:00.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj22]: success 2023-10-12T01:12:00.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj38]: success 2023-10-12T01:12:00.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj29]: success 2023-10-12T01:12:00.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj51]: success 2023-10-12T01:12:00.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj55]: success 2023-10-12T01:12:00.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj68]: success 2023-10-12T01:12:00.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj62]: success 2023-10-12T01:12:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.397+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj76]: success 2023-10-12T01:12:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj33]: success 2023-10-12T01:12:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj34 2023-10-12T01:12:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj37]: success 2023-10-12T01:12:00.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj74]: success 2023-10-12T01:12:00.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj57]: success 2023-10-12T01:12:00.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj69]: success 2023-10-12T01:12:00.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.399+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj92]: success 2023-10-12T01:12:00.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.401+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj93]: success 2023-10-12T01:12:00.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.401+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj87]: success 2023-10-12T01:12:00.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.401+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj95]: success 2023-10-12T01:12:00.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.401+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:entry[obj85]: success 2023-10-12T01:12:00.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.401+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: updating marker oid=bucket.full-sync-status.677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24 marker=obj95 2023-10-12T01:12:00.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.402+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: set bucket state=incremental 2023-10-12T01:12:00.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.403+0000 7f3f252f5780 5 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:full_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: bucket status objv={r=_fUh8fcuioO6yRDF17hsfgvr:154,w=:0} 2023-10-12T01:12:00.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.404+0000 7f3f252f5780 10 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-10-12T01:12:00.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:1]: sync status for source bucket shard: 2 2023-10-12T01:12:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:2]: sync status for source bucket shard: 2 2023-10-12T01:12:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:1]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:1]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:2]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:2]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: sync status for source bucket shard: 2 2023-10-12T01:12:00.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.406+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:0]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:5]: sync status for source bucket shard: 2 2023-10-12T01:12:00.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:5]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:5]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:4]: sync status for source bucket shard: 2 2023-10-12T01:12:00.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:4]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:4]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:3]: sync status for source bucket shard: 2 2023-10-12T01:12:00.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:7]: sync status for source bucket shard: 2 2023-10-12T01:12:00.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:3]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:3]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.407+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:7]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:7]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:9]: sync status for source bucket shard: 2 2023-10-12T01:12:00.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.408+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:9]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:9]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:6]: sync status for source bucket shard: 2 2023-10-12T01:12:00.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:6]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:6]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-12T01:12:00.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:10]: sync status for source bucket shard: 2 2023-10-12T01:12:00.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:8]: sync status for source bucket shard: 2 2023-10-12T01:12:00.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:10]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:10]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.409+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:8]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:8]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:12]: sync status for source bucket shard: 2 2023-10-12T01:12:00.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:12]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:12]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-12T01:12:00.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:16]: sync status for source bucket shard: 2 2023-10-12T01:12:00.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:16]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:16]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:11]: sync status for source bucket shard: 2 2023-10-12T01:12:00.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:11]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:11]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.410+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:14]: sync status for source bucket shard: 2 2023-10-12T01:12:00.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:14]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:14]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:13]: sync status for source bucket shard: 2 2023-10-12T01:12:00.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.411+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:13]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:13]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:15]: sync status for source bucket shard: 2 2023-10-12T01:12:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:15]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:15]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:17]: sync status for source bucket shard: 2 2023-10-12T01:12:00.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:17]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:17]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-12T01:12:00.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:18]: sync status for source bucket shard: 2 2023-10-12T01:12:00.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:21]: sync status for source bucket shard: 2 2023-10-12T01:12:00.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:23]: sync status for source bucket shard: 2 2023-10-12T01:12:00.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:18]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:18]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:21]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:21]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.412+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:23]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:23]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.413+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:26]: sync status for source bucket shard: 2 2023-10-12T01:12:00.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.413+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:27]: sync status for source bucket shard: 2 2023-10-12T01:12:00.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.413+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:26]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:26]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.413+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:27]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:27]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:24]: sync status for source bucket shard: 2 2023-10-12T01:12:00.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-12T01:12:00.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:24]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:24]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:22]: sync status for source bucket shard: 2 2023-10-12T01:12:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:20]: sync status for source bucket shard: 2 2023-10-12T01:12:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:19]: sync status for source bucket shard: 2 2023-10-12T01:12:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:28]: sync status for source bucket shard: 2 2023-10-12T01:12:00.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:30]: sync status for source bucket shard: 2 2023-10-12T01:12:00.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:25]: sync status for source bucket shard: 2 2023-10-12T01:12:00.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:22]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:22]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:20]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:20]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:19]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:19]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:28]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:28]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:30]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:30]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.415+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:25]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:25]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:31]: sync status for source bucket shard: 2 2023-10-12T01:12:00.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:31]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:31]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:35]: sync status for source bucket shard: 2 2023-10-12T01:12:00.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:32]: sync status for source bucket shard: 2 2023-10-12T01:12:00.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:29]: sync status for source bucket shard: 2 2023-10-12T01:12:00.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:34]: sync status for source bucket shard: 2 2023-10-12T01:12:00.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:33]: sync status for source bucket shard: 2 2023-10-12T01:12:00.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:35]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:35]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:32]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:32]: listing bilog for incremental sync; position=00000000001.103.6 2023-10-12T01:12:00.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:29]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:29]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:34]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:34]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.416+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:33]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:33]: listing bilog for incremental sync; position= 2023-10-12T01:12:00.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.417+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:36]: sync status for source bucket shard: 2 2023-10-12T01:12:00.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:00.417+0000 7f3f252f5780 20 RGW-SYNC:bucket[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24<-wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:36]:inc_sync[wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24:36]: listing bilog for incremental sync; position=00000000001.63.6 2023-10-12T01:12:00.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:00.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:00.483 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:00.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:00.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:00.606 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:12:00.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.606 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-10-12T01:12:00.607 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.607 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.607 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14 2023-10-12T01:12:00.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.608 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.608 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.608 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.608 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.608 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.609 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.157780Z", 2023-10-12T01:12:00.609 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.609 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.609 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.610 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.610 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.610 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.610 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.610 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.611 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2592139258007170756", 2023-10-12T01:12:00.611 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.611 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-10-12T01:12:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.024750Z", 2023-10-12T01:12:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11877022889081297194", 2023-10-12T01:12:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-10-12T01:12:00.620 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.620 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.620 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.622 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.622 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:06.039944Z", 2023-10-12T01:12:00.623 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.623 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.623 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.623 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.624 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.624 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.624 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.624 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.625 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.13933454649129462816", 2023-10-12T01:12:00.625 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.625 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.626 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.626 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-10-12T01:12:00.627 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.627 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.627 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.627 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:12:00.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.628 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.628 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.628 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.629 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.629 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.629 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.399566Z", 2023-10-12T01:12:00.629 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.630 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.630 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.630 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.630 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.631 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.631 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.631 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.631 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6694525580810272860", 2023-10-12T01:12:00.632 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.632 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.632 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.633 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-10-12T01:12:00.634 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.634 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.634 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.634 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 24 2023-10-12T01:12:00.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.635 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.635 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.636 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.636 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.636 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.636 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.408556Z", 2023-10-12T01:12:00.636 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.638 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.638 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.640 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12973748437009951978", 2023-10-12T01:12:00.640 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.640 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.640 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.641 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-10-12T01:12:00.641 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.641 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.641 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.642 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 15 2023-10-12T01:12:00.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.642 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.643 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.643 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.643 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.643 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.644 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.418345Z", 2023-10-12T01:12:00.644 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.644 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.644 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.645 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.645 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.645 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.645 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.645 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.646 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1271260889866353554", 2023-10-12T01:12:00.646 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.646 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.647 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.647 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-10-12T01:12:00.648 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.648 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.648 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.648 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T01:12:00.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.649 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.649 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.649 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.649 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.650 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.650 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.046351Z", 2023-10-12T01:12:00.650 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.651 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.651 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.651 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.651 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.651 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.652 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.652 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.652 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3485141866209445256", 2023-10-12T01:12:00.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.653 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.653 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.654 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-10-12T01:12:00.654 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.655 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.655 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.655 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:00.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.656 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.656 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.657 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.657 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.657 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.657 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.468156Z", 2023-10-12T01:12:00.658 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.658 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.658 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.658 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.659 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.659 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.659 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.659 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.660 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4221640778943778204", 2023-10-12T01:12:00.660 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.660 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.661 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-10-12T01:12:00.662 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.662 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.662 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.662 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:00.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.663 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.663 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.663 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.663 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.664 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.664 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.917926Z", 2023-10-12T01:12:00.664 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.665 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.665 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.665 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.666 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.666 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.666 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.667 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.668 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.14143311835852705565", 2023-10-12T01:12:00.668 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.668 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.669 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.670 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-10-12T01:12:00.671 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.671 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.671 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.672 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:00.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.672 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.673 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.673 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.674 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.674 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.674 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.107246Z", 2023-10-12T01:12:00.675 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.675 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.675 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.676 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.676 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.676 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.676 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.677 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.11611584838665253565", 2023-10-12T01:12:00.677 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.678 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.678 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-10-12T01:12:00.679 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.679 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.679 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T01:12:00.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.680 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.680 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.680 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.680 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.681 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.681 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.855778Z", 2023-10-12T01:12:00.681 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.681 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.681 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.682 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.682 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.682 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.682 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.682 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.683 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3560898687786948291", 2023-10-12T01:12:00.683 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.684 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.684 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-10-12T01:12:00.685 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.685 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.685 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.685 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:12:00.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.686 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.686 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.686 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.686 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.687 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.687 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.178423Z", 2023-10-12T01:12:00.687 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.687 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.687 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.688 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.688 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.688 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.688 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.688 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.689 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3619049624187039047", 2023-10-12T01:12:00.689 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.689 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.690 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.690 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-10-12T01:12:00.691 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.691 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.691 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.691 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:12:00.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.692 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.692 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.692 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.692 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.882871Z", 2023-10-12T01:12:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.694 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.694 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.694 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.694 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.12008031639559681034", 2023-10-12T01:12:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-10-12T01:12:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:12:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.261203Z", 2023-10-12T01:12:00.699 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.699 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.699 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4756152829973061865", 2023-10-12T01:12:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-10-12T01:12:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.702 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.703 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.310094Z", 2023-10-12T01:12:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.705 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.706 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1579236090235395604", 2023-10-12T01:12:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.707 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-10-12T01:12:00.708 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:12:00.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.710 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.711 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.241163Z", 2023-10-12T01:12:00.712 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.713 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.714 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.714 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.715 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.715 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.716 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1444114799079231560", 2023-10-12T01:12:00.716 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.716 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.717 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.718 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-10-12T01:12:00.718 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.719 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.719 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:00.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.720 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.720 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.720 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.720 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.720 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.721 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.976899Z", 2023-10-12T01:12:00.721 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.721 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.721 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.721 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.722 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.722 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.722 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.722 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.723 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.2645414554475994123", 2023-10-12T01:12:00.723 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.723 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.724 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.725 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-10-12T01:12:00.725 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.725 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.726 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:00.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.727 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.727 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.728 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.728 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.728 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.729 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.942974Z", 2023-10-12T01:12:00.729 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.730 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.730 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.14745539189000186545", 2023-10-12T01:12:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-10-12T01:12:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:12:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.911454Z", 2023-10-12T01:12:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10210260545199997619", 2023-10-12T01:12:00.740 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.740 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.741 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-10-12T01:12:00.741 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.742 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:00.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.742 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.742 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.743 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.743 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.743 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.743 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.262594Z", 2023-10-12T01:12:00.743 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.745 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.745 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.745 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16379494606549080812", 2023-10-12T01:12:00.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.746 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-10-12T01:12:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14 2023-10-12T01:12:00.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.748 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.748 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.749 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.749 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.749 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.749 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.902370Z", 2023-10-12T01:12:00.749 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.750 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.750 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.750 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.750 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.751 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.751 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.751 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.752 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.13610117564116482136", 2023-10-12T01:12:00.752 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.752 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.753 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-10-12T01:12:00.753 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.753 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.754 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.754 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:00.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.754 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.755 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.755 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.755 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.755 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.756 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.996512Z", 2023-10-12T01:12:00.756 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.756 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.756 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.757 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.757 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.757 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.758 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.758 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.759 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4559439387560688702", 2023-10-12T01:12:00.759 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.759 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.759 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.760 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-10-12T01:12:00.760 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.760 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.761 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.761 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14 2023-10-12T01:12:00.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.761 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.762 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.762 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.762 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.762 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.762 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.281973Z", 2023-10-12T01:12:00.763 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.763 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.763 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.763 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.764 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.764 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.764 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.764 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.765 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.4044585313011842691", 2023-10-12T01:12:00.765 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.765 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.766 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.766 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-10-12T01:12:00.767 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.767 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.767 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.767 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:00.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.768 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.768 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.768 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.768 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.769 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.769 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.929197Z", 2023-10-12T01:12:00.769 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.769 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.770 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.770 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.770 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.770 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.771 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.771 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.771 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.6529161792397836512", 2023-10-12T01:12:00.773 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-10-12T01:12:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.323573Z", 2023-10-12T01:12:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.781 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.781 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.781 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.781 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.782 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.782 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17480405257048305411", 2023-10-12T01:12:00.782 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.783 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.783 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.784 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-10-12T01:12:00.784 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.784 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.785 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:12:00.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.785 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.785 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.786 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.786 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.786 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.786 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.329916Z", 2023-10-12T01:12:00.786 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.787 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.787 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.787 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.787 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.787 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.788 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.788 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.788 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.7455523427823994896", 2023-10-12T01:12:00.789 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.789 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.789 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.790 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-10-12T01:12:00.790 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.791 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.791 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.792 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-10-12T01:12:00.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.792 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.793 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.793 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.794 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.794 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.794 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.058773Z", 2023-10-12T01:12:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.13963188991943526736", 2023-10-12T01:12:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-10-12T01:12:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:12:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.959193Z", 2023-10-12T01:12:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.3689626088403963099", 2023-10-12T01:12:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-10-12T01:12:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18 2023-10-12T01:12:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.962302Z", 2023-10-12T01:12:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.810 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.550021988649436209", 2023-10-12T01:12:00.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.811 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.811 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-10-12T01:12:00.811 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.812 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.812 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-10-12T01:12:00.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.813 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.813 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.813 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.813 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.813 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.814 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.992575Z", 2023-10-12T01:12:00.814 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.814 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.814 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.816 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.17690180922506835021", 2023-10-12T01:12:00.816 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.816 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.817 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-10-12T01:12:00.817 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.817 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.819 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-10-12T01:12:00.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.820 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.820 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.820 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.821 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.821 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.821 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.078457Z", 2023-10-12T01:12:00.822 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.822 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.822 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.823 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.823 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.823 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.824 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.824 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.825 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.1030257118933543736", 2023-10-12T01:12:00.825 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.826 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.827 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-10-12T01:12:00.827 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.827 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.828 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-10-12T01:12:00.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.828 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.828 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.829 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.829 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.829 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.829 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.341888Z", 2023-10-12T01:12:00.829 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.830 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.830 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.830 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.830 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.831 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.678214507033587789", 2023-10-12T01:12:00.832 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.832 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.832 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.833 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-10-12T01:12:00.833 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.833 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.834 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.834 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:12:00.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.834 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.834 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.835 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.835 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.835 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.835 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.979824Z", 2023-10-12T01:12:00.835 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.836 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.836 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.836 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.836 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.10571637572764553796", 2023-10-12T01:12:00.837 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.838 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.838 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.839 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-10-12T01:12:00.839 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.839 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.839 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.839 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:00.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.840 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.841 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.841 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.841 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.841 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.841 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.388674Z", 2023-10-12T01:12:00.842 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.842 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.842 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.842 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.843 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.843 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.843 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.843 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.844 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.16668725484505498093", 2023-10-12T01:12:00.844 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.844 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.845 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-10-12T01:12:00.845 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.846 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-10-12T01:12:00.846 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:00.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.847 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.847 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.847 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.847 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.847 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:07.016570Z", 2023-10-12T01:12:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "677f5a1d-3ab4-4b5f-af15-9890c2c79648.5773.8265801879011119376", 2023-10-12T01:12:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:00.851 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:12:00.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:00.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:00.852 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:00.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:00.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:00.938 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:12:00.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.939 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-10-12T01:12:00.940 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.940 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.941 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.941 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:00.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.942 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.942 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.942 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.943 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.943 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.943 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.157780Z", 2023-10-12T01:12:00.944 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.944 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.944 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.945 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.945 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.945 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.946 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.946 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.946 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fi3OJbxKX6Lg7SgliVFbpbzZ846bl6E", 2023-10-12T01:12:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.948 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-10-12T01:12:00.948 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.948 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.948 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.948 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T01:12:00.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.949 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.949 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.949 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.949 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.950 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.950 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.024750Z", 2023-10-12T01:12:00.950 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.950 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.951 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.951 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.951 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.951 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.951 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.952 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.952 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fWmo8wIbUyvD5Z0Pf6wws9Ctr5Rik-p", 2023-10-12T01:12:00.952 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.953 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-10-12T01:12:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.954 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.955 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.956 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:06.039944Z", 2023-10-12T01:12:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.957 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.958 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.958 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.960 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_l8XHEwERWSDjsfRW_KHuCAQH7APv6R0", 2023-10-12T01:12:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.961 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-10-12T01:12:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.963 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.964 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.965 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.966 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.399566Z", 2023-10-12T01:12:00.967 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.968 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.968 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.969 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.969 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.970 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.970 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.970 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.971 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ciw2cV2_sU_2J4oQlm7NGczq65ebeIV", 2023-10-12T01:12:00.971 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.972 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.972 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.973 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-10-12T01:12:00.974 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.974 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.975 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.975 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:00.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.977 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.978 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.978 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.979 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.979 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:00.979 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.408556Z", 2023-10-12T01:12:00.980 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.980 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.981 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.981 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.981 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.982 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.982 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.983 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.984 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_rTot6GYxF0Z3DTgrTZdNEnFkKuuUptl", 2023-10-12T01:12:00.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.985 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.985 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-10-12T01:12:00.986 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.986 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.986 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:00.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.987 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.987 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.987 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.987 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.988 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:00.988 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.418345Z", 2023-10-12T01:12:00.988 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.988 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.988 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.989 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.989 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.989 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.989 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.990 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.990 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_lzHdyROJt-JMa3kfwHr9JniuS6T083W", 2023-10-12T01:12:00.990 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.991 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.991 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-10-12T01:12:00.992 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.992 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.992 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:00.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.993 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.993 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.993 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:00.993 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:00.994 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:00.994 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.046351Z", 2023-10-12T01:12:00.994 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:00.994 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:00.995 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:00.995 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:00.995 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:00.995 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:00.996 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:00.996 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:00.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.996 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_R7hOMWpv9NkVAs5grTn31ClRtyPLPMw", 2023-10-12T01:12:00.997 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:00.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:00.997 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:00.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:00.998 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-10-12T01:12:00.998 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:00.998 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:00.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:00.999 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:00.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:00.999 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:00.999 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:00.999 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.000 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.000 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.000 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.468156Z", 2023-10-12T01:12:01.000 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.001 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.001 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.001 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.001 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.001 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.002 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.002 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.002 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_5KaAfgHVyZtnurAW5Elz1v0AsPq-SZk", 2023-10-12T01:12:01.003 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.003 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.003 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.004 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-10-12T01:12:01.004 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.004 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.004 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.005 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.005 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.005 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.005 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.006 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.006 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.006 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.917926Z", 2023-10-12T01:12:01.006 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.007 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.007 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.007 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.007 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.007 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.008 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.008 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.008 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_MRyy1MYob1mT6Rp2QtMs4_u7BSFNgU2", 2023-10-12T01:12:01.008 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.009 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.009 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-10-12T01:12:01.010 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.010 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.010 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.010 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.011 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:30.107246Z", 2023-10-12T01:12:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.012 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.013 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_J-oCm__-FqXkWbmd9-nFnVvijCMv8MV", 2023-10-12T01:12:01.014 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-10-12T01:12:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.016 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.017 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.855778Z", 2023-10-12T01:12:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.018 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.019 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.020 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_YZJnTcZfV2OVlWnnwfCdsfy2n7C-Cgg", 2023-10-12T01:12:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-10-12T01:12:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-10-12T01:12:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.023 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.024 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.178423Z", 2023-10-12T01:12:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.025 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.026 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.026 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.026 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.027 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_o4QkgnAj08YvXM3F6YjLsZmzwgwlJme", 2023-10-12T01:12:01.027 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.027 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-10-12T01:12:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:12:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.882871Z", 2023-10-12T01:12:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_cPpdnmlRXc2QcDbA1xDKim4ZUP_YL6c", 2023-10-12T01:12:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-10-12T01:12:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.261203Z", 2023-10-12T01:12:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_1_6BNtyrIgmE2eBHuccpleMNWymHMzr", 2023-10-12T01:12:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-10-12T01:12:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.310094Z", 2023-10-12T01:12:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.044 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.044 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.044 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.045 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_1QH7Z3LEbDlCNFTYRMggMH6O7WxGwJ9", 2023-10-12T01:12:01.045 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.045 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.046 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-10-12T01:12:01.046 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.046 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.047 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.047 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:01.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.047 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.047 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.048 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.048 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.048 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.048 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.241163Z", 2023-10-12T01:12:01.048 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.049 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.050 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.050 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.050 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fcwpZv9vwTxiUiZJP6wMuNj1kl4dAyR", 2023-10-12T01:12:01.051 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.051 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.051 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.052 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-10-12T01:12:01.052 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.052 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.053 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:01.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.053 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.053 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.053 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.976899Z", 2023-10-12T01:12:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.055 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.055 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.055 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.055 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.055 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.056 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.056 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.056 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_h8Wl61PH7kSgSiQ6nxpf_Q1zXuUrih5", 2023-10-12T01:12:01.057 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.057 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.058 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-10-12T01:12:01.058 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.058 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.059 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:01.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.059 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.059 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.059 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.060 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.060 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.060 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.942974Z", 2023-10-12T01:12:01.060 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.061 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.061 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.061 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.061 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.062 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.062 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.062 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Tpf3b4B2VeXshTh2_gSLvDoUaqkd_LM", 2023-10-12T01:12:01.063 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.063 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.064 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-10-12T01:12:01.064 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.064 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.064 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.064 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-10-12T01:12:01.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.065 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.065 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.065 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.066 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.066 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.066 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.911454Z", 2023-10-12T01:12:01.066 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.066 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.067 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.067 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.067 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.067 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.068 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.068 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.068 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_xr4EXL5a5M3fIY1i8ZeDe_08cWUN33O", 2023-10-12T01:12:01.068 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.069 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.069 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-10-12T01:12:01.070 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.070 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.070 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.071 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.071 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.071 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.071 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.072 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.072 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.262594Z", 2023-10-12T01:12:01.072 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.072 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.072 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.073 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.073 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.073 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.073 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Avl8pX-V73WTElZQ99PJYDih4TEwIPv", 2023-10-12T01:12:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.075 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.075 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-10-12T01:12:01.076 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.076 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.076 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.078 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.902370Z", 2023-10-12T01:12:01.078 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.078 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.078 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.079 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.079 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.079 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.079 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.079 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CZA9Bbk5jkHr_3C06ROTKT423B0GdSt", 2023-10-12T01:12:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.081 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-10-12T01:12:01.081 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.082 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.082 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:01.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.082 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.083 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.083 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.083 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.083 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.084 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:23.996512Z", 2023-10-12T01:12:01.084 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.084 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.084 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.084 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.085 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.085 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.085 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.085 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.086 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_28tMU1GBc9EmigUGNAi26RnFXxkueWg", 2023-10-12T01:12:01.086 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.086 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.086 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.087 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-10-12T01:12:01.087 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.087 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.088 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.088 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.088 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.088 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.089 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.089 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.089 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.089 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:57:47.281973Z", 2023-10-12T01:12:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.091 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.091 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.091 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.091 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ZimkhwCV13TEUvGXFGXRw0LO4d1m-Le", 2023-10-12T01:12:01.092 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.092 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.093 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-10-12T01:12:01.093 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.093 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.093 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.094 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-10-12T01:12:01.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.094 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.094 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.929197Z", 2023-10-12T01:12:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.096 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.096 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.096 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.096 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.097 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.097 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.097 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_T8acL2OHpZnk5YQqxE2zrQNKELDhXk3", 2023-10-12T01:12:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.099 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-10-12T01:12:01.099 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.099 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.100 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:01.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.100 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.101 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.101 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.101 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.101 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.101 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.323573Z", 2023-10-12T01:12:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kI3zTZVKN8ejoFE2CtDZnAqEvpp4rEi", 2023-10-12T01:12:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-10-12T01:12:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-10-12T01:12:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.329916Z", 2023-10-12T01:12:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_B7wExLhjqBGUCXmJGfFUvYYvJYAglLe", 2023-10-12T01:12:01.110 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.110 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.111 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-10-12T01:12:01.111 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.111 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.111 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-10-12T01:12:01.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.112 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.112 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.112 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.112 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.113 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-10-12T01:12:01.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.058773Z", 2023-10-12T01:12:01.113 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.113 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.113 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.114 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.114 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.114 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.114 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.115 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.115 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_varn0hwdzbYGeHNCbRba_1TasXiI-IT", 2023-10-12T01:12:01.115 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.115 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.116 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.116 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-10-12T01:12:01.116 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.117 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.117 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-10-12T01:12:01.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.118 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.118 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.118 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.118 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.118 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.119 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.959193Z", 2023-10-12T01:12:01.119 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.119 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.119 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.120 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.120 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.120 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.120 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.120 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.121 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_F09mxeRyTQE3oa9X3gxCEr6hoFdkXqD", 2023-10-12T01:12:01.121 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.121 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.121 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.122 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-10-12T01:12:01.122 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.122 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.123 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.123 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:01.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.123 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.123 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:09:29.962302Z", 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.124 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.125 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.125 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.125 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.125 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.126 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.126 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.126 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JRtJ9-weqKO15GvER4P5dcxhCAvcVL-", 2023-10-12T01:12:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-10-12T01:12:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:06.992575Z", 2023-10-12T01:12:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_GkTSok_OQ79WcqrqT-tkItl6MkiOUOq", 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-10-12T01:12:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T00:59:24.078457Z", 2023-10-12T01:12:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_P4KzOz75-yA6yxObGcQJCd635hQTPBT", 2023-10-12T01:12:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-10-12T01:12:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.140 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-10-12T01:12:01.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.140 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.140 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.142 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.142 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.142 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.142 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:04:26.341888Z", 2023-10-12T01:12:01.143 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.143 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.143 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.143 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.144 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.144 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.144 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.144 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.145 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_zOLhNS5QvBT-pKG0Ux8sIxR_TnrS5DI", 2023-10-12T01:12:01.145 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.145 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.146 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-10-12T01:12:01.146 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.147 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.147 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-10-12T01:12:01.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.147 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.148 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.148 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.148 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.148 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-10-12T01:12:01.149 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:03:05.979824Z", 2023-10-12T01:12:01.149 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.149 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.149 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.149 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.150 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.150 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.150 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.150 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.151 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_VRRb0vjulYKc5gTtgotstKtPAgW8kaN", 2023-10-12T01:12:01.151 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.151 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.152 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-10-12T01:12:01.152 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.152 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.153 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:01.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.153 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.153 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.154 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.154 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.154 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-10-12T01:12:01.154 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:06:27.388674Z", 2023-10-12T01:12:01.155 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.155 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.155 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.156 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.156 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.156 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.156 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.157 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.157 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_WaO5afuDeJYERhwKwyrqc44rhFW-8F1", 2023-10-12T01:12:01.158 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.158 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.158 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-10-12T01:12:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-10-12T01:12:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-10-12T01:12:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-10-12T01:12:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-10-12T01:12:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-10-12T01:12:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-10-12T01:12:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-10-12T01:12:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-10-12T01:12:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-10-12T01:12:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-10-12T01:01:07.016570Z", 2023-10-12T01:12:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-12T01:12:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-10-12T01:12:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-10-12T01:12:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-10-12T01:12:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-10-12T01:12:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-10-12T01:12:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-10-12T01:12:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-10-12T01:12:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_e94h6TPu_NZ0sz6EvXdZR1pHvyu9YnQ", 2023-10-12T01:12:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-10-12T01:12:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-10-12T01:12:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-10-12T01:12:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:01.164 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:12:01.165 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T01:12:01.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:12:01.168 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T01:12:01.169 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T01:12:01.193 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:12:01.193 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:12:01.193 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:12:02.194 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:02.218 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:12:02.219 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:12:02.219 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:12:02.220 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T01:12:02.221 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-59 2023-10-12T01:12:02.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:12:02.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:12:02.222 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:12:02.320 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:02.320 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:02.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:02.362+0000 7fbfaf522780 1 bucket sync caught up with source: 2023-10-12T01:12:02.363 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:12:02.363 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.63.6, , , , , , 00000000002.69.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.63.6] 2023-10-12T01:12:02.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:02.362+0000 7fbfaf522780 0 bucket checkpoint complete 2023-10-12T01:12:02.377 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-59 2023-10-12T01:12:02.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:12:02.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:12:02.378 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:12:02.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:02.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:02.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:02.472+0000 7f45401a6780 1 bucket sync caught up with source: 2023-10-12T01:12:02.474 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.356.6, 00000000006.343.6, 00000000006.205.6, 00000000015.302.6, 00000000012.307.6, 00000000003.150.6, 00000000003.315.6, 00000000012.345.6, 00000000021.286.6, 00000000018.311.6] 2023-10-12T01:12:02.474 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.356.6, 00000000006.343.6, 00000000006.205.6, 00000000015.302.6, 00000000012.307.6, 00000000003.150.6, 00000000003.315.6, 00000000012.345.6, 00000000021.286.6, 00000000018.311.6] 2023-10-12T01:12:02.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:02.472+0000 7f45401a6780 0 bucket checkpoint complete 2023-10-12T01:12:02.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:12:02.488 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:02.489 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:02.489 DEBUG:boto:Method: HEAD 2023-10-12T01:12:02.490 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:12:02.490 DEBUG:boto:Data: 2023-10-12T01:12:02.490 DEBUG:boto:Headers: {} 2023-10-12T01:12:02.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:02.491 DEBUG:boto:Port: 8001 2023-10-12T01:12:02.492 DEBUG:boto:Params: {} 2023-10-12T01:12:02.492 DEBUG:boto:Token: None 2023-10-12T01:12:02.493 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:02 GMT /wafmoa-59/ 2023-10-12T01:12:02.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bCKC7UvWMjK19PXkViJYz67QfRY= 2023-10-12T01:12:02.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bCKC7UvWMjK19PXkViJYz67QfRY=', 'Content-Length': '0'} 2023-10-12T01:12:02.495 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-12T01:12:02.495 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:12:03.437 DEBUG:boto:Token: None 2023-10-12T01:12:03.438 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:02 GMT /wafmoa-59/ 2023-10-12T01:12:03.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bCKC7UvWMjK19PXkViJYz67QfRY= 2023-10-12T01:12:03.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:02 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bCKC7UvWMjK19PXkViJYz67QfRY=', 'Content-Length': '0'} 2023-10-12T01:12:03.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000415c0cc67f268e84-00652747e3-1e19-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.446 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.446 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.447 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.447 DEBUG:boto:Path: /wafmoa-59/ 2023-10-12T01:12:03.448 DEBUG:boto:Data: 2023-10-12T01:12:03.448 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.449 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.450 DEBUG:boto:Params: {} 2023-10-12T01:12:03.450 DEBUG:boto:Token: None 2023-10-12T01:12:03.451 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.451 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KNMeCLYUhkLaB+K1qnZ/Z0bEieM= 2023-10-12T01:12:03.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KNMeCLYUhkLaB+K1qnZ/Z0bEieM=', 'Content-Length': '0'} 2023-10-12T01:12:03.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080a5a96ef598d4f0-00652747e3-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:12:03.457 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.457 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.458 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.458 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.459 DEBUG:boto:Method: GET 2023-10-12T01:12:03.459 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.459 DEBUG:boto:Data: 2023-10-12T01:12:03.460 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.461 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.461 DEBUG:boto:Params: {} 2023-10-12T01:12:03.462 DEBUG:boto:Token: None 2023-10-12T01:12:03.462 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/?versions 2023-10-12T01:12:03.463 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug= 2023-10-12T01:12:03.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug=', 'Content-Length': '0'} 2023-10-12T01:12:03.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b8eb92f400f96e-00652747e3-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.467 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:12:03.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:12:03.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:12:03.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:12:03.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:12:03.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:12:03.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:12:03.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:12:03.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:12:03.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:12:03.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:12:03.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:12:03.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:12:03.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:12:03.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:12:03.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:12:03.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:12:03.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:12:03.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:12:03.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:12:03.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:12:03.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:12:03.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:12:03.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:12:03.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:12:03.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:12:03.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:12:03.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:12:03.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:12:03.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:12:03.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:12:03.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:12:03.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:12:03.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:12:03.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:12:03.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:12:03.483 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.484 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.484 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.485 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.485 DEBUG:boto:Method: GET 2023-10-12T01:12:03.486 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.486 DEBUG:boto:Data: 2023-10-12T01:12:03.487 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.488 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.488 DEBUG:boto:Params: {} 2023-10-12T01:12:03.489 DEBUG:boto:Token: None 2023-10-12T01:12:03.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/?versions 2023-10-12T01:12:03.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug= 2023-10-12T01:12:03.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug=', 'Content-Length': '0'} 2023-10-12T01:12:03.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000333ca40ed77bd13-00652747e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.496 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:12:03.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:12:03.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:12:03.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:12:03.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:12:03.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:12:03.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:12:03.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:12:03.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:12:03.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:12:03.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:12:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:12:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:12:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:12:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:12:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:12:03.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:12:03.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:12:03.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:12:03.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:12:03.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:12:03.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:12:03.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:12:03.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:12:03.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:12:03.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:12:03.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:12:03.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:12:03.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:12:03.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:12:03.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:12:03.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:12:03.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:12:03.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:12:03.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:12:03.511 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.511 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.511 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.512 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.512 DEBUG:boto:Method: GET 2023-10-12T01:12:03.513 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.513 DEBUG:boto:Data: 2023-10-12T01:12:03.514 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.514 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.515 DEBUG:boto:Params: {} 2023-10-12T01:12:03.515 DEBUG:boto:Token: None 2023-10-12T01:12:03.516 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/?versions 2023-10-12T01:12:03.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug= 2023-10-12T01:12:03.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug=', 'Content-Length': '0'} 2023-10-12T01:12:03.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c9b907ddfe89d6a-00652747e3-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.521 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.523 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.523 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.524 DEBUG:boto:path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.524 DEBUG:boto:auth_path=/wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.525 DEBUG:boto:Method: GET 2023-10-12T01:12:03.525 DEBUG:boto:Path: /wafmoa-59/?versions&max-keys=999 2023-10-12T01:12:03.526 DEBUG:boto:Data: 2023-10-12T01:12:03.526 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.527 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.527 DEBUG:boto:Params: {} 2023-10-12T01:12:03.528 DEBUG:boto:Token: None 2023-10-12T01:12:03.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/?versions 2023-10-12T01:12:03.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug= 2023-10-12T01:12:03.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ji9uil0mwLbn03qXOo+9ZvuQPug=', 'Content-Length': '0'} 2023-10-12T01:12:03.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0116759acee6457-00652747e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.535 DEBUG:boto:b'wafmoa-59999falseobj11nulltrue2023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:12:03.537 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.537 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.538 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.538 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.539 DEBUG:boto:Method: GET 2023-10-12T01:12:03.539 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.540 DEBUG:boto:Data: 2023-10-12T01:12:03.540 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.541 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.541 DEBUG:boto:Params: {} 2023-10-12T01:12:03.542 DEBUG:boto:Token: None 2023-10-12T01:12:03.542 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs= 2023-10-12T01:12:03.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs=', 'Content-Length': '0'} 2023-10-12T01:12:03.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021699e36b47e44d2-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.547 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.548 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.548 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.549 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.549 DEBUG:boto:Method: GET 2023-10-12T01:12:03.550 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.550 DEBUG:boto:Data: 2023-10-12T01:12:03.550 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.551 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.552 DEBUG:boto:Params: {} 2023-10-12T01:12:03.552 DEBUG:boto:Token: None 2023-10-12T01:12:03.553 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs= 2023-10-12T01:12:03.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs=', 'Content-Length': '0'} 2023-10-12T01:12:03.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e436c0d57c84ba70-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.602 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.603 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.603 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.604 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.604 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.605 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.605 DEBUG:boto:Data: 2023-10-12T01:12:03.606 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.606 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.607 DEBUG:boto:Params: {} 2023-10-12T01:12:03.607 DEBUG:boto:Token: None 2023-10-12T01:12:03.608 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+oFb2fuqEjbQ82JXac7VNG9yF2Y= 2023-10-12T01:12:03.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+oFb2fuqEjbQ82JXac7VNG9yF2Y=', 'Content-Length': '0'} 2023-10-12T01:12:03.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a0ea02e64336b80-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.611 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.612 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.612 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.613 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.613 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.614 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.614 DEBUG:boto:Data: 2023-10-12T01:12:03.615 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.616 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.616 DEBUG:boto:Params: {} 2023-10-12T01:12:03.616 DEBUG:boto:Token: None 2023-10-12T01:12:03.617 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+oFb2fuqEjbQ82JXac7VNG9yF2Y= 2023-10-12T01:12:03.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+oFb2fuqEjbQ82JXac7VNG9yF2Y=', 'Content-Length': '0'} 2023-10-12T01:12:03.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e19194a5b32961d9-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:12:03.621 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.622 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.622 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.623 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.623 DEBUG:boto:Method: GET 2023-10-12T01:12:03.624 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.624 DEBUG:boto:Data: 2023-10-12T01:12:03.624 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.625 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.626 DEBUG:boto:Params: {} 2023-10-12T01:12:03.627 DEBUG:boto:Token: None 2023-10-12T01:12:03.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs= 2023-10-12T01:12:03.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs=', 'Content-Length': '0'} 2023-10-12T01:12:03.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e84744a99e53f8-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.674 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.675 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.676 DEBUG:boto:path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.676 DEBUG:boto:auth_path=/wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.676 DEBUG:boto:Method: GET 2023-10-12T01:12:03.677 DEBUG:boto:Path: /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.677 DEBUG:boto:Data: 2023-10-12T01:12:03.678 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.679 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.679 DEBUG:boto:Params: {} 2023-10-12T01:12:03.680 DEBUG:boto:Token: None 2023-10-12T01:12:03.680 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11?versionId=null 2023-10-12T01:12:03.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs= 2023-10-12T01:12:03.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AukLHTEMfeVgjUFaQgCiJpN3FRs=', 'Content-Length': '0'} 2023-10-12T01:12:03.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f9634265e4cd81-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.684 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.684 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.685 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.685 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.686 DEBUG:boto:Method: GET 2023-10-12T01:12:03.686 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.687 DEBUG:boto:Data: 2023-10-12T01:12:03.687 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.688 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.688 DEBUG:boto:Params: {} 2023-10-12T01:12:03.689 DEBUG:boto:Token: None 2023-10-12T01:12:03.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.690 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71dcc73ae3f3e8d-00652747e3-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.693 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.694 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.695 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.695 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.695 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.696 DEBUG:boto:Method: GET 2023-10-12T01:12:03.696 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.697 DEBUG:boto:Data: 2023-10-12T01:12:03.697 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.698 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.698 DEBUG:boto:Params: {} 2023-10-12T01:12:03.699 DEBUG:boto:Token: None 2023-10-12T01:12:03.699 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002160d0eeb0e715f9-00652747e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.705 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.705 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.706 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.706 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.706 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:12:03.707 DEBUG:boto:Data: 2023-10-12T01:12:03.707 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.708 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.708 DEBUG:boto:Params: {} 2023-10-12T01:12:03.709 DEBUG:boto:Token: None 2023-10-12T01:12:03.709 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11 2023-10-12T01:12:03.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f6PXZUFwulgJ1vyqe09/pWe2C3k= 2023-10-12T01:12:03.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f6PXZUFwulgJ1vyqe09/pWe2C3k=', 'Content-Length': '0'} 2023-10-12T01:12:03.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0fe4c4caa8fe5f0-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.712 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.713 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.713 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.714 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:12:03.714 DEBUG:boto:Data: 2023-10-12T01:12:03.715 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.715 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.716 DEBUG:boto:Params: {} 2023-10-12T01:12:03.716 DEBUG:boto:Token: None 2023-10-12T01:12:03.717 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11 2023-10-12T01:12:03.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f6PXZUFwulgJ1vyqe09/pWe2C3k= 2023-10-12T01:12:03.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f6PXZUFwulgJ1vyqe09/pWe2C3k=', 'Content-Length': '0'} 2023-10-12T01:12:03.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcae9dc3e538bbfa-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:12:03.720 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.721 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.721 DEBUG:boto:Method: GET 2023-10-12T01:12:03.721 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:12:03.722 DEBUG:boto:Data: 2023-10-12T01:12:03.722 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.723 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.724 DEBUG:boto:Params: {} 2023-10-12T01:12:03.724 DEBUG:boto:Token: None 2023-10-12T01:12:03.724 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11 2023-10-12T01:12:03.725 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TIoch7ZEEmi80I/UT/QwlfFGOsE= 2023-10-12T01:12:03.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TIoch7ZEEmi80I/UT/QwlfFGOsE=', 'Content-Length': '0'} 2023-10-12T01:12:03.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0fd98178e2553bd-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.728 DEBUG:boto:path=/wafmoa-59/obj11 2023-10-12T01:12:03.728 DEBUG:boto:auth_path=/wafmoa-59/obj11 2023-10-12T01:12:03.729 DEBUG:boto:Method: GET 2023-10-12T01:12:03.729 DEBUG:boto:Path: /wafmoa-59/obj11 2023-10-12T01:12:03.730 DEBUG:boto:Data: 2023-10-12T01:12:03.730 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.731 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.731 DEBUG:boto:Params: {} 2023-10-12T01:12:03.732 DEBUG:boto:Token: None 2023-10-12T01:12:03.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj11 2023-10-12T01:12:03.732 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TIoch7ZEEmi80I/UT/QwlfFGOsE= 2023-10-12T01:12:03.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TIoch7ZEEmi80I/UT/QwlfFGOsE=', 'Content-Length': '0'} 2023-10-12T01:12:03.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e59b22b8c1c20a1-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.778 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.779 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.779 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.780 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.780 DEBUG:boto:Method: GET 2023-10-12T01:12:03.780 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.781 DEBUG:boto:Data: 2023-10-12T01:12:03.781 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.782 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.782 DEBUG:boto:Params: {} 2023-10-12T01:12:03.783 DEBUG:boto:Token: None 2023-10-12T01:12:03.783 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000886fa3aa1107a4a1-00652747e3-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.787 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.788 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.788 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.789 DEBUG:boto:path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.789 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.790 DEBUG:boto:Method: GET 2023-10-12T01:12:03.790 DEBUG:boto:Path: /wafmoa-59/?prefix=obj11 2023-10-12T01:12:03.791 DEBUG:boto:Data: 2023-10-12T01:12:03.791 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.792 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.792 DEBUG:boto:Params: {} 2023-10-12T01:12:03.793 DEBUG:boto:Token: None 2023-10-12T01:12:03.793 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f75975edc738b0a6-00652747e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.799 DEBUG:boto:b'wafmoa-59obj111000falseobj112023-10-12T00:57:47.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:12:03.800 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.800 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.800 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.801 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.801 DEBUG:boto:Method: GET 2023-10-12T01:12:03.802 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.802 DEBUG:boto:Data: 2023-10-12T01:12:03.803 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.803 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.804 DEBUG:boto:Params: {} 2023-10-12T01:12:03.804 DEBUG:boto:Token: None 2023-10-12T01:12:03.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.805 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg= 2023-10-12T01:12:03.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg=', 'Content-Length': '0'} 2023-10-12T01:12:03.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c356a973cfd1759-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.809 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.810 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.810 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.811 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.811 DEBUG:boto:Method: GET 2023-10-12T01:12:03.811 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.812 DEBUG:boto:Data: 2023-10-12T01:12:03.812 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.813 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.814 DEBUG:boto:Params: {} 2023-10-12T01:12:03.814 DEBUG:boto:Token: None 2023-10-12T01:12:03.815 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg= 2023-10-12T01:12:03.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg=', 'Content-Length': '0'} 2023-10-12T01:12:03.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113d6cf16a8165a4-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.818 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.819 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.819 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.819 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.820 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.820 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.821 DEBUG:boto:Data: 2023-10-12T01:12:03.821 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.822 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.822 DEBUG:boto:Params: {} 2023-10-12T01:12:03.823 DEBUG:boto:Token: None 2023-10-12T01:12:03.823 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MPgLvgLLflMBaX1TBBgr4F95j4Q= 2023-10-12T01:12:03.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MPgLvgLLflMBaX1TBBgr4F95j4Q=', 'Content-Length': '0'} 2023-10-12T01:12:03.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff12517c1b2c299-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.827 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.827 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.827 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.828 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.828 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.829 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.829 DEBUG:boto:Data: 2023-10-12T01:12:03.829 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.830 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.831 DEBUG:boto:Params: {} 2023-10-12T01:12:03.831 DEBUG:boto:Token: None 2023-10-12T01:12:03.832 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MPgLvgLLflMBaX1TBBgr4F95j4Q= 2023-10-12T01:12:03.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MPgLvgLLflMBaX1TBBgr4F95j4Q=', 'Content-Length': '0'} 2023-10-12T01:12:03.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eea6c9285530d44-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:12:03.835 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.835 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.836 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.836 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.837 DEBUG:boto:Method: GET 2023-10-12T01:12:03.837 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.838 DEBUG:boto:Data: 2023-10-12T01:12:03.838 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.839 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.839 DEBUG:boto:Params: {} 2023-10-12T01:12:03.840 DEBUG:boto:Token: None 2023-10-12T01:12:03.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg= 2023-10-12T01:12:03.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg=', 'Content-Length': '0'} 2023-10-12T01:12:03.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000034b61f206e96afb-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.844 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.844 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.844 DEBUG:boto:path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.845 DEBUG:boto:auth_path=/wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.845 DEBUG:boto:Method: GET 2023-10-12T01:12:03.846 DEBUG:boto:Path: /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.846 DEBUG:boto:Data: 2023-10-12T01:12:03.847 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.850 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.850 DEBUG:boto:Params: {} 2023-10-12T01:12:03.850 DEBUG:boto:Token: None 2023-10-12T01:12:03.851 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111?versionId=null 2023-10-12T01:12:03.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg= 2023-10-12T01:12:03.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yIDUWKO2PuvpsylzHZUTdSmq4Kg=', 'Content-Length': '0'} 2023-10-12T01:12:03.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaac595e078fab6d-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.894 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.895 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.895 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.896 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.896 DEBUG:boto:Method: GET 2023-10-12T01:12:03.897 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.897 DEBUG:boto:Data: 2023-10-12T01:12:03.898 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.898 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.899 DEBUG:boto:Params: {} 2023-10-12T01:12:03.899 DEBUG:boto:Token: None 2023-10-12T01:12:03.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45c6ab72d55e836-00652747e3-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.904 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.905 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.905 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.906 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.906 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.907 DEBUG:boto:Method: GET 2023-10-12T01:12:03.907 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.908 DEBUG:boto:Data: 2023-10-12T01:12:03.908 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.909 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.909 DEBUG:boto:Params: {} 2023-10-12T01:12:03.910 DEBUG:boto:Token: None 2023-10-12T01:12:03.910 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/ 2023-10-12T01:12:03.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA= 2023-10-12T01:12:03.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:R6yM7G7YSg/jXKWI0cSf3x0iCwA=', 'Content-Length': '0'} 2023-10-12T01:12:03.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1500a3636605f44-00652747e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.916 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:03.917 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.917 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.917 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.918 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:12:03.918 DEBUG:boto:Data: 2023-10-12T01:12:03.919 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.919 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.920 DEBUG:boto:Params: {} 2023-10-12T01:12:03.920 DEBUG:boto:Token: None 2023-10-12T01:12:03.921 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111 2023-10-12T01:12:03.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FzxOT+o5H7g6M0sdgY0JEgv88iA= 2023-10-12T01:12:03.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FzxOT+o5H7g6M0sdgY0JEgv88iA=', 'Content-Length': '0'} 2023-10-12T01:12:03.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075bc74e0f84aad0c-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.924 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.925 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.925 DEBUG:boto:Method: HEAD 2023-10-12T01:12:03.925 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:12:03.926 DEBUG:boto:Data: 2023-10-12T01:12:03.926 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.927 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.927 DEBUG:boto:Params: {} 2023-10-12T01:12:03.928 DEBUG:boto:Token: None 2023-10-12T01:12:03.928 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111 2023-10-12T01:12:03.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FzxOT+o5H7g6M0sdgY0JEgv88iA= 2023-10-12T01:12:03.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FzxOT+o5H7g6M0sdgY0JEgv88iA=', 'Content-Length': '0'} 2023-10-12T01:12:03.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000710898a8129ea6e-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:12:03.932 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.933 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.933 DEBUG:boto:Method: GET 2023-10-12T01:12:03.934 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:12:03.934 DEBUG:boto:Data: 2023-10-12T01:12:03.934 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.935 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.936 DEBUG:boto:Params: {} 2023-10-12T01:12:03.936 DEBUG:boto:Token: None 2023-10-12T01:12:03.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111 2023-10-12T01:12:03.937 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3uOIq/ldEn2tvBtj6v4rEZTnLAc= 2023-10-12T01:12:03.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3uOIq/ldEn2tvBtj6v4rEZTnLAc=', 'Content-Length': '0'} 2023-10-12T01:12:03.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c72cf54b94f981-00652747e3-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.986 DEBUG:boto:path=/wafmoa-59/obj111 2023-10-12T01:12:03.987 DEBUG:boto:auth_path=/wafmoa-59/obj111 2023-10-12T01:12:03.987 DEBUG:boto:Method: GET 2023-10-12T01:12:03.988 DEBUG:boto:Path: /wafmoa-59/obj111 2023-10-12T01:12:03.988 DEBUG:boto:Data: 2023-10-12T01:12:03.989 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:03.989 DEBUG:boto:Port: 8000 2023-10-12T01:12:03.990 DEBUG:boto:Params: {} 2023-10-12T01:12:03.990 DEBUG:boto:Token: None 2023-10-12T01:12:03.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:03 GMT /wafmoa-59/obj111 2023-10-12T01:12:03.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3uOIq/ldEn2tvBtj6v4rEZTnLAc= 2023-10-12T01:12:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:03 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3uOIq/ldEn2tvBtj6v4rEZTnLAc=', 'Content-Length': '0'} 2023-10-12T01:12:03.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5c869ae1a2f6c4-00652747e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:03.995 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:03.995 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:03.996 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.996 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.997 DEBUG:boto:Method: GET 2023-10-12T01:12:03.997 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:12:03.998 DEBUG:boto:Data: 2023-10-12T01:12:03.998 DEBUG:boto:Headers: {} 2023-10-12T01:12:03.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:03.999 DEBUG:boto:Port: 8001 2023-10-12T01:12:03.999 DEBUG:boto:Params: {} 2023-10-12T01:12:04.000 DEBUG:boto:Token: None 2023-10-12T01:12:04.000 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.001 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021580649aeace56-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.005 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.005 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.006 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.006 DEBUG:boto:path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:04.007 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj111 2023-10-12T01:12:04.007 DEBUG:boto:Method: GET 2023-10-12T01:12:04.008 DEBUG:boto:Path: /wafmoa-59/?prefix=obj111 2023-10-12T01:12:04.008 DEBUG:boto:Data: 2023-10-12T01:12:04.008 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.009 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.010 DEBUG:boto:Params: {} 2023-10-12T01:12:04.010 DEBUG:boto:Token: None 2023-10-12T01:12:04.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a170f69ef8dd1819-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.058 DEBUG:boto:b'wafmoa-59obj1111000falseobj1112023-10-12T01:09:30.024Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:12:04.060 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.060 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.060 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.061 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.061 DEBUG:boto:Method: GET 2023-10-12T01:12:04.062 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.062 DEBUG:boto:Data: 2023-10-12T01:12:04.063 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.064 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.064 DEBUG:boto:Params: {} 2023-10-12T01:12:04.065 DEBUG:boto:Token: None 2023-10-12T01:12:04.065 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU= 2023-10-12T01:12:04.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU=', 'Content-Length': '0'} 2023-10-12T01:12:04.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e5865fce1bf4917-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.069 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.070 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.070 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.071 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.071 DEBUG:boto:Method: GET 2023-10-12T01:12:04.071 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.072 DEBUG:boto:Data: 2023-10-12T01:12:04.072 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.073 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.074 DEBUG:boto:Params: {} 2023-10-12T01:12:04.074 DEBUG:boto:Token: None 2023-10-12T01:12:04.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU= 2023-10-12T01:12:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU=', 'Content-Length': '0'} 2023-10-12T01:12:04.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cca6d0d08791a3-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.078 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.079 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.079 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.080 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.080 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.080 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.081 DEBUG:boto:Data: 2023-10-12T01:12:04.081 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.082 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.082 DEBUG:boto:Params: {} 2023-10-12T01:12:04.083 DEBUG:boto:Token: None 2023-10-12T01:12:04.083 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rQJsZcbAIEdoJhWU46bAxgcMd5Q= 2023-10-12T01:12:04.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rQJsZcbAIEdoJhWU46bAxgcMd5Q=', 'Content-Length': '0'} 2023-10-12T01:12:04.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005167aa88943c7f0d-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.087 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.087 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.088 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.088 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.089 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.089 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.089 DEBUG:boto:Data: 2023-10-12T01:12:04.090 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.091 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.091 DEBUG:boto:Params: {} 2023-10-12T01:12:04.092 DEBUG:boto:Token: None 2023-10-12T01:12:04.092 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rQJsZcbAIEdoJhWU46bAxgcMd5Q= 2023-10-12T01:12:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rQJsZcbAIEdoJhWU46bAxgcMd5Q=', 'Content-Length': '0'} 2023-10-12T01:12:04.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9dc3692699789ae-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:12:04.096 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.096 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.097 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.097 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.098 DEBUG:boto:Method: GET 2023-10-12T01:12:04.098 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.098 DEBUG:boto:Data: 2023-10-12T01:12:04.099 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.100 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.100 DEBUG:boto:Params: {} 2023-10-12T01:12:04.101 DEBUG:boto:Token: None 2023-10-12T01:12:04.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU= 2023-10-12T01:12:04.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU=', 'Content-Length': '0'} 2023-10-12T01:12:04.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1fcd2f4f30b6fa4-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.105 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.105 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.106 DEBUG:boto:path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.106 DEBUG:boto:auth_path=/wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.107 DEBUG:boto:Method: GET 2023-10-12T01:12:04.107 DEBUG:boto:Path: /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.107 DEBUG:boto:Data: 2023-10-12T01:12:04.108 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.109 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.109 DEBUG:boto:Params: {} 2023-10-12T01:12:04.109 DEBUG:boto:Token: None 2023-10-12T01:12:04.110 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115?versionId=null 2023-10-12T01:12:04.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU= 2023-10-12T01:12:04.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:e5Z11sOzLcqhJlL++myzMWAcpbU=', 'Content-Length': '0'} 2023-10-12T01:12:04.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008799104c0597f099-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.114 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.114 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.115 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.115 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.116 DEBUG:boto:Method: GET 2023-10-12T01:12:04.116 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.117 DEBUG:boto:Data: 2023-10-12T01:12:04.117 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.118 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.118 DEBUG:boto:Params: {} 2023-10-12T01:12:04.119 DEBUG:boto:Token: None 2023-10-12T01:12:04.119 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa464beb312deec6-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.123 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.124 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.125 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.125 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.125 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.126 DEBUG:boto:Method: GET 2023-10-12T01:12:04.126 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.127 DEBUG:boto:Data: 2023-10-12T01:12:04.127 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.128 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.128 DEBUG:boto:Params: {} 2023-10-12T01:12:04.129 DEBUG:boto:Token: None 2023-10-12T01:12:04.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.130 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f1375a5d0d65249-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.135 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.136 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.136 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.137 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.137 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:12:04.137 DEBUG:boto:Data: 2023-10-12T01:12:04.138 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.139 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.139 DEBUG:boto:Params: {} 2023-10-12T01:12:04.140 DEBUG:boto:Token: None 2023-10-12T01:12:04.140 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115 2023-10-12T01:12:04.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gBhGXTDMdvql/lz6YoBy6p/gCmU= 2023-10-12T01:12:04.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gBhGXTDMdvql/lz6YoBy6p/gCmU=', 'Content-Length': '0'} 2023-10-12T01:12:04.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d23869a9a3ffc8-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.144 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.144 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.144 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.145 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:12:04.145 DEBUG:boto:Data: 2023-10-12T01:12:04.146 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.146 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.147 DEBUG:boto:Params: {} 2023-10-12T01:12:04.147 DEBUG:boto:Token: None 2023-10-12T01:12:04.148 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115 2023-10-12T01:12:04.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gBhGXTDMdvql/lz6YoBy6p/gCmU= 2023-10-12T01:12:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gBhGXTDMdvql/lz6YoBy6p/gCmU=', 'Content-Length': '0'} 2023-10-12T01:12:04.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc8732a64e27625-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:12:04.151 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.152 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.152 DEBUG:boto:Method: GET 2023-10-12T01:12:04.153 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:12:04.153 DEBUG:boto:Data: 2023-10-12T01:12:04.153 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.154 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.155 DEBUG:boto:Params: {} 2023-10-12T01:12:04.155 DEBUG:boto:Token: None 2023-10-12T01:12:04.156 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115 2023-10-12T01:12:04.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P86S2P2Fka2AdUlJHrQYKeIt8ms= 2023-10-12T01:12:04.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P86S2P2Fka2AdUlJHrQYKeIt8ms=', 'Content-Length': '0'} 2023-10-12T01:12:04.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37f5fb0f0ccb601-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.206 DEBUG:boto:path=/wafmoa-59/obj115 2023-10-12T01:12:04.207 DEBUG:boto:auth_path=/wafmoa-59/obj115 2023-10-12T01:12:04.207 DEBUG:boto:Method: GET 2023-10-12T01:12:04.208 DEBUG:boto:Path: /wafmoa-59/obj115 2023-10-12T01:12:04.208 DEBUG:boto:Data: 2023-10-12T01:12:04.209 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.210 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.210 DEBUG:boto:Params: {} 2023-10-12T01:12:04.211 DEBUG:boto:Token: None 2023-10-12T01:12:04.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj115 2023-10-12T01:12:04.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P86S2P2Fka2AdUlJHrQYKeIt8ms= 2023-10-12T01:12:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P86S2P2Fka2AdUlJHrQYKeIt8ms=', 'Content-Length': '0'} 2023-10-12T01:12:04.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014545171860cf5cf-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.215 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.215 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.216 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.216 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.217 DEBUG:boto:Method: GET 2023-10-12T01:12:04.217 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.217 DEBUG:boto:Data: 2023-10-12T01:12:04.218 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.219 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.219 DEBUG:boto:Params: {} 2023-10-12T01:12:04.220 DEBUG:boto:Token: None 2023-10-12T01:12:04.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbf3146e37be553-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.224 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.225 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.225 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.226 DEBUG:boto:path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.226 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.226 DEBUG:boto:Method: GET 2023-10-12T01:12:04.227 DEBUG:boto:Path: /wafmoa-59/?prefix=obj115 2023-10-12T01:12:04.227 DEBUG:boto:Data: 2023-10-12T01:12:04.228 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.228 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.229 DEBUG:boto:Params: {} 2023-10-12T01:12:04.229 DEBUG:boto:Token: None 2023-10-12T01:12:04.230 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.230 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e6376170e30453e-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.236 DEBUG:boto:b'wafmoa-59obj1151000falseobj1152023-10-12T01:03:06.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:12:04.237 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.237 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.238 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.238 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.239 DEBUG:boto:Method: GET 2023-10-12T01:12:04.239 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.239 DEBUG:boto:Data: 2023-10-12T01:12:04.240 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.241 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.241 DEBUG:boto:Params: {} 2023-10-12T01:12:04.242 DEBUG:boto:Token: None 2023-10-12T01:12:04.242 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.243 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro= 2023-10-12T01:12:04.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro=', 'Content-Length': '0'} 2023-10-12T01:12:04.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddcdaf965e8cf27-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.246 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.247 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.247 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.248 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.248 DEBUG:boto:Method: GET 2023-10-12T01:12:04.248 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.249 DEBUG:boto:Data: 2023-10-12T01:12:04.249 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.250 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.250 DEBUG:boto:Params: {} 2023-10-12T01:12:04.251 DEBUG:boto:Token: None 2023-10-12T01:12:04.251 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro= 2023-10-12T01:12:04.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro=', 'Content-Length': '0'} 2023-10-12T01:12:04.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048995979f51e9c54-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.255 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.255 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.256 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.256 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.257 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.257 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.257 DEBUG:boto:Data: 2023-10-12T01:12:04.258 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.259 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.259 DEBUG:boto:Params: {} 2023-10-12T01:12:04.259 DEBUG:boto:Token: None 2023-10-12T01:12:04.260 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xo7sl8s9zxEk4Qm1zLYy71+bOdg= 2023-10-12T01:12:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xo7sl8s9zxEk4Qm1zLYy71+bOdg=', 'Content-Length': '0'} 2023-10-12T01:12:04.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891a977ebd37de3d-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.263 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.264 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.266 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.266 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.267 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.267 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.267 DEBUG:boto:Data: 2023-10-12T01:12:04.268 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.269 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.269 DEBUG:boto:Params: {} 2023-10-12T01:12:04.269 DEBUG:boto:Token: None 2023-10-12T01:12:04.270 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Xo7sl8s9zxEk4Qm1zLYy71+bOdg= 2023-10-12T01:12:04.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Xo7sl8s9zxEk4Qm1zLYy71+bOdg=', 'Content-Length': '0'} 2023-10-12T01:12:04.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069d3f0ce0b7e78b-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:12:04.274 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.274 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.275 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.275 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.276 DEBUG:boto:Method: GET 2023-10-12T01:12:04.276 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.276 DEBUG:boto:Data: 2023-10-12T01:12:04.277 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.278 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.278 DEBUG:boto:Params: {} 2023-10-12T01:12:04.279 DEBUG:boto:Token: None 2023-10-12T01:12:04.279 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro= 2023-10-12T01:12:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro=', 'Content-Length': '0'} 2023-10-12T01:12:04.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2722183dc26ae53-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.326 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.327 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.327 DEBUG:boto:path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.328 DEBUG:boto:auth_path=/wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.328 DEBUG:boto:Method: GET 2023-10-12T01:12:04.329 DEBUG:boto:Path: /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.329 DEBUG:boto:Data: 2023-10-12T01:12:04.330 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.331 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.331 DEBUG:boto:Params: {} 2023-10-12T01:12:04.331 DEBUG:boto:Token: None 2023-10-12T01:12:04.332 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116?versionId=null 2023-10-12T01:12:04.333 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro= 2023-10-12T01:12:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6QHQAkyj3wvD6yT2zNyplYQhLro=', 'Content-Length': '0'} 2023-10-12T01:12:04.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bac8f638d421dc4-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.336 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.337 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.337 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.337 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.338 DEBUG:boto:Method: GET 2023-10-12T01:12:04.339 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.339 DEBUG:boto:Data: 2023-10-12T01:12:04.339 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.340 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.341 DEBUG:boto:Params: {} 2023-10-12T01:12:04.341 DEBUG:boto:Token: None 2023-10-12T01:12:04.342 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.342 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e93d3c41c1935ba-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.346 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.347 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.347 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.348 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.348 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.349 DEBUG:boto:Method: GET 2023-10-12T01:12:04.349 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.350 DEBUG:boto:Data: 2023-10-12T01:12:04.350 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.351 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.351 DEBUG:boto:Params: {} 2023-10-12T01:12:04.352 DEBUG:boto:Token: None 2023-10-12T01:12:04.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a08cd33c79abfe-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.358 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.359 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.360 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.360 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.360 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:12:04.361 DEBUG:boto:Data: 2023-10-12T01:12:04.361 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.362 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.363 DEBUG:boto:Params: {} 2023-10-12T01:12:04.363 DEBUG:boto:Token: None 2023-10-12T01:12:04.364 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116 2023-10-12T01:12:04.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8wnr0eZt66t414Ir5uKmwTLc/x8= 2023-10-12T01:12:04.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8wnr0eZt66t414Ir5uKmwTLc/x8=', 'Content-Length': '0'} 2023-10-12T01:12:04.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073af894443fdfd8-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.368 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.368 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.369 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.369 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:12:04.369 DEBUG:boto:Data: 2023-10-12T01:12:04.370 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.371 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.371 DEBUG:boto:Params: {} 2023-10-12T01:12:04.372 DEBUG:boto:Token: None 2023-10-12T01:12:04.372 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116 2023-10-12T01:12:04.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8wnr0eZt66t414Ir5uKmwTLc/x8= 2023-10-12T01:12:04.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8wnr0eZt66t414Ir5uKmwTLc/x8=', 'Content-Length': '0'} 2023-10-12T01:12:04.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f6afe0c94f64ae-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:12:04.376 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.376 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.377 DEBUG:boto:Method: GET 2023-10-12T01:12:04.377 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:12:04.378 DEBUG:boto:Data: 2023-10-12T01:12:04.378 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.379 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.379 DEBUG:boto:Params: {} 2023-10-12T01:12:04.380 DEBUG:boto:Token: None 2023-10-12T01:12:04.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116 2023-10-12T01:12:04.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hr1Oj1iqp3uEvsuq4hvT5e6sLIE= 2023-10-12T01:12:04.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hr1Oj1iqp3uEvsuq4hvT5e6sLIE=', 'Content-Length': '0'} 2023-10-12T01:12:04.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5068a1d4cc974d-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.384 DEBUG:boto:path=/wafmoa-59/obj116 2023-10-12T01:12:04.385 DEBUG:boto:auth_path=/wafmoa-59/obj116 2023-10-12T01:12:04.385 DEBUG:boto:Method: GET 2023-10-12T01:12:04.385 DEBUG:boto:Path: /wafmoa-59/obj116 2023-10-12T01:12:04.386 DEBUG:boto:Data: 2023-10-12T01:12:04.386 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.387 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.388 DEBUG:boto:Params: {} 2023-10-12T01:12:04.388 DEBUG:boto:Token: None 2023-10-12T01:12:04.389 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj116 2023-10-12T01:12:04.389 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hr1Oj1iqp3uEvsuq4hvT5e6sLIE= 2023-10-12T01:12:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hr1Oj1iqp3uEvsuq4hvT5e6sLIE=', 'Content-Length': '0'} 2023-10-12T01:12:04.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a04edf48b59020-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.438 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.439 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.439 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.440 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.440 DEBUG:boto:Method: GET 2023-10-12T01:12:04.441 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.441 DEBUG:boto:Data: 2023-10-12T01:12:04.442 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.443 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.443 DEBUG:boto:Params: {} 2023-10-12T01:12:04.444 DEBUG:boto:Token: None 2023-10-12T01:12:04.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.445 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aed4c8d36c5ed44-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.448 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.449 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.450 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.450 DEBUG:boto:path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.450 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.451 DEBUG:boto:Method: GET 2023-10-12T01:12:04.451 DEBUG:boto:Path: /wafmoa-59/?prefix=obj116 2023-10-12T01:12:04.452 DEBUG:boto:Data: 2023-10-12T01:12:04.452 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.453 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.453 DEBUG:boto:Params: {} 2023-10-12T01:12:04.454 DEBUG:boto:Token: None 2023-10-12T01:12:04.454 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007624d59a7ff58f96-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.460 DEBUG:boto:b'wafmoa-59obj1161000falseobj1162023-10-12T01:04:26.399Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:12:04.461 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.461 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.462 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.462 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.463 DEBUG:boto:Method: GET 2023-10-12T01:12:04.463 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.464 DEBUG:boto:Data: 2023-10-12T01:12:04.464 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.465 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.465 DEBUG:boto:Params: {} 2023-10-12T01:12:04.466 DEBUG:boto:Token: None 2023-10-12T01:12:04.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8= 2023-10-12T01:12:04.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8=', 'Content-Length': '0'} 2023-10-12T01:12:04.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ef969e6f54b2960-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.514 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.515 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.515 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.516 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.516 DEBUG:boto:Method: GET 2023-10-12T01:12:04.517 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.517 DEBUG:boto:Data: 2023-10-12T01:12:04.517 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.518 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.519 DEBUG:boto:Params: {} 2023-10-12T01:12:04.519 DEBUG:boto:Token: None 2023-10-12T01:12:04.520 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8= 2023-10-12T01:12:04.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8=', 'Content-Length': '0'} 2023-10-12T01:12:04.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1463e9b905b64a8-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.524 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.524 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.525 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.525 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.525 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.526 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.526 DEBUG:boto:Data: 2023-10-12T01:12:04.527 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.527 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.528 DEBUG:boto:Params: {} 2023-10-12T01:12:04.528 DEBUG:boto:Token: None 2023-10-12T01:12:04.529 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xeUklDbyk1pCwdDwFAh710cnj38= 2023-10-12T01:12:04.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xeUklDbyk1pCwdDwFAh710cnj38=', 'Content-Length': '0'} 2023-10-12T01:12:04.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d2cf139ae703168-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.532 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.533 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.533 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.534 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.534 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.534 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.535 DEBUG:boto:Data: 2023-10-12T01:12:04.535 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.536 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.537 DEBUG:boto:Params: {} 2023-10-12T01:12:04.537 DEBUG:boto:Token: None 2023-10-12T01:12:04.538 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xeUklDbyk1pCwdDwFAh710cnj38= 2023-10-12T01:12:04.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xeUklDbyk1pCwdDwFAh710cnj38=', 'Content-Length': '0'} 2023-10-12T01:12:04.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c9939b6ecd51bf2-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:12:04.542 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.542 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.543 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.543 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.543 DEBUG:boto:Method: GET 2023-10-12T01:12:04.544 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.544 DEBUG:boto:Data: 2023-10-12T01:12:04.545 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.546 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.546 DEBUG:boto:Params: {} 2023-10-12T01:12:04.547 DEBUG:boto:Token: None 2023-10-12T01:12:04.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8= 2023-10-12T01:12:04.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8=', 'Content-Length': '0'} 2023-10-12T01:12:04.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be340115979d7d0-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.551 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.551 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.552 DEBUG:boto:path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.552 DEBUG:boto:auth_path=/wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.553 DEBUG:boto:Method: GET 2023-10-12T01:12:04.553 DEBUG:boto:Path: /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.553 DEBUG:boto:Data: 2023-10-12T01:12:04.554 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.555 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.555 DEBUG:boto:Params: {} 2023-10-12T01:12:04.556 DEBUG:boto:Token: None 2023-10-12T01:12:04.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124?versionId=null 2023-10-12T01:12:04.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8= 2023-10-12T01:12:04.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HtZqSIzi0ouj3AkVSEUF3FmHVj8=', 'Content-Length': '0'} 2023-10-12T01:12:04.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f970a556bf2f2225-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.560 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.560 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.561 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.561 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.562 DEBUG:boto:Method: GET 2023-10-12T01:12:04.562 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.563 DEBUG:boto:Data: 2023-10-12T01:12:04.563 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.564 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.564 DEBUG:boto:Params: {} 2023-10-12T01:12:04.565 DEBUG:boto:Token: None 2023-10-12T01:12:04.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d9001edbcb449d-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.569 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.570 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.571 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.571 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.571 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.572 DEBUG:boto:Method: GET 2023-10-12T01:12:04.572 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.573 DEBUG:boto:Data: 2023-10-12T01:12:04.573 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.574 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.575 DEBUG:boto:Params: {} 2023-10-12T01:12:04.575 DEBUG:boto:Token: None 2023-10-12T01:12:04.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35663a20bf777f8-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.581 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.582 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.582 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.583 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.583 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:12:04.584 DEBUG:boto:Data: 2023-10-12T01:12:04.584 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.585 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.585 DEBUG:boto:Params: {} 2023-10-12T01:12:04.586 DEBUG:boto:Token: None 2023-10-12T01:12:04.586 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124 2023-10-12T01:12:04.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sfSzIlBBnbNY5+pi9kIoOZopbDM= 2023-10-12T01:12:04.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sfSzIlBBnbNY5+pi9kIoOZopbDM=', 'Content-Length': '0'} 2023-10-12T01:12:04.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e9e0ae05964e1f-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.592 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.592 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.593 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.593 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:12:04.594 DEBUG:boto:Data: 2023-10-12T01:12:04.594 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.595 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.595 DEBUG:boto:Params: {} 2023-10-12T01:12:04.596 DEBUG:boto:Token: None 2023-10-12T01:12:04.597 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124 2023-10-12T01:12:04.597 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sfSzIlBBnbNY5+pi9kIoOZopbDM= 2023-10-12T01:12:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sfSzIlBBnbNY5+pi9kIoOZopbDM=', 'Content-Length': '0'} 2023-10-12T01:12:04.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e7c8b2d8b392638-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:12:04.601 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.602 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.602 DEBUG:boto:Method: GET 2023-10-12T01:12:04.603 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:12:04.603 DEBUG:boto:Data: 2023-10-12T01:12:04.604 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.604 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.605 DEBUG:boto:Params: {} 2023-10-12T01:12:04.605 DEBUG:boto:Token: None 2023-10-12T01:12:04.606 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124 2023-10-12T01:12:04.606 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gkn77Auu/nplU5to8jlYO2GetKY= 2023-10-12T01:12:04.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gkn77Auu/nplU5to8jlYO2GetKY=', 'Content-Length': '0'} 2023-10-12T01:12:04.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d569ca54d346545-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.610 DEBUG:boto:path=/wafmoa-59/obj124 2023-10-12T01:12:04.610 DEBUG:boto:auth_path=/wafmoa-59/obj124 2023-10-12T01:12:04.611 DEBUG:boto:Method: GET 2023-10-12T01:12:04.611 DEBUG:boto:Path: /wafmoa-59/obj124 2023-10-12T01:12:04.612 DEBUG:boto:Data: 2023-10-12T01:12:04.612 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.613 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.614 DEBUG:boto:Params: {} 2023-10-12T01:12:04.614 DEBUG:boto:Token: None 2023-10-12T01:12:04.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj124 2023-10-12T01:12:04.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gkn77Auu/nplU5to8jlYO2GetKY= 2023-10-12T01:12:04.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gkn77Auu/nplU5to8jlYO2GetKY=', 'Content-Length': '0'} 2023-10-12T01:12:04.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050353514e98c6ff0-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.658 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.659 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.660 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.660 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.661 DEBUG:boto:Method: GET 2023-10-12T01:12:04.661 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.662 DEBUG:boto:Data: 2023-10-12T01:12:04.662 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.663 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.664 DEBUG:boto:Params: {} 2023-10-12T01:12:04.664 DEBUG:boto:Token: None 2023-10-12T01:12:04.665 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.666 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfdb8ca24e70a075-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.669 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.670 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.671 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.671 DEBUG:boto:path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.672 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.672 DEBUG:boto:Method: GET 2023-10-12T01:12:04.673 DEBUG:boto:Path: /wafmoa-59/?prefix=obj124 2023-10-12T01:12:04.673 DEBUG:boto:Data: 2023-10-12T01:12:04.674 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.675 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.675 DEBUG:boto:Params: {} 2023-10-12T01:12:04.675 DEBUG:boto:Token: None 2023-10-12T01:12:04.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0b8ad6d963dd31-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.682 DEBUG:boto:b'wafmoa-59obj1241000falseobj1242023-10-12T01:06:27.408Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:12:04.683 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.684 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.684 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.685 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.685 DEBUG:boto:Method: GET 2023-10-12T01:12:04.686 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.686 DEBUG:boto:Data: 2023-10-12T01:12:04.687 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.687 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.688 DEBUG:boto:Params: {} 2023-10-12T01:12:04.689 DEBUG:boto:Token: None 2023-10-12T01:12:04.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.690 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY= 2023-10-12T01:12:04.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY=', 'Content-Length': '0'} 2023-10-12T01:12:04.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fea003557c0c888-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.694 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.694 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.694 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.695 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.695 DEBUG:boto:Method: GET 2023-10-12T01:12:04.696 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.696 DEBUG:boto:Data: 2023-10-12T01:12:04.697 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.698 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.698 DEBUG:boto:Params: {} 2023-10-12T01:12:04.699 DEBUG:boto:Token: None 2023-10-12T01:12:04.699 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY= 2023-10-12T01:12:04.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY=', 'Content-Length': '0'} 2023-10-12T01:12:04.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2b06e886fcafc1-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.703 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.704 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.704 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.705 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.705 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.706 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.706 DEBUG:boto:Data: 2023-10-12T01:12:04.707 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.708 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.708 DEBUG:boto:Params: {} 2023-10-12T01:12:04.708 DEBUG:boto:Token: None 2023-10-12T01:12:04.709 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.710 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HRfFCLiXGjdUHkjv54Nd6EJjVU= 2023-10-12T01:12:04.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HRfFCLiXGjdUHkjv54Nd6EJjVU=', 'Content-Length': '0'} 2023-10-12T01:12:04.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177c5307686d3068-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.713 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.713 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.714 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.714 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.715 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.715 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.716 DEBUG:boto:Data: 2023-10-12T01:12:04.716 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.717 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.718 DEBUG:boto:Params: {} 2023-10-12T01:12:04.718 DEBUG:boto:Token: None 2023-10-12T01:12:04.719 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.719 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9HRfFCLiXGjdUHkjv54Nd6EJjVU= 2023-10-12T01:12:04.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9HRfFCLiXGjdUHkjv54Nd6EJjVU=', 'Content-Length': '0'} 2023-10-12T01:12:04.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f87ca4159c4da4-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:12:04.723 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.723 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.724 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.725 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.725 DEBUG:boto:Method: GET 2023-10-12T01:12:04.726 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.726 DEBUG:boto:Data: 2023-10-12T01:12:04.727 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.728 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.728 DEBUG:boto:Params: {} 2023-10-12T01:12:04.728 DEBUG:boto:Token: None 2023-10-12T01:12:04.729 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.730 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY= 2023-10-12T01:12:04.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY=', 'Content-Length': '0'} 2023-10-12T01:12:04.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007769c9bba6d14713-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.733 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.734 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.734 DEBUG:boto:path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.735 DEBUG:boto:auth_path=/wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.735 DEBUG:boto:Method: GET 2023-10-12T01:12:04.736 DEBUG:boto:Path: /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.736 DEBUG:boto:Data: 2023-10-12T01:12:04.736 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.737 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.738 DEBUG:boto:Params: {} 2023-10-12T01:12:04.739 DEBUG:boto:Token: None 2023-10-12T01:12:04.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145?versionId=null 2023-10-12T01:12:04.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY= 2023-10-12T01:12:04.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a1ge+YXeSrzxgWOzyYaVDR52XwY=', 'Content-Length': '0'} 2023-10-12T01:12:04.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d53148c9d8887e6-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.786 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.787 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.787 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.788 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.788 DEBUG:boto:Method: GET 2023-10-12T01:12:04.789 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.789 DEBUG:boto:Data: 2023-10-12T01:12:04.790 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.790 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.791 DEBUG:boto:Params: {} 2023-10-12T01:12:04.791 DEBUG:boto:Token: None 2023-10-12T01:12:04.792 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e58185939eef246-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.796 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.797 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.797 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.798 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.798 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.799 DEBUG:boto:Method: GET 2023-10-12T01:12:04.799 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.800 DEBUG:boto:Data: 2023-10-12T01:12:04.800 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.801 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.801 DEBUG:boto:Params: {} 2023-10-12T01:12:04.802 DEBUG:boto:Token: None 2023-10-12T01:12:04.802 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deed0baabd815595-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.808 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.809 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.809 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.810 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.810 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:12:04.811 DEBUG:boto:Data: 2023-10-12T01:12:04.811 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.812 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.812 DEBUG:boto:Params: {} 2023-10-12T01:12:04.813 DEBUG:boto:Token: None 2023-10-12T01:12:04.813 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145 2023-10-12T01:12:04.814 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6LhK74Fnza8vD1yAOOH1cVtxAFg= 2023-10-12T01:12:04.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6LhK74Fnza8vD1yAOOH1cVtxAFg=', 'Content-Length': '0'} 2023-10-12T01:12:04.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a7d1934ec2ce03-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.817 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.817 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.818 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.818 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:12:04.819 DEBUG:boto:Data: 2023-10-12T01:12:04.819 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.820 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.820 DEBUG:boto:Params: {} 2023-10-12T01:12:04.821 DEBUG:boto:Token: None 2023-10-12T01:12:04.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145 2023-10-12T01:12:04.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6LhK74Fnza8vD1yAOOH1cVtxAFg= 2023-10-12T01:12:04.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6LhK74Fnza8vD1yAOOH1cVtxAFg=', 'Content-Length': '0'} 2023-10-12T01:12:04.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009407f3a75bfd884d-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:12:04.825 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.825 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.826 DEBUG:boto:Method: GET 2023-10-12T01:12:04.826 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:12:04.827 DEBUG:boto:Data: 2023-10-12T01:12:04.827 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.828 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.828 DEBUG:boto:Params: {} 2023-10-12T01:12:04.829 DEBUG:boto:Token: None 2023-10-12T01:12:04.829 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145 2023-10-12T01:12:04.830 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2JmZgjzIrA2i9G4AtbsskpYz50E= 2023-10-12T01:12:04.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2JmZgjzIrA2i9G4AtbsskpYz50E=', 'Content-Length': '0'} 2023-10-12T01:12:04.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2ba1490e285afd-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.878 DEBUG:boto:path=/wafmoa-59/obj145 2023-10-12T01:12:04.879 DEBUG:boto:auth_path=/wafmoa-59/obj145 2023-10-12T01:12:04.879 DEBUG:boto:Method: GET 2023-10-12T01:12:04.880 DEBUG:boto:Path: /wafmoa-59/obj145 2023-10-12T01:12:04.880 DEBUG:boto:Data: 2023-10-12T01:12:04.881 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.882 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.882 DEBUG:boto:Params: {} 2023-10-12T01:12:04.883 DEBUG:boto:Token: None 2023-10-12T01:12:04.883 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj145 2023-10-12T01:12:04.884 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2JmZgjzIrA2i9G4AtbsskpYz50E= 2023-10-12T01:12:04.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2JmZgjzIrA2i9G4AtbsskpYz50E=', 'Content-Length': '0'} 2023-10-12T01:12:04.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313d22a98469774c-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.887 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.888 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.888 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.888 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.889 DEBUG:boto:Method: GET 2023-10-12T01:12:04.889 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.890 DEBUG:boto:Data: 2023-10-12T01:12:04.890 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.891 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.892 DEBUG:boto:Params: {} 2023-10-12T01:12:04.892 DEBUG:boto:Token: None 2023-10-12T01:12:04.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.893 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c2da0075e512d7-00652747e4-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.898 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.899 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:04.899 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:04.900 DEBUG:boto:path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.900 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.900 DEBUG:boto:Method: GET 2023-10-12T01:12:04.901 DEBUG:boto:Path: /wafmoa-59/?prefix=obj145 2023-10-12T01:12:04.901 DEBUG:boto:Data: 2023-10-12T01:12:04.902 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.902 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.903 DEBUG:boto:Params: {} 2023-10-12T01:12:04.903 DEBUG:boto:Token: None 2023-10-12T01:12:04.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/ 2023-10-12T01:12:04.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY= 2023-10-12T01:12:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/Qo65SBBdwFMHFASPsn5F1S/tY=', 'Content-Length': '0'} 2023-10-12T01:12:04.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb7e8029ce41208-00652747e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.954 DEBUG:boto:b'wafmoa-59obj1451000falseobj1452023-10-12T01:04:26.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:04.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:12:04.956 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:04.956 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:04.956 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.957 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.957 DEBUG:boto:Method: GET 2023-10-12T01:12:04.958 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.958 DEBUG:boto:Data: 2023-10-12T01:12:04.959 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.959 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.960 DEBUG:boto:Params: {} 2023-10-12T01:12:04.960 DEBUG:boto:Token: None 2023-10-12T01:12:04.961 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY= 2023-10-12T01:12:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY=', 'Content-Length': '0'} 2023-10-12T01:12:04.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2296ef3b3c370d2-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.965 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:04.965 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:04.966 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.966 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.967 DEBUG:boto:Method: GET 2023-10-12T01:12:04.967 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.968 DEBUG:boto:Data: 2023-10-12T01:12:04.968 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.969 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.969 DEBUG:boto:Params: {} 2023-10-12T01:12:04.970 DEBUG:boto:Token: None 2023-10-12T01:12:04.971 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY= 2023-10-12T01:12:04.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY=', 'Content-Length': '0'} 2023-10-12T01:12:04.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5cbee3b281afed-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.974 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:04.975 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:04.975 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.976 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.976 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.977 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.977 DEBUG:boto:Data: 2023-10-12T01:12:04.977 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.978 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.979 DEBUG:boto:Params: {} 2023-10-12T01:12:04.979 DEBUG:boto:Token: None 2023-10-12T01:12:04.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:10Ob+1qFUAse3orE0mRL3MuIcuY= 2023-10-12T01:12:04.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:10Ob+1qFUAse3orE0mRL3MuIcuY=', 'Content-Length': '0'} 2023-10-12T01:12:04.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f7812e48a7963b-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.983 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:04.984 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:04.984 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.984 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.985 DEBUG:boto:Method: HEAD 2023-10-12T01:12:04.985 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.986 DEBUG:boto:Data: 2023-10-12T01:12:04.986 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:04.987 DEBUG:boto:Port: 8000 2023-10-12T01:12:04.987 DEBUG:boto:Params: {} 2023-10-12T01:12:04.988 DEBUG:boto:Token: None 2023-10-12T01:12:04.988 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:10Ob+1qFUAse3orE0mRL3MuIcuY= 2023-10-12T01:12:04.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:10Ob+1qFUAse3orE0mRL3MuIcuY=', 'Content-Length': '0'} 2023-10-12T01:12:04.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3fd1c7b2cc0861-00652747e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:04.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:12:04.992 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:04.993 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:04.993 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.993 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.994 DEBUG:boto:Method: GET 2023-10-12T01:12:04.994 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.995 DEBUG:boto:Data: 2023-10-12T01:12:04.995 DEBUG:boto:Headers: {} 2023-10-12T01:12:04.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:04.996 DEBUG:boto:Port: 8001 2023-10-12T01:12:04.997 DEBUG:boto:Params: {} 2023-10-12T01:12:04.997 DEBUG:boto:Token: None 2023-10-12T01:12:04.998 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:04 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:04.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY= 2023-10-12T01:12:04.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:04 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aXL3sx4X6Y3rSk4xX5VsyoYOdVY=', 'Content-Length': '0'} 2023-10-12T01:12:05.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac1ffa881ea9115-00652747e4-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.001 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:05.002 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:05.002 DEBUG:boto:path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:05.003 DEBUG:boto:auth_path=/wafmoa-59/obj148?versionId=null 2023-10-12T01:12:05.003 DEBUG:boto:Method: GET 2023-10-12T01:12:05.004 DEBUG:boto:Path: /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:05.004 DEBUG:boto:Data: 2023-10-12T01:12:05.004 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.005 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.006 DEBUG:boto:Params: {} 2023-10-12T01:12:05.006 DEBUG:boto:Token: None 2023-10-12T01:12:05.007 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj148?versionId=null 2023-10-12T01:12:05.007 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0IyFEGWxGMz5ioZrzyU8cnmFgEY= 2023-10-12T01:12:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0IyFEGWxGMz5ioZrzyU8cnmFgEY=', 'Content-Length': '0'} 2023-10-12T01:12:05.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915647bfa9aa1c79-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.011 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.011 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.012 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.012 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.012 DEBUG:boto:Method: GET 2023-10-12T01:12:05.013 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.013 DEBUG:boto:Data: 2023-10-12T01:12:05.014 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.015 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.015 DEBUG:boto:Params: {} 2023-10-12T01:12:05.016 DEBUG:boto:Token: None 2023-10-12T01:12:05.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bc00fcfad9e1b5e-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.021 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.021 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.022 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.022 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.023 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.023 DEBUG:boto:Method: GET 2023-10-12T01:12:05.024 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.024 DEBUG:boto:Data: 2023-10-12T01:12:05.025 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.025 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.026 DEBUG:boto:Params: {} 2023-10-12T01:12:05.026 DEBUG:boto:Token: None 2023-10-12T01:12:05.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf94e9fb3506fdf-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.032 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.033 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:05.034 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:05.034 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.035 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:12:05.035 DEBUG:boto:Data: 2023-10-12T01:12:05.035 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.036 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.037 DEBUG:boto:Params: {} 2023-10-12T01:12:05.037 DEBUG:boto:Token: None 2023-10-12T01:12:05.038 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj148 2023-10-12T01:12:05.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g5lqLyP0VHaYC3E9xHMRSG5/9YA= 2023-10-12T01:12:05.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g5lqLyP0VHaYC3E9xHMRSG5/9YA=', 'Content-Length': '0'} 2023-10-12T01:12:05.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8df32a827b23b3-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.041 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:05.041 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:05.042 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.042 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:12:05.043 DEBUG:boto:Data: 2023-10-12T01:12:05.043 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.044 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.044 DEBUG:boto:Params: {} 2023-10-12T01:12:05.045 DEBUG:boto:Token: None 2023-10-12T01:12:05.045 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj148 2023-10-12T01:12:05.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:g5lqLyP0VHaYC3E9xHMRSG5/9YA= 2023-10-12T01:12:05.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:g5lqLyP0VHaYC3E9xHMRSG5/9YA=', 'Content-Length': '0'} 2023-10-12T01:12:05.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be8742844a30df4-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:12:05.050 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:05.050 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:05.051 DEBUG:boto:Method: GET 2023-10-12T01:12:05.051 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:12:05.052 DEBUG:boto:Data: 2023-10-12T01:12:05.052 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.053 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.053 DEBUG:boto:Params: {} 2023-10-12T01:12:05.054 DEBUG:boto:Token: None 2023-10-12T01:12:05.054 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj148 2023-10-12T01:12:05.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jzgje9eKvaP8Zsg7O3zSH/+27V8= 2023-10-12T01:12:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jzgje9eKvaP8Zsg7O3zSH/+27V8=', 'Content-Length': '0'} 2023-10-12T01:12:05.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab83d2eb890c37b1-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.098 DEBUG:boto:path=/wafmoa-59/obj148 2023-10-12T01:12:05.099 DEBUG:boto:auth_path=/wafmoa-59/obj148 2023-10-12T01:12:05.100 DEBUG:boto:Method: GET 2023-10-12T01:12:05.100 DEBUG:boto:Path: /wafmoa-59/obj148 2023-10-12T01:12:05.101 DEBUG:boto:Data: 2023-10-12T01:12:05.101 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.102 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.103 DEBUG:boto:Params: {} 2023-10-12T01:12:05.103 DEBUG:boto:Token: None 2023-10-12T01:12:05.104 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj148 2023-10-12T01:12:05.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jzgje9eKvaP8Zsg7O3zSH/+27V8= 2023-10-12T01:12:05.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jzgje9eKvaP8Zsg7O3zSH/+27V8=', 'Content-Length': '0'} 2023-10-12T01:12:05.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed50a2d9dca86693-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.108 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.109 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.109 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.109 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.110 DEBUG:boto:Method: GET 2023-10-12T01:12:05.110 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.111 DEBUG:boto:Data: 2023-10-12T01:12:05.111 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.112 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.113 DEBUG:boto:Params: {} 2023-10-12T01:12:05.113 DEBUG:boto:Token: None 2023-10-12T01:12:05.114 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb73aabedfd2ec45-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.118 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.119 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.119 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.119 DEBUG:boto:path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.120 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.120 DEBUG:boto:Method: GET 2023-10-12T01:12:05.121 DEBUG:boto:Path: /wafmoa-59/?prefix=obj148 2023-10-12T01:12:05.121 DEBUG:boto:Data: 2023-10-12T01:12:05.122 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.123 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.123 DEBUG:boto:Params: {} 2023-10-12T01:12:05.123 DEBUG:boto:Token: None 2023-10-12T01:12:05.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d299863f74009d-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.130 DEBUG:boto:b'wafmoa-59obj1481000falseobj1482023-10-12T01:09:30.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:12:05.131 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.132 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.132 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.133 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.133 DEBUG:boto:Method: GET 2023-10-12T01:12:05.134 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.134 DEBUG:boto:Data: 2023-10-12T01:12:05.135 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.136 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.136 DEBUG:boto:Params: {} 2023-10-12T01:12:05.137 DEBUG:boto:Token: None 2023-10-12T01:12:05.137 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ= 2023-10-12T01:12:05.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ=', 'Content-Length': '0'} 2023-10-12T01:12:05.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f42fd87711a5371-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.141 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.142 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.142 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.143 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.143 DEBUG:boto:Method: GET 2023-10-12T01:12:05.144 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.144 DEBUG:boto:Data: 2023-10-12T01:12:05.145 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.145 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.146 DEBUG:boto:Params: {} 2023-10-12T01:12:05.146 DEBUG:boto:Token: None 2023-10-12T01:12:05.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ= 2023-10-12T01:12:05.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ=', 'Content-Length': '0'} 2023-10-12T01:12:05.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356c3a2e530f5709-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.151 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.152 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.152 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.152 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.153 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.153 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.153 DEBUG:boto:Data: 2023-10-12T01:12:05.154 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.155 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.155 DEBUG:boto:Params: {} 2023-10-12T01:12:05.156 DEBUG:boto:Token: None 2023-10-12T01:12:05.156 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m637GjCsbG4h7202Scd628gAJ1E= 2023-10-12T01:12:05.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m637GjCsbG4h7202Scd628gAJ1E=', 'Content-Length': '0'} 2023-10-12T01:12:05.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab2ad31a64910ad-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.159 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.160 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.160 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.161 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.161 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.161 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.162 DEBUG:boto:Data: 2023-10-12T01:12:05.162 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.163 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.163 DEBUG:boto:Params: {} 2023-10-12T01:12:05.164 DEBUG:boto:Token: None 2023-10-12T01:12:05.164 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m637GjCsbG4h7202Scd628gAJ1E= 2023-10-12T01:12:05.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m637GjCsbG4h7202Scd628gAJ1E=', 'Content-Length': '0'} 2023-10-12T01:12:05.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009048bfec6f3fdf67-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:12:05.168 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.168 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.169 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.169 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.169 DEBUG:boto:Method: GET 2023-10-12T01:12:05.170 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.170 DEBUG:boto:Data: 2023-10-12T01:12:05.171 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.171 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.172 DEBUG:boto:Params: {} 2023-10-12T01:12:05.172 DEBUG:boto:Token: None 2023-10-12T01:12:05.173 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.173 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ= 2023-10-12T01:12:05.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ=', 'Content-Length': '0'} 2023-10-12T01:12:05.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6912c0b8f846cdf-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.177 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.177 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.177 DEBUG:boto:path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.178 DEBUG:boto:auth_path=/wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.178 DEBUG:boto:Method: GET 2023-10-12T01:12:05.179 DEBUG:boto:Path: /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.179 DEBUG:boto:Data: 2023-10-12T01:12:05.180 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.180 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.181 DEBUG:boto:Params: {} 2023-10-12T01:12:05.181 DEBUG:boto:Token: None 2023-10-12T01:12:05.182 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155?versionId=null 2023-10-12T01:12:05.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ= 2023-10-12T01:12:05.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j1KHADo6Dw1zar+TYCWkLdOrpJQ=', 'Content-Length': '0'} 2023-10-12T01:12:05.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f1d6cdacd3dbc2-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.226 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.227 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.227 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.228 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.228 DEBUG:boto:Method: GET 2023-10-12T01:12:05.229 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.229 DEBUG:boto:Data: 2023-10-12T01:12:05.229 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.230 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.231 DEBUG:boto:Params: {} 2023-10-12T01:12:05.231 DEBUG:boto:Token: None 2023-10-12T01:12:05.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e32f4b0adf26353-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.235 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.236 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.237 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.237 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.237 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.238 DEBUG:boto:Method: GET 2023-10-12T01:12:05.238 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.239 DEBUG:boto:Data: 2023-10-12T01:12:05.239 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.240 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.240 DEBUG:boto:Params: {} 2023-10-12T01:12:05.241 DEBUG:boto:Token: None 2023-10-12T01:12:05.241 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.242 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434f4db109af53cd-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.247 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.247 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.248 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.248 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.249 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:12:05.249 DEBUG:boto:Data: 2023-10-12T01:12:05.249 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.250 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.251 DEBUG:boto:Params: {} 2023-10-12T01:12:05.251 DEBUG:boto:Token: None 2023-10-12T01:12:05.252 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155 2023-10-12T01:12:05.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cyD23gkrFpOnGbGindF/G7Xl7CU= 2023-10-12T01:12:05.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cyD23gkrFpOnGbGindF/G7Xl7CU=', 'Content-Length': '0'} 2023-10-12T01:12:05.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d8f1d4a078fe4d-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.255 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.255 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.256 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.256 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:12:05.257 DEBUG:boto:Data: 2023-10-12T01:12:05.257 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.258 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.258 DEBUG:boto:Params: {} 2023-10-12T01:12:05.259 DEBUG:boto:Token: None 2023-10-12T01:12:05.259 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155 2023-10-12T01:12:05.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cyD23gkrFpOnGbGindF/G7Xl7CU= 2023-10-12T01:12:05.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cyD23gkrFpOnGbGindF/G7Xl7CU=', 'Content-Length': '0'} 2023-10-12T01:12:05.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b68c6d9ea0c5cf6-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:12:05.263 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.264 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.264 DEBUG:boto:Method: GET 2023-10-12T01:12:05.265 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:12:05.265 DEBUG:boto:Data: 2023-10-12T01:12:05.265 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.266 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.267 DEBUG:boto:Params: {} 2023-10-12T01:12:05.267 DEBUG:boto:Token: None 2023-10-12T01:12:05.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155 2023-10-12T01:12:05.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dqQVNjX3Wue1OZ21Iiv89AtzXkA= 2023-10-12T01:12:05.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dqQVNjX3Wue1OZ21Iiv89AtzXkA=', 'Content-Length': '0'} 2023-10-12T01:12:05.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ae4e5a807bb10b-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.314 DEBUG:boto:path=/wafmoa-59/obj155 2023-10-12T01:12:05.315 DEBUG:boto:auth_path=/wafmoa-59/obj155 2023-10-12T01:12:05.315 DEBUG:boto:Method: GET 2023-10-12T01:12:05.316 DEBUG:boto:Path: /wafmoa-59/obj155 2023-10-12T01:12:05.316 DEBUG:boto:Data: 2023-10-12T01:12:05.317 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.318 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.318 DEBUG:boto:Params: {} 2023-10-12T01:12:05.319 DEBUG:boto:Token: None 2023-10-12T01:12:05.319 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj155 2023-10-12T01:12:05.320 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dqQVNjX3Wue1OZ21Iiv89AtzXkA= 2023-10-12T01:12:05.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dqQVNjX3Wue1OZ21Iiv89AtzXkA=', 'Content-Length': '0'} 2023-10-12T01:12:05.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef07b5628a0f9d4a-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.323 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.323 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.324 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.324 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.325 DEBUG:boto:Method: GET 2023-10-12T01:12:05.325 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.326 DEBUG:boto:Data: 2023-10-12T01:12:05.326 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.327 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.327 DEBUG:boto:Params: {} 2023-10-12T01:12:05.328 DEBUG:boto:Token: None 2023-10-12T01:12:05.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb42f9f7e749fd8d-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.332 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.333 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.334 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.334 DEBUG:boto:path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.335 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.335 DEBUG:boto:Method: GET 2023-10-12T01:12:05.335 DEBUG:boto:Path: /wafmoa-59/?prefix=obj155 2023-10-12T01:12:05.336 DEBUG:boto:Data: 2023-10-12T01:12:05.336 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.337 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.338 DEBUG:boto:Params: {} 2023-10-12T01:12:05.338 DEBUG:boto:Token: None 2023-10-12T01:12:05.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b4e581f3bbe57bc-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.390 DEBUG:boto:b'wafmoa-59obj1551000falseobj1552023-10-12T01:06:27.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:12:05.392 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.392 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.393 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.393 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.394 DEBUG:boto:Method: GET 2023-10-12T01:12:05.394 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.394 DEBUG:boto:Data: 2023-10-12T01:12:05.395 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.396 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.396 DEBUG:boto:Params: {} 2023-10-12T01:12:05.397 DEBUG:boto:Token: None 2023-10-12T01:12:05.397 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM= 2023-10-12T01:12:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM=', 'Content-Length': '0'} 2023-10-12T01:12:05.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a2a6cb5261d455-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.401 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.402 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.402 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.403 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.403 DEBUG:boto:Method: GET 2023-10-12T01:12:05.404 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.404 DEBUG:boto:Data: 2023-10-12T01:12:05.405 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.405 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.406 DEBUG:boto:Params: {} 2023-10-12T01:12:05.407 DEBUG:boto:Token: None 2023-10-12T01:12:05.407 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM= 2023-10-12T01:12:05.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM=', 'Content-Length': '0'} 2023-10-12T01:12:05.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9fce53c8c6af48a-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.411 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.411 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.412 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.412 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.412 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.413 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.413 DEBUG:boto:Data: 2023-10-12T01:12:05.414 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.414 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.415 DEBUG:boto:Params: {} 2023-10-12T01:12:05.415 DEBUG:boto:Token: None 2023-10-12T01:12:05.416 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9RkxWa8YsWVXXTIjT51LOQB/03M= 2023-10-12T01:12:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9RkxWa8YsWVXXTIjT51LOQB/03M=', 'Content-Length': '0'} 2023-10-12T01:12:05.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068218ca4c4ac881a-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.419 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.420 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.420 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.421 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.421 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.421 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.422 DEBUG:boto:Data: 2023-10-12T01:12:05.422 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.423 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.423 DEBUG:boto:Params: {} 2023-10-12T01:12:05.424 DEBUG:boto:Token: None 2023-10-12T01:12:05.424 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9RkxWa8YsWVXXTIjT51LOQB/03M= 2023-10-12T01:12:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9RkxWa8YsWVXXTIjT51LOQB/03M=', 'Content-Length': '0'} 2023-10-12T01:12:05.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab49662a12b9271-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:12:05.428 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.429 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.429 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.429 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.430 DEBUG:boto:Method: GET 2023-10-12T01:12:05.430 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.431 DEBUG:boto:Data: 2023-10-12T01:12:05.431 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.432 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.433 DEBUG:boto:Params: {} 2023-10-12T01:12:05.433 DEBUG:boto:Token: None 2023-10-12T01:12:05.434 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.434 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM= 2023-10-12T01:12:05.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM=', 'Content-Length': '0'} 2023-10-12T01:12:05.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f218dbc8b623d0-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.437 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.438 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.438 DEBUG:boto:path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.439 DEBUG:boto:auth_path=/wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.439 DEBUG:boto:Method: GET 2023-10-12T01:12:05.439 DEBUG:boto:Path: /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.440 DEBUG:boto:Data: 2023-10-12T01:12:05.440 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.441 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.442 DEBUG:boto:Params: {} 2023-10-12T01:12:05.442 DEBUG:boto:Token: None 2023-10-12T01:12:05.443 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17?versionId=null 2023-10-12T01:12:05.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM= 2023-10-12T01:12:05.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IjRg+PxAv6NfscsFG/l8WvDKRDM=', 'Content-Length': '0'} 2023-10-12T01:12:05.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085437cb70e560622-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.487 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.487 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.488 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.489 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.489 DEBUG:boto:Method: GET 2023-10-12T01:12:05.490 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.490 DEBUG:boto:Data: 2023-10-12T01:12:05.490 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.491 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.492 DEBUG:boto:Params: {} 2023-10-12T01:12:05.492 DEBUG:boto:Token: None 2023-10-12T01:12:05.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240e469d3a4368a9-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.497 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.498 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.499 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.499 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.500 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.500 DEBUG:boto:Method: GET 2023-10-12T01:12:05.500 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.501 DEBUG:boto:Data: 2023-10-12T01:12:05.501 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.502 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.503 DEBUG:boto:Params: {} 2023-10-12T01:12:05.503 DEBUG:boto:Token: None 2023-10-12T01:12:05.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb590cf03960a2f9-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.509 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.510 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.511 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.511 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.511 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:12:05.512 DEBUG:boto:Data: 2023-10-12T01:12:05.512 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.513 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.513 DEBUG:boto:Params: {} 2023-10-12T01:12:05.514 DEBUG:boto:Token: None 2023-10-12T01:12:05.514 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17 2023-10-12T01:12:05.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tqMDnC0JTJHKgJQEp2YPSFTQ/WM= 2023-10-12T01:12:05.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tqMDnC0JTJHKgJQEp2YPSFTQ/WM=', 'Content-Length': '0'} 2023-10-12T01:12:05.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082378b9d90824523-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.518 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.518 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.519 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.519 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:12:05.520 DEBUG:boto:Data: 2023-10-12T01:12:05.520 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.521 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.521 DEBUG:boto:Params: {} 2023-10-12T01:12:05.522 DEBUG:boto:Token: None 2023-10-12T01:12:05.522 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17 2023-10-12T01:12:05.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tqMDnC0JTJHKgJQEp2YPSFTQ/WM= 2023-10-12T01:12:05.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tqMDnC0JTJHKgJQEp2YPSFTQ/WM=', 'Content-Length': '0'} 2023-10-12T01:12:05.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b769917489f5727f-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:12:05.526 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.527 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.527 DEBUG:boto:Method: GET 2023-10-12T01:12:05.527 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:12:05.528 DEBUG:boto:Data: 2023-10-12T01:12:05.528 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.529 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.530 DEBUG:boto:Params: {} 2023-10-12T01:12:05.530 DEBUG:boto:Token: None 2023-10-12T01:12:05.531 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17 2023-10-12T01:12:05.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4fl8qNoPz8AfXCbc02zk07lVrBM= 2023-10-12T01:12:05.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4fl8qNoPz8AfXCbc02zk07lVrBM=', 'Content-Length': '0'} 2023-10-12T01:12:05.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f15318d947dc7a5-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.534 DEBUG:boto:path=/wafmoa-59/obj17 2023-10-12T01:12:05.535 DEBUG:boto:auth_path=/wafmoa-59/obj17 2023-10-12T01:12:05.535 DEBUG:boto:Method: GET 2023-10-12T01:12:05.536 DEBUG:boto:Path: /wafmoa-59/obj17 2023-10-12T01:12:05.536 DEBUG:boto:Data: 2023-10-12T01:12:05.536 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.537 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.538 DEBUG:boto:Params: {} 2023-10-12T01:12:05.538 DEBUG:boto:Token: None 2023-10-12T01:12:05.539 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj17 2023-10-12T01:12:05.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4fl8qNoPz8AfXCbc02zk07lVrBM= 2023-10-12T01:12:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4fl8qNoPz8AfXCbc02zk07lVrBM=', 'Content-Length': '0'} 2023-10-12T01:12:05.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cacfbad4ecbee4f-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.543 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.543 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.543 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.544 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.544 DEBUG:boto:Method: GET 2023-10-12T01:12:05.545 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.545 DEBUG:boto:Data: 2023-10-12T01:12:05.546 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.546 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.547 DEBUG:boto:Params: {} 2023-10-12T01:12:05.547 DEBUG:boto:Token: None 2023-10-12T01:12:05.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59cc1a5d66b0586-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.552 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.553 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.554 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.554 DEBUG:boto:path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.555 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.555 DEBUG:boto:Method: GET 2023-10-12T01:12:05.556 DEBUG:boto:Path: /wafmoa-59/?prefix=obj17 2023-10-12T01:12:05.556 DEBUG:boto:Data: 2023-10-12T01:12:05.557 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.557 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.558 DEBUG:boto:Params: {} 2023-10-12T01:12:05.558 DEBUG:boto:Token: None 2023-10-12T01:12:05.559 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aadfa88dd803c45-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.565 DEBUG:boto:b'wafmoa-59obj171000falseobj172023-10-12T00:59:23.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:12:05.566 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.567 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.567 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.568 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.568 DEBUG:boto:Method: GET 2023-10-12T01:12:05.569 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.569 DEBUG:boto:Data: 2023-10-12T01:12:05.570 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.572 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.572 DEBUG:boto:Params: {} 2023-10-12T01:12:05.573 DEBUG:boto:Token: None 2023-10-12T01:12:05.574 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.574 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY= 2023-10-12T01:12:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY=', 'Content-Length': '0'} 2023-10-12T01:12:05.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7a9e603a7ff7ea-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.626 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.627 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.627 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.628 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.628 DEBUG:boto:Method: GET 2023-10-12T01:12:05.629 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.629 DEBUG:boto:Data: 2023-10-12T01:12:05.630 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.631 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.631 DEBUG:boto:Params: {} 2023-10-12T01:12:05.632 DEBUG:boto:Token: None 2023-10-12T01:12:05.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY= 2023-10-12T01:12:05.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY=', 'Content-Length': '0'} 2023-10-12T01:12:05.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87a3ff4a2144a5c-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.636 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.637 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.637 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.638 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.638 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.639 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.639 DEBUG:boto:Data: 2023-10-12T01:12:05.640 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.640 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.641 DEBUG:boto:Params: {} 2023-10-12T01:12:05.641 DEBUG:boto:Token: None 2023-10-12T01:12:05.642 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.643 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gzFn4n6/QAUI4N9aTmPqvG+KVXE= 2023-10-12T01:12:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gzFn4n6/QAUI4N9aTmPqvG+KVXE=', 'Content-Length': '0'} 2023-10-12T01:12:05.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009481073273fd4ff7-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.646 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.647 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.648 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.649 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.649 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.650 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.651 DEBUG:boto:Data: 2023-10-12T01:12:05.652 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.653 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.654 DEBUG:boto:Params: {} 2023-10-12T01:12:05.655 DEBUG:boto:Token: None 2023-10-12T01:12:05.656 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.657 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gzFn4n6/QAUI4N9aTmPqvG+KVXE= 2023-10-12T01:12:05.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gzFn4n6/QAUI4N9aTmPqvG+KVXE=', 'Content-Length': '0'} 2023-10-12T01:12:05.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a450abcac86427-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:12:05.662 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.663 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.664 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.664 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.665 DEBUG:boto:Method: GET 2023-10-12T01:12:05.666 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.667 DEBUG:boto:Data: 2023-10-12T01:12:05.668 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.669 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.670 DEBUG:boto:Params: {} 2023-10-12T01:12:05.671 DEBUG:boto:Token: None 2023-10-12T01:12:05.671 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.672 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY= 2023-10-12T01:12:05.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY=', 'Content-Length': '0'} 2023-10-12T01:12:05.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51a307220601416-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.676 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.676 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.677 DEBUG:boto:path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.677 DEBUG:boto:auth_path=/wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.678 DEBUG:boto:Method: GET 2023-10-12T01:12:05.678 DEBUG:boto:Path: /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.679 DEBUG:boto:Data: 2023-10-12T01:12:05.679 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.680 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.680 DEBUG:boto:Params: {} 2023-10-12T01:12:05.681 DEBUG:boto:Token: None 2023-10-12T01:12:05.681 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185?versionId=null 2023-10-12T01:12:05.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY= 2023-10-12T01:12:05.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5crLmNvVIHPHFsPdXzekFsZJqY=', 'Content-Length': '0'} 2023-10-12T01:12:05.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a85d01f84441e8-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.685 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.686 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.686 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.687 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.687 DEBUG:boto:Method: GET 2023-10-12T01:12:05.688 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.688 DEBUG:boto:Data: 2023-10-12T01:12:05.688 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.689 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.690 DEBUG:boto:Params: {} 2023-10-12T01:12:05.690 DEBUG:boto:Token: None 2023-10-12T01:12:05.691 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.691 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642f4d2a2a2a7b25-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.695 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.696 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.696 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.697 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.697 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.697 DEBUG:boto:Method: GET 2023-10-12T01:12:05.698 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.698 DEBUG:boto:Data: 2023-10-12T01:12:05.699 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.700 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.700 DEBUG:boto:Params: {} 2023-10-12T01:12:05.701 DEBUG:boto:Token: None 2023-10-12T01:12:05.701 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.702 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ece708c4b46386d-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.707 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.708 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.708 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.709 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.709 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:12:05.710 DEBUG:boto:Data: 2023-10-12T01:12:05.710 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.711 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.711 DEBUG:boto:Params: {} 2023-10-12T01:12:05.712 DEBUG:boto:Token: None 2023-10-12T01:12:05.712 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185 2023-10-12T01:12:05.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cYARo9uva3/i5J8le4BzF1wFgBk= 2023-10-12T01:12:05.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cYARo9uva3/i5J8le4BzF1wFgBk=', 'Content-Length': '0'} 2023-10-12T01:12:05.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008224267c1e26e4e0-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.716 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.716 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.717 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.717 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:12:05.718 DEBUG:boto:Data: 2023-10-12T01:12:05.718 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.719 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.719 DEBUG:boto:Params: {} 2023-10-12T01:12:05.720 DEBUG:boto:Token: None 2023-10-12T01:12:05.720 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185 2023-10-12T01:12:05.721 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cYARo9uva3/i5J8le4BzF1wFgBk= 2023-10-12T01:12:05.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cYARo9uva3/i5J8le4BzF1wFgBk=', 'Content-Length': '0'} 2023-10-12T01:12:05.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7938e8f529e1198-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:12:05.724 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.724 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.725 DEBUG:boto:Method: GET 2023-10-12T01:12:05.725 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:12:05.726 DEBUG:boto:Data: 2023-10-12T01:12:05.726 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.727 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.727 DEBUG:boto:Params: {} 2023-10-12T01:12:05.728 DEBUG:boto:Token: None 2023-10-12T01:12:05.728 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185 2023-10-12T01:12:05.729 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJWnNJvYoe6sC5+kELbeZVsp3xU= 2023-10-12T01:12:05.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJWnNJvYoe6sC5+kELbeZVsp3xU=', 'Content-Length': '0'} 2023-10-12T01:12:05.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705fa696ede72828-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.774 DEBUG:boto:path=/wafmoa-59/obj185 2023-10-12T01:12:05.775 DEBUG:boto:auth_path=/wafmoa-59/obj185 2023-10-12T01:12:05.776 DEBUG:boto:Method: GET 2023-10-12T01:12:05.776 DEBUG:boto:Path: /wafmoa-59/obj185 2023-10-12T01:12:05.776 DEBUG:boto:Data: 2023-10-12T01:12:05.777 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.778 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.778 DEBUG:boto:Params: {} 2023-10-12T01:12:05.779 DEBUG:boto:Token: None 2023-10-12T01:12:05.779 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj185 2023-10-12T01:12:05.780 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pJWnNJvYoe6sC5+kELbeZVsp3xU= 2023-10-12T01:12:05.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pJWnNJvYoe6sC5+kELbeZVsp3xU=', 'Content-Length': '0'} 2023-10-12T01:12:05.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029cb109cfe33d28c-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.783 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.783 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.784 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.784 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.785 DEBUG:boto:Method: GET 2023-10-12T01:12:05.785 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.786 DEBUG:boto:Data: 2023-10-12T01:12:05.786 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.787 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.787 DEBUG:boto:Params: {} 2023-10-12T01:12:05.788 DEBUG:boto:Token: None 2023-10-12T01:12:05.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006296a75b9731e051-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.793 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.794 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.794 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.795 DEBUG:boto:path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.795 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.795 DEBUG:boto:Method: GET 2023-10-12T01:12:05.796 DEBUG:boto:Path: /wafmoa-59/?prefix=obj185 2023-10-12T01:12:05.796 DEBUG:boto:Data: 2023-10-12T01:12:05.797 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.797 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.798 DEBUG:boto:Params: {} 2023-10-12T01:12:05.798 DEBUG:boto:Token: None 2023-10-12T01:12:05.799 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.800 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100a1ab1ce951f8e-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.805 DEBUG:boto:b'wafmoa-59obj1851000falseobj1852023-10-12T01:09:30.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:12:05.806 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.807 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.807 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.808 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.808 DEBUG:boto:Method: GET 2023-10-12T01:12:05.808 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.809 DEBUG:boto:Data: 2023-10-12T01:12:05.809 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.810 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.811 DEBUG:boto:Params: {} 2023-10-12T01:12:05.811 DEBUG:boto:Token: None 2023-10-12T01:12:05.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.812 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48= 2023-10-12T01:12:05.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48=', 'Content-Length': '0'} 2023-10-12T01:12:05.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb4ba081d719508-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.816 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.817 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.817 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.817 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.818 DEBUG:boto:Method: GET 2023-10-12T01:12:05.818 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.819 DEBUG:boto:Data: 2023-10-12T01:12:05.820 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.821 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.821 DEBUG:boto:Params: {} 2023-10-12T01:12:05.822 DEBUG:boto:Token: None 2023-10-12T01:12:05.822 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48= 2023-10-12T01:12:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48=', 'Content-Length': '0'} 2023-10-12T01:12:05.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f7d33b98bbcaef-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.826 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.826 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.827 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.827 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.828 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.828 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.829 DEBUG:boto:Data: 2023-10-12T01:12:05.829 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.830 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.830 DEBUG:boto:Params: {} 2023-10-12T01:12:05.831 DEBUG:boto:Token: None 2023-10-12T01:12:05.831 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:761HiTkWXztj8qfm6STywWEQlxU= 2023-10-12T01:12:05.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:761HiTkWXztj8qfm6STywWEQlxU=', 'Content-Length': '0'} 2023-10-12T01:12:05.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000462e5b988246d22a-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.835 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.836 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.836 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.836 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.837 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.837 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.838 DEBUG:boto:Data: 2023-10-12T01:12:05.838 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.839 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.840 DEBUG:boto:Params: {} 2023-10-12T01:12:05.840 DEBUG:boto:Token: None 2023-10-12T01:12:05.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:761HiTkWXztj8qfm6STywWEQlxU= 2023-10-12T01:12:05.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:761HiTkWXztj8qfm6STywWEQlxU=', 'Content-Length': '0'} 2023-10-12T01:12:05.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009991a6d56c7be347-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:12:05.845 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.845 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.846 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.846 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.847 DEBUG:boto:Method: GET 2023-10-12T01:12:05.847 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.848 DEBUG:boto:Data: 2023-10-12T01:12:05.848 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.849 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.849 DEBUG:boto:Params: {} 2023-10-12T01:12:05.850 DEBUG:boto:Token: None 2023-10-12T01:12:05.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48= 2023-10-12T01:12:05.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48=', 'Content-Length': '0'} 2023-10-12T01:12:05.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4f34fc1705a6e6-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.894 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.895 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.895 DEBUG:boto:path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.896 DEBUG:boto:auth_path=/wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.896 DEBUG:boto:Method: GET 2023-10-12T01:12:05.897 DEBUG:boto:Path: /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.897 DEBUG:boto:Data: 2023-10-12T01:12:05.898 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.898 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.899 DEBUG:boto:Params: {} 2023-10-12T01:12:05.899 DEBUG:boto:Token: None 2023-10-12T01:12:05.900 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19?versionId=null 2023-10-12T01:12:05.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48= 2023-10-12T01:12:05.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIhzTZLDxvcI+XJ/N8V+b9Twp48=', 'Content-Length': '0'} 2023-10-12T01:12:05.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f98f76f6912ff1-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.904 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.904 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.905 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.905 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.905 DEBUG:boto:Method: GET 2023-10-12T01:12:05.906 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.906 DEBUG:boto:Data: 2023-10-12T01:12:05.907 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.909 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.909 DEBUG:boto:Params: {} 2023-10-12T01:12:05.910 DEBUG:boto:Token: None 2023-10-12T01:12:05.910 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e29b5a0b0015b22-00652747e5-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.915 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.916 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:05.916 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:05.916 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.917 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.917 DEBUG:boto:Method: GET 2023-10-12T01:12:05.918 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:12:05.918 DEBUG:boto:Data: 2023-10-12T01:12:05.919 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.919 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.920 DEBUG:boto:Params: {} 2023-10-12T01:12:05.920 DEBUG:boto:Token: None 2023-10-12T01:12:05.921 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/ 2023-10-12T01:12:05.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE= 2023-10-12T01:12:05.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KZ+Y4m/DUPZ2inNROeYylQ9U1eE=', 'Content-Length': '0'} 2023-10-12T01:12:05.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037e14d09f18d1f81-00652747e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.966 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:05.967 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.968 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.968 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.969 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:12:05.969 DEBUG:boto:Data: 2023-10-12T01:12:05.969 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.970 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.971 DEBUG:boto:Params: {} 2023-10-12T01:12:05.971 DEBUG:boto:Token: None 2023-10-12T01:12:05.972 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19 2023-10-12T01:12:05.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BR1kRcabiKwgM476y2xO9BwOGS4= 2023-10-12T01:12:05.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BR1kRcabiKwgM476y2xO9BwOGS4=', 'Content-Length': '0'} 2023-10-12T01:12:05.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006331ffe4426751e5-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.975 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.976 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.976 DEBUG:boto:Method: HEAD 2023-10-12T01:12:05.977 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:12:05.977 DEBUG:boto:Data: 2023-10-12T01:12:05.978 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.978 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.979 DEBUG:boto:Params: {} 2023-10-12T01:12:05.979 DEBUG:boto:Token: None 2023-10-12T01:12:05.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19 2023-10-12T01:12:05.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BR1kRcabiKwgM476y2xO9BwOGS4= 2023-10-12T01:12:05.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BR1kRcabiKwgM476y2xO9BwOGS4=', 'Content-Length': '0'} 2023-10-12T01:12:05.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044bb5643bb4cc71b-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:12:05.983 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.984 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.984 DEBUG:boto:Method: GET 2023-10-12T01:12:05.985 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:12:05.985 DEBUG:boto:Data: 2023-10-12T01:12:05.986 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:05.986 DEBUG:boto:Port: 8001 2023-10-12T01:12:05.987 DEBUG:boto:Params: {} 2023-10-12T01:12:05.987 DEBUG:boto:Token: None 2023-10-12T01:12:05.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19 2023-10-12T01:12:05.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WkW+ZpaloGxlXC6LwdZ575ito9w= 2023-10-12T01:12:05.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WkW+ZpaloGxlXC6LwdZ575ito9w=', 'Content-Length': '0'} 2023-10-12T01:12:05.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49ddbd67d241c8d-00652747e5-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.991 DEBUG:boto:path=/wafmoa-59/obj19 2023-10-12T01:12:05.992 DEBUG:boto:auth_path=/wafmoa-59/obj19 2023-10-12T01:12:05.992 DEBUG:boto:Method: GET 2023-10-12T01:12:05.993 DEBUG:boto:Path: /wafmoa-59/obj19 2023-10-12T01:12:05.993 DEBUG:boto:Data: 2023-10-12T01:12:05.994 DEBUG:boto:Headers: {} 2023-10-12T01:12:05.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:05.994 DEBUG:boto:Port: 8000 2023-10-12T01:12:05.995 DEBUG:boto:Params: {} 2023-10-12T01:12:05.995 DEBUG:boto:Token: None 2023-10-12T01:12:05.996 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:05 GMT /wafmoa-59/obj19 2023-10-12T01:12:05.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WkW+ZpaloGxlXC6LwdZ575ito9w= 2023-10-12T01:12:05.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WkW+ZpaloGxlXC6LwdZ575ito9w=', 'Content-Length': '0'} 2023-10-12T01:12:05.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842dc795a25f84d6-00652747e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:05.999 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.000 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.000 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.001 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.001 DEBUG:boto:Method: GET 2023-10-12T01:12:06.002 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.002 DEBUG:boto:Data: 2023-10-12T01:12:06.003 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.003 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.004 DEBUG:boto:Params: {} 2023-10-12T01:12:06.004 DEBUG:boto:Token: None 2023-10-12T01:12:06.005 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eedea540faefcef1-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.050 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.051 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.052 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.052 DEBUG:boto:path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.053 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.053 DEBUG:boto:Method: GET 2023-10-12T01:12:06.053 DEBUG:boto:Path: /wafmoa-59/?prefix=obj19 2023-10-12T01:12:06.054 DEBUG:boto:Data: 2023-10-12T01:12:06.054 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.055 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.056 DEBUG:boto:Params: {} 2023-10-12T01:12:06.056 DEBUG:boto:Token: None 2023-10-12T01:12:06.057 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.057 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8e80e0eb24b5a5-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.063 DEBUG:boto:b'wafmoa-59obj191000falseobj192023-10-12T01:01:06.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:12:06.064 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.064 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.065 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.065 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.065 DEBUG:boto:Method: GET 2023-10-12T01:12:06.066 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.066 DEBUG:boto:Data: 2023-10-12T01:12:06.067 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.068 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.068 DEBUG:boto:Params: {} 2023-10-12T01:12:06.068 DEBUG:boto:Token: None 2023-10-12T01:12:06.069 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY= 2023-10-12T01:12:06.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY=', 'Content-Length': '0'} 2023-10-12T01:12:06.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765544bd56d498e6-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.073 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.074 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.074 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.075 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.075 DEBUG:boto:Method: GET 2023-10-12T01:12:06.075 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.076 DEBUG:boto:Data: 2023-10-12T01:12:06.076 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.077 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.078 DEBUG:boto:Params: {} 2023-10-12T01:12:06.078 DEBUG:boto:Token: None 2023-10-12T01:12:06.079 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.079 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY= 2023-10-12T01:12:06.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY=', 'Content-Length': '0'} 2023-10-12T01:12:06.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65294c9d604d689-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.083 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.083 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.084 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.084 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.085 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.085 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.085 DEBUG:boto:Data: 2023-10-12T01:12:06.086 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.087 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.087 DEBUG:boto:Params: {} 2023-10-12T01:12:06.088 DEBUG:boto:Token: None 2023-10-12T01:12:06.088 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.089 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a5GP3jiUkMObu+iYoJJFG+egFEw= 2023-10-12T01:12:06.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a5GP3jiUkMObu+iYoJJFG+egFEw=', 'Content-Length': '0'} 2023-10-12T01:12:06.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000166062ecca7c381c-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.092 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.092 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.093 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.093 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.093 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.094 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.094 DEBUG:boto:Data: 2023-10-12T01:12:06.095 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.095 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.096 DEBUG:boto:Params: {} 2023-10-12T01:12:06.096 DEBUG:boto:Token: None 2023-10-12T01:12:06.097 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.097 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:a5GP3jiUkMObu+iYoJJFG+egFEw= 2023-10-12T01:12:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:a5GP3jiUkMObu+iYoJJFG+egFEw=', 'Content-Length': '0'} 2023-10-12T01:12:06.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b7e465cb0bb4a1-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:12:06.101 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.101 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.102 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.102 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.103 DEBUG:boto:Method: GET 2023-10-12T01:12:06.103 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.104 DEBUG:boto:Data: 2023-10-12T01:12:06.104 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.105 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.105 DEBUG:boto:Params: {} 2023-10-12T01:12:06.106 DEBUG:boto:Token: None 2023-10-12T01:12:06.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY= 2023-10-12T01:12:06.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY=', 'Content-Length': '0'} 2023-10-12T01:12:06.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000394eb163d587bae-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.110 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.111 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.115 DEBUG:boto:path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.116 DEBUG:boto:auth_path=/wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.116 DEBUG:boto:Method: GET 2023-10-12T01:12:06.117 DEBUG:boto:Path: /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.117 DEBUG:boto:Data: 2023-10-12T01:12:06.118 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.119 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.119 DEBUG:boto:Params: {} 2023-10-12T01:12:06.120 DEBUG:boto:Token: None 2023-10-12T01:12:06.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22?versionId=null 2023-10-12T01:12:06.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY= 2023-10-12T01:12:06.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ILTAW2uY1QY/4+hkdPxv2TsWkCY=', 'Content-Length': '0'} 2023-10-12T01:12:06.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02790bfea857d80-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.125 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.125 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.126 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.126 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.127 DEBUG:boto:Method: GET 2023-10-12T01:12:06.127 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.128 DEBUG:boto:Data: 2023-10-12T01:12:06.128 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.129 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.130 DEBUG:boto:Params: {} 2023-10-12T01:12:06.131 DEBUG:boto:Token: None 2023-10-12T01:12:06.132 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.133 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7c23b28e26f580-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.137 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.138 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.139 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.139 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.140 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.140 DEBUG:boto:Method: GET 2023-10-12T01:12:06.140 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.141 DEBUG:boto:Data: 2023-10-12T01:12:06.142 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.142 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.143 DEBUG:boto:Params: {} 2023-10-12T01:12:06.143 DEBUG:boto:Token: None 2023-10-12T01:12:06.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000663c691677422c3-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.150 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.152 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.152 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.153 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.154 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:12:06.155 DEBUG:boto:Data: 2023-10-12T01:12:06.156 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.157 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.158 DEBUG:boto:Params: {} 2023-10-12T01:12:06.159 DEBUG:boto:Token: None 2023-10-12T01:12:06.159 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22 2023-10-12T01:12:06.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qZjqy13gjPcCsGX8ZCKG8szI4C0= 2023-10-12T01:12:06.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qZjqy13gjPcCsGX8ZCKG8szI4C0=', 'Content-Length': '0'} 2023-10-12T01:12:06.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc7c946b7bf91a9-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.165 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.165 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.166 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.167 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:12:06.168 DEBUG:boto:Data: 2023-10-12T01:12:06.168 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.170 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.170 DEBUG:boto:Params: {} 2023-10-12T01:12:06.171 DEBUG:boto:Token: None 2023-10-12T01:12:06.172 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22 2023-10-12T01:12:06.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qZjqy13gjPcCsGX8ZCKG8szI4C0= 2023-10-12T01:12:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qZjqy13gjPcCsGX8ZCKG8szI4C0=', 'Content-Length': '0'} 2023-10-12T01:12:06.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ea95ef488aab07-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:12:06.177 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.178 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.178 DEBUG:boto:Method: GET 2023-10-12T01:12:06.179 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:12:06.179 DEBUG:boto:Data: 2023-10-12T01:12:06.180 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.181 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.181 DEBUG:boto:Params: {} 2023-10-12T01:12:06.182 DEBUG:boto:Token: None 2023-10-12T01:12:06.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22 2023-10-12T01:12:06.183 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5+SgdO6QeDEEVANPOvZ5PBw3AQA= 2023-10-12T01:12:06.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5+SgdO6QeDEEVANPOvZ5PBw3AQA=', 'Content-Length': '0'} 2023-10-12T01:12:06.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1814a4b7867e62e-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.186 DEBUG:boto:path=/wafmoa-59/obj22 2023-10-12T01:12:06.187 DEBUG:boto:auth_path=/wafmoa-59/obj22 2023-10-12T01:12:06.187 DEBUG:boto:Method: GET 2023-10-12T01:12:06.188 DEBUG:boto:Path: /wafmoa-59/obj22 2023-10-12T01:12:06.188 DEBUG:boto:Data: 2023-10-12T01:12:06.189 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.189 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.190 DEBUG:boto:Params: {} 2023-10-12T01:12:06.191 DEBUG:boto:Token: None 2023-10-12T01:12:06.191 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj22 2023-10-12T01:12:06.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5+SgdO6QeDEEVANPOvZ5PBw3AQA= 2023-10-12T01:12:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5+SgdO6QeDEEVANPOvZ5PBw3AQA=', 'Content-Length': '0'} 2023-10-12T01:12:06.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36bc846e4a7b349-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.234 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.235 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.236 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.236 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.236 DEBUG:boto:Method: GET 2023-10-12T01:12:06.237 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.237 DEBUG:boto:Data: 2023-10-12T01:12:06.238 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.239 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.239 DEBUG:boto:Params: {} 2023-10-12T01:12:06.240 DEBUG:boto:Token: None 2023-10-12T01:12:06.240 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051426541f7036506-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.245 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.245 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.246 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.246 DEBUG:boto:path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.247 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.247 DEBUG:boto:Method: GET 2023-10-12T01:12:06.248 DEBUG:boto:Path: /wafmoa-59/?prefix=obj22 2023-10-12T01:12:06.248 DEBUG:boto:Data: 2023-10-12T01:12:06.248 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.249 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.249 DEBUG:boto:Params: {} 2023-10-12T01:12:06.250 DEBUG:boto:Token: None 2023-10-12T01:12:06.251 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.251 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d72bdaada73779-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.256 DEBUG:boto:b'wafmoa-59obj221000falseobj222023-10-12T00:57:47.178Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:12:06.257 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.258 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.258 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.259 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.259 DEBUG:boto:Method: GET 2023-10-12T01:12:06.260 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.260 DEBUG:boto:Data: 2023-10-12T01:12:06.261 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.261 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.262 DEBUG:boto:Params: {} 2023-10-12T01:12:06.262 DEBUG:boto:Token: None 2023-10-12T01:12:06.263 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s= 2023-10-12T01:12:06.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s=', 'Content-Length': '0'} 2023-10-12T01:12:06.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4617c1806ba287-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.310 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.311 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.311 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.312 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.312 DEBUG:boto:Method: GET 2023-10-12T01:12:06.313 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.313 DEBUG:boto:Data: 2023-10-12T01:12:06.314 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.314 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.315 DEBUG:boto:Params: {} 2023-10-12T01:12:06.317 DEBUG:boto:Token: None 2023-10-12T01:12:06.318 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.318 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s= 2023-10-12T01:12:06.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s=', 'Content-Length': '0'} 2023-10-12T01:12:06.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ee34137b81c2c0-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.322 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.322 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.323 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.323 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.324 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.324 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.324 DEBUG:boto:Data: 2023-10-12T01:12:06.325 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.326 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.326 DEBUG:boto:Params: {} 2023-10-12T01:12:06.327 DEBUG:boto:Token: None 2023-10-12T01:12:06.327 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0f3OFVJbf3VQ9fNj+vnclZ0Oxv0= 2023-10-12T01:12:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0f3OFVJbf3VQ9fNj+vnclZ0Oxv0=', 'Content-Length': '0'} 2023-10-12T01:12:06.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce513e9d36c552e-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.331 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.331 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.331 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.332 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.332 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.333 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.333 DEBUG:boto:Data: 2023-10-12T01:12:06.333 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.334 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.335 DEBUG:boto:Params: {} 2023-10-12T01:12:06.335 DEBUG:boto:Token: None 2023-10-12T01:12:06.336 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0f3OFVJbf3VQ9fNj+vnclZ0Oxv0= 2023-10-12T01:12:06.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0f3OFVJbf3VQ9fNj+vnclZ0Oxv0=', 'Content-Length': '0'} 2023-10-12T01:12:06.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e827f0ff8c39dc-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:12:06.340 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.340 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.341 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.341 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.341 DEBUG:boto:Method: GET 2023-10-12T01:12:06.342 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.342 DEBUG:boto:Data: 2023-10-12T01:12:06.343 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.344 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.344 DEBUG:boto:Params: {} 2023-10-12T01:12:06.344 DEBUG:boto:Token: None 2023-10-12T01:12:06.345 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.345 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s= 2023-10-12T01:12:06.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s=', 'Content-Length': '0'} 2023-10-12T01:12:06.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f751477217de08-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.349 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.349 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.350 DEBUG:boto:path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.350 DEBUG:boto:auth_path=/wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.350 DEBUG:boto:Method: GET 2023-10-12T01:12:06.351 DEBUG:boto:Path: /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.351 DEBUG:boto:Data: 2023-10-12T01:12:06.352 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.352 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.353 DEBUG:boto:Params: {} 2023-10-12T01:12:06.353 DEBUG:boto:Token: None 2023-10-12T01:12:06.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23?versionId=null 2023-10-12T01:12:06.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s= 2023-10-12T01:12:06.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WQDSu4jSLzFLz5v8f0X1Ip7G80s=', 'Content-Length': '0'} 2023-10-12T01:12:06.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e0e07233ddf992-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.358 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.358 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.359 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.359 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.360 DEBUG:boto:Method: GET 2023-10-12T01:12:06.360 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.361 DEBUG:boto:Data: 2023-10-12T01:12:06.361 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.362 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.362 DEBUG:boto:Params: {} 2023-10-12T01:12:06.363 DEBUG:boto:Token: None 2023-10-12T01:12:06.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e233b11dd84659-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.367 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.368 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.368 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.369 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.369 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.370 DEBUG:boto:Method: GET 2023-10-12T01:12:06.370 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.371 DEBUG:boto:Data: 2023-10-12T01:12:06.371 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.372 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.372 DEBUG:boto:Params: {} 2023-10-12T01:12:06.373 DEBUG:boto:Token: None 2023-10-12T01:12:06.373 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.374 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006178658848c39d88-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.379 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.380 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.380 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.381 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.381 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:12:06.382 DEBUG:boto:Data: 2023-10-12T01:12:06.382 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.383 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.383 DEBUG:boto:Params: {} 2023-10-12T01:12:06.384 DEBUG:boto:Token: None 2023-10-12T01:12:06.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23 2023-10-12T01:12:06.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7vI37rJR2Ad30Gw1XX9b9PkWF5w= 2023-10-12T01:12:06.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7vI37rJR2Ad30Gw1XX9b9PkWF5w=', 'Content-Length': '0'} 2023-10-12T01:12:06.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c75119141e9bd5-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.388 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.388 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.389 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.389 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:12:06.390 DEBUG:boto:Data: 2023-10-12T01:12:06.390 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.391 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.391 DEBUG:boto:Params: {} 2023-10-12T01:12:06.392 DEBUG:boto:Token: None 2023-10-12T01:12:06.392 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23 2023-10-12T01:12:06.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7vI37rJR2Ad30Gw1XX9b9PkWF5w= 2023-10-12T01:12:06.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7vI37rJR2Ad30Gw1XX9b9PkWF5w=', 'Content-Length': '0'} 2023-10-12T01:12:06.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f531e67f6963acff-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:12:06.396 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.397 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.397 DEBUG:boto:Method: GET 2023-10-12T01:12:06.398 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:12:06.398 DEBUG:boto:Data: 2023-10-12T01:12:06.399 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.399 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.400 DEBUG:boto:Params: {} 2023-10-12T01:12:06.400 DEBUG:boto:Token: None 2023-10-12T01:12:06.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23 2023-10-12T01:12:06.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mGB77UPXzOMd+lsD+12dMoMu4ho= 2023-10-12T01:12:06.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mGB77UPXzOMd+lsD+12dMoMu4ho=', 'Content-Length': '0'} 2023-10-12T01:12:06.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3333f8ea659a367-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.450 DEBUG:boto:path=/wafmoa-59/obj23 2023-10-12T01:12:06.451 DEBUG:boto:auth_path=/wafmoa-59/obj23 2023-10-12T01:12:06.452 DEBUG:boto:Method: GET 2023-10-12T01:12:06.452 DEBUG:boto:Path: /wafmoa-59/obj23 2023-10-12T01:12:06.453 DEBUG:boto:Data: 2023-10-12T01:12:06.453 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.454 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.454 DEBUG:boto:Params: {} 2023-10-12T01:12:06.455 DEBUG:boto:Token: None 2023-10-12T01:12:06.456 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj23 2023-10-12T01:12:06.456 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mGB77UPXzOMd+lsD+12dMoMu4ho= 2023-10-12T01:12:06.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mGB77UPXzOMd+lsD+12dMoMu4ho=', 'Content-Length': '0'} 2023-10-12T01:12:06.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbc8e20e13d4cd2-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.461 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.461 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.462 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.462 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.462 DEBUG:boto:Method: GET 2023-10-12T01:12:06.463 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.463 DEBUG:boto:Data: 2023-10-12T01:12:06.464 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.465 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.465 DEBUG:boto:Params: {} 2023-10-12T01:12:06.465 DEBUG:boto:Token: None 2023-10-12T01:12:06.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e3e27b08c832ce-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.470 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.471 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.472 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.472 DEBUG:boto:path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.472 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.473 DEBUG:boto:Method: GET 2023-10-12T01:12:06.473 DEBUG:boto:Path: /wafmoa-59/?prefix=obj23 2023-10-12T01:12:06.474 DEBUG:boto:Data: 2023-10-12T01:12:06.474 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.475 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.476 DEBUG:boto:Params: {} 2023-10-12T01:12:06.476 DEBUG:boto:Token: None 2023-10-12T01:12:06.477 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85469bffa869bc3-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.482 DEBUG:boto:b'wafmoa-59obj231000falseobj232023-10-12T01:03:05.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:12:06.483 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.484 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.484 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.485 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.485 DEBUG:boto:Method: GET 2023-10-12T01:12:06.486 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.486 DEBUG:boto:Data: 2023-10-12T01:12:06.487 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.487 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.488 DEBUG:boto:Params: {} 2023-10-12T01:12:06.488 DEBUG:boto:Token: None 2023-10-12T01:12:06.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE= 2023-10-12T01:12:06.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE=', 'Content-Length': '0'} 2023-10-12T01:12:06.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327ad5d272ed3d6c-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.493 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.493 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.494 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.494 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.495 DEBUG:boto:Method: GET 2023-10-12T01:12:06.495 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.495 DEBUG:boto:Data: 2023-10-12T01:12:06.496 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.497 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.497 DEBUG:boto:Params: {} 2023-10-12T01:12:06.498 DEBUG:boto:Token: None 2023-10-12T01:12:06.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE= 2023-10-12T01:12:06.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE=', 'Content-Length': '0'} 2023-10-12T01:12:06.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b99e257f94f35de-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.502 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.502 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.503 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.503 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.504 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.504 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.505 DEBUG:boto:Data: 2023-10-12T01:12:06.505 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.506 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.507 DEBUG:boto:Params: {} 2023-10-12T01:12:06.507 DEBUG:boto:Token: None 2023-10-12T01:12:06.508 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8q028DyLiD2EaYvSgbONSLTiatI= 2023-10-12T01:12:06.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8q028DyLiD2EaYvSgbONSLTiatI=', 'Content-Length': '0'} 2023-10-12T01:12:06.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000642e560cab6ef41d-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.511 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.512 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.512 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.512 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.513 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.513 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.514 DEBUG:boto:Data: 2023-10-12T01:12:06.514 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.515 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.515 DEBUG:boto:Params: {} 2023-10-12T01:12:06.516 DEBUG:boto:Token: None 2023-10-12T01:12:06.516 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8q028DyLiD2EaYvSgbONSLTiatI= 2023-10-12T01:12:06.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8q028DyLiD2EaYvSgbONSLTiatI=', 'Content-Length': '0'} 2023-10-12T01:12:06.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007de085de476cee4-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:12:06.522 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.523 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.523 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.524 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.524 DEBUG:boto:Method: GET 2023-10-12T01:12:06.525 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.525 DEBUG:boto:Data: 2023-10-12T01:12:06.526 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.526 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.527 DEBUG:boto:Params: {} 2023-10-12T01:12:06.527 DEBUG:boto:Token: None 2023-10-12T01:12:06.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE= 2023-10-12T01:12:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE=', 'Content-Length': '0'} 2023-10-12T01:12:06.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba93a81b7db77704-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.532 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.532 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.533 DEBUG:boto:path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.533 DEBUG:boto:auth_path=/wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.534 DEBUG:boto:Method: GET 2023-10-12T01:12:06.534 DEBUG:boto:Path: /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.534 DEBUG:boto:Data: 2023-10-12T01:12:06.535 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.536 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.536 DEBUG:boto:Params: {} 2023-10-12T01:12:06.537 DEBUG:boto:Token: None 2023-10-12T01:12:06.537 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29?versionId=null 2023-10-12T01:12:06.538 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE= 2023-10-12T01:12:06.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j8d4k6K92uCLFg7ZG3tCUpJ26AE=', 'Content-Length': '0'} 2023-10-12T01:12:06.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043b64e41dcc1be5-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.541 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.542 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.542 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.543 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.543 DEBUG:boto:Method: GET 2023-10-12T01:12:06.543 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.544 DEBUG:boto:Data: 2023-10-12T01:12:06.544 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.545 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.546 DEBUG:boto:Params: {} 2023-10-12T01:12:06.546 DEBUG:boto:Token: None 2023-10-12T01:12:06.547 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.547 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ee20a2138b198d-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.594 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.595 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.596 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.596 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.597 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.597 DEBUG:boto:Method: GET 2023-10-12T01:12:06.598 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.598 DEBUG:boto:Data: 2023-10-12T01:12:06.599 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.599 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.600 DEBUG:boto:Params: {} 2023-10-12T01:12:06.600 DEBUG:boto:Token: None 2023-10-12T01:12:06.601 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.601 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000536abb073a9042d4-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.607 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.608 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.608 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.609 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.609 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:12:06.610 DEBUG:boto:Data: 2023-10-12T01:12:06.610 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.611 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.611 DEBUG:boto:Params: {} 2023-10-12T01:12:06.612 DEBUG:boto:Token: None 2023-10-12T01:12:06.612 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29 2023-10-12T01:12:06.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PN5zH8FmViiiU6+0nqa8sWC9VNM= 2023-10-12T01:12:06.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PN5zH8FmViiiU6+0nqa8sWC9VNM=', 'Content-Length': '0'} 2023-10-12T01:12:06.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a175b909eb0185-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.616 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.616 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.617 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.617 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:12:06.618 DEBUG:boto:Data: 2023-10-12T01:12:06.618 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.619 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.619 DEBUG:boto:Params: {} 2023-10-12T01:12:06.620 DEBUG:boto:Token: None 2023-10-12T01:12:06.620 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29 2023-10-12T01:12:06.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PN5zH8FmViiiU6+0nqa8sWC9VNM= 2023-10-12T01:12:06.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PN5zH8FmViiiU6+0nqa8sWC9VNM=', 'Content-Length': '0'} 2023-10-12T01:12:06.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fad89357578456-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:12:06.624 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.625 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.625 DEBUG:boto:Method: GET 2023-10-12T01:12:06.625 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:12:06.626 DEBUG:boto:Data: 2023-10-12T01:12:06.626 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.627 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.628 DEBUG:boto:Params: {} 2023-10-12T01:12:06.628 DEBUG:boto:Token: None 2023-10-12T01:12:06.629 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29 2023-10-12T01:12:06.629 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3BwNlYoO1Fp2HfHLs1RzXWcrr7s= 2023-10-12T01:12:06.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3BwNlYoO1Fp2HfHLs1RzXWcrr7s=', 'Content-Length': '0'} 2023-10-12T01:12:06.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbf1d661b4d660c-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.633 DEBUG:boto:path=/wafmoa-59/obj29 2023-10-12T01:12:06.633 DEBUG:boto:auth_path=/wafmoa-59/obj29 2023-10-12T01:12:06.633 DEBUG:boto:Method: GET 2023-10-12T01:12:06.634 DEBUG:boto:Path: /wafmoa-59/obj29 2023-10-12T01:12:06.634 DEBUG:boto:Data: 2023-10-12T01:12:06.635 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.636 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.636 DEBUG:boto:Params: {} 2023-10-12T01:12:06.637 DEBUG:boto:Token: None 2023-10-12T01:12:06.637 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj29 2023-10-12T01:12:06.638 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3BwNlYoO1Fp2HfHLs1RzXWcrr7s= 2023-10-12T01:12:06.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3BwNlYoO1Fp2HfHLs1RzXWcrr7s=', 'Content-Length': '0'} 2023-10-12T01:12:06.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e056a9d3358979-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.686 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.687 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.687 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.688 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.688 DEBUG:boto:Method: GET 2023-10-12T01:12:06.689 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.689 DEBUG:boto:Data: 2023-10-12T01:12:06.689 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.690 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.691 DEBUG:boto:Params: {} 2023-10-12T01:12:06.691 DEBUG:boto:Token: None 2023-10-12T01:12:06.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e14cf2d582a65c-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.696 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.697 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.697 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.698 DEBUG:boto:path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.698 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.698 DEBUG:boto:Method: GET 2023-10-12T01:12:06.699 DEBUG:boto:Path: /wafmoa-59/?prefix=obj29 2023-10-12T01:12:06.699 DEBUG:boto:Data: 2023-10-12T01:12:06.700 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.701 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.701 DEBUG:boto:Params: {} 2023-10-12T01:12:06.702 DEBUG:boto:Token: None 2023-10-12T01:12:06.702 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb5ed97e32e375c-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.708 DEBUG:boto:b'wafmoa-59obj291000falseobj292023-10-12T01:04:26.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:12:06.709 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.709 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.709 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.710 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.710 DEBUG:boto:Method: GET 2023-10-12T01:12:06.711 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.711 DEBUG:boto:Data: 2023-10-12T01:12:06.712 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.713 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.713 DEBUG:boto:Params: {} 2023-10-12T01:12:06.714 DEBUG:boto:Token: None 2023-10-12T01:12:06.714 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8= 2023-10-12T01:12:06.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8=', 'Content-Length': '0'} 2023-10-12T01:12:06.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6d702f220c3312-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.762 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.763 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.763 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.764 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.764 DEBUG:boto:Method: GET 2023-10-12T01:12:06.765 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.765 DEBUG:boto:Data: 2023-10-12T01:12:06.766 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.767 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.767 DEBUG:boto:Params: {} 2023-10-12T01:12:06.767 DEBUG:boto:Token: None 2023-10-12T01:12:06.768 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.768 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8= 2023-10-12T01:12:06.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8=', 'Content-Length': '0'} 2023-10-12T01:12:06.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007610ad24296d7b66-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.772 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.772 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.773 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.773 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.774 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.774 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.775 DEBUG:boto:Data: 2023-10-12T01:12:06.775 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.776 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.776 DEBUG:boto:Params: {} 2023-10-12T01:12:06.777 DEBUG:boto:Token: None 2023-10-12T01:12:06.777 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.778 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t2P9/Dw8DSd9xXSzBj2v+apdhHs= 2023-10-12T01:12:06.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t2P9/Dw8DSd9xXSzBj2v+apdhHs=', 'Content-Length': '0'} 2023-10-12T01:12:06.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000057132dc7fbfd078-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.781 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.781 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.782 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.782 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.783 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.783 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.783 DEBUG:boto:Data: 2023-10-12T01:12:06.784 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.785 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.785 DEBUG:boto:Params: {} 2023-10-12T01:12:06.785 DEBUG:boto:Token: None 2023-10-12T01:12:06.786 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t2P9/Dw8DSd9xXSzBj2v+apdhHs= 2023-10-12T01:12:06.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t2P9/Dw8DSd9xXSzBj2v+apdhHs=', 'Content-Length': '0'} 2023-10-12T01:12:06.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aaa14e93e03140-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:12:06.790 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.790 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.791 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.791 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.792 DEBUG:boto:Method: GET 2023-10-12T01:12:06.792 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.793 DEBUG:boto:Data: 2023-10-12T01:12:06.793 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.794 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.794 DEBUG:boto:Params: {} 2023-10-12T01:12:06.795 DEBUG:boto:Token: None 2023-10-12T01:12:06.795 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.796 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8= 2023-10-12T01:12:06.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8=', 'Content-Length': '0'} 2023-10-12T01:12:06.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fef2a37fdd797f-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.799 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.799 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.800 DEBUG:boto:path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.800 DEBUG:boto:auth_path=/wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.801 DEBUG:boto:Method: GET 2023-10-12T01:12:06.801 DEBUG:boto:Path: /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.802 DEBUG:boto:Data: 2023-10-12T01:12:06.802 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.803 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.803 DEBUG:boto:Params: {} 2023-10-12T01:12:06.804 DEBUG:boto:Token: None 2023-10-12T01:12:06.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31?versionId=null 2023-10-12T01:12:06.805 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8= 2023-10-12T01:12:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bGgktBYJ/CPVErHBOez79yCHUI8=', 'Content-Length': '0'} 2023-10-12T01:12:06.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6c0881f396529d-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.809 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.809 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.810 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.810 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.810 DEBUG:boto:Method: GET 2023-10-12T01:12:06.811 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.811 DEBUG:boto:Data: 2023-10-12T01:12:06.812 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.813 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.813 DEBUG:boto:Params: {} 2023-10-12T01:12:06.813 DEBUG:boto:Token: None 2023-10-12T01:12:06.814 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.814 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a9f975c5cb9536-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.818 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.819 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.819 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.820 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.820 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.821 DEBUG:boto:Method: GET 2023-10-12T01:12:06.821 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.822 DEBUG:boto:Data: 2023-10-12T01:12:06.822 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.823 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.823 DEBUG:boto:Params: {} 2023-10-12T01:12:06.824 DEBUG:boto:Token: None 2023-10-12T01:12:06.824 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6dac33e2d625a1-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.830 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.831 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.831 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.832 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.832 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:12:06.832 DEBUG:boto:Data: 2023-10-12T01:12:06.833 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.834 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.834 DEBUG:boto:Params: {} 2023-10-12T01:12:06.835 DEBUG:boto:Token: None 2023-10-12T01:12:06.835 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31 2023-10-12T01:12:06.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:daOl9lAsqfpVLtxgs834kuHV77s= 2023-10-12T01:12:06.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:daOl9lAsqfpVLtxgs834kuHV77s=', 'Content-Length': '0'} 2023-10-12T01:12:06.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0af00827fdb9f3b-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.839 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.839 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.840 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.840 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:12:06.840 DEBUG:boto:Data: 2023-10-12T01:12:06.841 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.842 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.842 DEBUG:boto:Params: {} 2023-10-12T01:12:06.843 DEBUG:boto:Token: None 2023-10-12T01:12:06.843 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31 2023-10-12T01:12:06.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:daOl9lAsqfpVLtxgs834kuHV77s= 2023-10-12T01:12:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:daOl9lAsqfpVLtxgs834kuHV77s=', 'Content-Length': '0'} 2023-10-12T01:12:06.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b946b94c3f4680d9-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:12:06.847 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.848 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.848 DEBUG:boto:Method: GET 2023-10-12T01:12:06.849 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:12:06.849 DEBUG:boto:Data: 2023-10-12T01:12:06.849 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.850 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.851 DEBUG:boto:Params: {} 2023-10-12T01:12:06.851 DEBUG:boto:Token: None 2023-10-12T01:12:06.852 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31 2023-10-12T01:12:06.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uFetwp4uVX7FbpsIUEtIl5Veaag= 2023-10-12T01:12:06.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uFetwp4uVX7FbpsIUEtIl5Veaag=', 'Content-Length': '0'} 2023-10-12T01:12:06.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091464832f545584c-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.855 DEBUG:boto:path=/wafmoa-59/obj31 2023-10-12T01:12:06.856 DEBUG:boto:auth_path=/wafmoa-59/obj31 2023-10-12T01:12:06.856 DEBUG:boto:Method: GET 2023-10-12T01:12:06.857 DEBUG:boto:Path: /wafmoa-59/obj31 2023-10-12T01:12:06.857 DEBUG:boto:Data: 2023-10-12T01:12:06.858 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.858 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.859 DEBUG:boto:Params: {} 2023-10-12T01:12:06.859 DEBUG:boto:Token: None 2023-10-12T01:12:06.860 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj31 2023-10-12T01:12:06.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uFetwp4uVX7FbpsIUEtIl5Veaag= 2023-10-12T01:12:06.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uFetwp4uVX7FbpsIUEtIl5Veaag=', 'Content-Length': '0'} 2023-10-12T01:12:06.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24c0abad37b3d12-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.906 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.909 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.909 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.910 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.910 DEBUG:boto:Method: GET 2023-10-12T01:12:06.910 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.911 DEBUG:boto:Data: 2023-10-12T01:12:06.911 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.912 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.913 DEBUG:boto:Params: {} 2023-10-12T01:12:06.913 DEBUG:boto:Token: None 2023-10-12T01:12:06.914 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.914 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c900039b1bdcba77-00652747e6-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.918 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.919 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:06.919 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:06.920 DEBUG:boto:path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.920 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.921 DEBUG:boto:Method: GET 2023-10-12T01:12:06.921 DEBUG:boto:Path: /wafmoa-59/?prefix=obj31 2023-10-12T01:12:06.921 DEBUG:boto:Data: 2023-10-12T01:12:06.922 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.923 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.923 DEBUG:boto:Params: {} 2023-10-12T01:12:06.924 DEBUG:boto:Token: None 2023-10-12T01:12:06.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/ 2023-10-12T01:12:06.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw= 2023-10-12T01:12:06.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CzOdaSLn8BCjjxF9Hzvi1K6jzKw=', 'Content-Length': '0'} 2023-10-12T01:12:06.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031ab2b40f29d7a08-00652747e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.930 DEBUG:boto:b'wafmoa-59obj311000falseobj312023-10-12T01:06:27.310Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:06.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:12:06.932 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:06.932 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:06.932 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.933 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.933 DEBUG:boto:Method: GET 2023-10-12T01:12:06.934 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.934 DEBUG:boto:Data: 2023-10-12T01:12:06.935 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.936 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.936 DEBUG:boto:Params: {} 2023-10-12T01:12:06.936 DEBUG:boto:Token: None 2023-10-12T01:12:06.937 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.938 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA= 2023-10-12T01:12:06.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA=', 'Content-Length': '0'} 2023-10-12T01:12:06.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0ed1ae7b975495-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.941 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:06.942 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:06.942 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.943 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.943 DEBUG:boto:Method: GET 2023-10-12T01:12:06.944 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.944 DEBUG:boto:Data: 2023-10-12T01:12:06.944 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.945 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.946 DEBUG:boto:Params: {} 2023-10-12T01:12:06.946 DEBUG:boto:Token: None 2023-10-12T01:12:06.947 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.947 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA= 2023-10-12T01:12:06.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA=', 'Content-Length': '0'} 2023-10-12T01:12:06.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f03630b12abe21-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.951 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:06.951 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:06.952 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.952 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.953 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.953 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.953 DEBUG:boto:Data: 2023-10-12T01:12:06.954 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.955 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.955 DEBUG:boto:Params: {} 2023-10-12T01:12:06.956 DEBUG:boto:Token: None 2023-10-12T01:12:06.956 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7evCs6jr24PCsFcJkx88B/Fje4= 2023-10-12T01:12:06.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7evCs6jr24PCsFcJkx88B/Fje4=', 'Content-Length': '0'} 2023-10-12T01:12:06.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f586914cd568e0f-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.960 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:06.960 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:06.960 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.961 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.961 DEBUG:boto:Method: HEAD 2023-10-12T01:12:06.962 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.962 DEBUG:boto:Data: 2023-10-12T01:12:06.963 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:06.963 DEBUG:boto:Port: 8000 2023-10-12T01:12:06.964 DEBUG:boto:Params: {} 2023-10-12T01:12:06.964 DEBUG:boto:Token: None 2023-10-12T01:12:06.965 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.965 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L7evCs6jr24PCsFcJkx88B/Fje4= 2023-10-12T01:12:06.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L7evCs6jr24PCsFcJkx88B/Fje4=', 'Content-Length': '0'} 2023-10-12T01:12:06.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013bdf5a3699c01a9-00652747e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:06.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:12:06.969 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:06.969 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:06.970 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.970 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.971 DEBUG:boto:Method: GET 2023-10-12T01:12:06.971 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.971 DEBUG:boto:Data: 2023-10-12T01:12:06.972 DEBUG:boto:Headers: {} 2023-10-12T01:12:06.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:06.973 DEBUG:boto:Port: 8001 2023-10-12T01:12:06.973 DEBUG:boto:Params: {} 2023-10-12T01:12:06.974 DEBUG:boto:Token: None 2023-10-12T01:12:06.974 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:06 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:06.975 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA= 2023-10-12T01:12:06.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fmdfz99HESmmeUBE5ufYtoDyEpA=', 'Content-Length': '0'} 2023-10-12T01:12:06.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047833542502f7b06-00652747e6-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.022 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:07.023 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:07.023 DEBUG:boto:path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:07.024 DEBUG:boto:auth_path=/wafmoa-59/obj33?versionId=null 2023-10-12T01:12:07.024 DEBUG:boto:Method: GET 2023-10-12T01:12:07.025 DEBUG:boto:Path: /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:07.025 DEBUG:boto:Data: 2023-10-12T01:12:07.026 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.026 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.027 DEBUG:boto:Params: {} 2023-10-12T01:12:07.028 DEBUG:boto:Token: None 2023-10-12T01:12:07.028 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj33?versionId=null 2023-10-12T01:12:07.029 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6XRyoX/IxTCqRscPqK+4jm8MoUs= 2023-10-12T01:12:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6XRyoX/IxTCqRscPqK+4jm8MoUs=', 'Content-Length': '0'} 2023-10-12T01:12:07.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1baa476c21ee0d5-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.032 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.033 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.033 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.033 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.034 DEBUG:boto:Method: GET 2023-10-12T01:12:07.034 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.035 DEBUG:boto:Data: 2023-10-12T01:12:07.035 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.036 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.036 DEBUG:boto:Params: {} 2023-10-12T01:12:07.037 DEBUG:boto:Token: None 2023-10-12T01:12:07.037 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f003920b388987-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.042 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.042 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.043 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.043 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.044 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.044 DEBUG:boto:Method: GET 2023-10-12T01:12:07.045 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.045 DEBUG:boto:Data: 2023-10-12T01:12:07.045 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.046 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.047 DEBUG:boto:Params: {} 2023-10-12T01:12:07.047 DEBUG:boto:Token: None 2023-10-12T01:12:07.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4968ab05c7f8102-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.054 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.054 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:07.055 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:07.055 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.056 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:12:07.056 DEBUG:boto:Data: 2023-10-12T01:12:07.057 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.057 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.058 DEBUG:boto:Params: {} 2023-10-12T01:12:07.058 DEBUG:boto:Token: None 2023-10-12T01:12:07.059 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj33 2023-10-12T01:12:07.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JoovkGnYpV5Aen4IAbC6YQ7+f08= 2023-10-12T01:12:07.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JoovkGnYpV5Aen4IAbC6YQ7+f08=', 'Content-Length': '0'} 2023-10-12T01:12:07.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3e19fa9d41bfc8-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.062 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:07.063 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:07.063 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.064 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:12:07.064 DEBUG:boto:Data: 2023-10-12T01:12:07.065 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.065 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.066 DEBUG:boto:Params: {} 2023-10-12T01:12:07.066 DEBUG:boto:Token: None 2023-10-12T01:12:07.067 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj33 2023-10-12T01:12:07.067 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JoovkGnYpV5Aen4IAbC6YQ7+f08= 2023-10-12T01:12:07.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JoovkGnYpV5Aen4IAbC6YQ7+f08=', 'Content-Length': '0'} 2023-10-12T01:12:07.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064742d207cae5c2d-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:12:07.071 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:07.071 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:07.072 DEBUG:boto:Method: GET 2023-10-12T01:12:07.072 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:12:07.073 DEBUG:boto:Data: 2023-10-12T01:12:07.073 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.074 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.074 DEBUG:boto:Params: {} 2023-10-12T01:12:07.075 DEBUG:boto:Token: None 2023-10-12T01:12:07.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj33 2023-10-12T01:12:07.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qVx+L1w+oaaM+jMzwAYJb/oWr4o= 2023-10-12T01:12:07.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qVx+L1w+oaaM+jMzwAYJb/oWr4o=', 'Content-Length': '0'} 2023-10-12T01:12:07.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1f75188604b535-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.079 DEBUG:boto:path=/wafmoa-59/obj33 2023-10-12T01:12:07.080 DEBUG:boto:auth_path=/wafmoa-59/obj33 2023-10-12T01:12:07.080 DEBUG:boto:Method: GET 2023-10-12T01:12:07.081 DEBUG:boto:Path: /wafmoa-59/obj33 2023-10-12T01:12:07.081 DEBUG:boto:Data: 2023-10-12T01:12:07.081 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.082 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.083 DEBUG:boto:Params: {} 2023-10-12T01:12:07.083 DEBUG:boto:Token: None 2023-10-12T01:12:07.084 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj33 2023-10-12T01:12:07.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qVx+L1w+oaaM+jMzwAYJb/oWr4o= 2023-10-12T01:12:07.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qVx+L1w+oaaM+jMzwAYJb/oWr4o=', 'Content-Length': '0'} 2023-10-12T01:12:07.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000731e5531cc60c05-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.087 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.088 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.088 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.089 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.089 DEBUG:boto:Method: GET 2023-10-12T01:12:07.090 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.090 DEBUG:boto:Data: 2023-10-12T01:12:07.090 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.091 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.092 DEBUG:boto:Params: {} 2023-10-12T01:12:07.092 DEBUG:boto:Token: None 2023-10-12T01:12:07.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b7941835d9c3ff-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.097 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.098 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.098 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.099 DEBUG:boto:path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.099 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.099 DEBUG:boto:Method: GET 2023-10-12T01:12:07.100 DEBUG:boto:Path: /wafmoa-59/?prefix=obj33 2023-10-12T01:12:07.100 DEBUG:boto:Data: 2023-10-12T01:12:07.101 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.102 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.102 DEBUG:boto:Params: {} 2023-10-12T01:12:07.103 DEBUG:boto:Token: None 2023-10-12T01:12:07.103 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df322d7931e0167-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.150 DEBUG:boto:b'wafmoa-59obj331000falseobj332023-10-12T00:57:47.241Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:12:07.152 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.153 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.153 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.153 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.154 DEBUG:boto:Method: GET 2023-10-12T01:12:07.154 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.155 DEBUG:boto:Data: 2023-10-12T01:12:07.155 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.156 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.158 DEBUG:boto:Params: {} 2023-10-12T01:12:07.158 DEBUG:boto:Token: None 2023-10-12T01:12:07.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y= 2023-10-12T01:12:07.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y=', 'Content-Length': '0'} 2023-10-12T01:12:07.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9f827a39e337a1-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.164 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.164 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.165 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.165 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.166 DEBUG:boto:Method: GET 2023-10-12T01:12:07.166 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.167 DEBUG:boto:Data: 2023-10-12T01:12:07.167 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.169 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.169 DEBUG:boto:Params: {} 2023-10-12T01:12:07.170 DEBUG:boto:Token: None 2023-10-12T01:12:07.170 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.171 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y= 2023-10-12T01:12:07.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y=', 'Content-Length': '0'} 2023-10-12T01:12:07.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056961d20d84dadf3-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.174 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.174 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.175 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.175 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.176 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.176 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.176 DEBUG:boto:Data: 2023-10-12T01:12:07.177 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.178 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.178 DEBUG:boto:Params: {} 2023-10-12T01:12:07.179 DEBUG:boto:Token: None 2023-10-12T01:12:07.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8MeGZt0TAjrJ+8HBLK5AwTGIVRE= 2023-10-12T01:12:07.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8MeGZt0TAjrJ+8HBLK5AwTGIVRE=', 'Content-Length': '0'} 2023-10-12T01:12:07.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002956a1ec47e73fea-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.182 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.183 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.183 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.184 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.184 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.185 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.185 DEBUG:boto:Data: 2023-10-12T01:12:07.185 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.186 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.187 DEBUG:boto:Params: {} 2023-10-12T01:12:07.187 DEBUG:boto:Token: None 2023-10-12T01:12:07.188 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8MeGZt0TAjrJ+8HBLK5AwTGIVRE= 2023-10-12T01:12:07.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8MeGZt0TAjrJ+8HBLK5AwTGIVRE=', 'Content-Length': '0'} 2023-10-12T01:12:07.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201e0e32a9f49656-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:12:07.191 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.192 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.192 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.193 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.193 DEBUG:boto:Method: GET 2023-10-12T01:12:07.194 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.194 DEBUG:boto:Data: 2023-10-12T01:12:07.195 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.195 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.196 DEBUG:boto:Params: {} 2023-10-12T01:12:07.196 DEBUG:boto:Token: None 2023-10-12T01:12:07.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y= 2023-10-12T01:12:07.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y=', 'Content-Length': '0'} 2023-10-12T01:12:07.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cf154751bfc52de-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.246 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.247 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.247 DEBUG:boto:path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.248 DEBUG:boto:auth_path=/wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.248 DEBUG:boto:Method: GET 2023-10-12T01:12:07.249 DEBUG:boto:Path: /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.249 DEBUG:boto:Data: 2023-10-12T01:12:07.250 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.251 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.251 DEBUG:boto:Params: {} 2023-10-12T01:12:07.252 DEBUG:boto:Token: None 2023-10-12T01:12:07.252 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34?versionId=null 2023-10-12T01:12:07.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y= 2023-10-12T01:12:07.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hmGWbxPpUkU9piSkoMgV2GTOW+Y=', 'Content-Length': '0'} 2023-10-12T01:12:07.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007080c9f8ec96889e-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.256 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.257 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.257 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.257 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.258 DEBUG:boto:Method: GET 2023-10-12T01:12:07.258 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.259 DEBUG:boto:Data: 2023-10-12T01:12:07.259 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.260 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.260 DEBUG:boto:Params: {} 2023-10-12T01:12:07.261 DEBUG:boto:Token: None 2023-10-12T01:12:07.261 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.262 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b255f8458b0c41-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.265 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.266 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.267 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.267 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.268 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.268 DEBUG:boto:Method: GET 2023-10-12T01:12:07.268 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.269 DEBUG:boto:Data: 2023-10-12T01:12:07.269 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.270 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.271 DEBUG:boto:Params: {} 2023-10-12T01:12:07.271 DEBUG:boto:Token: None 2023-10-12T01:12:07.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000710693f909a0ed27-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.277 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.278 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.279 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.279 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.280 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:12:07.280 DEBUG:boto:Data: 2023-10-12T01:12:07.280 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.281 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.282 DEBUG:boto:Params: {} 2023-10-12T01:12:07.282 DEBUG:boto:Token: None 2023-10-12T01:12:07.283 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34 2023-10-12T01:12:07.283 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFxoSLXcpmJ2xXyxM3JvgK1Q8+s= 2023-10-12T01:12:07.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFxoSLXcpmJ2xXyxM3JvgK1Q8+s=', 'Content-Length': '0'} 2023-10-12T01:12:07.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e122baf3da1118-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.286 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.287 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.287 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.288 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:12:07.288 DEBUG:boto:Data: 2023-10-12T01:12:07.288 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.289 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.290 DEBUG:boto:Params: {} 2023-10-12T01:12:07.290 DEBUG:boto:Token: None 2023-10-12T01:12:07.291 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34 2023-10-12T01:12:07.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFxoSLXcpmJ2xXyxM3JvgK1Q8+s= 2023-10-12T01:12:07.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFxoSLXcpmJ2xXyxM3JvgK1Q8+s=', 'Content-Length': '0'} 2023-10-12T01:12:07.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f517e23011e372-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:12:07.294 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.295 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.295 DEBUG:boto:Method: GET 2023-10-12T01:12:07.296 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:12:07.296 DEBUG:boto:Data: 2023-10-12T01:12:07.297 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.297 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.298 DEBUG:boto:Params: {} 2023-10-12T01:12:07.299 DEBUG:boto:Token: None 2023-10-12T01:12:07.299 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34 2023-10-12T01:12:07.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/r/vVm0cjG1Qjn3UjnBLx3mvcms= 2023-10-12T01:12:07.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/r/vVm0cjG1Qjn3UjnBLx3mvcms=', 'Content-Length': '0'} 2023-10-12T01:12:07.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e18be721c9d6e2f-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.303 DEBUG:boto:path=/wafmoa-59/obj34 2023-10-12T01:12:07.303 DEBUG:boto:auth_path=/wafmoa-59/obj34 2023-10-12T01:12:07.304 DEBUG:boto:Method: GET 2023-10-12T01:12:07.304 DEBUG:boto:Path: /wafmoa-59/obj34 2023-10-12T01:12:07.305 DEBUG:boto:Data: 2023-10-12T01:12:07.305 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.306 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.307 DEBUG:boto:Params: {} 2023-10-12T01:12:07.307 DEBUG:boto:Token: None 2023-10-12T01:12:07.307 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj34 2023-10-12T01:12:07.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/r/vVm0cjG1Qjn3UjnBLx3mvcms= 2023-10-12T01:12:07.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/r/vVm0cjG1Qjn3UjnBLx3mvcms=', 'Content-Length': '0'} 2023-10-12T01:12:07.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ebbe12f06a1587-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.354 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.355 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.356 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.357 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.357 DEBUG:boto:Method: GET 2023-10-12T01:12:07.358 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.359 DEBUG:boto:Data: 2023-10-12T01:12:07.359 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.360 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.361 DEBUG:boto:Params: {} 2023-10-12T01:12:07.362 DEBUG:boto:Token: None 2023-10-12T01:12:07.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009877d46f645adeaa-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.368 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.369 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.370 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.371 DEBUG:boto:path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.371 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.372 DEBUG:boto:Method: GET 2023-10-12T01:12:07.373 DEBUG:boto:Path: /wafmoa-59/?prefix=obj34 2023-10-12T01:12:07.373 DEBUG:boto:Data: 2023-10-12T01:12:07.374 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.375 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.375 DEBUG:boto:Params: {} 2023-10-12T01:12:07.375 DEBUG:boto:Token: None 2023-10-12T01:12:07.376 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a06b9fcd75c94f-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.382 DEBUG:boto:b'wafmoa-59obj341000falseobj342023-10-12T00:59:23.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:12:07.384 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.384 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.385 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.385 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.386 DEBUG:boto:Method: GET 2023-10-12T01:12:07.386 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.387 DEBUG:boto:Data: 2023-10-12T01:12:07.387 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.388 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.388 DEBUG:boto:Params: {} 2023-10-12T01:12:07.389 DEBUG:boto:Token: None 2023-10-12T01:12:07.389 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A= 2023-10-12T01:12:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A=', 'Content-Length': '0'} 2023-10-12T01:12:07.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee172e01af41d749-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.393 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.393 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.394 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.395 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.395 DEBUG:boto:Method: GET 2023-10-12T01:12:07.396 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.396 DEBUG:boto:Data: 2023-10-12T01:12:07.397 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.399 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.400 DEBUG:boto:Params: {} 2023-10-12T01:12:07.400 DEBUG:boto:Token: None 2023-10-12T01:12:07.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A= 2023-10-12T01:12:07.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A=', 'Content-Length': '0'} 2023-10-12T01:12:07.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ba0e3fd3155302-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.406 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.407 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.408 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.408 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.409 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.410 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.410 DEBUG:boto:Data: 2023-10-12T01:12:07.411 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.411 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.412 DEBUG:boto:Params: {} 2023-10-12T01:12:07.412 DEBUG:boto:Token: None 2023-10-12T01:12:07.413 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vDwHOo95mC4BQCmEzF8X7Hj3DJc= 2023-10-12T01:12:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vDwHOo95mC4BQCmEzF8X7Hj3DJc=', 'Content-Length': '0'} 2023-10-12T01:12:07.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661f1fd8724caf13-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.416 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.417 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.417 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.418 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.418 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.419 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.420 DEBUG:boto:Data: 2023-10-12T01:12:07.420 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.421 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.421 DEBUG:boto:Params: {} 2023-10-12T01:12:07.421 DEBUG:boto:Token: None 2023-10-12T01:12:07.422 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vDwHOo95mC4BQCmEzF8X7Hj3DJc= 2023-10-12T01:12:07.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vDwHOo95mC4BQCmEzF8X7Hj3DJc=', 'Content-Length': '0'} 2023-10-12T01:12:07.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00d579f17b568ea-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:12:07.426 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.426 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.427 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.427 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.428 DEBUG:boto:Method: GET 2023-10-12T01:12:07.428 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.428 DEBUG:boto:Data: 2023-10-12T01:12:07.429 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.430 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.431 DEBUG:boto:Params: {} 2023-10-12T01:12:07.431 DEBUG:boto:Token: None 2023-10-12T01:12:07.432 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A= 2023-10-12T01:12:07.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A=', 'Content-Length': '0'} 2023-10-12T01:12:07.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009286c34a50e5f4e0-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.437 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.437 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.437 DEBUG:boto:path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.438 DEBUG:boto:auth_path=/wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.438 DEBUG:boto:Method: GET 2023-10-12T01:12:07.439 DEBUG:boto:Path: /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.439 DEBUG:boto:Data: 2023-10-12T01:12:07.439 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.440 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.441 DEBUG:boto:Params: {} 2023-10-12T01:12:07.441 DEBUG:boto:Token: None 2023-10-12T01:12:07.442 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37?versionId=null 2023-10-12T01:12:07.443 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A= 2023-10-12T01:12:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:amHvcayV6ee0zi1BmQZCPIKWa4A=', 'Content-Length': '0'} 2023-10-12T01:12:07.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bff3a17528181b8-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.446 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.446 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.447 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.447 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.448 DEBUG:boto:Method: GET 2023-10-12T01:12:07.448 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.448 DEBUG:boto:Data: 2023-10-12T01:12:07.449 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.450 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.450 DEBUG:boto:Params: {} 2023-10-12T01:12:07.451 DEBUG:boto:Token: None 2023-10-12T01:12:07.451 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1359fc7aa4f861-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.498 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.500 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.500 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.501 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.502 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.503 DEBUG:boto:Method: GET 2023-10-12T01:12:07.504 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.504 DEBUG:boto:Data: 2023-10-12T01:12:07.505 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.506 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.506 DEBUG:boto:Params: {} 2023-10-12T01:12:07.507 DEBUG:boto:Token: None 2023-10-12T01:12:07.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a4628b4ff1cbfeb-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.514 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.515 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.515 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.516 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.516 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:12:07.517 DEBUG:boto:Data: 2023-10-12T01:12:07.517 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.519 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.519 DEBUG:boto:Params: {} 2023-10-12T01:12:07.520 DEBUG:boto:Token: None 2023-10-12T01:12:07.521 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37 2023-10-12T01:12:07.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E/tmTTaC8I0Yw6pgYVMXXNgjIYc= 2023-10-12T01:12:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E/tmTTaC8I0Yw6pgYVMXXNgjIYc=', 'Content-Length': '0'} 2023-10-12T01:12:07.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c71ae40a50969ae-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.526 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.526 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.527 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.527 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:12:07.528 DEBUG:boto:Data: 2023-10-12T01:12:07.528 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.529 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.529 DEBUG:boto:Params: {} 2023-10-12T01:12:07.530 DEBUG:boto:Token: None 2023-10-12T01:12:07.530 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37 2023-10-12T01:12:07.531 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:E/tmTTaC8I0Yw6pgYVMXXNgjIYc= 2023-10-12T01:12:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:E/tmTTaC8I0Yw6pgYVMXXNgjIYc=', 'Content-Length': '0'} 2023-10-12T01:12:07.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2b228a81284ee2-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:12:07.536 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.537 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.538 DEBUG:boto:Method: GET 2023-10-12T01:12:07.539 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:12:07.539 DEBUG:boto:Data: 2023-10-12T01:12:07.540 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.542 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.542 DEBUG:boto:Params: {} 2023-10-12T01:12:07.543 DEBUG:boto:Token: None 2023-10-12T01:12:07.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37 2023-10-12T01:12:07.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d8tXKVZ2dukx++TkGORbD4+whik= 2023-10-12T01:12:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d8tXKVZ2dukx++TkGORbD4+whik=', 'Content-Length': '0'} 2023-10-12T01:12:07.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000165a49bea9d6a87-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.547 DEBUG:boto:path=/wafmoa-59/obj37 2023-10-12T01:12:07.548 DEBUG:boto:auth_path=/wafmoa-59/obj37 2023-10-12T01:12:07.549 DEBUG:boto:Method: GET 2023-10-12T01:12:07.549 DEBUG:boto:Path: /wafmoa-59/obj37 2023-10-12T01:12:07.550 DEBUG:boto:Data: 2023-10-12T01:12:07.551 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.552 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.553 DEBUG:boto:Params: {} 2023-10-12T01:12:07.553 DEBUG:boto:Token: None 2023-10-12T01:12:07.554 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj37 2023-10-12T01:12:07.555 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d8tXKVZ2dukx++TkGORbD4+whik= 2023-10-12T01:12:07.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d8tXKVZ2dukx++TkGORbD4+whik=', 'Content-Length': '0'} 2023-10-12T01:12:07.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076142929f2e62580-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.602 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.603 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.603 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.604 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.604 DEBUG:boto:Method: GET 2023-10-12T01:12:07.605 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.605 DEBUG:boto:Data: 2023-10-12T01:12:07.606 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.607 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.607 DEBUG:boto:Params: {} 2023-10-12T01:12:07.608 DEBUG:boto:Token: None 2023-10-12T01:12:07.609 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.610 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd35d82bdf3263b3-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.615 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.615 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.616 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.616 DEBUG:boto:path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.617 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.617 DEBUG:boto:Method: GET 2023-10-12T01:12:07.618 DEBUG:boto:Path: /wafmoa-59/?prefix=obj37 2023-10-12T01:12:07.618 DEBUG:boto:Data: 2023-10-12T01:12:07.618 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.619 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.620 DEBUG:boto:Params: {} 2023-10-12T01:12:07.620 DEBUG:boto:Token: None 2023-10-12T01:12:07.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041841de08b7bce31-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.627 DEBUG:boto:b'wafmoa-59obj371000falseobj372023-10-12T01:09:29.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:12:07.628 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.629 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.629 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.629 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.630 DEBUG:boto:Method: GET 2023-10-12T01:12:07.630 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.631 DEBUG:boto:Data: 2023-10-12T01:12:07.631 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.632 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.632 DEBUG:boto:Params: {} 2023-10-12T01:12:07.633 DEBUG:boto:Token: None 2023-10-12T01:12:07.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.634 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg= 2023-10-12T01:12:07.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg=', 'Content-Length': '0'} 2023-10-12T01:12:07.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033429b11e69bc114-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.638 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.638 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.639 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.639 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.639 DEBUG:boto:Method: GET 2023-10-12T01:12:07.640 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.640 DEBUG:boto:Data: 2023-10-12T01:12:07.641 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.642 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.642 DEBUG:boto:Params: {} 2023-10-12T01:12:07.643 DEBUG:boto:Token: None 2023-10-12T01:12:07.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg= 2023-10-12T01:12:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg=', 'Content-Length': '0'} 2023-10-12T01:12:07.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3212e30223c9e64-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.647 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.647 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.648 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.648 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.648 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.649 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.649 DEBUG:boto:Data: 2023-10-12T01:12:07.650 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.650 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.651 DEBUG:boto:Params: {} 2023-10-12T01:12:07.651 DEBUG:boto:Token: None 2023-10-12T01:12:07.652 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GDimnrsHrOsdpUje8U5lAh/uJL8= 2023-10-12T01:12:07.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GDimnrsHrOsdpUje8U5lAh/uJL8=', 'Content-Length': '0'} 2023-10-12T01:12:07.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c5f8cb918cc073-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.655 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.656 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.656 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.656 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.657 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.657 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.658 DEBUG:boto:Data: 2023-10-12T01:12:07.658 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.659 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.659 DEBUG:boto:Params: {} 2023-10-12T01:12:07.660 DEBUG:boto:Token: None 2023-10-12T01:12:07.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GDimnrsHrOsdpUje8U5lAh/uJL8= 2023-10-12T01:12:07.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GDimnrsHrOsdpUje8U5lAh/uJL8=', 'Content-Length': '0'} 2023-10-12T01:12:07.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303088f6c41eb120-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:12:07.664 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.665 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.665 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.666 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.666 DEBUG:boto:Method: GET 2023-10-12T01:12:07.667 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.667 DEBUG:boto:Data: 2023-10-12T01:12:07.668 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.668 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.669 DEBUG:boto:Params: {} 2023-10-12T01:12:07.669 DEBUG:boto:Token: None 2023-10-12T01:12:07.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.670 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg= 2023-10-12T01:12:07.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg=', 'Content-Length': '0'} 2023-10-12T01:12:07.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2101e30181f03e4-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.674 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.674 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.674 DEBUG:boto:path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.675 DEBUG:boto:auth_path=/wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.675 DEBUG:boto:Method: GET 2023-10-12T01:12:07.676 DEBUG:boto:Path: /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.676 DEBUG:boto:Data: 2023-10-12T01:12:07.677 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.678 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.678 DEBUG:boto:Params: {} 2023-10-12T01:12:07.678 DEBUG:boto:Token: None 2023-10-12T01:12:07.679 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38?versionId=null 2023-10-12T01:12:07.679 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg= 2023-10-12T01:12:07.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0Dp1KvYkg0cOrBwjztUx71slzPg=', 'Content-Length': '0'} 2023-10-12T01:12:07.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee90316d3efcd758-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.726 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.727 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.727 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.728 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.728 DEBUG:boto:Method: GET 2023-10-12T01:12:07.729 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.729 DEBUG:boto:Data: 2023-10-12T01:12:07.730 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.731 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.731 DEBUG:boto:Params: {} 2023-10-12T01:12:07.732 DEBUG:boto:Token: None 2023-10-12T01:12:07.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.733 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a04b518ae09e9a1-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.737 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.738 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.738 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.739 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.739 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.740 DEBUG:boto:Method: GET 2023-10-12T01:12:07.740 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.741 DEBUG:boto:Data: 2023-10-12T01:12:07.741 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.742 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.743 DEBUG:boto:Params: {} 2023-10-12T01:12:07.745 DEBUG:boto:Token: None 2023-10-12T01:12:07.745 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c896c2b1214ec97-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.751 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.752 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.752 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.753 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.753 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:12:07.754 DEBUG:boto:Data: 2023-10-12T01:12:07.754 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.755 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.756 DEBUG:boto:Params: {} 2023-10-12T01:12:07.756 DEBUG:boto:Token: None 2023-10-12T01:12:07.757 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38 2023-10-12T01:12:07.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PHX42RNuvclF0YcRIgA50kgNrnc= 2023-10-12T01:12:07.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PHX42RNuvclF0YcRIgA50kgNrnc=', 'Content-Length': '0'} 2023-10-12T01:12:07.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34a3bdeb01ade7f-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.760 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.761 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.762 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.762 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:12:07.763 DEBUG:boto:Data: 2023-10-12T01:12:07.763 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.764 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.765 DEBUG:boto:Params: {} 2023-10-12T01:12:07.765 DEBUG:boto:Token: None 2023-10-12T01:12:07.765 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38 2023-10-12T01:12:07.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PHX42RNuvclF0YcRIgA50kgNrnc= 2023-10-12T01:12:07.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PHX42RNuvclF0YcRIgA50kgNrnc=', 'Content-Length': '0'} 2023-10-12T01:12:07.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cda745a00affed9-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:12:07.770 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.770 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.771 DEBUG:boto:Method: GET 2023-10-12T01:12:07.771 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:12:07.772 DEBUG:boto:Data: 2023-10-12T01:12:07.772 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.773 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.774 DEBUG:boto:Params: {} 2023-10-12T01:12:07.774 DEBUG:boto:Token: None 2023-10-12T01:12:07.775 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38 2023-10-12T01:12:07.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VdfLCUSz9+pXGkb5bmhszdsoepI= 2023-10-12T01:12:07.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VdfLCUSz9+pXGkb5bmhszdsoepI=', 'Content-Length': '0'} 2023-10-12T01:12:07.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace84b165994f68d-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.779 DEBUG:boto:path=/wafmoa-59/obj38 2023-10-12T01:12:07.779 DEBUG:boto:auth_path=/wafmoa-59/obj38 2023-10-12T01:12:07.780 DEBUG:boto:Method: GET 2023-10-12T01:12:07.780 DEBUG:boto:Path: /wafmoa-59/obj38 2023-10-12T01:12:07.780 DEBUG:boto:Data: 2023-10-12T01:12:07.781 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.782 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.782 DEBUG:boto:Params: {} 2023-10-12T01:12:07.783 DEBUG:boto:Token: None 2023-10-12T01:12:07.783 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj38 2023-10-12T01:12:07.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VdfLCUSz9+pXGkb5bmhszdsoepI= 2023-10-12T01:12:07.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VdfLCUSz9+pXGkb5bmhszdsoepI=', 'Content-Length': '0'} 2023-10-12T01:12:07.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63404014919bb9c-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.788 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.788 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.789 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.789 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.789 DEBUG:boto:Method: GET 2023-10-12T01:12:07.790 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.790 DEBUG:boto:Data: 2023-10-12T01:12:07.791 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.792 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.792 DEBUG:boto:Params: {} 2023-10-12T01:12:07.793 DEBUG:boto:Token: None 2023-10-12T01:12:07.793 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397b2ca02571c627-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.798 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.799 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.800 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.800 DEBUG:boto:path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.801 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.801 DEBUG:boto:Method: GET 2023-10-12T01:12:07.802 DEBUG:boto:Path: /wafmoa-59/?prefix=obj38 2023-10-12T01:12:07.802 DEBUG:boto:Data: 2023-10-12T01:12:07.803 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.804 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.804 DEBUG:boto:Params: {} 2023-10-12T01:12:07.805 DEBUG:boto:Token: None 2023-10-12T01:12:07.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d2191fa5975fce-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.854 DEBUG:boto:b'wafmoa-59obj381000falseobj382023-10-12T01:01:06.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:12:07.856 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.857 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.857 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.857 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.858 DEBUG:boto:Method: GET 2023-10-12T01:12:07.858 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.859 DEBUG:boto:Data: 2023-10-12T01:12:07.859 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.860 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.861 DEBUG:boto:Params: {} 2023-10-12T01:12:07.861 DEBUG:boto:Token: None 2023-10-12T01:12:07.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.862 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo= 2023-10-12T01:12:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo=', 'Content-Length': '0'} 2023-10-12T01:12:07.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015bd4baa26703516-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.866 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.867 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.867 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.868 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.868 DEBUG:boto:Method: GET 2023-10-12T01:12:07.869 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.870 DEBUG:boto:Data: 2023-10-12T01:12:07.871 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.872 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.873 DEBUG:boto:Params: {} 2023-10-12T01:12:07.873 DEBUG:boto:Token: None 2023-10-12T01:12:07.874 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo= 2023-10-12T01:12:07.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo=', 'Content-Length': '0'} 2023-10-12T01:12:07.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ee04f0af583af3-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.879 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.879 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.880 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.880 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.881 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.881 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.882 DEBUG:boto:Data: 2023-10-12T01:12:07.882 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.883 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.884 DEBUG:boto:Params: {} 2023-10-12T01:12:07.884 DEBUG:boto:Token: None 2023-10-12T01:12:07.885 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z0Sf4VcQGihttNZBOYatCaFA3ow= 2023-10-12T01:12:07.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z0Sf4VcQGihttNZBOYatCaFA3ow=', 'Content-Length': '0'} 2023-10-12T01:12:07.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4f5ae45e76a812-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.889 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.890 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.890 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.890 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.891 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.891 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.892 DEBUG:boto:Data: 2023-10-12T01:12:07.892 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.893 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.893 DEBUG:boto:Params: {} 2023-10-12T01:12:07.894 DEBUG:boto:Token: None 2023-10-12T01:12:07.894 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z0Sf4VcQGihttNZBOYatCaFA3ow= 2023-10-12T01:12:07.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z0Sf4VcQGihttNZBOYatCaFA3ow=', 'Content-Length': '0'} 2023-10-12T01:12:07.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1491ef10a0df14-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:12:07.898 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.899 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.899 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.900 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.900 DEBUG:boto:Method: GET 2023-10-12T01:12:07.900 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.901 DEBUG:boto:Data: 2023-10-12T01:12:07.901 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.902 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.903 DEBUG:boto:Params: {} 2023-10-12T01:12:07.903 DEBUG:boto:Token: None 2023-10-12T01:12:07.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo= 2023-10-12T01:12:07.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo=', 'Content-Length': '0'} 2023-10-12T01:12:07.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7514491739805f7-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.908 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.909 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.910 DEBUG:boto:path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.911 DEBUG:boto:auth_path=/wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.911 DEBUG:boto:Method: GET 2023-10-12T01:12:07.912 DEBUG:boto:Path: /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.913 DEBUG:boto:Data: 2023-10-12T01:12:07.914 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.915 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.916 DEBUG:boto:Params: {} 2023-10-12T01:12:07.917 DEBUG:boto:Token: None 2023-10-12T01:12:07.918 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44?versionId=null 2023-10-12T01:12:07.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo= 2023-10-12T01:12:07.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GZEktihXHEQxuiHLYclR0vJrYQo=', 'Content-Length': '0'} 2023-10-12T01:12:07.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e2d964a8d44df4-00652747e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.923 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.923 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.924 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.924 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.925 DEBUG:boto:Method: GET 2023-10-12T01:12:07.925 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.926 DEBUG:boto:Data: 2023-10-12T01:12:07.926 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.927 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.928 DEBUG:boto:Params: {} 2023-10-12T01:12:07.928 DEBUG:boto:Token: None 2023-10-12T01:12:07.929 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce70183f08d2a373-00652747e7-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.974 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.975 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:07.976 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:07.976 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.977 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.977 DEBUG:boto:Method: GET 2023-10-12T01:12:07.978 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:12:07.978 DEBUG:boto:Data: 2023-10-12T01:12:07.978 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:07.980 DEBUG:boto:Port: 8000 2023-10-12T01:12:07.980 DEBUG:boto:Params: {} 2023-10-12T01:12:07.980 DEBUG:boto:Token: None 2023-10-12T01:12:07.981 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/ 2023-10-12T01:12:07.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE= 2023-10-12T01:12:07.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FGdwcVbfiSgzCHK7dBPjdl1DOE=', 'Content-Length': '0'} 2023-10-12T01:12:07.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113d4e355abe4f4c-00652747e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.988 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:07.988 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.989 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.989 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.990 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:12:07.991 DEBUG:boto:Data: 2023-10-12T01:12:07.992 DEBUG:boto:Headers: {} 2023-10-12T01:12:07.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:07.992 DEBUG:boto:Port: 8001 2023-10-12T01:12:07.993 DEBUG:boto:Params: {} 2023-10-12T01:12:07.993 DEBUG:boto:Token: None 2023-10-12T01:12:07.994 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:07 GMT /wafmoa-59/obj44 2023-10-12T01:12:07.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iWDveYrVyTAUq5cMEIkGCKTHCWQ= 2023-10-12T01:12:07.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iWDveYrVyTAUq5cMEIkGCKTHCWQ=', 'Content-Length': '0'} 2023-10-12T01:12:07.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a1109e6c11aa37-00652747e7-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:07.998 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:07.998 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:07.999 DEBUG:boto:Method: HEAD 2023-10-12T01:12:07.999 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:12:08.000 DEBUG:boto:Data: 2023-10-12T01:12:08.000 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.002 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.002 DEBUG:boto:Params: {} 2023-10-12T01:12:08.003 DEBUG:boto:Token: None 2023-10-12T01:12:08.003 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj44 2023-10-12T01:12:08.004 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OJEd6kgOE7bfis+KLSAE6qReicM= 2023-10-12T01:12:08.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OJEd6kgOE7bfis+KLSAE6qReicM=', 'Content-Length': '0'} 2023-10-12T01:12:08.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba8d85e65895206-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:12:08.008 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:08.008 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:08.009 DEBUG:boto:Method: GET 2023-10-12T01:12:08.009 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:12:08.010 DEBUG:boto:Data: 2023-10-12T01:12:08.010 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.011 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.012 DEBUG:boto:Params: {} 2023-10-12T01:12:08.012 DEBUG:boto:Token: None 2023-10-12T01:12:08.013 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj44 2023-10-12T01:12:08.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NHBENKP6YV4SnB/RmKpyijuUFQA= 2023-10-12T01:12:08.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NHBENKP6YV4SnB/RmKpyijuUFQA=', 'Content-Length': '0'} 2023-10-12T01:12:08.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df27888dbc5a507-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.017 DEBUG:boto:path=/wafmoa-59/obj44 2023-10-12T01:12:08.017 DEBUG:boto:auth_path=/wafmoa-59/obj44 2023-10-12T01:12:08.018 DEBUG:boto:Method: GET 2023-10-12T01:12:08.019 DEBUG:boto:Path: /wafmoa-59/obj44 2023-10-12T01:12:08.019 DEBUG:boto:Data: 2023-10-12T01:12:08.020 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.022 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.022 DEBUG:boto:Params: {} 2023-10-12T01:12:08.023 DEBUG:boto:Token: None 2023-10-12T01:12:08.024 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj44 2023-10-12T01:12:08.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NHBENKP6YV4SnB/RmKpyijuUFQA= 2023-10-12T01:12:08.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NHBENKP6YV4SnB/RmKpyijuUFQA=', 'Content-Length': '0'} 2023-10-12T01:12:08.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd174ab6342dea7-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.074 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.075 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.076 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.076 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.077 DEBUG:boto:Method: GET 2023-10-12T01:12:08.077 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.077 DEBUG:boto:Data: 2023-10-12T01:12:08.078 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.079 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.079 DEBUG:boto:Params: {} 2023-10-12T01:12:08.080 DEBUG:boto:Token: None 2023-10-12T01:12:08.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e54e4d2d5d61a8-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.085 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.086 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.087 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.087 DEBUG:boto:path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.087 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.088 DEBUG:boto:Method: GET 2023-10-12T01:12:08.088 DEBUG:boto:Path: /wafmoa-59/?prefix=obj44 2023-10-12T01:12:08.089 DEBUG:boto:Data: 2023-10-12T01:12:08.089 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.090 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.091 DEBUG:boto:Params: {} 2023-10-12T01:12:08.091 DEBUG:boto:Token: None 2023-10-12T01:12:08.092 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f9635b4bee8f06f-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.098 DEBUG:boto:b'wafmoa-59obj441000falseobj442023-10-12T00:57:47.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:12:08.099 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.099 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.100 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.100 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.101 DEBUG:boto:Method: GET 2023-10-12T01:12:08.102 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.103 DEBUG:boto:Data: 2023-10-12T01:12:08.103 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.104 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.105 DEBUG:boto:Params: {} 2023-10-12T01:12:08.105 DEBUG:boto:Token: None 2023-10-12T01:12:08.106 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.106 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY= 2023-10-12T01:12:08.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY=', 'Content-Length': '0'} 2023-10-12T01:12:08.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eed203dd80b33da1-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.110 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.110 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.111 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.112 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.113 DEBUG:boto:Method: GET 2023-10-12T01:12:08.113 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.114 DEBUG:boto:Data: 2023-10-12T01:12:08.114 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.115 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.115 DEBUG:boto:Params: {} 2023-10-12T01:12:08.116 DEBUG:boto:Token: None 2023-10-12T01:12:08.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.117 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY= 2023-10-12T01:12:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY=', 'Content-Length': '0'} 2023-10-12T01:12:08.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d50f01e64477139-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.120 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.120 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.121 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.121 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.122 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.122 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.122 DEBUG:boto:Data: 2023-10-12T01:12:08.123 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.124 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.124 DEBUG:boto:Params: {} 2023-10-12T01:12:08.125 DEBUG:boto:Token: None 2023-10-12T01:12:08.125 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KJ9fxCilsw8xh1DvLKBcm3tTeoA= 2023-10-12T01:12:08.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KJ9fxCilsw8xh1DvLKBcm3tTeoA=', 'Content-Length': '0'} 2023-10-12T01:12:08.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd4bf69737312b0-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.129 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.129 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.130 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.130 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.130 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.131 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.131 DEBUG:boto:Data: 2023-10-12T01:12:08.132 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.133 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.133 DEBUG:boto:Params: {} 2023-10-12T01:12:08.134 DEBUG:boto:Token: None 2023-10-12T01:12:08.134 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.134 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KJ9fxCilsw8xh1DvLKBcm3tTeoA= 2023-10-12T01:12:08.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KJ9fxCilsw8xh1DvLKBcm3tTeoA=', 'Content-Length': '0'} 2023-10-12T01:12:08.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ab9467888a8240-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:12:08.138 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.138 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.139 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.139 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.140 DEBUG:boto:Method: GET 2023-10-12T01:12:08.140 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.141 DEBUG:boto:Data: 2023-10-12T01:12:08.141 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.142 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.142 DEBUG:boto:Params: {} 2023-10-12T01:12:08.143 DEBUG:boto:Token: None 2023-10-12T01:12:08.143 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.144 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY= 2023-10-12T01:12:08.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY=', 'Content-Length': '0'} 2023-10-12T01:12:08.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000468c38462555653-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.147 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.148 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.148 DEBUG:boto:path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.148 DEBUG:boto:auth_path=/wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.149 DEBUG:boto:Method: GET 2023-10-12T01:12:08.149 DEBUG:boto:Path: /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.150 DEBUG:boto:Data: 2023-10-12T01:12:08.150 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.151 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.152 DEBUG:boto:Params: {} 2023-10-12T01:12:08.152 DEBUG:boto:Token: None 2023-10-12T01:12:08.153 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46?versionId=null 2023-10-12T01:12:08.153 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY= 2023-10-12T01:12:08.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:27woLjbCUQ4wZgKqXlzi3XaHQnY=', 'Content-Length': '0'} 2023-10-12T01:12:08.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f7dee0c8b75a5f-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.158 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.158 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.159 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.159 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.160 DEBUG:boto:Method: GET 2023-10-12T01:12:08.160 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.161 DEBUG:boto:Data: 2023-10-12T01:12:08.161 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.162 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.162 DEBUG:boto:Params: {} 2023-10-12T01:12:08.163 DEBUG:boto:Token: None 2023-10-12T01:12:08.163 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.164 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000251f77f8557d3f0f-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.168 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.168 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.169 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.169 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.170 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.170 DEBUG:boto:Method: GET 2023-10-12T01:12:08.171 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.171 DEBUG:boto:Data: 2023-10-12T01:12:08.172 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.173 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.173 DEBUG:boto:Params: {} 2023-10-12T01:12:08.174 DEBUG:boto:Token: None 2023-10-12T01:12:08.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d1b5961c8aec86-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.180 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.181 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.181 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.182 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.182 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:12:08.183 DEBUG:boto:Data: 2023-10-12T01:12:08.183 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.184 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.185 DEBUG:boto:Params: {} 2023-10-12T01:12:08.185 DEBUG:boto:Token: None 2023-10-12T01:12:08.186 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46 2023-10-12T01:12:08.186 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GSBlxl44CXDDPWbpmATHByod24I= 2023-10-12T01:12:08.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GSBlxl44CXDDPWbpmATHByod24I=', 'Content-Length': '0'} 2023-10-12T01:12:08.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185eb37a0137a9e9-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.189 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.190 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.190 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.191 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:12:08.191 DEBUG:boto:Data: 2023-10-12T01:12:08.192 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.193 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.193 DEBUG:boto:Params: {} 2023-10-12T01:12:08.194 DEBUG:boto:Token: None 2023-10-12T01:12:08.194 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46 2023-10-12T01:12:08.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GSBlxl44CXDDPWbpmATHByod24I= 2023-10-12T01:12:08.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GSBlxl44CXDDPWbpmATHByod24I=', 'Content-Length': '0'} 2023-10-12T01:12:08.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008244a3a2479583fd-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:12:08.198 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.199 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.199 DEBUG:boto:Method: GET 2023-10-12T01:12:08.200 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:12:08.200 DEBUG:boto:Data: 2023-10-12T01:12:08.201 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.202 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.202 DEBUG:boto:Params: {} 2023-10-12T01:12:08.203 DEBUG:boto:Token: None 2023-10-12T01:12:08.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46 2023-10-12T01:12:08.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rsS39fZBJzSm386WO/9g32yUR+A= 2023-10-12T01:12:08.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rsS39fZBJzSm386WO/9g32yUR+A=', 'Content-Length': '0'} 2023-10-12T01:12:08.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b33ff4938e07ea-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.254 DEBUG:boto:path=/wafmoa-59/obj46 2023-10-12T01:12:08.255 DEBUG:boto:auth_path=/wafmoa-59/obj46 2023-10-12T01:12:08.256 DEBUG:boto:Method: GET 2023-10-12T01:12:08.256 DEBUG:boto:Path: /wafmoa-59/obj46 2023-10-12T01:12:08.256 DEBUG:boto:Data: 2023-10-12T01:12:08.257 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.258 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.258 DEBUG:boto:Params: {} 2023-10-12T01:12:08.259 DEBUG:boto:Token: None 2023-10-12T01:12:08.260 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj46 2023-10-12T01:12:08.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rsS39fZBJzSm386WO/9g32yUR+A= 2023-10-12T01:12:08.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rsS39fZBJzSm386WO/9g32yUR+A=', 'Content-Length': '0'} 2023-10-12T01:12:08.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615a7eb673016eb5-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.264 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.264 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.265 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.265 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.266 DEBUG:boto:Method: GET 2023-10-12T01:12:08.266 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.267 DEBUG:boto:Data: 2023-10-12T01:12:08.267 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.268 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.268 DEBUG:boto:Params: {} 2023-10-12T01:12:08.269 DEBUG:boto:Token: None 2023-10-12T01:12:08.269 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf6f8b380ab93ff-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.274 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.274 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.275 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.275 DEBUG:boto:path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.276 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.276 DEBUG:boto:Method: GET 2023-10-12T01:12:08.277 DEBUG:boto:Path: /wafmoa-59/?prefix=obj46 2023-10-12T01:12:08.277 DEBUG:boto:Data: 2023-10-12T01:12:08.278 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.279 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.279 DEBUG:boto:Params: {} 2023-10-12T01:12:08.279 DEBUG:boto:Token: None 2023-10-12T01:12:08.280 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aedd8b2bea4fc7c-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.285 DEBUG:boto:b'wafmoa-59obj461000falseobj462023-10-12T01:03:05.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:12:08.287 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.287 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.287 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.288 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.288 DEBUG:boto:Method: GET 2023-10-12T01:12:08.289 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.289 DEBUG:boto:Data: 2023-10-12T01:12:08.290 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.291 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.291 DEBUG:boto:Params: {} 2023-10-12T01:12:08.292 DEBUG:boto:Token: None 2023-10-12T01:12:08.292 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA= 2023-10-12T01:12:08.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA=', 'Content-Length': '0'} 2023-10-12T01:12:08.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022deb8ed8d03a437-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.296 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.296 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.297 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.297 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.298 DEBUG:boto:Method: GET 2023-10-12T01:12:08.298 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.299 DEBUG:boto:Data: 2023-10-12T01:12:08.299 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.300 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.300 DEBUG:boto:Params: {} 2023-10-12T01:12:08.301 DEBUG:boto:Token: None 2023-10-12T01:12:08.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.302 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA= 2023-10-12T01:12:08.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA=', 'Content-Length': '0'} 2023-10-12T01:12:08.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd9afb091d7de93-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.305 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.306 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.306 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.307 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.307 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.308 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.308 DEBUG:boto:Data: 2023-10-12T01:12:08.308 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.310 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.310 DEBUG:boto:Params: {} 2023-10-12T01:12:08.310 DEBUG:boto:Token: None 2023-10-12T01:12:08.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AB6ogBTj39C8yrF1q0juLt4N9r8= 2023-10-12T01:12:08.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AB6ogBTj39C8yrF1q0juLt4N9r8=', 'Content-Length': '0'} 2023-10-12T01:12:08.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9aa95f9f6380e7-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.314 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.315 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.315 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.316 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.316 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.317 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.317 DEBUG:boto:Data: 2023-10-12T01:12:08.317 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.318 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.318 DEBUG:boto:Params: {} 2023-10-12T01:12:08.319 DEBUG:boto:Token: None 2023-10-12T01:12:08.319 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.320 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AB6ogBTj39C8yrF1q0juLt4N9r8= 2023-10-12T01:12:08.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AB6ogBTj39C8yrF1q0juLt4N9r8=', 'Content-Length': '0'} 2023-10-12T01:12:08.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9c793643455dfb-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:12:08.324 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.324 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.325 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.325 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.326 DEBUG:boto:Method: GET 2023-10-12T01:12:08.326 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.327 DEBUG:boto:Data: 2023-10-12T01:12:08.327 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.328 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.328 DEBUG:boto:Params: {} 2023-10-12T01:12:08.329 DEBUG:boto:Token: None 2023-10-12T01:12:08.329 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.330 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA= 2023-10-12T01:12:08.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA=', 'Content-Length': '0'} 2023-10-12T01:12:08.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1027c47f0538791-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.333 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.333 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.334 DEBUG:boto:path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.334 DEBUG:boto:auth_path=/wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.335 DEBUG:boto:Method: GET 2023-10-12T01:12:08.335 DEBUG:boto:Path: /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.336 DEBUG:boto:Data: 2023-10-12T01:12:08.336 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.337 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.337 DEBUG:boto:Params: {} 2023-10-12T01:12:08.338 DEBUG:boto:Token: None 2023-10-12T01:12:08.338 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51?versionId=null 2023-10-12T01:12:08.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA= 2023-10-12T01:12:08.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZK6RBudGeg6wsyX5dFyIWhGLolA=', 'Content-Length': '0'} 2023-10-12T01:12:08.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b6c8528358cb61-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.386 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.387 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.388 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.388 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.388 DEBUG:boto:Method: GET 2023-10-12T01:12:08.389 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.389 DEBUG:boto:Data: 2023-10-12T01:12:08.390 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.391 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.391 DEBUG:boto:Params: {} 2023-10-12T01:12:08.392 DEBUG:boto:Token: None 2023-10-12T01:12:08.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009018063e722628e9-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.397 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.397 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.398 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.398 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.399 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.399 DEBUG:boto:Method: GET 2023-10-12T01:12:08.400 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.400 DEBUG:boto:Data: 2023-10-12T01:12:08.401 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.402 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.402 DEBUG:boto:Params: {} 2023-10-12T01:12:08.402 DEBUG:boto:Token: None 2023-10-12T01:12:08.403 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097583278e154e8c2-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.409 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.409 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.410 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.410 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.411 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:12:08.411 DEBUG:boto:Data: 2023-10-12T01:12:08.412 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.413 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.413 DEBUG:boto:Params: {} 2023-10-12T01:12:08.414 DEBUG:boto:Token: None 2023-10-12T01:12:08.414 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51 2023-10-12T01:12:08.414 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:05fKD64PUqT1Mpmgg9weKRF7AY0= 2023-10-12T01:12:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:05fKD64PUqT1Mpmgg9weKRF7AY0=', 'Content-Length': '0'} 2023-10-12T01:12:08.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005610b7336e96f68-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.418 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.418 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.418 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.419 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:12:08.419 DEBUG:boto:Data: 2023-10-12T01:12:08.420 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.420 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.421 DEBUG:boto:Params: {} 2023-10-12T01:12:08.421 DEBUG:boto:Token: None 2023-10-12T01:12:08.422 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51 2023-10-12T01:12:08.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:05fKD64PUqT1Mpmgg9weKRF7AY0= 2023-10-12T01:12:08.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:05fKD64PUqT1Mpmgg9weKRF7AY0=', 'Content-Length': '0'} 2023-10-12T01:12:08.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000551fd9ef67bbeba4-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:12:08.426 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.426 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.427 DEBUG:boto:Method: GET 2023-10-12T01:12:08.427 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:12:08.428 DEBUG:boto:Data: 2023-10-12T01:12:08.428 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.429 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.429 DEBUG:boto:Params: {} 2023-10-12T01:12:08.430 DEBUG:boto:Token: None 2023-10-12T01:12:08.430 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51 2023-10-12T01:12:08.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:czgOLhY745PiszFZADOrdrn2YLo= 2023-10-12T01:12:08.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:czgOLhY745PiszFZADOrdrn2YLo=', 'Content-Length': '0'} 2023-10-12T01:12:08.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd4f485eeb305cc-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.474 DEBUG:boto:path=/wafmoa-59/obj51 2023-10-12T01:12:08.475 DEBUG:boto:auth_path=/wafmoa-59/obj51 2023-10-12T01:12:08.475 DEBUG:boto:Method: GET 2023-10-12T01:12:08.476 DEBUG:boto:Path: /wafmoa-59/obj51 2023-10-12T01:12:08.476 DEBUG:boto:Data: 2023-10-12T01:12:08.477 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.477 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.478 DEBUG:boto:Params: {} 2023-10-12T01:12:08.478 DEBUG:boto:Token: None 2023-10-12T01:12:08.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj51 2023-10-12T01:12:08.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:czgOLhY745PiszFZADOrdrn2YLo= 2023-10-12T01:12:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:czgOLhY745PiszFZADOrdrn2YLo=', 'Content-Length': '0'} 2023-10-12T01:12:08.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1fabfd6c2cca06-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.483 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.483 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.484 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.484 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.484 DEBUG:boto:Method: GET 2023-10-12T01:12:08.485 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.485 DEBUG:boto:Data: 2023-10-12T01:12:08.486 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.487 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.487 DEBUG:boto:Params: {} 2023-10-12T01:12:08.488 DEBUG:boto:Token: None 2023-10-12T01:12:08.488 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba635b42386e3519-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.492 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.493 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.493 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.494 DEBUG:boto:path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.494 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.494 DEBUG:boto:Method: GET 2023-10-12T01:12:08.495 DEBUG:boto:Path: /wafmoa-59/?prefix=obj51 2023-10-12T01:12:08.495 DEBUG:boto:Data: 2023-10-12T01:12:08.496 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.497 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.497 DEBUG:boto:Params: {} 2023-10-12T01:12:08.497 DEBUG:boto:Token: None 2023-10-12T01:12:08.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa5daefd4fa420b9-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.504 DEBUG:boto:b'wafmoa-59obj511000falseobj512023-10-12T00:59:23.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:12:08.505 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.505 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.506 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.506 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.507 DEBUG:boto:Method: GET 2023-10-12T01:12:08.507 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.508 DEBUG:boto:Data: 2023-10-12T01:12:08.508 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.509 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.509 DEBUG:boto:Params: {} 2023-10-12T01:12:08.510 DEBUG:boto:Token: None 2023-10-12T01:12:08.510 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.511 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc= 2023-10-12T01:12:08.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc=', 'Content-Length': '0'} 2023-10-12T01:12:08.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003822a2fff36cdae0-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.514 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.515 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.515 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.515 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.516 DEBUG:boto:Method: GET 2023-10-12T01:12:08.516 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.517 DEBUG:boto:Data: 2023-10-12T01:12:08.517 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.518 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.519 DEBUG:boto:Params: {} 2023-10-12T01:12:08.519 DEBUG:boto:Token: None 2023-10-12T01:12:08.519 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc= 2023-10-12T01:12:08.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc=', 'Content-Length': '0'} 2023-10-12T01:12:08.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d4dc2fbc359483-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.570 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.571 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.674 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.675 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.676 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.676 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.677 DEBUG:boto:Data: 2023-10-12T01:12:08.677 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.678 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.678 DEBUG:boto:Params: {} 2023-10-12T01:12:08.679 DEBUG:boto:Token: None 2023-10-12T01:12:08.679 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z/QJlNUmr/I00/r2lTi1lbh2cKw= 2023-10-12T01:12:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z/QJlNUmr/I00/r2lTi1lbh2cKw=', 'Content-Length': '0'} 2023-10-12T01:12:08.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f386c9d05e4701d-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.683 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.684 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.684 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.685 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.685 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.686 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.686 DEBUG:boto:Data: 2023-10-12T01:12:08.687 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.687 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.688 DEBUG:boto:Params: {} 2023-10-12T01:12:08.688 DEBUG:boto:Token: None 2023-10-12T01:12:08.689 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:z/QJlNUmr/I00/r2lTi1lbh2cKw= 2023-10-12T01:12:08.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:z/QJlNUmr/I00/r2lTi1lbh2cKw=', 'Content-Length': '0'} 2023-10-12T01:12:08.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4beb0bd13933857-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:12:08.693 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.694 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.694 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.695 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.695 DEBUG:boto:Method: GET 2023-10-12T01:12:08.696 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.696 DEBUG:boto:Data: 2023-10-12T01:12:08.697 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.697 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.698 DEBUG:boto:Params: {} 2023-10-12T01:12:08.698 DEBUG:boto:Token: None 2023-10-12T01:12:08.699 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc= 2023-10-12T01:12:08.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc=', 'Content-Length': '0'} 2023-10-12T01:12:08.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee06f67a5e381eb-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.742 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.743 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.744 DEBUG:boto:path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.744 DEBUG:boto:auth_path=/wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.745 DEBUG:boto:Method: GET 2023-10-12T01:12:08.745 DEBUG:boto:Path: /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.745 DEBUG:boto:Data: 2023-10-12T01:12:08.746 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.747 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.747 DEBUG:boto:Params: {} 2023-10-12T01:12:08.748 DEBUG:boto:Token: None 2023-10-12T01:12:08.748 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55?versionId=null 2023-10-12T01:12:08.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc= 2023-10-12T01:12:08.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uUQ7i5upR9hRwF/j8F5jNRuNMUc=', 'Content-Length': '0'} 2023-10-12T01:12:08.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db50b319b11edb4-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.752 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.753 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.753 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.754 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.754 DEBUG:boto:Method: GET 2023-10-12T01:12:08.755 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.755 DEBUG:boto:Data: 2023-10-12T01:12:08.756 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.757 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.757 DEBUG:boto:Params: {} 2023-10-12T01:12:08.758 DEBUG:boto:Token: None 2023-10-12T01:12:08.758 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab46fc8155bdfecf-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.763 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.763 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.764 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.764 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.765 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.765 DEBUG:boto:Method: GET 2023-10-12T01:12:08.766 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.767 DEBUG:boto:Data: 2023-10-12T01:12:08.767 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.768 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.769 DEBUG:boto:Params: {} 2023-10-12T01:12:08.769 DEBUG:boto:Token: None 2023-10-12T01:12:08.769 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c7d35394e6f3496-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.776 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.777 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.777 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.777 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.778 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:12:08.778 DEBUG:boto:Data: 2023-10-12T01:12:08.779 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.780 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.780 DEBUG:boto:Params: {} 2023-10-12T01:12:08.780 DEBUG:boto:Token: None 2023-10-12T01:12:08.781 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55 2023-10-12T01:12:08.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im332AL4sY1xrOWGh2r/f5pl8XM= 2023-10-12T01:12:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im332AL4sY1xrOWGh2r/f5pl8XM=', 'Content-Length': '0'} 2023-10-12T01:12:08.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006afcb1a7bbf5c6e6-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.785 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.785 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.785 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.786 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:12:08.786 DEBUG:boto:Data: 2023-10-12T01:12:08.787 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.788 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.788 DEBUG:boto:Params: {} 2023-10-12T01:12:08.788 DEBUG:boto:Token: None 2023-10-12T01:12:08.789 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55 2023-10-12T01:12:08.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Im332AL4sY1xrOWGh2r/f5pl8XM= 2023-10-12T01:12:08.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Im332AL4sY1xrOWGh2r/f5pl8XM=', 'Content-Length': '0'} 2023-10-12T01:12:08.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef4f5f02cac4c25-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:12:08.793 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.794 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.794 DEBUG:boto:Method: GET 2023-10-12T01:12:08.795 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:12:08.795 DEBUG:boto:Data: 2023-10-12T01:12:08.795 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.796 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.797 DEBUG:boto:Params: {} 2023-10-12T01:12:08.797 DEBUG:boto:Token: None 2023-10-12T01:12:08.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55 2023-10-12T01:12:08.798 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pk1hTJdvUWkHtmd9ipdyBArmyeo= 2023-10-12T01:12:08.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pk1hTJdvUWkHtmd9ipdyBArmyeo=', 'Content-Length': '0'} 2023-10-12T01:12:08.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a316bdebc7c55eb-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.801 DEBUG:boto:path=/wafmoa-59/obj55 2023-10-12T01:12:08.802 DEBUG:boto:auth_path=/wafmoa-59/obj55 2023-10-12T01:12:08.802 DEBUG:boto:Method: GET 2023-10-12T01:12:08.803 DEBUG:boto:Path: /wafmoa-59/obj55 2023-10-12T01:12:08.803 DEBUG:boto:Data: 2023-10-12T01:12:08.804 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.805 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.805 DEBUG:boto:Params: {} 2023-10-12T01:12:08.806 DEBUG:boto:Token: None 2023-10-12T01:12:08.806 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj55 2023-10-12T01:12:08.807 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pk1hTJdvUWkHtmd9ipdyBArmyeo= 2023-10-12T01:12:08.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pk1hTJdvUWkHtmd9ipdyBArmyeo=', 'Content-Length': '0'} 2023-10-12T01:12:08.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:57:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061885468749b4a58-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.810 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.811 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.811 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.812 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.812 DEBUG:boto:Method: GET 2023-10-12T01:12:08.812 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.813 DEBUG:boto:Data: 2023-10-12T01:12:08.813 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.814 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.815 DEBUG:boto:Params: {} 2023-10-12T01:12:08.815 DEBUG:boto:Token: None 2023-10-12T01:12:08.816 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.816 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009deb05e3e9f5cd6e-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.820 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.820 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.821 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.821 DEBUG:boto:path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.822 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.822 DEBUG:boto:Method: GET 2023-10-12T01:12:08.823 DEBUG:boto:Path: /wafmoa-59/?prefix=obj55 2023-10-12T01:12:08.823 DEBUG:boto:Data: 2023-10-12T01:12:08.823 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.824 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.825 DEBUG:boto:Params: {} 2023-10-12T01:12:08.825 DEBUG:boto:Token: None 2023-10-12T01:12:08.826 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c0de37c976dcc18-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.831 DEBUG:boto:b'wafmoa-59obj551000falseobj552023-10-12T00:57:47.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:12:08.833 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.833 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.833 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.834 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.834 DEBUG:boto:Method: GET 2023-10-12T01:12:08.835 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.835 DEBUG:boto:Data: 2023-10-12T01:12:08.836 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.837 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.837 DEBUG:boto:Params: {} 2023-10-12T01:12:08.838 DEBUG:boto:Token: None 2023-10-12T01:12:08.838 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI= 2023-10-12T01:12:08.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI=', 'Content-Length': '0'} 2023-10-12T01:12:08.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f8fce801641dea-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.882 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.883 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.884 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.884 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.884 DEBUG:boto:Method: GET 2023-10-12T01:12:08.885 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.885 DEBUG:boto:Data: 2023-10-12T01:12:08.886 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.887 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.887 DEBUG:boto:Params: {} 2023-10-12T01:12:08.888 DEBUG:boto:Token: None 2023-10-12T01:12:08.888 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.889 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI= 2023-10-12T01:12:08.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI=', 'Content-Length': '0'} 2023-10-12T01:12:08.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ac9bf5a5b61567-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.893 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.893 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.894 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.894 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.894 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.895 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.895 DEBUG:boto:Data: 2023-10-12T01:12:08.896 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.897 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.897 DEBUG:boto:Params: {} 2023-10-12T01:12:08.897 DEBUG:boto:Token: None 2023-10-12T01:12:08.898 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0HZ2CuIiJuetQclLlK+hYYFsncw= 2023-10-12T01:12:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0HZ2CuIiJuetQclLlK+hYYFsncw=', 'Content-Length': '0'} 2023-10-12T01:12:08.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026287e3f404f65a5-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.901 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.902 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.902 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.903 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.903 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.904 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.904 DEBUG:boto:Data: 2023-10-12T01:12:08.904 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.905 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.906 DEBUG:boto:Params: {} 2023-10-12T01:12:08.906 DEBUG:boto:Token: None 2023-10-12T01:12:08.907 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0HZ2CuIiJuetQclLlK+hYYFsncw= 2023-10-12T01:12:08.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0HZ2CuIiJuetQclLlK+hYYFsncw=', 'Content-Length': '0'} 2023-10-12T01:12:08.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b69905f916f4a3-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:12:08.911 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.911 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.912 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.912 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.913 DEBUG:boto:Method: GET 2023-10-12T01:12:08.913 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.913 DEBUG:boto:Data: 2023-10-12T01:12:08.914 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.915 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.915 DEBUG:boto:Params: {} 2023-10-12T01:12:08.916 DEBUG:boto:Token: None 2023-10-12T01:12:08.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI= 2023-10-12T01:12:08.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI=', 'Content-Length': '0'} 2023-10-12T01:12:08.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b1d1c7c1bff037c-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.920 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.920 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.921 DEBUG:boto:path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.921 DEBUG:boto:auth_path=/wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.921 DEBUG:boto:Method: GET 2023-10-12T01:12:08.922 DEBUG:boto:Path: /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.922 DEBUG:boto:Data: 2023-10-12T01:12:08.925 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.927 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.927 DEBUG:boto:Params: {} 2023-10-12T01:12:08.928 DEBUG:boto:Token: None 2023-10-12T01:12:08.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57?versionId=null 2023-10-12T01:12:08.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI= 2023-10-12T01:12:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4FZQ1MpOTuVUN/DlN7/Ja/2qqHI=', 'Content-Length': '0'} 2023-10-12T01:12:08.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b44a36609749df83-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.933 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.934 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.935 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.936 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.936 DEBUG:boto:Method: GET 2023-10-12T01:12:08.937 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.938 DEBUG:boto:Data: 2023-10-12T01:12:08.938 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.939 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.940 DEBUG:boto:Params: {} 2023-10-12T01:12:08.940 DEBUG:boto:Token: None 2023-10-12T01:12:08.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.941 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f0c6989b3982f54-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.945 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.946 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.947 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.947 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.947 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.948 DEBUG:boto:Method: GET 2023-10-12T01:12:08.948 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.949 DEBUG:boto:Data: 2023-10-12T01:12:08.949 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.950 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.950 DEBUG:boto:Params: {} 2023-10-12T01:12:08.951 DEBUG:boto:Token: None 2023-10-12T01:12:08.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acdda92865a55db6-00652747e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.957 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:08.958 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.958 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.959 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.959 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:12:08.959 DEBUG:boto:Data: 2023-10-12T01:12:08.960 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.961 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.961 DEBUG:boto:Params: {} 2023-10-12T01:12:08.961 DEBUG:boto:Token: None 2023-10-12T01:12:08.962 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57 2023-10-12T01:12:08.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m+30H5fa4+9UV4VlhUEDCtB2ti0= 2023-10-12T01:12:08.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m+30H5fa4+9UV4VlhUEDCtB2ti0=', 'Content-Length': '0'} 2023-10-12T01:12:08.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009700e06f191c1384-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.966 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.966 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.967 DEBUG:boto:Method: HEAD 2023-10-12T01:12:08.967 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:12:08.967 DEBUG:boto:Data: 2023-10-12T01:12:08.968 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.968 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.969 DEBUG:boto:Params: {} 2023-10-12T01:12:08.969 DEBUG:boto:Token: None 2023-10-12T01:12:08.970 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57 2023-10-12T01:12:08.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m+30H5fa4+9UV4VlhUEDCtB2ti0= 2023-10-12T01:12:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m+30H5fa4+9UV4VlhUEDCtB2ti0=', 'Content-Length': '0'} 2023-10-12T01:12:08.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049053b6c48ecc2b6-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:12:08.974 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.974 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.975 DEBUG:boto:Method: GET 2023-10-12T01:12:08.975 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:12:08.976 DEBUG:boto:Data: 2023-10-12T01:12:08.976 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.977 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.977 DEBUG:boto:Params: {} 2023-10-12T01:12:08.978 DEBUG:boto:Token: None 2023-10-12T01:12:08.978 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57 2023-10-12T01:12:08.979 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/dmw9V25DSXUVK2O/lmrquVL5E= 2023-10-12T01:12:08.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/dmw9V25DSXUVK2O/lmrquVL5E=', 'Content-Length': '0'} 2023-10-12T01:12:08.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caed4a1b9586df16-00652747e8-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.982 DEBUG:boto:path=/wafmoa-59/obj57 2023-10-12T01:12:08.982 DEBUG:boto:auth_path=/wafmoa-59/obj57 2023-10-12T01:12:08.983 DEBUG:boto:Method: GET 2023-10-12T01:12:08.983 DEBUG:boto:Path: /wafmoa-59/obj57 2023-10-12T01:12:08.983 DEBUG:boto:Data: 2023-10-12T01:12:08.984 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:08.985 DEBUG:boto:Port: 8000 2023-10-12T01:12:08.985 DEBUG:boto:Params: {} 2023-10-12T01:12:08.986 DEBUG:boto:Token: None 2023-10-12T01:12:08.986 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/obj57 2023-10-12T01:12:08.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2/dmw9V25DSXUVK2O/lmrquVL5E= 2023-10-12T01:12:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2/dmw9V25DSXUVK2O/lmrquVL5E=', 'Content-Length': '0'} 2023-10-12T01:12:08.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbfa169e901bd0e-00652747e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.990 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:08.990 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:08.991 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.991 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.992 DEBUG:boto:Method: GET 2023-10-12T01:12:08.992 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:12:08.993 DEBUG:boto:Data: 2023-10-12T01:12:08.993 DEBUG:boto:Headers: {} 2023-10-12T01:12:08.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:08.994 DEBUG:boto:Port: 8001 2023-10-12T01:12:08.994 DEBUG:boto:Params: {} 2023-10-12T01:12:08.995 DEBUG:boto:Token: None 2023-10-12T01:12:08.995 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:08 GMT /wafmoa-59/ 2023-10-12T01:12:08.996 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA= 2023-10-12T01:12:08.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:08 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:n9VQ4RkmHVOxYHiEFO2KDny8rfA=', 'Content-Length': '0'} 2023-10-12T01:12:08.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c7af63627a626b-00652747e8-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:08.999 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.000 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.000 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.001 DEBUG:boto:path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:09.001 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj57 2023-10-12T01:12:09.002 DEBUG:boto:Method: GET 2023-10-12T01:12:09.002 DEBUG:boto:Path: /wafmoa-59/?prefix=obj57 2023-10-12T01:12:09.003 DEBUG:boto:Data: 2023-10-12T01:12:09.003 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.004 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.004 DEBUG:boto:Params: {} 2023-10-12T01:12:09.005 DEBUG:boto:Token: None 2023-10-12T01:12:09.005 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030c89fbc43bc5753-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.011 DEBUG:boto:b'wafmoa-59obj571000falseobj572023-10-12T01:01:06.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:12:09.012 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.012 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.013 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.013 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.014 DEBUG:boto:Method: GET 2023-10-12T01:12:09.014 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.015 DEBUG:boto:Data: 2023-10-12T01:12:09.015 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.016 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.017 DEBUG:boto:Params: {} 2023-10-12T01:12:09.017 DEBUG:boto:Token: None 2023-10-12T01:12:09.018 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.018 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4= 2023-10-12T01:12:09.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4=', 'Content-Length': '0'} 2023-10-12T01:12:09.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004228da3e877e5c06-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.062 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.063 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.063 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.064 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.064 DEBUG:boto:Method: GET 2023-10-12T01:12:09.065 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.065 DEBUG:boto:Data: 2023-10-12T01:12:09.066 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.067 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.067 DEBUG:boto:Params: {} 2023-10-12T01:12:09.068 DEBUG:boto:Token: None 2023-10-12T01:12:09.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.069 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4= 2023-10-12T01:12:09.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4=', 'Content-Length': '0'} 2023-10-12T01:12:09.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c76763638c201f-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.072 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.073 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.073 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.073 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.074 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.074 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.075 DEBUG:boto:Data: 2023-10-12T01:12:09.075 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.076 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.076 DEBUG:boto:Params: {} 2023-10-12T01:12:09.077 DEBUG:boto:Token: None 2023-10-12T01:12:09.077 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.078 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TG5dZSiEcfcfK6LxT8RouHvfWs= 2023-10-12T01:12:09.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TG5dZSiEcfcfK6LxT8RouHvfWs=', 'Content-Length': '0'} 2023-10-12T01:12:09.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a69b2dfb920b1c-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.081 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.081 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.082 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.082 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.082 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.083 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.083 DEBUG:boto:Data: 2023-10-12T01:12:09.084 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.084 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.085 DEBUG:boto:Params: {} 2023-10-12T01:12:09.085 DEBUG:boto:Token: None 2023-10-12T01:12:09.086 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.086 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TG5dZSiEcfcfK6LxT8RouHvfWs= 2023-10-12T01:12:09.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TG5dZSiEcfcfK6LxT8RouHvfWs=', 'Content-Length': '0'} 2023-10-12T01:12:09.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f688cca1446c117-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:12:09.090 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.090 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.091 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.091 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.092 DEBUG:boto:Method: GET 2023-10-12T01:12:09.092 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.093 DEBUG:boto:Data: 2023-10-12T01:12:09.093 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.094 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.094 DEBUG:boto:Params: {} 2023-10-12T01:12:09.095 DEBUG:boto:Token: None 2023-10-12T01:12:09.095 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4= 2023-10-12T01:12:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4=', 'Content-Length': '0'} 2023-10-12T01:12:09.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c349b29ea9b8161e-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.099 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.100 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.100 DEBUG:boto:path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.100 DEBUG:boto:auth_path=/wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.101 DEBUG:boto:Method: GET 2023-10-12T01:12:09.101 DEBUG:boto:Path: /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.102 DEBUG:boto:Data: 2023-10-12T01:12:09.102 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.103 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.104 DEBUG:boto:Params: {} 2023-10-12T01:12:09.104 DEBUG:boto:Token: None 2023-10-12T01:12:09.105 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58?versionId=null 2023-10-12T01:12:09.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4= 2023-10-12T01:12:09.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y3hIAxDTLNR8W5tLbMEajI8DRq4=', 'Content-Length': '0'} 2023-10-12T01:12:09.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75bd1b75cdf0358-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.109 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.109 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.109 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.110 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.110 DEBUG:boto:Method: GET 2023-10-12T01:12:09.111 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.111 DEBUG:boto:Data: 2023-10-12T01:12:09.112 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.112 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.113 DEBUG:boto:Params: {} 2023-10-12T01:12:09.113 DEBUG:boto:Token: None 2023-10-12T01:12:09.114 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.114 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c38016ce8eb59c36-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.118 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.119 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.120 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.120 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.120 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.121 DEBUG:boto:Method: GET 2023-10-12T01:12:09.122 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.122 DEBUG:boto:Data: 2023-10-12T01:12:09.122 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.123 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.124 DEBUG:boto:Params: {} 2023-10-12T01:12:09.124 DEBUG:boto:Token: None 2023-10-12T01:12:09.125 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98dfbdecaf1bd93-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.130 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.131 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.132 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.132 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.133 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:12:09.133 DEBUG:boto:Data: 2023-10-12T01:12:09.133 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.134 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.135 DEBUG:boto:Params: {} 2023-10-12T01:12:09.135 DEBUG:boto:Token: None 2023-10-12T01:12:09.136 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58 2023-10-12T01:12:09.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8xjkwNkVH9pE+aqnHFN5z/K5GE= 2023-10-12T01:12:09.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8xjkwNkVH9pE+aqnHFN5z/K5GE=', 'Content-Length': '0'} 2023-10-12T01:12:09.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b54470066ccc05-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.140 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.141 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.141 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.142 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:12:09.142 DEBUG:boto:Data: 2023-10-12T01:12:09.143 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.143 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.144 DEBUG:boto:Params: {} 2023-10-12T01:12:09.144 DEBUG:boto:Token: None 2023-10-12T01:12:09.145 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58 2023-10-12T01:12:09.145 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K8xjkwNkVH9pE+aqnHFN5z/K5GE= 2023-10-12T01:12:09.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K8xjkwNkVH9pE+aqnHFN5z/K5GE=', 'Content-Length': '0'} 2023-10-12T01:12:09.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a214d3c90267c68b-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:12:09.150 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.150 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.151 DEBUG:boto:Method: GET 2023-10-12T01:12:09.151 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:12:09.152 DEBUG:boto:Data: 2023-10-12T01:12:09.152 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.153 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.154 DEBUG:boto:Params: {} 2023-10-12T01:12:09.154 DEBUG:boto:Token: None 2023-10-12T01:12:09.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58 2023-10-12T01:12:09.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rs0MEnu6K+LLQiQxpcie4o1MLKI= 2023-10-12T01:12:09.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rs0MEnu6K+LLQiQxpcie4o1MLKI=', 'Content-Length': '0'} 2023-10-12T01:12:09.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004729460ab0c6ee1f-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.198 DEBUG:boto:path=/wafmoa-59/obj58 2023-10-12T01:12:09.199 DEBUG:boto:auth_path=/wafmoa-59/obj58 2023-10-12T01:12:09.199 DEBUG:boto:Method: GET 2023-10-12T01:12:09.200 DEBUG:boto:Path: /wafmoa-59/obj58 2023-10-12T01:12:09.200 DEBUG:boto:Data: 2023-10-12T01:12:09.201 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.202 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.202 DEBUG:boto:Params: {} 2023-10-12T01:12:09.203 DEBUG:boto:Token: None 2023-10-12T01:12:09.203 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj58 2023-10-12T01:12:09.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rs0MEnu6K+LLQiQxpcie4o1MLKI= 2023-10-12T01:12:09.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rs0MEnu6K+LLQiQxpcie4o1MLKI=', 'Content-Length': '0'} 2023-10-12T01:12:09.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686708dae75253a3-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.207 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.208 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.208 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.209 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.209 DEBUG:boto:Method: GET 2023-10-12T01:12:09.209 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.210 DEBUG:boto:Data: 2023-10-12T01:12:09.210 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.212 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.212 DEBUG:boto:Params: {} 2023-10-12T01:12:09.213 DEBUG:boto:Token: None 2023-10-12T01:12:09.213 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4e9c33bcaa936d-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.218 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.219 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.219 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.220 DEBUG:boto:path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.220 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.220 DEBUG:boto:Method: GET 2023-10-12T01:12:09.221 DEBUG:boto:Path: /wafmoa-59/?prefix=obj58 2023-10-12T01:12:09.221 DEBUG:boto:Data: 2023-10-12T01:12:09.222 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.223 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.223 DEBUG:boto:Params: {} 2023-10-12T01:12:09.224 DEBUG:boto:Token: None 2023-10-12T01:12:09.224 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.225 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069dd98067edafa36-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.270 DEBUG:boto:b'wafmoa-59obj581000falseobj582023-10-12T01:04:26.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:12:09.271 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.272 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.272 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.273 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.273 DEBUG:boto:Method: GET 2023-10-12T01:12:09.274 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.274 DEBUG:boto:Data: 2023-10-12T01:12:09.275 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.276 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.276 DEBUG:boto:Params: {} 2023-10-12T01:12:09.277 DEBUG:boto:Token: None 2023-10-12T01:12:09.277 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE= 2023-10-12T01:12:09.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE=', 'Content-Length': '0'} 2023-10-12T01:12:09.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac372b2e030dc322-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.281 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.282 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.282 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.283 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.283 DEBUG:boto:Method: GET 2023-10-12T01:12:09.284 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.284 DEBUG:boto:Data: 2023-10-12T01:12:09.284 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.285 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.286 DEBUG:boto:Params: {} 2023-10-12T01:12:09.286 DEBUG:boto:Token: None 2023-10-12T01:12:09.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE= 2023-10-12T01:12:09.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE=', 'Content-Length': '0'} 2023-10-12T01:12:09.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266b46297fea2f33-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.291 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.291 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.292 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.292 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.292 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.293 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.293 DEBUG:boto:Data: 2023-10-12T01:12:09.294 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.295 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.295 DEBUG:boto:Params: {} 2023-10-12T01:12:09.295 DEBUG:boto:Token: None 2023-10-12T01:12:09.296 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:atiGDmbEwkRyZCMTADaRPJgbfMk= 2023-10-12T01:12:09.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:atiGDmbEwkRyZCMTADaRPJgbfMk=', 'Content-Length': '0'} 2023-10-12T01:12:09.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7f6f5690d07894-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.299 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.300 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.300 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.301 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.301 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.302 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.302 DEBUG:boto:Data: 2023-10-12T01:12:09.302 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.303 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.304 DEBUG:boto:Params: {} 2023-10-12T01:12:09.304 DEBUG:boto:Token: None 2023-10-12T01:12:09.305 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:atiGDmbEwkRyZCMTADaRPJgbfMk= 2023-10-12T01:12:09.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:atiGDmbEwkRyZCMTADaRPJgbfMk=', 'Content-Length': '0'} 2023-10-12T01:12:09.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a830092fb298afef-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:12:09.309 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.309 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.310 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.310 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.311 DEBUG:boto:Method: GET 2023-10-12T01:12:09.311 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.312 DEBUG:boto:Data: 2023-10-12T01:12:09.312 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.313 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.313 DEBUG:boto:Params: {} 2023-10-12T01:12:09.314 DEBUG:boto:Token: None 2023-10-12T01:12:09.314 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE= 2023-10-12T01:12:09.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE=', 'Content-Length': '0'} 2023-10-12T01:12:09.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056df08bb3df1d867-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.318 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.319 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.319 DEBUG:boto:path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.320 DEBUG:boto:auth_path=/wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.320 DEBUG:boto:Method: GET 2023-10-12T01:12:09.321 DEBUG:boto:Path: /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.321 DEBUG:boto:Data: 2023-10-12T01:12:09.321 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.322 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.323 DEBUG:boto:Params: {} 2023-10-12T01:12:09.323 DEBUG:boto:Token: None 2023-10-12T01:12:09.324 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62?versionId=null 2023-10-12T01:12:09.324 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE= 2023-10-12T01:12:09.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CvMUCQqv4Z65AFHMMTAeKWdoQBE=', 'Content-Length': '0'} 2023-10-12T01:12:09.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002542da407b02c1a9-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.328 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.328 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.328 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.329 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.329 DEBUG:boto:Method: GET 2023-10-12T01:12:09.330 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.330 DEBUG:boto:Data: 2023-10-12T01:12:09.331 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.331 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.332 DEBUG:boto:Params: {} 2023-10-12T01:12:09.332 DEBUG:boto:Token: None 2023-10-12T01:12:09.333 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.333 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceaa8a490212b893-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.337 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.338 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.338 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.339 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.339 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.340 DEBUG:boto:Method: GET 2023-10-12T01:12:09.340 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.340 DEBUG:boto:Data: 2023-10-12T01:12:09.341 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.342 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.342 DEBUG:boto:Params: {} 2023-10-12T01:12:09.343 DEBUG:boto:Token: None 2023-10-12T01:12:09.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068260f6b94e62aa5-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.390 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.391 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.392 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.392 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.393 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:12:09.393 DEBUG:boto:Data: 2023-10-12T01:12:09.393 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.394 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.395 DEBUG:boto:Params: {} 2023-10-12T01:12:09.395 DEBUG:boto:Token: None 2023-10-12T01:12:09.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62 2023-10-12T01:12:09.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XUPA8CB4EBQ7zNRnGVVnRIiAwp8= 2023-10-12T01:12:09.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XUPA8CB4EBQ7zNRnGVVnRIiAwp8=', 'Content-Length': '0'} 2023-10-12T01:12:09.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c689a7ad549b5b24-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.400 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.400 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.401 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.401 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:12:09.401 DEBUG:boto:Data: 2023-10-12T01:12:09.402 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.403 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.403 DEBUG:boto:Params: {} 2023-10-12T01:12:09.404 DEBUG:boto:Token: None 2023-10-12T01:12:09.404 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62 2023-10-12T01:12:09.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XUPA8CB4EBQ7zNRnGVVnRIiAwp8= 2023-10-12T01:12:09.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XUPA8CB4EBQ7zNRnGVVnRIiAwp8=', 'Content-Length': '0'} 2023-10-12T01:12:09.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f1f131965c400a1-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:12:09.408 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.409 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.409 DEBUG:boto:Method: GET 2023-10-12T01:12:09.410 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:12:09.410 DEBUG:boto:Data: 2023-10-12T01:12:09.410 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.411 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.412 DEBUG:boto:Params: {} 2023-10-12T01:12:09.412 DEBUG:boto:Token: None 2023-10-12T01:12:09.413 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62 2023-10-12T01:12:09.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eHwJCTxx5uoIG5CMWZ955ZJCYVI= 2023-10-12T01:12:09.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eHwJCTxx5uoIG5CMWZ955ZJCYVI=', 'Content-Length': '0'} 2023-10-12T01:12:09.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000a3a18b6af7157b-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.458 DEBUG:boto:path=/wafmoa-59/obj62 2023-10-12T01:12:09.459 DEBUG:boto:auth_path=/wafmoa-59/obj62 2023-10-12T01:12:09.459 DEBUG:boto:Method: GET 2023-10-12T01:12:09.460 DEBUG:boto:Path: /wafmoa-59/obj62 2023-10-12T01:12:09.460 DEBUG:boto:Data: 2023-10-12T01:12:09.461 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.462 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.462 DEBUG:boto:Params: {} 2023-10-12T01:12:09.463 DEBUG:boto:Token: None 2023-10-12T01:12:09.463 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj62 2023-10-12T01:12:09.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eHwJCTxx5uoIG5CMWZ955ZJCYVI= 2023-10-12T01:12:09.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eHwJCTxx5uoIG5CMWZ955ZJCYVI=', 'Content-Length': '0'} 2023-10-12T01:12:09.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bdf0733af280f45-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.468 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.469 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.469 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.469 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.470 DEBUG:boto:Method: GET 2023-10-12T01:12:09.470 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.471 DEBUG:boto:Data: 2023-10-12T01:12:09.471 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.472 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.472 DEBUG:boto:Params: {} 2023-10-12T01:12:09.473 DEBUG:boto:Token: None 2023-10-12T01:12:09.473 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.474 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016b1677b573e825-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.478 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.479 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.479 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.480 DEBUG:boto:path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.480 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.480 DEBUG:boto:Method: GET 2023-10-12T01:12:09.481 DEBUG:boto:Path: /wafmoa-59/?prefix=obj62 2023-10-12T01:12:09.481 DEBUG:boto:Data: 2023-10-12T01:12:09.482 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.483 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.483 DEBUG:boto:Params: {} 2023-10-12T01:12:09.484 DEBUG:boto:Token: None 2023-10-12T01:12:09.484 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373120f99813f087-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.490 DEBUG:boto:b'wafmoa-59obj621000falseobj622023-10-12T01:06:27.329Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:12:09.491 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.491 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.492 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.492 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.493 DEBUG:boto:Method: GET 2023-10-12T01:12:09.493 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.494 DEBUG:boto:Data: 2023-10-12T01:12:09.494 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.495 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.495 DEBUG:boto:Params: {} 2023-10-12T01:12:09.496 DEBUG:boto:Token: None 2023-10-12T01:12:09.496 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ= 2023-10-12T01:12:09.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ=', 'Content-Length': '0'} 2023-10-12T01:12:09.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d2de9304dc520e-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.500 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.501 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.501 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.502 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.502 DEBUG:boto:Method: GET 2023-10-12T01:12:09.503 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.503 DEBUG:boto:Data: 2023-10-12T01:12:09.503 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.504 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.505 DEBUG:boto:Params: {} 2023-10-12T01:12:09.505 DEBUG:boto:Token: None 2023-10-12T01:12:09.506 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ= 2023-10-12T01:12:09.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ=', 'Content-Length': '0'} 2023-10-12T01:12:09.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8b94e2978cb6c4-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.554 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.555 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.556 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.556 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.557 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.557 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.557 DEBUG:boto:Data: 2023-10-12T01:12:09.558 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.559 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.559 DEBUG:boto:Params: {} 2023-10-12T01:12:09.560 DEBUG:boto:Token: None 2023-10-12T01:12:09.560 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.561 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2sFdq1rrc7Cmt2HendiwYJPXK0g= 2023-10-12T01:12:09.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2sFdq1rrc7Cmt2HendiwYJPXK0g=', 'Content-Length': '0'} 2023-10-12T01:12:09.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531b64aa8a6fb4a0-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.564 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.564 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.565 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.565 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.566 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.566 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.567 DEBUG:boto:Data: 2023-10-12T01:12:09.567 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.568 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.568 DEBUG:boto:Params: {} 2023-10-12T01:12:09.569 DEBUG:boto:Token: None 2023-10-12T01:12:09.569 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2sFdq1rrc7Cmt2HendiwYJPXK0g= 2023-10-12T01:12:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2sFdq1rrc7Cmt2HendiwYJPXK0g=', 'Content-Length': '0'} 2023-10-12T01:12:09.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877d688856c025d4-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:12:09.573 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.573 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.574 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.574 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.575 DEBUG:boto:Method: GET 2023-10-12T01:12:09.575 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.576 DEBUG:boto:Data: 2023-10-12T01:12:09.576 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.577 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.577 DEBUG:boto:Params: {} 2023-10-12T01:12:09.578 DEBUG:boto:Token: None 2023-10-12T01:12:09.578 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.579 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ= 2023-10-12T01:12:09.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ=', 'Content-Length': '0'} 2023-10-12T01:12:09.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a04cd169a7cb00-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.582 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.583 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.583 DEBUG:boto:path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.584 DEBUG:boto:auth_path=/wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.584 DEBUG:boto:Method: GET 2023-10-12T01:12:09.584 DEBUG:boto:Path: /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.585 DEBUG:boto:Data: 2023-10-12T01:12:09.585 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.586 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.586 DEBUG:boto:Params: {} 2023-10-12T01:12:09.587 DEBUG:boto:Token: None 2023-10-12T01:12:09.589 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68?versionId=null 2023-10-12T01:12:09.589 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ= 2023-10-12T01:12:09.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xpIe4INXgNZxT1Lyh8TEQatWfHQ=', 'Content-Length': '0'} 2023-10-12T01:12:09.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7570a9469aa492-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.593 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.593 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.593 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.594 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.594 DEBUG:boto:Method: GET 2023-10-12T01:12:09.595 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.595 DEBUG:boto:Data: 2023-10-12T01:12:09.596 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.597 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.597 DEBUG:boto:Params: {} 2023-10-12T01:12:09.597 DEBUG:boto:Token: None 2023-10-12T01:12:09.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e0e85781e9acd8-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.602 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.603 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.603 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.604 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.604 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.605 DEBUG:boto:Method: GET 2023-10-12T01:12:09.605 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.606 DEBUG:boto:Data: 2023-10-12T01:12:09.606 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.607 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.607 DEBUG:boto:Params: {} 2023-10-12T01:12:09.608 DEBUG:boto:Token: None 2023-10-12T01:12:09.608 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.609 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d910441c9191b4e0-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.614 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.615 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.615 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.615 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.616 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:12:09.616 DEBUG:boto:Data: 2023-10-12T01:12:09.617 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.618 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.618 DEBUG:boto:Params: {} 2023-10-12T01:12:09.618 DEBUG:boto:Token: None 2023-10-12T01:12:09.619 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68 2023-10-12T01:12:09.619 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gpf6pV9Q4jkTWPDQ1wrJJCdfMA8= 2023-10-12T01:12:09.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gpf6pV9Q4jkTWPDQ1wrJJCdfMA8=', 'Content-Length': '0'} 2023-10-12T01:12:09.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26a2e6cc876ab3d-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.623 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.623 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.624 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.624 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:12:09.624 DEBUG:boto:Data: 2023-10-12T01:12:09.625 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.626 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.626 DEBUG:boto:Params: {} 2023-10-12T01:12:09.627 DEBUG:boto:Token: None 2023-10-12T01:12:09.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68 2023-10-12T01:12:09.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Gpf6pV9Q4jkTWPDQ1wrJJCdfMA8= 2023-10-12T01:12:09.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Gpf6pV9Q4jkTWPDQ1wrJJCdfMA8=', 'Content-Length': '0'} 2023-10-12T01:12:09.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9cb4e69fde8dbd-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:12:09.631 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.632 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.632 DEBUG:boto:Method: GET 2023-10-12T01:12:09.633 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:12:09.633 DEBUG:boto:Data: 2023-10-12T01:12:09.633 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.634 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.635 DEBUG:boto:Params: {} 2023-10-12T01:12:09.635 DEBUG:boto:Token: None 2023-10-12T01:12:09.636 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68 2023-10-12T01:12:09.636 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nHvFKHDle27XVbTe31k8E7GDAAA= 2023-10-12T01:12:09.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nHvFKHDle27XVbTe31k8E7GDAAA=', 'Content-Length': '0'} 2023-10-12T01:12:09.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990e769ba4e285a2-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.640 DEBUG:boto:path=/wafmoa-59/obj68 2023-10-12T01:12:09.640 DEBUG:boto:auth_path=/wafmoa-59/obj68 2023-10-12T01:12:09.641 DEBUG:boto:Method: GET 2023-10-12T01:12:09.641 DEBUG:boto:Path: /wafmoa-59/obj68 2023-10-12T01:12:09.641 DEBUG:boto:Data: 2023-10-12T01:12:09.642 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.643 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.643 DEBUG:boto:Params: {} 2023-10-12T01:12:09.644 DEBUG:boto:Token: None 2023-10-12T01:12:09.644 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj68 2023-10-12T01:12:09.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nHvFKHDle27XVbTe31k8E7GDAAA= 2023-10-12T01:12:09.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nHvFKHDle27XVbTe31k8E7GDAAA=', 'Content-Length': '0'} 2023-10-12T01:12:09.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d551c84e84ab7b-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.648 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.648 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.649 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.649 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.650 DEBUG:boto:Method: GET 2023-10-12T01:12:09.650 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.651 DEBUG:boto:Data: 2023-10-12T01:12:09.651 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.652 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.652 DEBUG:boto:Params: {} 2023-10-12T01:12:09.653 DEBUG:boto:Token: None 2023-10-12T01:12:09.653 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009534c59da2631c9d-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.698 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.699 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.700 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.700 DEBUG:boto:path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.701 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.701 DEBUG:boto:Method: GET 2023-10-12T01:12:09.702 DEBUG:boto:Path: /wafmoa-59/?prefix=obj68 2023-10-12T01:12:09.702 DEBUG:boto:Data: 2023-10-12T01:12:09.702 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.703 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.704 DEBUG:boto:Params: {} 2023-10-12T01:12:09.704 DEBUG:boto:Token: None 2023-10-12T01:12:09.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.705 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25a221b6d8556b4-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.711 DEBUG:boto:b'wafmoa-59obj681000falseobj682023-10-12T00:59:24.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:12:09.712 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.712 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.713 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.713 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.714 DEBUG:boto:Method: GET 2023-10-12T01:12:09.714 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.715 DEBUG:boto:Data: 2023-10-12T01:12:09.715 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.716 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.716 DEBUG:boto:Params: {} 2023-10-12T01:12:09.717 DEBUG:boto:Token: None 2023-10-12T01:12:09.717 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk= 2023-10-12T01:12:09.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk=', 'Content-Length': '0'} 2023-10-12T01:12:09.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043fee49c16ab356f-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.722 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.722 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.723 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.723 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.724 DEBUG:boto:Method: GET 2023-10-12T01:12:09.724 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.724 DEBUG:boto:Data: 2023-10-12T01:12:09.725 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.726 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.726 DEBUG:boto:Params: {} 2023-10-12T01:12:09.727 DEBUG:boto:Token: None 2023-10-12T01:12:09.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk= 2023-10-12T01:12:09.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk=', 'Content-Length': '0'} 2023-10-12T01:12:09.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c939aa9aae9a68-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.778 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.779 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.779 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.780 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.780 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.781 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.781 DEBUG:boto:Data: 2023-10-12T01:12:09.781 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.782 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.783 DEBUG:boto:Params: {} 2023-10-12T01:12:09.783 DEBUG:boto:Token: None 2023-10-12T01:12:09.784 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3UxKP8Ri3rdCaBn8r/saYBcbFNw= 2023-10-12T01:12:09.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3UxKP8Ri3rdCaBn8r/saYBcbFNw=', 'Content-Length': '0'} 2023-10-12T01:12:09.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892c89b1f1583bb3-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.787 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.788 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.788 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.789 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.789 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.790 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.790 DEBUG:boto:Data: 2023-10-12T01:12:09.790 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.791 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.793 DEBUG:boto:Params: {} 2023-10-12T01:12:09.793 DEBUG:boto:Token: None 2023-10-12T01:12:09.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3UxKP8Ri3rdCaBn8r/saYBcbFNw= 2023-10-12T01:12:09.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3UxKP8Ri3rdCaBn8r/saYBcbFNw=', 'Content-Length': '0'} 2023-10-12T01:12:09.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061af68937f396194-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:12:09.798 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.798 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.799 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.799 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.800 DEBUG:boto:Method: GET 2023-10-12T01:12:09.800 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.801 DEBUG:boto:Data: 2023-10-12T01:12:09.801 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.802 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.802 DEBUG:boto:Params: {} 2023-10-12T01:12:09.803 DEBUG:boto:Token: None 2023-10-12T01:12:09.803 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk= 2023-10-12T01:12:09.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk=', 'Content-Length': '0'} 2023-10-12T01:12:09.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041179c2ae5be2d25-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.807 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.808 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.808 DEBUG:boto:path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.808 DEBUG:boto:auth_path=/wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.809 DEBUG:boto:Method: GET 2023-10-12T01:12:09.809 DEBUG:boto:Path: /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.810 DEBUG:boto:Data: 2023-10-12T01:12:09.810 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.811 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.812 DEBUG:boto:Params: {} 2023-10-12T01:12:09.812 DEBUG:boto:Token: None 2023-10-12T01:12:09.813 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69?versionId=null 2023-10-12T01:12:09.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk= 2023-10-12T01:12:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eeKwLe5KTjQiXAln9/A/i9exXyk=', 'Content-Length': '0'} 2023-10-12T01:12:09.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006704fd68d51f6b9e-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.816 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.817 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.817 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.818 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.818 DEBUG:boto:Method: GET 2023-10-12T01:12:09.819 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.819 DEBUG:boto:Data: 2023-10-12T01:12:09.820 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.820 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.821 DEBUG:boto:Params: {} 2023-10-12T01:12:09.821 DEBUG:boto:Token: None 2023-10-12T01:12:09.822 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.822 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000721f5d9709f70ff3-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.870 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.871 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.872 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.872 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.873 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.873 DEBUG:boto:Method: GET 2023-10-12T01:12:09.873 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.874 DEBUG:boto:Data: 2023-10-12T01:12:09.874 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.875 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.876 DEBUG:boto:Params: {} 2023-10-12T01:12:09.876 DEBUG:boto:Token: None 2023-10-12T01:12:09.877 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f56a231640ca5ca-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.882 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.883 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.884 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.884 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.884 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:12:09.885 DEBUG:boto:Data: 2023-10-12T01:12:09.885 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.886 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.887 DEBUG:boto:Params: {} 2023-10-12T01:12:09.887 DEBUG:boto:Token: None 2023-10-12T01:12:09.888 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69 2023-10-12T01:12:09.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CcOk7hX9uDEaNgddRxttX+rY1bY= 2023-10-12T01:12:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CcOk7hX9uDEaNgddRxttX+rY1bY=', 'Content-Length': '0'} 2023-10-12T01:12:09.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a18d764a17e943-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.892 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.892 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.892 DEBUG:boto:Method: HEAD 2023-10-12T01:12:09.893 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:12:09.893 DEBUG:boto:Data: 2023-10-12T01:12:09.894 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.895 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.895 DEBUG:boto:Params: {} 2023-10-12T01:12:09.896 DEBUG:boto:Token: None 2023-10-12T01:12:09.896 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69 2023-10-12T01:12:09.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CcOk7hX9uDEaNgddRxttX+rY1bY= 2023-10-12T01:12:09.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CcOk7hX9uDEaNgddRxttX+rY1bY=', 'Content-Length': '0'} 2023-10-12T01:12:09.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aa23d62984f7287-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:12:09.900 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.900 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.901 DEBUG:boto:Method: GET 2023-10-12T01:12:09.901 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:12:09.902 DEBUG:boto:Data: 2023-10-12T01:12:09.902 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.903 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.904 DEBUG:boto:Params: {} 2023-10-12T01:12:09.904 DEBUG:boto:Token: None 2023-10-12T01:12:09.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69 2023-10-12T01:12:09.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LaWeh4n++RgTmqbj3IQDyZ1JpFM= 2023-10-12T01:12:09.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LaWeh4n++RgTmqbj3IQDyZ1JpFM=', 'Content-Length': '0'} 2023-10-12T01:12:09.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d281032ede71a6-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.908 DEBUG:boto:path=/wafmoa-59/obj69 2023-10-12T01:12:09.909 DEBUG:boto:auth_path=/wafmoa-59/obj69 2023-10-12T01:12:09.909 DEBUG:boto:Method: GET 2023-10-12T01:12:09.910 DEBUG:boto:Path: /wafmoa-59/obj69 2023-10-12T01:12:09.910 DEBUG:boto:Data: 2023-10-12T01:12:09.910 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.911 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.912 DEBUG:boto:Params: {} 2023-10-12T01:12:09.912 DEBUG:boto:Token: None 2023-10-12T01:12:09.913 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj69 2023-10-12T01:12:09.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LaWeh4n++RgTmqbj3IQDyZ1JpFM= 2023-10-12T01:12:09.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LaWeh4n++RgTmqbj3IQDyZ1JpFM=', 'Content-Length': '0'} 2023-10-12T01:12:09.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005830e5a703a83854-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.916 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.917 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.917 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.918 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.918 DEBUG:boto:Method: GET 2023-10-12T01:12:09.919 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.919 DEBUG:boto:Data: 2023-10-12T01:12:09.920 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.920 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.921 DEBUG:boto:Params: {} 2023-10-12T01:12:09.921 DEBUG:boto:Token: None 2023-10-12T01:12:09.922 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.922 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b6500531a65a6c-00652747e9-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.926 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.927 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:09.927 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:09.927 DEBUG:boto:path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.928 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.928 DEBUG:boto:Method: GET 2023-10-12T01:12:09.929 DEBUG:boto:Path: /wafmoa-59/?prefix=obj69 2023-10-12T01:12:09.929 DEBUG:boto:Data: 2023-10-12T01:12:09.930 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.931 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.931 DEBUG:boto:Params: {} 2023-10-12T01:12:09.931 DEBUG:boto:Token: None 2023-10-12T01:12:09.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/ 2023-10-12T01:12:09.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA= 2023-10-12T01:12:09.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FX+D0fi57MSOgxKknvan0DIZNvA=', 'Content-Length': '0'} 2023-10-12T01:12:09.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026eda3205b2b3279-00652747e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.937 DEBUG:boto:b'wafmoa-59obj691000falseobj692023-10-12T01:03:05.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:09.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:12:09.939 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:09.939 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:09.939 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.940 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.940 DEBUG:boto:Method: GET 2023-10-12T01:12:09.941 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.941 DEBUG:boto:Data: 2023-10-12T01:12:09.942 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:09.943 DEBUG:boto:Port: 8001 2023-10-12T01:12:09.943 DEBUG:boto:Params: {} 2023-10-12T01:12:09.943 DEBUG:boto:Token: None 2023-10-12T01:12:09.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qqcssTYisyYR2j7DlHBmNXsTCgA= 2023-10-12T01:12:09.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qqcssTYisyYR2j7DlHBmNXsTCgA=', 'Content-Length': '0'} 2023-10-12T01:12:09.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea686ea832d3b723-00652747e9-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.948 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:09.949 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:09.949 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.949 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.950 DEBUG:boto:Method: GET 2023-10-12T01:12:09.950 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.951 DEBUG:boto:Data: 2023-10-12T01:12:09.951 DEBUG:boto:Headers: {} 2023-10-12T01:12:09.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:09.952 DEBUG:boto:Port: 8000 2023-10-12T01:12:09.952 DEBUG:boto:Params: {} 2023-10-12T01:12:09.953 DEBUG:boto:Token: None 2023-10-12T01:12:09.953 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:09 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:09.954 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qqcssTYisyYR2j7DlHBmNXsTCgA= 2023-10-12T01:12:09.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:09 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qqcssTYisyYR2j7DlHBmNXsTCgA=', 'Content-Length': '0'} 2023-10-12T01:12:09.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed896b9b7bcd733c-00652747e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:09.998 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:09.999 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:09.999 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.000 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.000 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.001 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.001 DEBUG:boto:Data: 2023-10-12T01:12:10.001 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.002 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.003 DEBUG:boto:Params: {} 2023-10-12T01:12:10.003 DEBUG:boto:Token: None 2023-10-12T01:12:10.010 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35d/PFlnyQMDP7cVDpncq4gaJ/M= 2023-10-12T01:12:10.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35d/PFlnyQMDP7cVDpncq4gaJ/M=', 'Content-Length': '0'} 2023-10-12T01:12:10.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046954f7dced781c5-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.015 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.015 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.016 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.016 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.017 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.017 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.017 DEBUG:boto:Data: 2023-10-12T01:12:10.018 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.019 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.019 DEBUG:boto:Params: {} 2023-10-12T01:12:10.020 DEBUG:boto:Token: None 2023-10-12T01:12:10.020 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:35d/PFlnyQMDP7cVDpncq4gaJ/M= 2023-10-12T01:12:10.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:35d/PFlnyQMDP7cVDpncq4gaJ/M=', 'Content-Length': '0'} 2023-10-12T01:12:10.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000824aab09df1f285-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:12:10.024 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.025 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.025 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.025 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.026 DEBUG:boto:Method: GET 2023-10-12T01:12:10.026 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.027 DEBUG:boto:Data: 2023-10-12T01:12:10.027 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.028 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.029 DEBUG:boto:Params: {} 2023-10-12T01:12:10.029 DEBUG:boto:Token: None 2023-10-12T01:12:10.030 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sKWWzNPgLayHWzC3MzwNkxWgmqQ= 2023-10-12T01:12:10.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sKWWzNPgLayHWzC3MzwNkxWgmqQ=', 'Content-Length': '0'} 2023-10-12T01:12:10.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb121c3557af5ba2-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.034 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.034 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.035 DEBUG:boto:path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.035 DEBUG:boto:auth_path=/wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.036 DEBUG:boto:Method: GET 2023-10-12T01:12:10.036 DEBUG:boto:Path: /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.037 DEBUG:boto:Data: 2023-10-12T01:12:10.037 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.038 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.038 DEBUG:boto:Params: {} 2023-10-12T01:12:10.039 DEBUG:boto:Token: None 2023-10-12T01:12:10.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74?versionId=null 2023-10-12T01:12:10.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sKWWzNPgLayHWzC3MzwNkxWgmqQ= 2023-10-12T01:12:10.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sKWWzNPgLayHWzC3MzwNkxWgmqQ=', 'Content-Length': '0'} 2023-10-12T01:12:10.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee38d089ef63730-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.043 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.044 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.044 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.045 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.045 DEBUG:boto:Method: GET 2023-10-12T01:12:10.046 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.046 DEBUG:boto:Data: 2023-10-12T01:12:10.047 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.048 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.048 DEBUG:boto:Params: {} 2023-10-12T01:12:10.049 DEBUG:boto:Token: None 2023-10-12T01:12:10.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a07432f5d7b68a3-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.053 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.054 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.055 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.055 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.055 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.056 DEBUG:boto:Method: GET 2023-10-12T01:12:10.056 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.057 DEBUG:boto:Data: 2023-10-12T01:12:10.057 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.058 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.058 DEBUG:boto:Params: {} 2023-10-12T01:12:10.059 DEBUG:boto:Token: None 2023-10-12T01:12:10.059 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aaa36610f081a82-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.106 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.107 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.108 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.109 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.109 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:12:10.110 DEBUG:boto:Data: 2023-10-12T01:12:10.110 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.111 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.112 DEBUG:boto:Params: {} 2023-10-12T01:12:10.112 DEBUG:boto:Token: None 2023-10-12T01:12:10.113 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74 2023-10-12T01:12:10.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pVkJKiCBu3ChNfdevGzxnZ6/x+s= 2023-10-12T01:12:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pVkJKiCBu3ChNfdevGzxnZ6/x+s=', 'Content-Length': '0'} 2023-10-12T01:12:10.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367800b9270fdd4a-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.117 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.117 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.118 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.118 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:12:10.119 DEBUG:boto:Data: 2023-10-12T01:12:10.120 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.121 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.121 DEBUG:boto:Params: {} 2023-10-12T01:12:10.122 DEBUG:boto:Token: None 2023-10-12T01:12:10.122 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74 2023-10-12T01:12:10.123 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pVkJKiCBu3ChNfdevGzxnZ6/x+s= 2023-10-12T01:12:10.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pVkJKiCBu3ChNfdevGzxnZ6/x+s=', 'Content-Length': '0'} 2023-10-12T01:12:10.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2d5f713cf6c4fd-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:12:10.126 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.127 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.127 DEBUG:boto:Method: GET 2023-10-12T01:12:10.127 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:12:10.128 DEBUG:boto:Data: 2023-10-12T01:12:10.128 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.129 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.129 DEBUG:boto:Params: {} 2023-10-12T01:12:10.130 DEBUG:boto:Token: None 2023-10-12T01:12:10.130 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74 2023-10-12T01:12:10.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ozxl+vQrocWoox0rlaqiFWN3MlU= 2023-10-12T01:12:10.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ozxl+vQrocWoox0rlaqiFWN3MlU=', 'Content-Length': '0'} 2023-10-12T01:12:10.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93468445a85030e-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.136 DEBUG:boto:path=/wafmoa-59/obj74 2023-10-12T01:12:10.137 DEBUG:boto:auth_path=/wafmoa-59/obj74 2023-10-12T01:12:10.138 DEBUG:boto:Method: GET 2023-10-12T01:12:10.139 DEBUG:boto:Path: /wafmoa-59/obj74 2023-10-12T01:12:10.140 DEBUG:boto:Data: 2023-10-12T01:12:10.140 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.142 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.143 DEBUG:boto:Params: {} 2023-10-12T01:12:10.144 DEBUG:boto:Token: None 2023-10-12T01:12:10.144 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj74 2023-10-12T01:12:10.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ozxl+vQrocWoox0rlaqiFWN3MlU= 2023-10-12T01:12:10.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ozxl+vQrocWoox0rlaqiFWN3MlU=', 'Content-Length': '0'} 2023-10-12T01:12:10.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:09:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ac29fa6b40ab41-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.150 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.151 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.152 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.153 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.154 DEBUG:boto:Method: GET 2023-10-12T01:12:10.154 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.155 DEBUG:boto:Data: 2023-10-12T01:12:10.156 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.157 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.158 DEBUG:boto:Params: {} 2023-10-12T01:12:10.158 DEBUG:boto:Token: None 2023-10-12T01:12:10.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.160 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce611211545191bd-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.210 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.212 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.212 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.213 DEBUG:boto:path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.214 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.214 DEBUG:boto:Method: GET 2023-10-12T01:12:10.215 DEBUG:boto:Path: /wafmoa-59/?prefix=obj74 2023-10-12T01:12:10.216 DEBUG:boto:Data: 2023-10-12T01:12:10.216 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.218 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.218 DEBUG:boto:Params: {} 2023-10-12T01:12:10.219 DEBUG:boto:Token: None 2023-10-12T01:12:10.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.220 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc8d44c94b44bb4-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.226 DEBUG:boto:b'wafmoa-59obj741000falseobj742023-10-12T01:09:29.962Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:12:10.227 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.228 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.228 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.229 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.229 DEBUG:boto:Method: GET 2023-10-12T01:12:10.230 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.230 DEBUG:boto:Data: 2023-10-12T01:12:10.231 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.231 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.232 DEBUG:boto:Params: {} 2023-10-12T01:12:10.232 DEBUG:boto:Token: None 2023-10-12T01:12:10.233 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik= 2023-10-12T01:12:10.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik=', 'Content-Length': '0'} 2023-10-12T01:12:10.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df9f15b10c2fdf1-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.237 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.237 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.238 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.238 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.239 DEBUG:boto:Method: GET 2023-10-12T01:12:10.239 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.240 DEBUG:boto:Data: 2023-10-12T01:12:10.240 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.241 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.241 DEBUG:boto:Params: {} 2023-10-12T01:12:10.242 DEBUG:boto:Token: None 2023-10-12T01:12:10.242 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.243 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik= 2023-10-12T01:12:10.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik=', 'Content-Length': '0'} 2023-10-12T01:12:10.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e8c67d5cdd08ee-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.246 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.247 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.247 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.248 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.248 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.248 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.249 DEBUG:boto:Data: 2023-10-12T01:12:10.249 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.250 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.250 DEBUG:boto:Params: {} 2023-10-12T01:12:10.251 DEBUG:boto:Token: None 2023-10-12T01:12:10.251 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.252 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wd16uBq//c/QFPyG4kYch8usXE0= 2023-10-12T01:12:10.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wd16uBq//c/QFPyG4kYch8usXE0=', 'Content-Length': '0'} 2023-10-12T01:12:10.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a6d78c5a8942ff-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.255 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.255 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.256 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.256 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.257 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.257 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.257 DEBUG:boto:Data: 2023-10-12T01:12:10.258 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.259 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.259 DEBUG:boto:Params: {} 2023-10-12T01:12:10.259 DEBUG:boto:Token: None 2023-10-12T01:12:10.260 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.260 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wd16uBq//c/QFPyG4kYch8usXE0= 2023-10-12T01:12:10.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wd16uBq//c/QFPyG4kYch8usXE0=', 'Content-Length': '0'} 2023-10-12T01:12:10.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e7844cc5a31d77-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:12:10.264 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.264 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.265 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.265 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.266 DEBUG:boto:Method: GET 2023-10-12T01:12:10.266 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.267 DEBUG:boto:Data: 2023-10-12T01:12:10.267 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.268 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.268 DEBUG:boto:Params: {} 2023-10-12T01:12:10.269 DEBUG:boto:Token: None 2023-10-12T01:12:10.269 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik= 2023-10-12T01:12:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik=', 'Content-Length': '0'} 2023-10-12T01:12:10.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9d78a9ba46ffdd-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.273 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.274 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.274 DEBUG:boto:path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.275 DEBUG:boto:auth_path=/wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.275 DEBUG:boto:Method: GET 2023-10-12T01:12:10.275 DEBUG:boto:Path: /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.276 DEBUG:boto:Data: 2023-10-12T01:12:10.276 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.277 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.277 DEBUG:boto:Params: {} 2023-10-12T01:12:10.278 DEBUG:boto:Token: None 2023-10-12T01:12:10.279 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76?versionId=null 2023-10-12T01:12:10.279 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik= 2023-10-12T01:12:10.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xNVNgtIWV5qzAOHGT15psI2hhik=', 'Content-Length': '0'} 2023-10-12T01:12:10.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005163291d44b596c-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.322 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.323 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.324 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.324 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.324 DEBUG:boto:Method: GET 2023-10-12T01:12:10.325 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.325 DEBUG:boto:Data: 2023-10-12T01:12:10.326 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.327 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.327 DEBUG:boto:Params: {} 2023-10-12T01:12:10.327 DEBUG:boto:Token: None 2023-10-12T01:12:10.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041564e8b0d251ecb-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.333 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.333 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.334 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.334 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.335 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.335 DEBUG:boto:Method: GET 2023-10-12T01:12:10.335 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.336 DEBUG:boto:Data: 2023-10-12T01:12:10.336 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.337 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.338 DEBUG:boto:Params: {} 2023-10-12T01:12:10.338 DEBUG:boto:Token: None 2023-10-12T01:12:10.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.339 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77022e2e84707f7-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.344 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.345 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.345 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.346 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.346 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:12:10.346 DEBUG:boto:Data: 2023-10-12T01:12:10.347 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.348 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.348 DEBUG:boto:Params: {} 2023-10-12T01:12:10.349 DEBUG:boto:Token: None 2023-10-12T01:12:10.349 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76 2023-10-12T01:12:10.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:22kKYgK1lPXG3jhcU8OCzyF78dw= 2023-10-12T01:12:10.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:22kKYgK1lPXG3jhcU8OCzyF78dw=', 'Content-Length': '0'} 2023-10-12T01:12:10.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d093d9fbf352018-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.353 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.353 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.354 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.354 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:12:10.355 DEBUG:boto:Data: 2023-10-12T01:12:10.355 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.356 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.356 DEBUG:boto:Params: {} 2023-10-12T01:12:10.357 DEBUG:boto:Token: None 2023-10-12T01:12:10.357 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76 2023-10-12T01:12:10.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:22kKYgK1lPXG3jhcU8OCzyF78dw= 2023-10-12T01:12:10.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:22kKYgK1lPXG3jhcU8OCzyF78dw=', 'Content-Length': '0'} 2023-10-12T01:12:10.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571f28132084ae73-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:12:10.361 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.362 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.362 DEBUG:boto:Method: GET 2023-10-12T01:12:10.363 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:12:10.363 DEBUG:boto:Data: 2023-10-12T01:12:10.363 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.364 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.365 DEBUG:boto:Params: {} 2023-10-12T01:12:10.365 DEBUG:boto:Token: None 2023-10-12T01:12:10.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76 2023-10-12T01:12:10.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:shlc9a46y4RTQBk8eWGmew2dNkA= 2023-10-12T01:12:10.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:shlc9a46y4RTQBk8eWGmew2dNkA=', 'Content-Length': '0'} 2023-10-12T01:12:10.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9bb2454d3b37fce-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.410 DEBUG:boto:path=/wafmoa-59/obj76 2023-10-12T01:12:10.411 DEBUG:boto:auth_path=/wafmoa-59/obj76 2023-10-12T01:12:10.412 DEBUG:boto:Method: GET 2023-10-12T01:12:10.412 DEBUG:boto:Path: /wafmoa-59/obj76 2023-10-12T01:12:10.413 DEBUG:boto:Data: 2023-10-12T01:12:10.413 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.414 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.414 DEBUG:boto:Params: {} 2023-10-12T01:12:10.415 DEBUG:boto:Token: None 2023-10-12T01:12:10.415 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj76 2023-10-12T01:12:10.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:shlc9a46y4RTQBk8eWGmew2dNkA= 2023-10-12T01:12:10.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:shlc9a46y4RTQBk8eWGmew2dNkA=', 'Content-Length': '0'} 2023-10-12T01:12:10.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e6b6cdb43c22acd-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.419 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.419 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.420 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.420 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.421 DEBUG:boto:Method: GET 2023-10-12T01:12:10.421 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.422 DEBUG:boto:Data: 2023-10-12T01:12:10.422 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.423 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.423 DEBUG:boto:Params: {} 2023-10-12T01:12:10.424 DEBUG:boto:Token: None 2023-10-12T01:12:10.424 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df7ac1156a4d466f-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.429 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.429 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.430 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.430 DEBUG:boto:path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.431 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.431 DEBUG:boto:Method: GET 2023-10-12T01:12:10.432 DEBUG:boto:Path: /wafmoa-59/?prefix=obj76 2023-10-12T01:12:10.432 DEBUG:boto:Data: 2023-10-12T01:12:10.433 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.434 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.434 DEBUG:boto:Params: {} 2023-10-12T01:12:10.435 DEBUG:boto:Token: None 2023-10-12T01:12:10.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1c0ddde73705ca-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.482 DEBUG:boto:b'wafmoa-59obj761000falseobj762023-10-12T01:01:06.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:12:10.484 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.484 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.484 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.485 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.485 DEBUG:boto:Method: GET 2023-10-12T01:12:10.486 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.486 DEBUG:boto:Data: 2023-10-12T01:12:10.487 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.487 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.488 DEBUG:boto:Params: {} 2023-10-12T01:12:10.488 DEBUG:boto:Token: None 2023-10-12T01:12:10.489 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8= 2023-10-12T01:12:10.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8=', 'Content-Length': '0'} 2023-10-12T01:12:10.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ba02060ece5e09-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.493 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.493 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.494 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.494 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.495 DEBUG:boto:Method: GET 2023-10-12T01:12:10.495 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.496 DEBUG:boto:Data: 2023-10-12T01:12:10.496 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.497 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.497 DEBUG:boto:Params: {} 2023-10-12T01:12:10.498 DEBUG:boto:Token: None 2023-10-12T01:12:10.498 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8= 2023-10-12T01:12:10.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8=', 'Content-Length': '0'} 2023-10-12T01:12:10.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476795b6a58efa17-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.502 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.503 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.503 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.504 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.504 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.504 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.505 DEBUG:boto:Data: 2023-10-12T01:12:10.505 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.506 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.506 DEBUG:boto:Params: {} 2023-10-12T01:12:10.507 DEBUG:boto:Token: None 2023-10-12T01:12:10.507 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jm87dTC4qsZweWk4To1IEaBTLaA= 2023-10-12T01:12:10.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jm87dTC4qsZweWk4To1IEaBTLaA=', 'Content-Length': '0'} 2023-10-12T01:12:10.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97c38374eafc40e-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.511 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.511 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.512 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.512 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.512 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.513 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.513 DEBUG:boto:Data: 2023-10-12T01:12:10.514 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.514 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.515 DEBUG:boto:Params: {} 2023-10-12T01:12:10.515 DEBUG:boto:Token: None 2023-10-12T01:12:10.516 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jm87dTC4qsZweWk4To1IEaBTLaA= 2023-10-12T01:12:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jm87dTC4qsZweWk4To1IEaBTLaA=', 'Content-Length': '0'} 2023-10-12T01:12:10.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f877fa0337bd0f3-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:12:10.520 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.520 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.521 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.521 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.522 DEBUG:boto:Method: GET 2023-10-12T01:12:10.522 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.522 DEBUG:boto:Data: 2023-10-12T01:12:10.523 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.524 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.524 DEBUG:boto:Params: {} 2023-10-12T01:12:10.525 DEBUG:boto:Token: None 2023-10-12T01:12:10.525 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.526 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8= 2023-10-12T01:12:10.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8=', 'Content-Length': '0'} 2023-10-12T01:12:10.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba8b79854b8799a-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.529 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.530 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.530 DEBUG:boto:path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.531 DEBUG:boto:auth_path=/wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.531 DEBUG:boto:Method: GET 2023-10-12T01:12:10.532 DEBUG:boto:Path: /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.532 DEBUG:boto:Data: 2023-10-12T01:12:10.533 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.534 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.534 DEBUG:boto:Params: {} 2023-10-12T01:12:10.535 DEBUG:boto:Token: None 2023-10-12T01:12:10.535 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85?versionId=null 2023-10-12T01:12:10.536 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8= 2023-10-12T01:12:10.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fDrLhR/XvyF3sKGjT6lEHSdmbA8=', 'Content-Length': '0'} 2023-10-12T01:12:10.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc7b4efb95388b5-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.539 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.539 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.540 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.540 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.541 DEBUG:boto:Method: GET 2023-10-12T01:12:10.541 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.542 DEBUG:boto:Data: 2023-10-12T01:12:10.542 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.543 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.544 DEBUG:boto:Params: {} 2023-10-12T01:12:10.544 DEBUG:boto:Token: None 2023-10-12T01:12:10.545 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.545 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c979adad84b697-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.549 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.550 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.550 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.551 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.551 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.552 DEBUG:boto:Method: GET 2023-10-12T01:12:10.552 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.553 DEBUG:boto:Data: 2023-10-12T01:12:10.553 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.554 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.555 DEBUG:boto:Params: {} 2023-10-12T01:12:10.555 DEBUG:boto:Token: None 2023-10-12T01:12:10.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5f6d02f25ebdb4-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.606 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.607 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.609 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.609 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.610 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:12:10.610 DEBUG:boto:Data: 2023-10-12T01:12:10.611 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.612 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.612 DEBUG:boto:Params: {} 2023-10-12T01:12:10.614 DEBUG:boto:Token: None 2023-10-12T01:12:10.615 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85 2023-10-12T01:12:10.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aZa3LMuh3V77LU6o9wNyxfPCVr8= 2023-10-12T01:12:10.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aZa3LMuh3V77LU6o9wNyxfPCVr8=', 'Content-Length': '0'} 2023-10-12T01:12:10.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65394de595a608f-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.619 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.619 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.619 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.620 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:12:10.620 DEBUG:boto:Data: 2023-10-12T01:12:10.621 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.622 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.622 DEBUG:boto:Params: {} 2023-10-12T01:12:10.623 DEBUG:boto:Token: None 2023-10-12T01:12:10.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85 2023-10-12T01:12:10.624 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aZa3LMuh3V77LU6o9wNyxfPCVr8= 2023-10-12T01:12:10.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aZa3LMuh3V77LU6o9wNyxfPCVr8=', 'Content-Length': '0'} 2023-10-12T01:12:10.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46411e55a186310-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:12:10.628 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.628 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.629 DEBUG:boto:Method: GET 2023-10-12T01:12:10.629 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:12:10.630 DEBUG:boto:Data: 2023-10-12T01:12:10.630 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.631 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.632 DEBUG:boto:Params: {} 2023-10-12T01:12:10.632 DEBUG:boto:Token: None 2023-10-12T01:12:10.633 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85 2023-10-12T01:12:10.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f9Yc4DLY3umBAwAaZL6ZH2uwAag= 2023-10-12T01:12:10.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f9Yc4DLY3umBAwAaZL6ZH2uwAag=', 'Content-Length': '0'} 2023-10-12T01:12:10.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d966003409a4dfb-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.637 DEBUG:boto:path=/wafmoa-59/obj85 2023-10-12T01:12:10.637 DEBUG:boto:auth_path=/wafmoa-59/obj85 2023-10-12T01:12:10.638 DEBUG:boto:Method: GET 2023-10-12T01:12:10.638 DEBUG:boto:Path: /wafmoa-59/obj85 2023-10-12T01:12:10.638 DEBUG:boto:Data: 2023-10-12T01:12:10.639 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.640 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.640 DEBUG:boto:Params: {} 2023-10-12T01:12:10.641 DEBUG:boto:Token: None 2023-10-12T01:12:10.641 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj85 2023-10-12T01:12:10.642 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f9Yc4DLY3umBAwAaZL6ZH2uwAag= 2023-10-12T01:12:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f9Yc4DLY3umBAwAaZL6ZH2uwAag=', 'Content-Length': '0'} 2023-10-12T01:12:10.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 00:59:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5ae5411598e34c-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.645 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.646 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.646 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.647 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.647 DEBUG:boto:Method: GET 2023-10-12T01:12:10.648 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.648 DEBUG:boto:Data: 2023-10-12T01:12:10.649 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.650 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.650 DEBUG:boto:Params: {} 2023-10-12T01:12:10.651 DEBUG:boto:Token: None 2023-10-12T01:12:10.651 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d97fe0684ecfc74b-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.656 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.656 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.657 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.657 DEBUG:boto:path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.658 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.658 DEBUG:boto:Method: GET 2023-10-12T01:12:10.659 DEBUG:boto:Path: /wafmoa-59/?prefix=obj85 2023-10-12T01:12:10.659 DEBUG:boto:Data: 2023-10-12T01:12:10.659 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.660 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.661 DEBUG:boto:Params: {} 2023-10-12T01:12:10.661 DEBUG:boto:Token: None 2023-10-12T01:12:10.662 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.662 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1d64ab37044f84-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.668 DEBUG:boto:b'wafmoa-59obj851000falseobj852023-10-12T00:59:24.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:12:10.669 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.669 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.670 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.670 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.670 DEBUG:boto:Method: GET 2023-10-12T01:12:10.671 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.671 DEBUG:boto:Data: 2023-10-12T01:12:10.672 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.673 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.673 DEBUG:boto:Params: {} 2023-10-12T01:12:10.674 DEBUG:boto:Token: None 2023-10-12T01:12:10.674 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.675 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0= 2023-10-12T01:12:10.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0=', 'Content-Length': '0'} 2023-10-12T01:12:10.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0486a7ce37035ad-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.678 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.679 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.679 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.679 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.680 DEBUG:boto:Method: GET 2023-10-12T01:12:10.680 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.681 DEBUG:boto:Data: 2023-10-12T01:12:10.681 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.682 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.682 DEBUG:boto:Params: {} 2023-10-12T01:12:10.683 DEBUG:boto:Token: None 2023-10-12T01:12:10.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.684 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0= 2023-10-12T01:12:10.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0=', 'Content-Length': '0'} 2023-10-12T01:12:10.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005448fdf6b7102b73-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.730 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.731 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.731 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.732 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.732 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.733 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.733 DEBUG:boto:Data: 2023-10-12T01:12:10.733 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.734 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.735 DEBUG:boto:Params: {} 2023-10-12T01:12:10.735 DEBUG:boto:Token: None 2023-10-12T01:12:10.736 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjUJbOoT+q4zPLp6DROkmNfw+Ac= 2023-10-12T01:12:10.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjUJbOoT+q4zPLp6DROkmNfw+Ac=', 'Content-Length': '0'} 2023-10-12T01:12:10.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620991ddbc6a2f54-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.739 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.740 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.740 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.741 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.741 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.741 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.742 DEBUG:boto:Data: 2023-10-12T01:12:10.742 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.743 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.743 DEBUG:boto:Params: {} 2023-10-12T01:12:10.744 DEBUG:boto:Token: None 2023-10-12T01:12:10.744 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjUJbOoT+q4zPLp6DROkmNfw+Ac= 2023-10-12T01:12:10.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjUJbOoT+q4zPLp6DROkmNfw+Ac=', 'Content-Length': '0'} 2023-10-12T01:12:10.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06c6ec53c0a3a38-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:12:10.748 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.749 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.749 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.750 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.750 DEBUG:boto:Method: GET 2023-10-12T01:12:10.751 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.751 DEBUG:boto:Data: 2023-10-12T01:12:10.751 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.752 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.753 DEBUG:boto:Params: {} 2023-10-12T01:12:10.753 DEBUG:boto:Token: None 2023-10-12T01:12:10.754 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.754 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0= 2023-10-12T01:12:10.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0=', 'Content-Length': '0'} 2023-10-12T01:12:10.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b105c85d235568-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.802 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.803 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.803 DEBUG:boto:path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.804 DEBUG:boto:auth_path=/wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.804 DEBUG:boto:Method: GET 2023-10-12T01:12:10.805 DEBUG:boto:Path: /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.805 DEBUG:boto:Data: 2023-10-12T01:12:10.806 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.807 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.807 DEBUG:boto:Params: {} 2023-10-12T01:12:10.807 DEBUG:boto:Token: None 2023-10-12T01:12:10.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87?versionId=null 2023-10-12T01:12:10.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0= 2023-10-12T01:12:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7r5jcoUCN7HEj6L1YCkr73sAN0=', 'Content-Length': '0'} 2023-10-12T01:12:10.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013374f015b6af04b-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.812 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.812 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.813 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.813 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.814 DEBUG:boto:Method: GET 2023-10-12T01:12:10.814 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.815 DEBUG:boto:Data: 2023-10-12T01:12:10.815 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.817 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.818 DEBUG:boto:Params: {} 2023-10-12T01:12:10.818 DEBUG:boto:Token: None 2023-10-12T01:12:10.819 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.819 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a45e3bca3bd85c2d-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.823 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.825 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.825 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.826 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.826 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.826 DEBUG:boto:Method: GET 2023-10-12T01:12:10.827 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.827 DEBUG:boto:Data: 2023-10-12T01:12:10.828 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.829 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.829 DEBUG:boto:Params: {} 2023-10-12T01:12:10.830 DEBUG:boto:Token: None 2023-10-12T01:12:10.830 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000817e1b1af654b586-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.836 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.837 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.837 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.838 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.838 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:12:10.838 DEBUG:boto:Data: 2023-10-12T01:12:10.839 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.840 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.840 DEBUG:boto:Params: {} 2023-10-12T01:12:10.840 DEBUG:boto:Token: None 2023-10-12T01:12:10.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87 2023-10-12T01:12:10.841 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p7sZPBvL7IWClvDRakIcMNU1WZU= 2023-10-12T01:12:10.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p7sZPBvL7IWClvDRakIcMNU1WZU=', 'Content-Length': '0'} 2023-10-12T01:12:10.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326d109be307581d-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.845 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.845 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.845 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.846 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:12:10.846 DEBUG:boto:Data: 2023-10-12T01:12:10.847 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.848 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.848 DEBUG:boto:Params: {} 2023-10-12T01:12:10.848 DEBUG:boto:Token: None 2023-10-12T01:12:10.849 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87 2023-10-12T01:12:10.849 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p7sZPBvL7IWClvDRakIcMNU1WZU= 2023-10-12T01:12:10.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p7sZPBvL7IWClvDRakIcMNU1WZU=', 'Content-Length': '0'} 2023-10-12T01:12:10.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135dc063928cdcc6-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:12:10.853 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.854 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.854 DEBUG:boto:Method: GET 2023-10-12T01:12:10.855 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:12:10.855 DEBUG:boto:Data: 2023-10-12T01:12:10.855 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.856 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.857 DEBUG:boto:Params: {} 2023-10-12T01:12:10.857 DEBUG:boto:Token: None 2023-10-12T01:12:10.858 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87 2023-10-12T01:12:10.858 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWC0+DyQJznpKcjMe8l1BJ47dZ0= 2023-10-12T01:12:10.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWC0+DyQJznpKcjMe8l1BJ47dZ0=', 'Content-Length': '0'} 2023-10-12T01:12:10.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078655f5b2d1b26b0-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.861 DEBUG:boto:path=/wafmoa-59/obj87 2023-10-12T01:12:10.862 DEBUG:boto:auth_path=/wafmoa-59/obj87 2023-10-12T01:12:10.862 DEBUG:boto:Method: GET 2023-10-12T01:12:10.863 DEBUG:boto:Path: /wafmoa-59/obj87 2023-10-12T01:12:10.863 DEBUG:boto:Data: 2023-10-12T01:12:10.864 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.865 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.865 DEBUG:boto:Params: {} 2023-10-12T01:12:10.865 DEBUG:boto:Token: None 2023-10-12T01:12:10.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj87 2023-10-12T01:12:10.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bWC0+DyQJznpKcjMe8l1BJ47dZ0= 2023-10-12T01:12:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bWC0+DyQJznpKcjMe8l1BJ47dZ0=', 'Content-Length': '0'} 2023-10-12T01:12:10.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:04:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff087e8d113e6c3-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.910 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.911 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.912 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.912 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.912 DEBUG:boto:Method: GET 2023-10-12T01:12:10.913 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.913 DEBUG:boto:Data: 2023-10-12T01:12:10.914 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.915 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.915 DEBUG:boto:Params: {} 2023-10-12T01:12:10.916 DEBUG:boto:Token: None 2023-10-12T01:12:10.916 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.917 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95741a799d57266-00652747ea-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.920 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.921 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:10.922 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:10.922 DEBUG:boto:path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.923 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.923 DEBUG:boto:Method: GET 2023-10-12T01:12:10.923 DEBUG:boto:Path: /wafmoa-59/?prefix=obj87 2023-10-12T01:12:10.924 DEBUG:boto:Data: 2023-10-12T01:12:10.924 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.925 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.926 DEBUG:boto:Params: {} 2023-10-12T01:12:10.926 DEBUG:boto:Token: None 2023-10-12T01:12:10.927 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/ 2023-10-12T01:12:10.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k= 2023-10-12T01:12:10.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IZMqJpkNb1w7HALe2+H6WrmFw8k=', 'Content-Length': '0'} 2023-10-12T01:12:10.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c215b986b83acbf2-00652747ea-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.932 DEBUG:boto:b'wafmoa-59obj871000falseobj872023-10-12T01:04:26.341Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:10.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:12:10.933 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:10.934 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:10.934 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.935 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.935 DEBUG:boto:Method: GET 2023-10-12T01:12:10.936 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.936 DEBUG:boto:Data: 2023-10-12T01:12:10.936 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:10.937 DEBUG:boto:Port: 8001 2023-10-12T01:12:10.938 DEBUG:boto:Params: {} 2023-10-12T01:12:10.938 DEBUG:boto:Token: None 2023-10-12T01:12:10.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.939 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QHXw4DLidOeUe4PzbV8YrrNK6hQ= 2023-10-12T01:12:10.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QHXw4DLidOeUe4PzbV8YrrNK6hQ=', 'Content-Length': '0'} 2023-10-12T01:12:10.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001420bfda13a4594f-00652747ea-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.986 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:10.987 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:10.987 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.988 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.988 DEBUG:boto:Method: GET 2023-10-12T01:12:10.989 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.989 DEBUG:boto:Data: 2023-10-12T01:12:10.990 DEBUG:boto:Headers: {} 2023-10-12T01:12:10.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:10.991 DEBUG:boto:Port: 8000 2023-10-12T01:12:10.991 DEBUG:boto:Params: {} 2023-10-12T01:12:10.991 DEBUG:boto:Token: None 2023-10-12T01:12:10.992 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:10 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.993 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QHXw4DLidOeUe4PzbV8YrrNK6hQ= 2023-10-12T01:12:10.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QHXw4DLidOeUe4PzbV8YrrNK6hQ=', 'Content-Length': '0'} 2023-10-12T01:12:10.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b79715616208be-00652747ea-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:10.996 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:10.997 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:10.998 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.998 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:10.999 DEBUG:boto:Method: HEAD 2023-10-12T01:12:10.999 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.000 DEBUG:boto:Data: 2023-10-12T01:12:11.000 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.001 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.001 DEBUG:boto:Params: {} 2023-10-12T01:12:11.002 DEBUG:boto:Token: None 2023-10-12T01:12:11.002 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jz6dwcluktoxhx5bOI8KbBy7jb8= 2023-10-12T01:12:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jz6dwcluktoxhx5bOI8KbBy7jb8=', 'Content-Length': '0'} 2023-10-12T01:12:11.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023bda1d25e9fe4a8-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.007 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.008 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.008 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.008 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.009 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.009 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.010 DEBUG:boto:Data: 2023-10-12T01:12:11.010 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.011 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.011 DEBUG:boto:Params: {} 2023-10-12T01:12:11.012 DEBUG:boto:Token: None 2023-10-12T01:12:11.012 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jz6dwcluktoxhx5bOI8KbBy7jb8= 2023-10-12T01:12:11.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jz6dwcluktoxhx5bOI8KbBy7jb8=', 'Content-Length': '0'} 2023-10-12T01:12:11.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f932615ed3e0a7-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:12:11.017 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.017 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.018 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.018 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.019 DEBUG:boto:Method: GET 2023-10-12T01:12:11.019 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.020 DEBUG:boto:Data: 2023-10-12T01:12:11.020 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.021 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.021 DEBUG:boto:Params: {} 2023-10-12T01:12:11.022 DEBUG:boto:Token: None 2023-10-12T01:12:11.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ybC7DLJKVlnK5Lv5Z48nb+ESUgg= 2023-10-12T01:12:11.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ybC7DLJKVlnK5Lv5Z48nb+ESUgg=', 'Content-Length': '0'} 2023-10-12T01:12:11.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc59be2a5e43862c-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.027 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.027 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.028 DEBUG:boto:path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.028 DEBUG:boto:auth_path=/wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.028 DEBUG:boto:Method: GET 2023-10-12T01:12:11.029 DEBUG:boto:Path: /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.029 DEBUG:boto:Data: 2023-10-12T01:12:11.030 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.031 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.031 DEBUG:boto:Params: {} 2023-10-12T01:12:11.032 DEBUG:boto:Token: None 2023-10-12T01:12:11.032 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92?versionId=null 2023-10-12T01:12:11.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ybC7DLJKVlnK5Lv5Z48nb+ESUgg= 2023-10-12T01:12:11.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ybC7DLJKVlnK5Lv5Z48nb+ESUgg=', 'Content-Length': '0'} 2023-10-12T01:12:11.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d1977edb1dd8d1-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.036 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.036 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.037 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.037 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.037 DEBUG:boto:Method: GET 2023-10-12T01:12:11.038 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.038 DEBUG:boto:Data: 2023-10-12T01:12:11.039 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.040 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.040 DEBUG:boto:Params: {} 2023-10-12T01:12:11.041 DEBUG:boto:Token: None 2023-10-12T01:12:11.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000267735651ec8c8f2-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.045 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.046 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.047 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.047 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.047 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.048 DEBUG:boto:Method: GET 2023-10-12T01:12:11.048 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.049 DEBUG:boto:Data: 2023-10-12T01:12:11.049 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.050 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.051 DEBUG:boto:Params: {} 2023-10-12T01:12:11.051 DEBUG:boto:Token: None 2023-10-12T01:12:11.051 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050358d3b16c15cec-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.098 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.099 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.099 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.100 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.100 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:12:11.101 DEBUG:boto:Data: 2023-10-12T01:12:11.101 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.102 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.103 DEBUG:boto:Params: {} 2023-10-12T01:12:11.103 DEBUG:boto:Token: None 2023-10-12T01:12:11.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92 2023-10-12T01:12:11.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJsvPy3Bwo5+gUbX9xdrb3rN71M= 2023-10-12T01:12:11.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJsvPy3Bwo5+gUbX9xdrb3rN71M=', 'Content-Length': '0'} 2023-10-12T01:12:11.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ff9dcf9da82a1d-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.108 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.108 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.109 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.109 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:12:11.109 DEBUG:boto:Data: 2023-10-12T01:12:11.110 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.111 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.111 DEBUG:boto:Params: {} 2023-10-12T01:12:11.112 DEBUG:boto:Token: None 2023-10-12T01:12:11.112 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92 2023-10-12T01:12:11.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJsvPy3Bwo5+gUbX9xdrb3rN71M= 2023-10-12T01:12:11.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJsvPy3Bwo5+gUbX9xdrb3rN71M=', 'Content-Length': '0'} 2023-10-12T01:12:11.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d345fd6de60a75-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:12:11.116 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.117 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.117 DEBUG:boto:Method: GET 2023-10-12T01:12:11.118 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:12:11.118 DEBUG:boto:Data: 2023-10-12T01:12:11.118 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.119 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.120 DEBUG:boto:Params: {} 2023-10-12T01:12:11.120 DEBUG:boto:Token: None 2023-10-12T01:12:11.121 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92 2023-10-12T01:12:11.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V0r6EidLZeemUZdggpJv35GXs1g= 2023-10-12T01:12:11.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V0r6EidLZeemUZdggpJv35GXs1g=', 'Content-Length': '0'} 2023-10-12T01:12:11.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e336638802ab1882-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.170 DEBUG:boto:path=/wafmoa-59/obj92 2023-10-12T01:12:11.171 DEBUG:boto:auth_path=/wafmoa-59/obj92 2023-10-12T01:12:11.171 DEBUG:boto:Method: GET 2023-10-12T01:12:11.172 DEBUG:boto:Path: /wafmoa-59/obj92 2023-10-12T01:12:11.172 DEBUG:boto:Data: 2023-10-12T01:12:11.173 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.174 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.174 DEBUG:boto:Params: {} 2023-10-12T01:12:11.175 DEBUG:boto:Token: None 2023-10-12T01:12:11.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj92 2023-10-12T01:12:11.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V0r6EidLZeemUZdggpJv35GXs1g= 2023-10-12T01:12:11.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V0r6EidLZeemUZdggpJv35GXs1g=', 'Content-Length': '0'} 2023-10-12T01:12:11.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:03:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aef6a69eca9affa-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.179 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.179 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.180 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.180 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.181 DEBUG:boto:Method: GET 2023-10-12T01:12:11.181 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.182 DEBUG:boto:Data: 2023-10-12T01:12:11.182 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.183 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.183 DEBUG:boto:Params: {} 2023-10-12T01:12:11.184 DEBUG:boto:Token: None 2023-10-12T01:12:11.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d664da05950d10-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.188 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.189 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.189 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.190 DEBUG:boto:path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.190 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.191 DEBUG:boto:Method: GET 2023-10-12T01:12:11.191 DEBUG:boto:Path: /wafmoa-59/?prefix=obj92 2023-10-12T01:12:11.192 DEBUG:boto:Data: 2023-10-12T01:12:11.192 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.193 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.193 DEBUG:boto:Params: {} 2023-10-12T01:12:11.194 DEBUG:boto:Token: None 2023-10-12T01:12:11.194 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062380a2f116aab93-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.199 DEBUG:boto:b'wafmoa-59obj921000falseobj922023-10-12T01:03:05.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:12:11.200 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.201 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.201 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.202 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.202 DEBUG:boto:Method: GET 2023-10-12T01:12:11.203 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.203 DEBUG:boto:Data: 2023-10-12T01:12:11.203 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.204 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.205 DEBUG:boto:Params: {} 2023-10-12T01:12:11.205 DEBUG:boto:Token: None 2023-10-12T01:12:11.206 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw= 2023-10-12T01:12:11.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw=', 'Content-Length': '0'} 2023-10-12T01:12:11.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11c931a518bd805-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.210 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.210 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.211 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.211 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.211 DEBUG:boto:Method: GET 2023-10-12T01:12:11.212 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.212 DEBUG:boto:Data: 2023-10-12T01:12:11.213 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.213 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.214 DEBUG:boto:Params: {} 2023-10-12T01:12:11.215 DEBUG:boto:Token: None 2023-10-12T01:12:11.215 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.216 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw= 2023-10-12T01:12:11.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw=', 'Content-Length': '0'} 2023-10-12T01:12:11.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698798f9647b991c-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.262 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.263 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.263 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.264 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.264 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.265 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.265 DEBUG:boto:Data: 2023-10-12T01:12:11.266 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.266 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.267 DEBUG:boto:Params: {} 2023-10-12T01:12:11.267 DEBUG:boto:Token: None 2023-10-12T01:12:11.268 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KkVGJNSVY9F3gNZBwB7bxG6dNYc= 2023-10-12T01:12:11.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KkVGJNSVY9F3gNZBwB7bxG6dNYc=', 'Content-Length': '0'} 2023-10-12T01:12:11.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb97684fd4287872-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.272 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.272 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.273 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.273 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.273 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.274 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.274 DEBUG:boto:Data: 2023-10-12T01:12:11.275 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.275 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.276 DEBUG:boto:Params: {} 2023-10-12T01:12:11.276 DEBUG:boto:Token: None 2023-10-12T01:12:11.277 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KkVGJNSVY9F3gNZBwB7bxG6dNYc= 2023-10-12T01:12:11.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KkVGJNSVY9F3gNZBwB7bxG6dNYc=', 'Content-Length': '0'} 2023-10-12T01:12:11.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abad382f43d507de-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:12:11.281 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.281 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.282 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.282 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.283 DEBUG:boto:Method: GET 2023-10-12T01:12:11.283 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.283 DEBUG:boto:Data: 2023-10-12T01:12:11.284 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.285 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.285 DEBUG:boto:Params: {} 2023-10-12T01:12:11.286 DEBUG:boto:Token: None 2023-10-12T01:12:11.286 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw= 2023-10-12T01:12:11.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw=', 'Content-Length': '0'} 2023-10-12T01:12:11.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc494740aca84e3c-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.330 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.331 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.331 DEBUG:boto:path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.332 DEBUG:boto:auth_path=/wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.332 DEBUG:boto:Method: GET 2023-10-12T01:12:11.333 DEBUG:boto:Path: /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.333 DEBUG:boto:Data: 2023-10-12T01:12:11.334 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.335 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.335 DEBUG:boto:Params: {} 2023-10-12T01:12:11.336 DEBUG:boto:Token: None 2023-10-12T01:12:11.336 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93?versionId=null 2023-10-12T01:12:11.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw= 2023-10-12T01:12:11.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F//7tuCRbvuZd3Z/dHhg4wCssUw=', 'Content-Length': '0'} 2023-10-12T01:12:11.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b05b9bdd5b40289d-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.340 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.340 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.341 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.341 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.342 DEBUG:boto:Method: GET 2023-10-12T01:12:11.342 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.343 DEBUG:boto:Data: 2023-10-12T01:12:11.343 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.344 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.344 DEBUG:boto:Params: {} 2023-10-12T01:12:11.345 DEBUG:boto:Token: None 2023-10-12T01:12:11.345 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a77ea40c389b90f-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.350 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.350 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.351 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.351 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.352 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.352 DEBUG:boto:Method: GET 2023-10-12T01:12:11.353 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.353 DEBUG:boto:Data: 2023-10-12T01:12:11.354 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.354 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.355 DEBUG:boto:Params: {} 2023-10-12T01:12:11.355 DEBUG:boto:Token: None 2023-10-12T01:12:11.356 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006effcc70fb90a5c3-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.362 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.362 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.363 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.363 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.364 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:12:11.364 DEBUG:boto:Data: 2023-10-12T01:12:11.364 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.365 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.366 DEBUG:boto:Params: {} 2023-10-12T01:12:11.366 DEBUG:boto:Token: None 2023-10-12T01:12:11.366 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93 2023-10-12T01:12:11.367 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGiGECjeMKFgaEChm6kl9DGjcqs= 2023-10-12T01:12:11.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGiGECjeMKFgaEChm6kl9DGjcqs=', 'Content-Length': '0'} 2023-10-12T01:12:11.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d7efbce41a6455-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.370 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.370 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.371 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.371 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:12:11.372 DEBUG:boto:Data: 2023-10-12T01:12:11.372 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.373 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.373 DEBUG:boto:Params: {} 2023-10-12T01:12:11.374 DEBUG:boto:Token: None 2023-10-12T01:12:11.374 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93 2023-10-12T01:12:11.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OGiGECjeMKFgaEChm6kl9DGjcqs= 2023-10-12T01:12:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OGiGECjeMKFgaEChm6kl9DGjcqs=', 'Content-Length': '0'} 2023-10-12T01:12:11.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113eb67e13838dc2-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:12:11.379 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.379 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.380 DEBUG:boto:Method: GET 2023-10-12T01:12:11.380 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:12:11.381 DEBUG:boto:Data: 2023-10-12T01:12:11.381 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.382 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.383 DEBUG:boto:Params: {} 2023-10-12T01:12:11.383 DEBUG:boto:Token: None 2023-10-12T01:12:11.384 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93 2023-10-12T01:12:11.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z6vZvF3JICNJ+6zrtePHfctg9Ys= 2023-10-12T01:12:11.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z6vZvF3JICNJ+6zrtePHfctg9Ys=', 'Content-Length': '0'} 2023-10-12T01:12:11.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c24a278e1bd4dae-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.387 DEBUG:boto:path=/wafmoa-59/obj93 2023-10-12T01:12:11.388 DEBUG:boto:auth_path=/wafmoa-59/obj93 2023-10-12T01:12:11.388 DEBUG:boto:Method: GET 2023-10-12T01:12:11.389 DEBUG:boto:Path: /wafmoa-59/obj93 2023-10-12T01:12:11.389 DEBUG:boto:Data: 2023-10-12T01:12:11.389 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.390 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.391 DEBUG:boto:Params: {} 2023-10-12T01:12:11.391 DEBUG:boto:Token: None 2023-10-12T01:12:11.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj93 2023-10-12T01:12:11.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Z6vZvF3JICNJ+6zrtePHfctg9Ys= 2023-10-12T01:12:11.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Z6vZvF3JICNJ+6zrtePHfctg9Ys=', 'Content-Length': '0'} 2023-10-12T01:12:11.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:06:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad3dbac2aa72288-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.438 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.439 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.439 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.440 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.440 DEBUG:boto:Method: GET 2023-10-12T01:12:11.441 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.441 DEBUG:boto:Data: 2023-10-12T01:12:11.441 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.442 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.443 DEBUG:boto:Params: {} 2023-10-12T01:12:11.443 DEBUG:boto:Token: None 2023-10-12T01:12:11.444 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.444 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29215127f05e6ea-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.448 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.449 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.449 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.450 DEBUG:boto:path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.450 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.451 DEBUG:boto:Method: GET 2023-10-12T01:12:11.451 DEBUG:boto:Path: /wafmoa-59/?prefix=obj93 2023-10-12T01:12:11.451 DEBUG:boto:Data: 2023-10-12T01:12:11.452 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.453 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.453 DEBUG:boto:Params: {} 2023-10-12T01:12:11.454 DEBUG:boto:Token: None 2023-10-12T01:12:11.454 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdb9ddf594522cf9-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.460 DEBUG:boto:b'wafmoa-59obj931000falseobj932023-10-12T01:06:27.388Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:12:11.461 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.462 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.462 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.463 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.463 DEBUG:boto:Method: GET 2023-10-12T01:12:11.464 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.464 DEBUG:boto:Data: 2023-10-12T01:12:11.464 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.465 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.466 DEBUG:boto:Params: {} 2023-10-12T01:12:11.466 DEBUG:boto:Token: None 2023-10-12T01:12:11.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE= 2023-10-12T01:12:11.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE=', 'Content-Length': '0'} 2023-10-12T01:12:11.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000a4fae61c147acb-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.471 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.471 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.472 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.472 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.473 DEBUG:boto:Method: GET 2023-10-12T01:12:11.473 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.474 DEBUG:boto:Data: 2023-10-12T01:12:11.474 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.475 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.475 DEBUG:boto:Params: {} 2023-10-12T01:12:11.476 DEBUG:boto:Token: None 2023-10-12T01:12:11.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.477 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE= 2023-10-12T01:12:11.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE=', 'Content-Length': '0'} 2023-10-12T01:12:11.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e41a43af6045ccc-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.480 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.481 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.481 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.481 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.482 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.482 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.483 DEBUG:boto:Data: 2023-10-12T01:12:11.483 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.484 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.484 DEBUG:boto:Params: {} 2023-10-12T01:12:11.485 DEBUG:boto:Token: None 2023-10-12T01:12:11.485 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.486 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qUx/xqz6F98c9zv98sPXelEl4MQ= 2023-10-12T01:12:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qUx/xqz6F98c9zv98sPXelEl4MQ=', 'Content-Length': '0'} 2023-10-12T01:12:11.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f7cfaf50e8e9a9a-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.489 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.489 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.490 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.490 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.490 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.491 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.491 DEBUG:boto:Data: 2023-10-12T01:12:11.492 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.493 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.493 DEBUG:boto:Params: {} 2023-10-12T01:12:11.493 DEBUG:boto:Token: None 2023-10-12T01:12:11.494 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qUx/xqz6F98c9zv98sPXelEl4MQ= 2023-10-12T01:12:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qUx/xqz6F98c9zv98sPXelEl4MQ=', 'Content-Length': '0'} 2023-10-12T01:12:11.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff20e5b29510d693-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:12:11.498 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.499 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.499 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.499 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.500 DEBUG:boto:Method: GET 2023-10-12T01:12:11.500 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.501 DEBUG:boto:Data: 2023-10-12T01:12:11.501 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.502 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.503 DEBUG:boto:Params: {} 2023-10-12T01:12:11.503 DEBUG:boto:Token: None 2023-10-12T01:12:11.504 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE= 2023-10-12T01:12:11.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE=', 'Content-Length': '0'} 2023-10-12T01:12:11.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea06f909377e464-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.550 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.551 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.551 DEBUG:boto:path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.552 DEBUG:boto:auth_path=/wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.552 DEBUG:boto:Method: GET 2023-10-12T01:12:11.553 DEBUG:boto:Path: /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.553 DEBUG:boto:Data: 2023-10-12T01:12:11.554 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.554 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.555 DEBUG:boto:Params: {} 2023-10-12T01:12:11.555 DEBUG:boto:Token: None 2023-10-12T01:12:11.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95?versionId=null 2023-10-12T01:12:11.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE= 2023-10-12T01:12:11.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DjtVLqIH5U3uu3yAnYdGZFYujEE=', 'Content-Length': '0'} 2023-10-12T01:12:11.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e401b8b0ca057087-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.560 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.560 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.561 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.561 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.562 DEBUG:boto:Method: GET 2023-10-12T01:12:11.562 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.562 DEBUG:boto:Data: 2023-10-12T01:12:11.563 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.564 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.564 DEBUG:boto:Params: {} 2023-10-12T01:12:11.565 DEBUG:boto:Token: None 2023-10-12T01:12:11.565 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.566 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f39f0768ccd7d9d-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.569 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.570 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.571 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.571 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.572 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.572 DEBUG:boto:Method: GET 2023-10-12T01:12:11.572 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.573 DEBUG:boto:Data: 2023-10-12T01:12:11.573 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.574 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.575 DEBUG:boto:Params: {} 2023-10-12T01:12:11.575 DEBUG:boto:Token: None 2023-10-12T01:12:11.575 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557fadbb8cd088fe-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.581 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.582 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.582 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.583 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.583 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:12:11.584 DEBUG:boto:Data: 2023-10-12T01:12:11.584 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.585 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.585 DEBUG:boto:Params: {} 2023-10-12T01:12:11.586 DEBUG:boto:Token: None 2023-10-12T01:12:11.586 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95 2023-10-12T01:12:11.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTYQwEHgWqDlRnHe+boa5cRoMx0= 2023-10-12T01:12:11.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTYQwEHgWqDlRnHe+boa5cRoMx0=', 'Content-Length': '0'} 2023-10-12T01:12:11.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceaf112dd468e5ca-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.590 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.590 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.591 DEBUG:boto:Method: HEAD 2023-10-12T01:12:11.591 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:12:11.591 DEBUG:boto:Data: 2023-10-12T01:12:11.592 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.593 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.593 DEBUG:boto:Params: {} 2023-10-12T01:12:11.594 DEBUG:boto:Token: None 2023-10-12T01:12:11.594 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95 2023-10-12T01:12:11.594 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JTYQwEHgWqDlRnHe+boa5cRoMx0= 2023-10-12T01:12:11.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JTYQwEHgWqDlRnHe+boa5cRoMx0=', 'Content-Length': '0'} 2023-10-12T01:12:11.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b3b99795ee8d49-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:12:11.598 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.599 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.599 DEBUG:boto:Method: GET 2023-10-12T01:12:11.599 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:12:11.600 DEBUG:boto:Data: 2023-10-12T01:12:11.600 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.601 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.602 DEBUG:boto:Params: {} 2023-10-12T01:12:11.602 DEBUG:boto:Token: None 2023-10-12T01:12:11.603 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95 2023-10-12T01:12:11.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dDmbDKduexmMwfU6WjqvT+QJkys= 2023-10-12T01:12:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dDmbDKduexmMwfU6WjqvT+QJkys=', 'Content-Length': '0'} 2023-10-12T01:12:11.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-59:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b9f34bad4354b4-00652747eb-1e19-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.606 DEBUG:boto:path=/wafmoa-59/obj95 2023-10-12T01:12:11.607 DEBUG:boto:auth_path=/wafmoa-59/obj95 2023-10-12T01:12:11.607 DEBUG:boto:Method: GET 2023-10-12T01:12:11.608 DEBUG:boto:Path: /wafmoa-59/obj95 2023-10-12T01:12:11.608 DEBUG:boto:Data: 2023-10-12T01:12:11.609 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.609 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.610 DEBUG:boto:Params: {} 2023-10-12T01:12:11.610 DEBUG:boto:Token: None 2023-10-12T01:12:11.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/obj95 2023-10-12T01:12:11.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dDmbDKduexmMwfU6WjqvT+QJkys= 2023-10-12T01:12:11.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dDmbDKduexmMwfU6WjqvT+QJkys=', 'Content-Length': '0'} 2023-10-12T01:12:11.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:01:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006882de58ad9d4286-00652747eb-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.615 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.615 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.616 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.616 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.617 DEBUG:boto:Method: GET 2023-10-12T01:12:11.617 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.617 DEBUG:boto:Data: 2023-10-12T01:12:11.618 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:12:11.619 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.619 DEBUG:boto:Params: {} 2023-10-12T01:12:11.620 DEBUG:boto:Token: None 2023-10-12T01:12:11.620 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.621 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035297d2f990e380e-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.624 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.625 DEBUG:boto:path=/wafmoa-59/ 2023-10-12T01:12:11.626 DEBUG:boto:auth_path=/wafmoa-59/ 2023-10-12T01:12:11.626 DEBUG:boto:path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.626 DEBUG:boto:auth_path=/wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.627 DEBUG:boto:Method: GET 2023-10-12T01:12:11.627 DEBUG:boto:Path: /wafmoa-59/?prefix=obj95 2023-10-12T01:12:11.628 DEBUG:boto:Data: 2023-10-12T01:12:11.628 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:12:11.629 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.629 DEBUG:boto:Params: {} 2023-10-12T01:12:11.630 DEBUG:boto:Token: None 2023-10-12T01:12:11.630 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:12:11 GMT /wafmoa-59/ 2023-10-12T01:12:11.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA= 2023-10-12T01:12:11.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:12:11 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yN/FqhjzCe94+koU3PWMfdS1XbA=', 'Content-Length': '0'} 2023-10-12T01:12:11.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15a0f52f9e2fc29-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.636 DEBUG:boto:b'wafmoa-59obj951000falseobj952023-10-12T01:01:07.016Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:12:11.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-59 zones={a2, a1} 2023-10-12T01:12:11.638 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-10-12T01:12:11.644 INFO:rgw_multi.tests:create role zone=a1 name=roles-wafmoa-1 2023-10-12T01:12:11.645 DEBUG:boto:Method: POST 2023-10-12T01:12:11.645 DEBUG:boto:Path: / 2023-10-12T01:12:11.646 DEBUG:boto:Data: 2023-10-12T01:12:11.646 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:12:11.647 DEBUG:boto:Port: 8000 2023-10-12T01:12:11.647 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-12T01:12:11.648 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:12:11.648 DEBUG:boto:Token: None 2023-10-12T01:12:11.649 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20231012T011211Z host;x-amz-date 37f0e8c24bb743940e7007229ddd9fcc34a18aba8c2411c6b9e8c62f468ad441 2023-10-12T01:12:11.649 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011211Z 20231012/front/smithi112/aws4_request fb2e08a0bd2c4f7bf84c925a0a9a5ac18cd16a76df359c7362a2399e23406b44 2023-10-12T01:12:11.650 DEBUG:boto:Signature: 17f85797d48f0a65fa10443cb65d0adba364d5a7c1632921006ab3d8901201e4 2023-10-12T01:12:11.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011211Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=17f85797d48f0a65fa10443cb65d0adba364d5a7c1632921006ab3d8901201e4', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-10-12T01:12:11.663 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d3be8ab5b62ccc6-00652747eb-168d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.664 DEBUG:boto:b'2220e0ff-1202-44a8-ad85-23ad100c3639roles-wafmoa-1/arn:aws:iam:::role/roles-wafmoa-12023-10-12T01:12:11.655Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000007d3be8ab5b62ccc6-00652747eb-168d-a1' 2023-10-12T01:12:11.665 INFO:rgw_multi.tests:create role zone=a2 name=roles-wafmoa-2 2023-10-12T01:12:11.665 DEBUG:boto:Method: POST 2023-10-12T01:12:11.666 DEBUG:boto:Path: / 2023-10-12T01:12:11.666 DEBUG:boto:Data: 2023-10-12T01:12:11.666 DEBUG:boto:Headers: {} 2023-10-12T01:12:11.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:12:11.667 DEBUG:boto:Port: 8001 2023-10-12T01:12:11.668 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-12T01:12:11.668 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:12:11.668 DEBUG:boto:Token: None 2023-10-12T01:12:11.669 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20231012T011211Z host;x-amz-date 5159f024c09c36fdb37c0a129b28b00670014f537c8bddf15f3fa86f88410cf5 2023-10-12T01:12:11.669 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011211Z 20231012/front/smithi112/aws4_request b0921b27f3cbf27e8fe98b4435e9639657cb4c144ff2c93ed92e996b3efa5e5e 2023-10-12T01:12:11.670 DEBUG:boto:Signature: e8ba636fc88db2f33ad66dcf50c0d40c48747eb4356e07327ba76f849928eab6 2023-10-12T01:12:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011211Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=e8ba636fc88db2f33ad66dcf50c0d40c48747eb4356e07327ba76f849928eab6', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-10-12T01:12:11.690 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c16415f61e2e91f7-00652747eb-1e19-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Thu, 12 Oct 2023 01:12:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:12:11.691 DEBUG:boto:b'0bb29fa0-d0be-4b27-9173-473187e6a355roles-wafmoa-2/arn:aws:iam:::role/roles-wafmoa-22023-10-12T01:12:11.685Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000c16415f61e2e91f7-00652747eb-1e19-a2' 2023-10-12T01:12:11.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:11.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:11.693 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:12:11.794 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:12:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:12:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:12:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.796 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:12:11.796 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:12:11.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:11.797 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:09:29.875315Z" 2023-10-12T01:12:11.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:11.798 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:12:11.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.798 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:12:11.799 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:11.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:12:11.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:11.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:11.801 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:11.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:11.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:11.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:11.879+0000 7f95ab05c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:11.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:11.880+0000 7f95ab05c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:11.892 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:11.900 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:11.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:11.902 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:11.903 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:11.904 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:11.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.905 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:11.905 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:11.905 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:11.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:11.906 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:11Z" 2023-10-12T01:12:11.906 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:11.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:11Z" } 2023-10-12T01:12:11.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:11.908 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:11.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:11.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:16.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:16.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:16.912 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:16.998+0000 7f95a3916780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:16.999+0000 7f95a3916780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:17.012 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:17.012 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:17.013 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:17Z" 2023-10-12T01:12:17.025 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:17.025 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:17Z" } 2023-10-12T01:12:17.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:17.027 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:17.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:17.027 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:22.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:22.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:22.030 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:22.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:22.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:22.122+0000 7f4643cae780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:22.123+0000 7f4643cae780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:22.137 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:22.137 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:22.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:22.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:22.140 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:22Z" 2023-10-12T01:12:22.152 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:22.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:22Z" } 2023-10-12T01:12:22.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:22.155 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:22.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:22.156 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:27.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:27.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:27.158 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:27.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:27.249+0000 7f4c2c9a3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:27.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:27.250+0000 7f4c2c9a3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:27.264 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:27.264 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:27Z" 2023-10-12T01:12:27.279 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:27.280 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:27Z" } 2023-10-12T01:12:27.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:27.281 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:27.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:27.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:32.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:32.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:32.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:32.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:32.374+0000 7f4a81e68780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:32.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:32.375+0000 7f4a81e68780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:32.390 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:32.390 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:32.391 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:32.392 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:32.393 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:32.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:32.395 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:32.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:32.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:32.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:32.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:32.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:32.402 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.403 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:32.404 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:32Z" 2023-10-12T01:12:32.405 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:32.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:32Z" } 2023-10-12T01:12:32.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:32.407 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:32.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:32.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:37.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:37.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:37.411 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:37.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:37.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:37.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:37.498+0000 7f4a09f94780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:37.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:37.499+0000 7f4a09f94780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:37.514 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:37.514 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:37.514 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:37.514 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:37.515 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:37.515 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:37.515 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:37.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.515 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:37.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:37.516 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:37.516 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:37.516 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:37.517 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:37.517 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:37.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:37.518 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:37.518 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:37.518 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:37.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:37.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:37.519 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:37.520 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:37.520 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:37.520 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:37.520 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:37.520 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:37.521 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:37.521 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:37.521 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:37.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:37.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:37.522 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:37.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:37.522 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:37.522 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:37.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:37.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:37.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:37.523 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:37.523 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:37.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:37.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:37.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:37.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:37.525 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:37.526 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:37.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:37.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:37.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:37.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.527 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:37.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:37.527 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:37.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:37.528 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:37Z" 2023-10-12T01:12:37.528 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:37.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:37Z" } 2023-10-12T01:12:37.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:37.530 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:37.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:37.530 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:42.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:42.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:42.533 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:42.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:42.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:42.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:42.621+0000 7f52de930780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:42.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:42.622+0000 7f52de930780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:42.636 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:42.636 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:42.636 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:42.637 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:42.637 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:42.637 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:42.637 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:42.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.638 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:42.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:42.638 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:42.638 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:42.639 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:42.639 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:42.639 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:42.639 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:42.639 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:42.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:42.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:42.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:42.648 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:42.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.649 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:42.649 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:42.649 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:42.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:42.650 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:42Z" 2023-10-12T01:12:42.650 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:42.651 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:42Z" } 2023-10-12T01:12:42.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:42.652 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:42.652 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:42.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:47.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:47.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:47.655 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:47.712 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:47.712 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:47.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:47.748+0000 7f1177e46780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:47.749+0000 7f1177e46780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:47.762 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:47.763 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:47.763 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:47.763 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:47.763 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:47Z" 2023-10-12T01:12:47.776 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:47.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:47Z" } 2023-10-12T01:12:47.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:47.779 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:47.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:47.779 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:52.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:52.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:52.782 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:52.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:52.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:52.871+0000 7f7525bfa780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:52.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:52.872+0000 7f7525bfa780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:52.886 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:52.886 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:52.886 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:52.887 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:52.887 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:52.887 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:52.887 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:52.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.888 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:52.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:52.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:52.888 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:52.889 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:52.889 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:52.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:52.889 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:52.890 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:52.890 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:52.890 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:52.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:52.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:52.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:52.891 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:52.891 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:52.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:52.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:52.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:52.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:52.893 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:52.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:52.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:52.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:52.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:52.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:52.894 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:52.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:52.894 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:52.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:52.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:52.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:52.895 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:52.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:52.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:52.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:52.896 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:52.897 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:52.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:52.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:52.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:52.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:52.898 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:52.898 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:52.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:52.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:52.899 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:52.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.899 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:52.899 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:52.899 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:52.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:52.900 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:52Z" 2023-10-12T01:12:52.900 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:52.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:52Z" } 2023-10-12T01:12:52.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:52.902 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:52.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:52.903 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:12:57.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:57.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:12:57.905 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:12:57.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:12:57.963 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:12:57.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:57.994+0000 7f73eeb77780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:57.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:12:57.995+0000 7f73eeb77780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:12:58.008 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:12:58.009 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:12:58.009 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:12:58.009 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:12:58.009 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:12:58.009 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:12:58.010 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:58.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.010 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:12:58.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:12:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:12:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:58.012 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:12:58.012 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:58.012 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:12:58.012 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:58.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:58.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:58.013 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:12:58.013 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:58.014 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:58.014 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:12:58.014 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:58.014 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:58.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:58.015 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:12:58.015 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:58.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:58.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:58.016 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:12:58.016 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:58.016 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:58.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:12:58.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:58.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:58.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:58.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:12:58.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:58.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:58.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:12:58.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:12:58.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:12:58.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:12:58.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:12:58.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:12:58.019 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:12:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:12:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:12:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:12:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:12:58.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:12:58.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.021 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:12:58.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:12:58.021 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:12:58.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:12:58.022 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:12:57Z" 2023-10-12T01:12:58.022 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:12:58.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:12:57Z" } 2023-10-12T01:12:58.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:12:58.024 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:58.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:12:58.025 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:13:03.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:03.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:03.027 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:03.084 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:03.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:03.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:03.115+0000 7f11d99b9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:03.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:03.116+0000 7f11d99b9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:03.132 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:03.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:03.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:03.134 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:13:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:03.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:13:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:03.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:13:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:03.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:03.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:03.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:13:03.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:03.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:03.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:03.139 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:03.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:03.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:03.142 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:03.142 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:03.142 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:03.142 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:03.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:03.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:03.143 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:03.143 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:03.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:03.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:03.144 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:03.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.144 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:03.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:03.145 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:03.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:03.145 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:03Z" 2023-10-12T01:13:03.145 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:03.146 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:03Z" } 2023-10-12T01:13:03.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:03.147 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:03.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:03.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:13:08.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:08.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:08.151 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:08.241+0000 7f7bd80ee780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:08.242+0000 7f7bd80ee780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:08.256 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:08.256 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:08.256 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:13:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:13:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:13:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:13:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:08.264 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:08.264 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:08.264 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:08.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:08.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:08.265 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:08.265 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:08.265 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:08.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:08.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:08.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:08.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:08.266 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:08.266 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:08.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:08.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:08.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:08.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:08.267 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:08.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:08.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:08.268 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:08.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.269 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:08.269 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:08.269 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:08.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:08.269 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:08Z" 2023-10-12T01:13:08.270 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:08.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:08Z" } 2023-10-12T01:13:08.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:08.272 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:08.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:08.272 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:13:13.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:13.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:13.275 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:13.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:13.364+0000 7f6854dda780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:13.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:13.365+0000 7f6854dda780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:13.379 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:13.379 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:13.380 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:13.380 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:13.380 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:13.380 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:13.381 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:13.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.381 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:13.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:13.381 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:13.382 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:13.382 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:13.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697069909.315956_77.1", 2023-10-12T01:13:13.382 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:13.383 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:13.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:13.383 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:18:29.315956Z", 2023-10-12T01:13:13.383 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:13.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:13.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:13.384 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:13.384 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:13.384 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:13.385 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-10-12T01:13:13.385 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:13.385 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:13.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:13.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-10-12T01:13:13.386 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:13.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:13.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:13.387 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:13.387 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:13.387 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:13.388 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:13.388 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:13.388 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:13.388 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:13.388 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:13.389 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:13.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:13.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:13.389 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:13.390 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:13.390 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:13.390 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:13.390 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:13.390 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:13.391 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:13.391 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:13.391 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:13.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:13.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:13.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:13.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.392 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:13.392 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:13.392 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:13.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:13.393 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:13Z" 2023-10-12T01:13:13.393 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:13.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697069909.315956_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T00:18:29.315956Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:13Z" } 2023-10-12T01:13:13.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697069909.315956_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T00:18:29.315956Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:13.395 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:13.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1697069909.315956_77.1', 1: '', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:13.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697073131.659356_324.1 target=1_1697069909.315956_77.1, shard=1 master=1_1697073131.679864_578.1 target= 2023-10-12T01:13:18.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.398 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:18.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:18.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:18.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:18.487+0000 7f82ec211780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:18.488+0000 7f82ec211780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:18.502 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:18Z" 2023-10-12T01:13:18.515 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:18.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:18Z" } 2023-10-12T01:13:18.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:18.517 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:18.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:18.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:13:18.518 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-wafmoa-1 zones={a2, a1} 2023-10-12T01:13:18.519 DEBUG:boto:Method: POST 2023-10-12T01:13:18.519 DEBUG:boto:Path: / 2023-10-12T01:13:18.520 DEBUG:boto:Data: 2023-10-12T01:13:18.520 DEBUG:boto:Headers: {} 2023-10-12T01:13:18.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:13:18.521 DEBUG:boto:Port: 8001 2023-10-12T01:13:18.521 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-12T01:13:18.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:13:18.523 DEBUG:boto:Token: None 2023-10-12T01:13:18.523 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20231012T011318Z host;x-amz-date 634f49c90c0be5d849ba6dc6c59df7ee75b205bf1178e5a173f0dfaff6d903c0 2023-10-12T01:13:18.524 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011318Z 20231012/front/smithi112/aws4_request 44fd651125d55ad3430c76f4ee61e023674ce5b8307f8e2941c9952310662da1 2023-10-12T01:13:18.524 DEBUG:boto:Signature: f8a4fd3c13d6d9c4fabe1a37a91d40caba44c21397b5afc2f0cb74557e21bed2 2023-10-12T01:13:18.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011318Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=f8a4fd3c13d6d9c4fabe1a37a91d40caba44c21397b5afc2f0cb74557e21bed2', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-10-12T01:13:18.528 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001041a7e64be651c-006527482e-1e19-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 12 Oct 2023 01:13:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:13:18.529 DEBUG:boto:b'tx0000001041a7e64be651c-006527482e-1e19-a22220e0ff-1202-44a8-ad85-23ad100c3639roles-wafmoa-1/arn:aws:iam:::role/roles-wafmoa-12023-10-12T01:13:16.404Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-12T01:13:18.529 DEBUG:boto:Method: POST 2023-10-12T01:13:18.530 DEBUG:boto:Path: / 2023-10-12T01:13:18.530 DEBUG:boto:Data: 2023-10-12T01:13:18.531 DEBUG:boto:Headers: {} 2023-10-12T01:13:18.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:13:18.532 DEBUG:boto:Port: 8000 2023-10-12T01:13:18.532 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-12T01:13:18.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:13:18.533 DEBUG:boto:Token: None 2023-10-12T01:13:18.533 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20231012T011318Z host;x-amz-date 634f49c90c0be5d849ba6dc6c59df7ee75b205bf1178e5a173f0dfaff6d903c0 2023-10-12T01:13:18.534 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011318Z 20231012/front/smithi112/aws4_request fa6c8e9af5cc7c326e30ac2705d60dbdc00dc10476376b33cd11138ac51e786e 2023-10-12T01:13:18.534 DEBUG:boto:Signature: 42b834ddcb21b882bc969e03a060ca0565331f6fcfdf9cacae6fb3ebac2342d9 2023-10-12T01:13:18.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011318Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=42b834ddcb21b882bc969e03a060ca0565331f6fcfdf9cacae6fb3ebac2342d9', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-10-12T01:13:18.537 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035e40448421b1108-006527482e-168d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 12 Oct 2023 01:13:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:13:18.538 DEBUG:boto:b'tx0000035e40448421b1108-006527482e-168d-a12220e0ff-1202-44a8-ad85-23ad100c3639roles-wafmoa-1/arn:aws:iam:::role/roles-wafmoa-12023-10-12T01:12:11.655Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-12T01:13:18.538 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-wafmoa-1 2023-10-12T01:13:18.539 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-wafmoa-1 zones={a2, a1} 2023-10-12T01:13:18.539 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-wafmoa-2 zones={a1, a2} 2023-10-12T01:13:18.540 DEBUG:boto:Method: POST 2023-10-12T01:13:18.540 DEBUG:boto:Path: / 2023-10-12T01:13:18.540 DEBUG:boto:Data: 2023-10-12T01:13:18.541 DEBUG:boto:Headers: {} 2023-10-12T01:13:18.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:13:18.542 DEBUG:boto:Port: 8000 2023-10-12T01:13:18.542 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-12T01:13:18.542 DEBUG:boto:Token: None 2023-10-12T01:13:18.543 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20231012T011318Z host;x-amz-date ff85dd07865ba1d3fd3f1095fd0dfd35900b793b6028d8e7fc1a45ac95921748 2023-10-12T01:13:18.543 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011318Z 20231012/front/smithi112/aws4_request 860b715a8c1a86662b4423cf402efae9c97968348f3748201ae9f6be94e48b3a 2023-10-12T01:13:18.544 DEBUG:boto:Signature: 2dacf657d078a5d60e2d11e4d362474e5be560eb416f44432c8f95edd1a5668a 2023-10-12T01:13:18.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011318Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=2dacf657d078a5d60e2d11e4d362474e5be560eb416f44432c8f95edd1a5668a', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-10-12T01:13:18.546 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af40dd3859f69cc9-006527482e-168d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 12 Oct 2023 01:13:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:13:18.546 DEBUG:boto:b'tx00000af40dd3859f69cc9-006527482e-168d-a10bb29fa0-d0be-4b27-9173-473187e6a355roles-wafmoa-2/arn:aws:iam:::role/roles-wafmoa-22023-10-12T01:12:11.676Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-12T01:13:18.547 DEBUG:boto:Method: POST 2023-10-12T01:13:18.547 DEBUG:boto:Path: / 2023-10-12T01:13:18.548 DEBUG:boto:Data: 2023-10-12T01:13:18.548 DEBUG:boto:Headers: {} 2023-10-12T01:13:18.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-10-12T01:13:18.549 DEBUG:boto:Port: 8001 2023-10-12T01:13:18.549 DEBUG:boto:Params: {'RoleName': 'roles-wafmoa-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-12T01:13:18.550 DEBUG:boto:Token: None 2023-10-12T01:13:18.550 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20231012T011318Z host;x-amz-date ff85dd07865ba1d3fd3f1095fd0dfd35900b793b6028d8e7fc1a45ac95921748 2023-10-12T01:13:18.551 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231012T011318Z 20231012/front/smithi112/aws4_request 272ea84020d29325b1c97f34a3469f54a3da6a027ff89ec640d41f80053ed054 2023-10-12T01:13:18.551 DEBUG:boto:Signature: 0f98b75c2618269419cc73e43a37a5d606279a41d98386e5e90b4d267c161ea5 2023-10-12T01:13:18.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231012T011318Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=L12R7HDAL0U7FKTOBSDN/20231012/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=0f98b75c2618269419cc73e43a37a5d606279a41d98386e5e90b4d267c161ea5', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-10-12T01:13:18.553 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000147df0d2dea3eca9-006527482e-1e19-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Thu, 12 Oct 2023 01:13:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:13:18.554 DEBUG:boto:b'tx00000147df0d2dea3eca9-006527482e-1e19-a20bb29fa0-d0be-4b27-9173-473187e6a355roles-wafmoa-2/arn:aws:iam:::role/roles-wafmoa-22023-10-12T01:12:11.676Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-12T01:13:18.555 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-wafmoa-2 2023-10-12T01:13:18.555 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-wafmoa-2 zones={a1, a2} 2023-10-12T01:13:18.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-10-12T01:13:18.557 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:13:18.557 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:13:18.557 DEBUG:boto:Method: PUT 2023-10-12T01:13:18.558 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:13:18.558 DEBUG:boto:Data: 2023-10-12T01:13:18.559 DEBUG:boto:Headers: {} 2023-10-12T01:13:18.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:13:18.560 DEBUG:boto:Port: 8000 2023-10-12T01:13:18.560 DEBUG:boto:Params: {} 2023-10-12T01:13:18.561 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:13:18.561 DEBUG:boto:Token: None 2023-10-12T01:13:18.562 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:13:18 GMT /wafmoa-60/ 2023-10-12T01:13:18.562 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZIbeBHDFuPlupqRvIbtKDxPa13I= 2023-10-12T01:13:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:13:18 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZIbeBHDFuPlupqRvIbtKDxPa13I=', 'Content-Length': '0'} 2023-10-12T01:13:18.578 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014335d7b1faf4cec-006527482e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:13:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:13:18.579 DEBUG:rgw_multi.tests:created bucket=wafmoa-60 2023-10-12T01:13:18.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.580 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:18.660 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:13:18.673 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:13:18.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:18.674 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:13:18.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:18.674 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:13:18.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073198.577963_651.1", 2023-10-12T01:13:18.675 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:13:18.577963Z" 2023-10-12T01:13:18.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:18.676 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:13:18.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.677 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:13:18.678 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:18.678 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:13:18.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:18.680 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:18.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:18.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:18.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:18.756+0000 7f3ce0dc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:18.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:18.757+0000 7f3ce0dc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:18.770 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:18.770 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.772 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:18.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.772 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:18.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.772 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:18.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.773 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:18.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.773 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:18.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.775 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:18.775 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.775 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:18.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.776 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:18.776 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.777 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:18.777 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.777 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.778 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:18.778 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.778 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.778 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.779 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:18.779 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:18.780 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:18.780 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:18.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:18.780 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:18.780 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:18.781 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:18.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:18.781 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:18.781 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:18.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:18.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:18.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.782 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:18.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:18.783 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:18.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:18.783 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:18Z" 2023-10-12T01:13:18.783 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:18.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:18Z" } 2023-10-12T01:13:18.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:18.785 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:18.786 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:18.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:23.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:23.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:23.789 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:23.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:23.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:23.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:23.878+0000 7fdbb8bd5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:23.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:23.879+0000 7fdbb8bd5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:23.893 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:23.893 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:23.893 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:23.894 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:23.894 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:23.894 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:23.894 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:23.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.895 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:23.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:23.895 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:23.896 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:23.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:23.898 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:23.898 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:23.898 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:23.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:23.899 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:23.899 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:23.899 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:23.899 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:23.900 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:23.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:23.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:23.901 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:23.901 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:23.901 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:23.901 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:23.902 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:23.902 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:23.902 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:23.902 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:23.902 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:23.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:23.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:23.903 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:23.903 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:23.904 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:23.904 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:23.904 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:23.904 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:23.905 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:23.905 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:23.905 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:23.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:23.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:23.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:23.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.906 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:23.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:23.907 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:23.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:23.907 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:23Z" 2023-10-12T01:13:23.907 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:23.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:23Z" } 2023-10-12T01:13:23.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:23.909 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:23.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:23.910 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:28.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:28.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:28.913 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:28.969 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:28.970 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:29.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:29.001+0000 7f1320993780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:29.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:29.002+0000 7f1320993780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:29.016 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:29.016 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:29.016 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:29.016 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:29.016 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:29.017 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:29.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:29.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.017 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:29.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:29.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:29.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:29.018 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:29.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:29.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:29.019 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:29.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:29.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:29.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:29.021 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:29.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:29.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:29.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:29.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:29.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:29.022 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:29.022 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:29.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:29.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:29.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:29.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:29.024 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:29.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:29.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:29.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:29.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:29.025 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:29.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:29.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:29.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:29.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:29.026 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:29.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:29.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:29.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:29.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:29.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:29.027 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:29.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:29.029 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:29.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:29.029 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:29Z" 2023-10-12T01:13:29.029 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:29.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:29Z" } 2023-10-12T01:13:29.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:29.031 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:29.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:29.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:34.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:34.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:34.035 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:34.091 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:34.092 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:34.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:34.124+0000 7f84018ac780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:34.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:34.125+0000 7f84018ac780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:34.138 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:34.139 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:34.139 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:34.139 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:34.139 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:34.141 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:34.141 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:34.141 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:34.141 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:34.141 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:34.142 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:34.142 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:34.142 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:34.142 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:34.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:34.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:34.143 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:34.143 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:34.143 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:34.144 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:34.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:34.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:34.145 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:34.145 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:34.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:34.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:34.146 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:34.146 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:34.146 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:34.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:34.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:34.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:34.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:34.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:34.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:34.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:34.149 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:34.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:34.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:34.150 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:34.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.150 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:34.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:34.151 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:34.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:34.151 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:34Z" 2023-10-12T01:13:34.151 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:34.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:34Z" } 2023-10-12T01:13:34.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:34.153 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:34.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:34.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:39.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:39.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:39.156 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:39.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:39.248+0000 7f7cee069780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:39.249+0000 7f7cee069780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:39.263 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:39Z" 2023-10-12T01:13:39.277 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:39.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:39Z" } 2023-10-12T01:13:39.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:39.279 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:39.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:39.280 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:44.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:44.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:44.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:44.339 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:44.339 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:44.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:44.371+0000 7fb7124cd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:44.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:44.372+0000 7fb7124cd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:44.386 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:44.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:44.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:44.391 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:44.391 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:44.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:44.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:44.392 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:44.392 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:44.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:44.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:44.394 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:44.394 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:44.394 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697072969.875315_647.1", 2023-10-12T01:13:44.394 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:44.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:44.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:44.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.875315Z", 2023-10-12T01:13:44.395 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:44.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:44.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:44.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:44.396 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:44.397 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:44.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:44.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:44.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:44.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:44.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:44.398 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:44.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:44.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:44.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:44.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.399 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:44.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:44.399 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:44.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:44.400 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:44Z" 2023-10-12T01:13:44.400 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:44.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697072969.875315_647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:09:29.875315Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:44Z" } 2023-10-12T01:13:44.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697072969.875315_647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.875315Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:44.402 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:44.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697072969.875315_647.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:44.403 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697072969.875315_647.1 2023-10-12T01:13:49.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:49.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:49.406 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:49.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:49.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:49.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:49.493+0000 7f27a1e2d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:49.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:49.494+0000 7f27a1e2d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:49.508 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:49.508 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:49.508 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:49.509 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:49.509 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:49.509 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:49.509 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:49.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.510 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:49.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:49.510 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:49.510 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:49.511 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:49.511 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:49.511 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:49.511 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:49.512 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:49.512 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:49.512 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:49.512 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:49.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:49.513 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:49.513 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:49.513 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:49.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:49.514 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:49.514 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:49.514 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:49.514 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:49.516 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:49.516 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:49.516 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073198.571815_648.1", 2023-10-12T01:13:49.516 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:49.516 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:49.517 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:49.517 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:13:18.571815Z", 2023-10-12T01:13:49.517 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:49.517 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:49.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:49.518 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:49.518 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:49.518 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:49.519 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:49.519 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:49.519 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:49.519 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:49.519 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:49.520 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:49.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:49.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:49.520 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:49.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.521 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:49.521 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:49.521 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:49.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:49.521 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:49Z" 2023-10-12T01:13:49.522 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:49.523 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073198.571815_648.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:13:18.571815Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:49Z" } 2023-10-12T01:13:49.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073198.571815_648.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:13:18.571815Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:49.524 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:49.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.571815_648.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:49.525 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073198.577963_651.1 target=1_1697073198.571815_648.1 2023-10-12T01:13:54.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:54.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:13:54.526 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:13:54.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:13:54.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:13:54.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:54.616+0000 7fef7cdbe780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:54.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:13:54.617+0000 7fef7cdbe780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:13:54.631 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:13:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:13:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:13:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:13:54.632 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:13:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:13:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:13:54.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:13:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:54.634 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:13:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:13:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:13:54.635 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:13:54.636 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:13:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:54.637 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:13:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:54.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:13:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:54.639 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073198.577963_651.1", 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:13:18.577963Z", 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:54.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:13:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:13:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:13:54.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:13:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:13:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:13:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:13:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:13:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:13:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:13:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:13:54.643 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:13:54.644 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:13:54Z" 2023-10-12T01:13:54.645 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:13:54.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073198.577963_651.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:13:18.577963Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:13:54Z" } 2023-10-12T01:13:54.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073198.577963_651.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:13:18.577963Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:13:54.647 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:54.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073198.577963_651.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:13:54.647 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:13:54.648 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:13:54.648 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:13:54.649 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:13:54.648+0000 7fcccebd8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 170701) UID: 0 2023-10-12T01:13:54.650 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:13:54.648+0000 7fccd2e417c0 -1 shutting down 2023-10-12T01:13:58.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:14:00.751 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:14:00.751 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:14:00.752 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:14:00.752 DEBUG:boto:Method: HEAD 2023-10-12T01:14:00.753 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:14:00.753 DEBUG:boto:Data: 2023-10-12T01:14:00.754 DEBUG:boto:Headers: {} 2023-10-12T01:14:00.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.755 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.755 DEBUG:boto:Params: {} 2023-10-12T01:14:00.755 DEBUG:boto:Token: None 2023-10-12T01:14:00.756 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/ 2023-10-12T01:14:00.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M= 2023-10-12T01:14:00.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M=', 'Content-Length': '0'} 2023-10-12T01:14:00.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad793d9caf95a3dd-0065274858-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.761 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:14:00.761 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:14:00.762 DEBUG:boto:Method: PUT 2023-10-12T01:14:00.762 DEBUG:boto:Path: /wafmoa-60/obj11 2023-10-12T01:14:00.763 DEBUG:boto:Data: 2023-10-12T01:14:00.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:14:00.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.764 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.764 DEBUG:boto:Params: {} 2023-10-12T01:14:00.765 DEBUG:boto:Token: None 2023-10-12T01:14:00.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/obj11 2023-10-12T01:14:00.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NlIvNQ//99klYR62vXseGgTMj/A= 2023-10-12T01:14:00.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NlIvNQ//99klYR62vXseGgTMj/A='} 2023-10-12T01:14:00.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000696bd583a76fb6e3-0065274858-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.776 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:14:00.776 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:14:00.777 DEBUG:boto:Method: HEAD 2023-10-12T01:14:00.777 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:14:00.778 DEBUG:boto:Data: 2023-10-12T01:14:00.778 DEBUG:boto:Headers: {} 2023-10-12T01:14:00.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.780 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.780 DEBUG:boto:Params: {} 2023-10-12T01:14:00.781 DEBUG:boto:Token: None 2023-10-12T01:14:00.782 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/ 2023-10-12T01:14:00.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M= 2023-10-12T01:14:00.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M=', 'Content-Length': '0'} 2023-10-12T01:14:00.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6e8981c8515ea41-0065274858-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.787 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:14:00.787 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:14:00.788 DEBUG:boto:Method: PUT 2023-10-12T01:14:00.788 DEBUG:boto:Path: /wafmoa-60/obj22 2023-10-12T01:14:00.789 DEBUG:boto:Data: 2023-10-12T01:14:00.789 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:14:00.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.790 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.790 DEBUG:boto:Params: {} 2023-10-12T01:14:00.791 DEBUG:boto:Token: None 2023-10-12T01:14:00.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/obj22 2023-10-12T01:14:00.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oHV0lFS/ll3RGPejlBNDFPdzykk= 2023-10-12T01:14:00.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oHV0lFS/ll3RGPejlBNDFPdzykk='} 2023-10-12T01:14:00.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f1864f458b7a3e7-0065274858-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.799 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:14:00.799 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:14:00.800 DEBUG:boto:Method: HEAD 2023-10-12T01:14:00.801 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:14:00.801 DEBUG:boto:Data: 2023-10-12T01:14:00.802 DEBUG:boto:Headers: {} 2023-10-12T01:14:00.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.803 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.804 DEBUG:boto:Params: {} 2023-10-12T01:14:00.805 DEBUG:boto:Token: None 2023-10-12T01:14:00.805 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/ 2023-10-12T01:14:00.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M= 2023-10-12T01:14:00.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M=', 'Content-Length': '0'} 2023-10-12T01:14:00.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b97b5de1bedd8f5-0065274858-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.809 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:14:00.810 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:14:00.810 DEBUG:boto:Method: PUT 2023-10-12T01:14:00.811 DEBUG:boto:Path: /wafmoa-60/obj33 2023-10-12T01:14:00.811 DEBUG:boto:Data: 2023-10-12T01:14:00.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:14:00.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.812 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.813 DEBUG:boto:Params: {} 2023-10-12T01:14:00.813 DEBUG:boto:Token: None 2023-10-12T01:14:00.813 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/obj33 2023-10-12T01:14:00.814 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UEu5DP+qJ9wJr8ledaFTPSCxbRs= 2023-10-12T01:14:00.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UEu5DP+qJ9wJr8ledaFTPSCxbRs='} 2023-10-12T01:14:00.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000123399992a84058b-0065274858-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.859 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:14:00.860 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:14:00.860 DEBUG:boto:Method: HEAD 2023-10-12T01:14:00.860 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:14:00.861 DEBUG:boto:Data: 2023-10-12T01:14:00.861 DEBUG:boto:Headers: {} 2023-10-12T01:14:00.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.862 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.862 DEBUG:boto:Params: {} 2023-10-12T01:14:00.863 DEBUG:boto:Token: None 2023-10-12T01:14:00.863 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/ 2023-10-12T01:14:00.864 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M= 2023-10-12T01:14:00.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M=', 'Content-Length': '0'} 2023-10-12T01:14:00.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ac291660ca6ffa5-0065274858-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.867 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:14:00.868 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:14:00.868 DEBUG:boto:Method: PUT 2023-10-12T01:14:00.869 DEBUG:boto:Path: /wafmoa-60/obj44 2023-10-12T01:14:00.869 DEBUG:boto:Data: 2023-10-12T01:14:00.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:14:00.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.870 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.871 DEBUG:boto:Params: {} 2023-10-12T01:14:00.871 DEBUG:boto:Token: None 2023-10-12T01:14:00.872 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/obj44 2023-10-12T01:14:00.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nrlQ9ewn6J22VZIkVbU6O6+5SxI= 2023-10-12T01:14:00.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nrlQ9ewn6J22VZIkVbU6O6+5SxI='} 2023-10-12T01:14:00.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df4d99f34239906d-0065274858-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.880 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:14:00.880 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:14:00.880 DEBUG:boto:Method: HEAD 2023-10-12T01:14:00.881 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:14:00.881 DEBUG:boto:Data: 2023-10-12T01:14:00.882 DEBUG:boto:Headers: {} 2023-10-12T01:14:00.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.882 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.883 DEBUG:boto:Params: {} 2023-10-12T01:14:00.883 DEBUG:boto:Token: None 2023-10-12T01:14:00.884 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/ 2023-10-12T01:14:00.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M= 2023-10-12T01:14:00.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/d8N6oJsfEQ87Qi9t1jBNDDxN9M=', 'Content-Length': '0'} 2023-10-12T01:14:00.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021c9ee1f193957ea-0065274858-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.888 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:14:00.889 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:14:00.889 DEBUG:boto:Method: PUT 2023-10-12T01:14:00.890 DEBUG:boto:Path: /wafmoa-60/obj55 2023-10-12T01:14:00.890 DEBUG:boto:Data: 2023-10-12T01:14:00.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:14:00.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:14:00.892 DEBUG:boto:Port: 8000 2023-10-12T01:14:00.892 DEBUG:boto:Params: {} 2023-10-12T01:14:00.893 DEBUG:boto:Token: None 2023-10-12T01:14:00.893 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:14:00 GMT /wafmoa-60/obj55 2023-10-12T01:14:00.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:t1phT+ugwoiiwl2EtCiLDOaO4MA= 2023-10-12T01:14:00.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:14:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:t1phT+ugwoiiwl2EtCiLDOaO4MA='} 2023-10-12T01:14:00.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e63f4620ef37d35-0065274858-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:14:00.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:14:00.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:14:00.944 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:14:01.002 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:14:01.003 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:14:01.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:01.036+0000 7fbae2a4c780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-12T01:14:01.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:01.038+0000 7fbae2a4c780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-12T01:14:01.060 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T01:14:01.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:14:01.063 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T01:14:01.063 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:14:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T01:14:01.086 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:14:01.086 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:14:01.086 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:14:02.087 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:14:02.109 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:14:02.109 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:14:02.110 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:14:02.110 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T01:14:02.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-60 2023-10-12T01:14:02.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:14:02.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:14:02.112 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:14:02.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:14:02.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:14:02.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:02.240+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:04.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:14:07.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:07.242+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:12.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:12.243+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:17.243+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:21.874 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:14:21.873+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:14:22.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:22.243+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:27.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:27.244+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:32.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:32.244+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:37.245+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:42.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:42.247+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:47.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:47.247+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:52.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:52.248+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:14:57.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:14:57.247+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:02.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:02.247+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:07.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:07.247+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:12.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:12.248+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:17.249+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:22.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:22.250+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:27.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:27.251+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:32.252+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:37.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:37.251+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:42.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:42.252+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:47.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:47.252+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:52.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:52.253+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:15:57.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:15:57.255+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:02.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:02.255+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:07.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:07.256+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:12.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:12.255+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:17.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:17.255+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:22.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:22.256+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:27.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:27.257+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:32.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:32.257+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:37.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:37.259+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:42.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:42.259+0000 7f058c9cb780 1 waiting to reach incremental sync.. 2023-10-12T01:16:47.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:47.261+0000 7f058c9cb780 1 bucket sync caught up with source: 2023-10-12T01:16:47.261 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.91.6, 00000000002.96.6, 00000000002.98.6] 2023-10-12T01:16:47.262 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.91.6, 00000000002.96.6, 00000000002.98.6] 2023-10-12T01:16:47.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:47.261+0000 7f058c9cb780 0 bucket checkpoint complete 2023-10-12T01:16:47.278 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-60 2023-10-12T01:16:47.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:16:47.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:16:47.280 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:16:47.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:47.342 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:47.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:47.382+0000 7f1cd37cf780 1 bucket sync caught up with source: 2023-10-12T01:16:47.382 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.230.6, 00000000002.353.6, 00000000002.192.6] 2023-10-12T01:16:47.382 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.230.6, 00000000002.353.6, 00000000002.192.6] 2023-10-12T01:16:47.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:47.382+0000 7f1cd37cf780 0 bucket checkpoint complete 2023-10-12T01:16:47.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-60 zones={a2, a1} 2023-10-12T01:16:47.396 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.396 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.397 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.397 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:16:47.398 DEBUG:boto:Data: 2023-10-12T01:16:47.398 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.399 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.399 DEBUG:boto:Params: {} 2023-10-12T01:16:47.400 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:16:47.400 DEBUG:boto:Token: None 2023-10-12T01:16:47.401 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q0oLDDRn+/2S60sR/by247uCb3A= 2023-10-12T01:16:47.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q0oLDDRn+/2S60sR/by247uCb3A=', 'Content-Length': '0'} 2023-10-12T01:16:47.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000de1ca580dcfd553-00652748ff-1ef8-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.408 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.408 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.409 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.409 DEBUG:boto:Path: /wafmoa-60/ 2023-10-12T01:16:47.410 DEBUG:boto:Data: 2023-10-12T01:16:47.410 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.411 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.411 DEBUG:boto:Params: {} 2023-10-12T01:16:47.412 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:16:47.412 DEBUG:boto:Token: None 2023-10-12T01:16:47.413 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q0oLDDRn+/2S60sR/by247uCb3A= 2023-10-12T01:16:47.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q0oLDDRn+/2S60sR/by247uCb3A=', 'Content-Length': '0'} 2023-10-12T01:16:47.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ad198cb81fa35d-00652748ff-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:16:47.418 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.419 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.419 DEBUG:boto:path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.420 DEBUG:boto:auth_path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.420 DEBUG:boto:Method: GET 2023-10-12T01:16:47.420 DEBUG:boto:Path: /wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.421 DEBUG:boto:Data: 2023-10-12T01:16:47.421 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.422 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.422 DEBUG:boto:Params: {} 2023-10-12T01:16:47.423 DEBUG:boto:Token: None 2023-10-12T01:16:47.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/?versions 2023-10-12T01:16:47.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c= 2023-10-12T01:16:47.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c=', 'Content-Length': '0'} 2023-10-12T01:16:47.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d51658080796e8c-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.427 DEBUG:boto:b'wafmoa-60999falseobj11nulltrue2023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:16:47.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:16:47.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:16:47.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:16:47.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:16:47.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:16:47.430 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.431 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.431 DEBUG:boto:path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.432 DEBUG:boto:auth_path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.432 DEBUG:boto:Method: GET 2023-10-12T01:16:47.433 DEBUG:boto:Path: /wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.433 DEBUG:boto:Data: 2023-10-12T01:16:47.433 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.434 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.434 DEBUG:boto:Params: {} 2023-10-12T01:16:47.435 DEBUG:boto:Token: None 2023-10-12T01:16:47.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/?versions 2023-10-12T01:16:47.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c= 2023-10-12T01:16:47.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c=', 'Content-Length': '0'} 2023-10-12T01:16:47.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1de9b56340d2e0-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.439 DEBUG:boto:b'wafmoa-60999falseobj11nulltrue2023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:16:47.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:16:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:16:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:16:47.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:16:47.442 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.442 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.443 DEBUG:boto:path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.443 DEBUG:boto:auth_path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.443 DEBUG:boto:Method: GET 2023-10-12T01:16:47.444 DEBUG:boto:Path: /wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.444 DEBUG:boto:Data: 2023-10-12T01:16:47.445 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.445 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.446 DEBUG:boto:Params: {} 2023-10-12T01:16:47.446 DEBUG:boto:Token: None 2023-10-12T01:16:47.447 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/?versions 2023-10-12T01:16:47.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c= 2023-10-12T01:16:47.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c=', 'Content-Length': '0'} 2023-10-12T01:16:47.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008250aff3755e8673-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.450 DEBUG:boto:b'wafmoa-60999falseobj11nulltrue2023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.451 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.452 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.452 DEBUG:boto:path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.452 DEBUG:boto:auth_path=/wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.453 DEBUG:boto:Method: GET 2023-10-12T01:16:47.453 DEBUG:boto:Path: /wafmoa-60/?versions&max-keys=999 2023-10-12T01:16:47.454 DEBUG:boto:Data: 2023-10-12T01:16:47.454 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.455 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.455 DEBUG:boto:Params: {} 2023-10-12T01:16:47.456 DEBUG:boto:Token: None 2023-10-12T01:16:47.456 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/?versions 2023-10-12T01:16:47.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c= 2023-10-12T01:16:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:si9J8FnakKa05lhITbKXls6Qi4c=', 'Content-Length': '0'} 2023-10-12T01:16:47.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc62fa253aa09fd4-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.460 DEBUG:boto:b'wafmoa-60999falseobj11nulltrue2023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:16:47.461 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.462 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.462 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.463 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.463 DEBUG:boto:Method: GET 2023-10-12T01:16:47.464 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.464 DEBUG:boto:Data: 2023-10-12T01:16:47.465 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.465 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.466 DEBUG:boto:Params: {} 2023-10-12T01:16:47.466 DEBUG:boto:Token: None 2023-10-12T01:16:47.467 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.467 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o= 2023-10-12T01:16:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o=', 'Content-Length': '0'} 2023-10-12T01:16:47.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e456c6626a068d-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.471 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.471 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.471 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.472 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.472 DEBUG:boto:Method: GET 2023-10-12T01:16:47.473 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.473 DEBUG:boto:Data: 2023-10-12T01:16:47.473 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.474 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.475 DEBUG:boto:Params: {} 2023-10-12T01:16:47.475 DEBUG:boto:Token: None 2023-10-12T01:16:47.476 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o= 2023-10-12T01:16:47.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o=', 'Content-Length': '0'} 2023-10-12T01:16:47.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ef4e68cc4cbd80-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.479 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.479 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.480 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.480 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.481 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.481 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.481 DEBUG:boto:Data: 2023-10-12T01:16:47.482 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.483 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.483 DEBUG:boto:Params: {} 2023-10-12T01:16:47.484 DEBUG:boto:Token: None 2023-10-12T01:16:47.484 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fu3TmLnjWvFqyYK63h8Yh4n1gSM= 2023-10-12T01:16:47.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fu3TmLnjWvFqyYK63h8Yh4n1gSM=', 'Content-Length': '0'} 2023-10-12T01:16:47.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b3efe6cdc79c45-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.487 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.488 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.488 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.488 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.489 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.489 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.489 DEBUG:boto:Data: 2023-10-12T01:16:47.490 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.491 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.491 DEBUG:boto:Params: {} 2023-10-12T01:16:47.491 DEBUG:boto:Token: None 2023-10-12T01:16:47.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fu3TmLnjWvFqyYK63h8Yh4n1gSM= 2023-10-12T01:16:47.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fu3TmLnjWvFqyYK63h8Yh4n1gSM=', 'Content-Length': '0'} 2023-10-12T01:16:47.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f51c794e6e7f06-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:16:47.495 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.496 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.496 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.496 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.497 DEBUG:boto:Method: GET 2023-10-12T01:16:47.497 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.498 DEBUG:boto:Data: 2023-10-12T01:16:47.498 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.499 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.499 DEBUG:boto:Params: {} 2023-10-12T01:16:47.500 DEBUG:boto:Token: None 2023-10-12T01:16:47.500 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o= 2023-10-12T01:16:47.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o=', 'Content-Length': '0'} 2023-10-12T01:16:47.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6064dbe57c2a9d-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.546 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.547 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.547 DEBUG:boto:path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.548 DEBUG:boto:auth_path=/wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.548 DEBUG:boto:Method: GET 2023-10-12T01:16:47.549 DEBUG:boto:Path: /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.549 DEBUG:boto:Data: 2023-10-12T01:16:47.550 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.551 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.551 DEBUG:boto:Params: {} 2023-10-12T01:16:47.551 DEBUG:boto:Token: None 2023-10-12T01:16:47.552 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11?versionId=null 2023-10-12T01:16:47.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o= 2023-10-12T01:16:47.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hzscCITCDisNbcmriTNwAUoSg0o=', 'Content-Length': '0'} 2023-10-12T01:16:47.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f9fdcf35259414-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.556 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.556 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.557 DEBUG:boto:path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.557 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.557 DEBUG:boto:Method: GET 2023-10-12T01:16:47.558 DEBUG:boto:Path: /wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.558 DEBUG:boto:Data: 2023-10-12T01:16:47.559 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.560 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.560 DEBUG:boto:Params: {} 2023-10-12T01:16:47.560 DEBUG:boto:Token: None 2023-10-12T01:16:47.561 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.561 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005565925d29434412-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.565 DEBUG:boto:b'wafmoa-60obj111000falseobj112023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.566 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.566 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.566 DEBUG:boto:path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.567 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.567 DEBUG:boto:Method: GET 2023-10-12T01:16:47.568 DEBUG:boto:Path: /wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.568 DEBUG:boto:Data: 2023-10-12T01:16:47.569 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.569 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.570 DEBUG:boto:Params: {} 2023-10-12T01:16:47.570 DEBUG:boto:Token: None 2023-10-12T01:16:47.571 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.571 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009887db2c9c6f885c-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.575 DEBUG:boto:b'wafmoa-60obj111000falseobj112023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.575 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.576 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.576 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.577 DEBUG:boto:Path: /wafmoa-60/obj11 2023-10-12T01:16:47.577 DEBUG:boto:Data: 2023-10-12T01:16:47.577 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.578 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.579 DEBUG:boto:Params: {} 2023-10-12T01:16:47.579 DEBUG:boto:Token: None 2023-10-12T01:16:47.579 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11 2023-10-12T01:16:47.580 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FLyChnaItDuGlSNlhsRcuRkG7LM= 2023-10-12T01:16:47.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FLyChnaItDuGlSNlhsRcuRkG7LM=', 'Content-Length': '0'} 2023-10-12T01:16:47.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002482279127e25f7-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.582 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.583 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.583 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.584 DEBUG:boto:Path: /wafmoa-60/obj11 2023-10-12T01:16:47.584 DEBUG:boto:Data: 2023-10-12T01:16:47.584 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.585 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.586 DEBUG:boto:Params: {} 2023-10-12T01:16:47.586 DEBUG:boto:Token: None 2023-10-12T01:16:47.587 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11 2023-10-12T01:16:47.587 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FLyChnaItDuGlSNlhsRcuRkG7LM= 2023-10-12T01:16:47.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FLyChnaItDuGlSNlhsRcuRkG7LM=', 'Content-Length': '0'} 2023-10-12T01:16:47.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098b35ebced5681ce-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:16:47.590 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.591 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.591 DEBUG:boto:Method: GET 2023-10-12T01:16:47.592 DEBUG:boto:Path: /wafmoa-60/obj11 2023-10-12T01:16:47.592 DEBUG:boto:Data: 2023-10-12T01:16:47.592 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.593 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.594 DEBUG:boto:Params: {} 2023-10-12T01:16:47.594 DEBUG:boto:Token: None 2023-10-12T01:16:47.595 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11 2023-10-12T01:16:47.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NwWPMa1xLMEFgfS+W/uDg5MA9W0= 2023-10-12T01:16:47.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NwWPMa1xLMEFgfS+W/uDg5MA9W0=', 'Content-Length': '0'} 2023-10-12T01:16:47.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89d51659e24282a-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.598 DEBUG:boto:path=/wafmoa-60/obj11 2023-10-12T01:16:47.598 DEBUG:boto:auth_path=/wafmoa-60/obj11 2023-10-12T01:16:47.599 DEBUG:boto:Method: GET 2023-10-12T01:16:47.599 DEBUG:boto:Path: /wafmoa-60/obj11 2023-10-12T01:16:47.600 DEBUG:boto:Data: 2023-10-12T01:16:47.600 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.601 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.601 DEBUG:boto:Params: {} 2023-10-12T01:16:47.602 DEBUG:boto:Token: None 2023-10-12T01:16:47.602 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj11 2023-10-12T01:16:47.603 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NwWPMa1xLMEFgfS+W/uDg5MA9W0= 2023-10-12T01:16:47.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NwWPMa1xLMEFgfS+W/uDg5MA9W0=', 'Content-Length': '0'} 2023-10-12T01:16:47.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6706f1c257a1da-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.606 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.606 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.606 DEBUG:boto:path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.607 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.607 DEBUG:boto:Method: GET 2023-10-12T01:16:47.608 DEBUG:boto:Path: /wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.608 DEBUG:boto:Data: 2023-10-12T01:16:47.608 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.609 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.610 DEBUG:boto:Params: {} 2023-10-12T01:16:47.610 DEBUG:boto:Token: None 2023-10-12T01:16:47.611 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.611 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ef09507cdea608-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.614 DEBUG:boto:b'wafmoa-60obj111000falseobj112023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.615 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.615 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.616 DEBUG:boto:path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.616 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.616 DEBUG:boto:Method: GET 2023-10-12T01:16:47.617 DEBUG:boto:Path: /wafmoa-60/?prefix=obj11 2023-10-12T01:16:47.617 DEBUG:boto:Data: 2023-10-12T01:16:47.618 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.618 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.619 DEBUG:boto:Params: {} 2023-10-12T01:16:47.619 DEBUG:boto:Token: None 2023-10-12T01:16:47.620 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.620 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b7fb4feec99e82-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.623 DEBUG:boto:b'wafmoa-60obj111000falseobj112023-10-12T01:14:00.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:16:47.624 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.625 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.625 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.626 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.626 DEBUG:boto:Method: GET 2023-10-12T01:16:47.627 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.627 DEBUG:boto:Data: 2023-10-12T01:16:47.627 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.628 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.629 DEBUG:boto:Params: {} 2023-10-12T01:16:47.629 DEBUG:boto:Token: None 2023-10-12T01:16:47.630 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.630 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA= 2023-10-12T01:16:47.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA=', 'Content-Length': '0'} 2023-10-12T01:16:47.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa62f510e40545e0-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.633 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.634 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.634 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.635 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.635 DEBUG:boto:Method: GET 2023-10-12T01:16:47.635 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.636 DEBUG:boto:Data: 2023-10-12T01:16:47.636 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.637 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.637 DEBUG:boto:Params: {} 2023-10-12T01:16:47.638 DEBUG:boto:Token: None 2023-10-12T01:16:47.638 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.639 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA= 2023-10-12T01:16:47.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA=', 'Content-Length': '0'} 2023-10-12T01:16:47.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c606fafad07c80b-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.682 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.683 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.683 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.683 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.684 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.684 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.685 DEBUG:boto:Data: 2023-10-12T01:16:47.685 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.686 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.686 DEBUG:boto:Params: {} 2023-10-12T01:16:47.687 DEBUG:boto:Token: None 2023-10-12T01:16:47.687 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.687 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tDR1xadg3+Lq4bZNYWruzF8wF+4= 2023-10-12T01:16:47.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tDR1xadg3+Lq4bZNYWruzF8wF+4=', 'Content-Length': '0'} 2023-10-12T01:16:47.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3db1f9bde8760e4-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.691 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.691 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.691 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.692 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.692 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.693 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.693 DEBUG:boto:Data: 2023-10-12T01:16:47.693 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.694 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.695 DEBUG:boto:Params: {} 2023-10-12T01:16:47.695 DEBUG:boto:Token: None 2023-10-12T01:16:47.696 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.696 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tDR1xadg3+Lq4bZNYWruzF8wF+4= 2023-10-12T01:16:47.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tDR1xadg3+Lq4bZNYWruzF8wF+4=', 'Content-Length': '0'} 2023-10-12T01:16:47.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53bd1b7bf9f3266-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:16:47.699 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.699 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.700 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.700 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.701 DEBUG:boto:Method: GET 2023-10-12T01:16:47.701 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.702 DEBUG:boto:Data: 2023-10-12T01:16:47.702 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.703 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.704 DEBUG:boto:Params: {} 2023-10-12T01:16:47.704 DEBUG:boto:Token: None 2023-10-12T01:16:47.704 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.705 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA= 2023-10-12T01:16:47.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA=', 'Content-Length': '0'} 2023-10-12T01:16:47.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53f37171d7a6eee-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.708 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.708 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.708 DEBUG:boto:path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.709 DEBUG:boto:auth_path=/wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.709 DEBUG:boto:Method: GET 2023-10-12T01:16:47.710 DEBUG:boto:Path: /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.710 DEBUG:boto:Data: 2023-10-12T01:16:47.711 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.711 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.712 DEBUG:boto:Params: {} 2023-10-12T01:16:47.712 DEBUG:boto:Token: None 2023-10-12T01:16:47.713 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22?versionId=null 2023-10-12T01:16:47.713 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA= 2023-10-12T01:16:47.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tIrsDbm0DluuiVFWegXzZn4IHGA=', 'Content-Length': '0'} 2023-10-12T01:16:47.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002129a029e2fe2dce-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.717 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.717 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.717 DEBUG:boto:path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.718 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.718 DEBUG:boto:Method: GET 2023-10-12T01:16:47.719 DEBUG:boto:Path: /wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.719 DEBUG:boto:Data: 2023-10-12T01:16:47.720 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.720 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.721 DEBUG:boto:Params: {} 2023-10-12T01:16:47.721 DEBUG:boto:Token: None 2023-10-12T01:16:47.722 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753e1feeae4e4897-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.725 DEBUG:boto:b'wafmoa-60obj221000falseobj222023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.726 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.727 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.727 DEBUG:boto:path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.727 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.728 DEBUG:boto:Method: GET 2023-10-12T01:16:47.728 DEBUG:boto:Path: /wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.729 DEBUG:boto:Data: 2023-10-12T01:16:47.729 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.730 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.730 DEBUG:boto:Params: {} 2023-10-12T01:16:47.731 DEBUG:boto:Token: None 2023-10-12T01:16:47.731 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3bd74d5c794590-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.735 DEBUG:boto:b'wafmoa-60obj221000falseobj222023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.735 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.736 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.736 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.737 DEBUG:boto:Path: /wafmoa-60/obj22 2023-10-12T01:16:47.737 DEBUG:boto:Data: 2023-10-12T01:16:47.737 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.738 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.739 DEBUG:boto:Params: {} 2023-10-12T01:16:47.739 DEBUG:boto:Token: None 2023-10-12T01:16:47.739 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22 2023-10-12T01:16:47.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mlazLBj6bNH9Sfafw3gMX/imXMo= 2023-10-12T01:16:47.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mlazLBj6bNH9Sfafw3gMX/imXMo=', 'Content-Length': '0'} 2023-10-12T01:16:47.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c8dccca1103465-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.743 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.743 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.743 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.744 DEBUG:boto:Path: /wafmoa-60/obj22 2023-10-12T01:16:47.744 DEBUG:boto:Data: 2023-10-12T01:16:47.745 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.745 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.746 DEBUG:boto:Params: {} 2023-10-12T01:16:47.746 DEBUG:boto:Token: None 2023-10-12T01:16:47.747 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22 2023-10-12T01:16:47.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mlazLBj6bNH9Sfafw3gMX/imXMo= 2023-10-12T01:16:47.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mlazLBj6bNH9Sfafw3gMX/imXMo=', 'Content-Length': '0'} 2023-10-12T01:16:47.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47700bb2fc94110-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:16:47.751 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.751 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.752 DEBUG:boto:Method: GET 2023-10-12T01:16:47.752 DEBUG:boto:Path: /wafmoa-60/obj22 2023-10-12T01:16:47.752 DEBUG:boto:Data: 2023-10-12T01:16:47.753 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.754 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.754 DEBUG:boto:Params: {} 2023-10-12T01:16:47.755 DEBUG:boto:Token: None 2023-10-12T01:16:47.755 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22 2023-10-12T01:16:47.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DpXtrciGY2AKdIUICBhYit9tT2o= 2023-10-12T01:16:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DpXtrciGY2AKdIUICBhYit9tT2o=', 'Content-Length': '0'} 2023-10-12T01:16:47.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde2adac62694feb-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.758 DEBUG:boto:path=/wafmoa-60/obj22 2023-10-12T01:16:47.759 DEBUG:boto:auth_path=/wafmoa-60/obj22 2023-10-12T01:16:47.759 DEBUG:boto:Method: GET 2023-10-12T01:16:47.760 DEBUG:boto:Path: /wafmoa-60/obj22 2023-10-12T01:16:47.760 DEBUG:boto:Data: 2023-10-12T01:16:47.760 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.761 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.762 DEBUG:boto:Params: {} 2023-10-12T01:16:47.762 DEBUG:boto:Token: None 2023-10-12T01:16:47.763 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj22 2023-10-12T01:16:47.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DpXtrciGY2AKdIUICBhYit9tT2o= 2023-10-12T01:16:47.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DpXtrciGY2AKdIUICBhYit9tT2o=', 'Content-Length': '0'} 2023-10-12T01:16:47.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbfe30deaa984dd-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.766 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.767 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.767 DEBUG:boto:path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.768 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.768 DEBUG:boto:Method: GET 2023-10-12T01:16:47.768 DEBUG:boto:Path: /wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.769 DEBUG:boto:Data: 2023-10-12T01:16:47.769 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.770 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.771 DEBUG:boto:Params: {} 2023-10-12T01:16:47.771 DEBUG:boto:Token: None 2023-10-12T01:16:47.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af92a92bccf92c00-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.775 DEBUG:boto:b'wafmoa-60obj221000falseobj222023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.776 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.776 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.777 DEBUG:boto:path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.777 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.778 DEBUG:boto:Method: GET 2023-10-12T01:16:47.778 DEBUG:boto:Path: /wafmoa-60/?prefix=obj22 2023-10-12T01:16:47.778 DEBUG:boto:Data: 2023-10-12T01:16:47.779 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.780 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.780 DEBUG:boto:Params: {} 2023-10-12T01:16:47.781 DEBUG:boto:Token: None 2023-10-12T01:16:47.781 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663ee9433f8e677c-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.785 DEBUG:boto:b'wafmoa-60obj221000falseobj222023-10-12T01:14:00.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:16:47.786 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.786 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.787 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.787 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.788 DEBUG:boto:Method: GET 2023-10-12T01:16:47.788 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.788 DEBUG:boto:Data: 2023-10-12T01:16:47.789 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.790 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.790 DEBUG:boto:Params: {} 2023-10-12T01:16:47.791 DEBUG:boto:Token: None 2023-10-12T01:16:47.791 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw= 2023-10-12T01:16:47.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw=', 'Content-Length': '0'} 2023-10-12T01:16:47.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d59899d8d4e63e6-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.794 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.795 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.795 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.796 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.796 DEBUG:boto:Method: GET 2023-10-12T01:16:47.796 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.797 DEBUG:boto:Data: 2023-10-12T01:16:47.797 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.798 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.799 DEBUG:boto:Params: {} 2023-10-12T01:16:47.799 DEBUG:boto:Token: None 2023-10-12T01:16:47.799 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.800 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw= 2023-10-12T01:16:47.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw=', 'Content-Length': '0'} 2023-10-12T01:16:47.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011c3a70c410b8ce-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.803 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.803 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.803 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.804 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.804 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.805 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.805 DEBUG:boto:Data: 2023-10-12T01:16:47.805 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.806 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.807 DEBUG:boto:Params: {} 2023-10-12T01:16:47.807 DEBUG:boto:Token: None 2023-10-12T01:16:47.808 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.808 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qBlF48tVqK2vH8lCYDUOpElfams= 2023-10-12T01:16:47.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qBlF48tVqK2vH8lCYDUOpElfams=', 'Content-Length': '0'} 2023-10-12T01:16:47.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85a7a59d6c6da6d-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.811 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.811 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.812 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.812 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.812 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.813 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.813 DEBUG:boto:Data: 2023-10-12T01:16:47.814 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.815 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.815 DEBUG:boto:Params: {} 2023-10-12T01:16:47.816 DEBUG:boto:Token: None 2023-10-12T01:16:47.816 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qBlF48tVqK2vH8lCYDUOpElfams= 2023-10-12T01:16:47.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qBlF48tVqK2vH8lCYDUOpElfams=', 'Content-Length': '0'} 2023-10-12T01:16:47.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006597699458ccf13a-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:16:47.820 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.820 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.821 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.821 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.821 DEBUG:boto:Method: GET 2023-10-12T01:16:47.822 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.822 DEBUG:boto:Data: 2023-10-12T01:16:47.823 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.824 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.824 DEBUG:boto:Params: {} 2023-10-12T01:16:47.825 DEBUG:boto:Token: None 2023-10-12T01:16:47.825 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.826 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw= 2023-10-12T01:16:47.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw=', 'Content-Length': '0'} 2023-10-12T01:16:47.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44a33ecb60861b1-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.870 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.871 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.871 DEBUG:boto:path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.872 DEBUG:boto:auth_path=/wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.872 DEBUG:boto:Method: GET 2023-10-12T01:16:47.873 DEBUG:boto:Path: /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.873 DEBUG:boto:Data: 2023-10-12T01:16:47.873 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.874 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.875 DEBUG:boto:Params: {} 2023-10-12T01:16:47.875 DEBUG:boto:Token: None 2023-10-12T01:16:47.876 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33?versionId=null 2023-10-12T01:16:47.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw= 2023-10-12T01:16:47.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8SET91Q1TEaLpy1dAmXnWBzLAYw=', 'Content-Length': '0'} 2023-10-12T01:16:47.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025612f89096b823d-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.879 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.880 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.880 DEBUG:boto:path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.881 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.881 DEBUG:boto:Method: GET 2023-10-12T01:16:47.881 DEBUG:boto:Path: /wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.882 DEBUG:boto:Data: 2023-10-12T01:16:47.882 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.883 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.884 DEBUG:boto:Params: {} 2023-10-12T01:16:47.884 DEBUG:boto:Token: None 2023-10-12T01:16:47.884 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d83d379ad529d219-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.888 DEBUG:boto:b'wafmoa-60obj331000falseobj332023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.889 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.889 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.890 DEBUG:boto:path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.890 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.891 DEBUG:boto:Method: GET 2023-10-12T01:16:47.891 DEBUG:boto:Path: /wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.891 DEBUG:boto:Data: 2023-10-12T01:16:47.892 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.893 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.893 DEBUG:boto:Params: {} 2023-10-12T01:16:47.894 DEBUG:boto:Token: None 2023-10-12T01:16:47.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea00fd9e263e353-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.898 DEBUG:boto:b'wafmoa-60obj331000falseobj332023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.899 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.899 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.900 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.900 DEBUG:boto:Path: /wafmoa-60/obj33 2023-10-12T01:16:47.900 DEBUG:boto:Data: 2023-10-12T01:16:47.901 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.902 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.902 DEBUG:boto:Params: {} 2023-10-12T01:16:47.903 DEBUG:boto:Token: None 2023-10-12T01:16:47.903 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33 2023-10-12T01:16:47.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nb2nLkbtcHhTetqutijIsmUpHl8= 2023-10-12T01:16:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nb2nLkbtcHhTetqutijIsmUpHl8=', 'Content-Length': '0'} 2023-10-12T01:16:47.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c8be8551866d91-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.906 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.907 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.907 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.908 DEBUG:boto:Path: /wafmoa-60/obj33 2023-10-12T01:16:47.908 DEBUG:boto:Data: 2023-10-12T01:16:47.909 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.909 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.910 DEBUG:boto:Params: {} 2023-10-12T01:16:47.910 DEBUG:boto:Token: None 2023-10-12T01:16:47.911 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33 2023-10-12T01:16:47.911 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nb2nLkbtcHhTetqutijIsmUpHl8= 2023-10-12T01:16:47.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nb2nLkbtcHhTetqutijIsmUpHl8=', 'Content-Length': '0'} 2023-10-12T01:16:47.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001667cc71979f85f0-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:16:47.914 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.915 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.915 DEBUG:boto:Method: GET 2023-10-12T01:16:47.915 DEBUG:boto:Path: /wafmoa-60/obj33 2023-10-12T01:16:47.916 DEBUG:boto:Data: 2023-10-12T01:16:47.916 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.917 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.918 DEBUG:boto:Params: {} 2023-10-12T01:16:47.918 DEBUG:boto:Token: None 2023-10-12T01:16:47.919 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33 2023-10-12T01:16:47.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H/i0cHZO7zxyZA19BeNHgKaQCNg= 2023-10-12T01:16:47.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H/i0cHZO7zxyZA19BeNHgKaQCNg=', 'Content-Length': '0'} 2023-10-12T01:16:47.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c5f0ce5f71a033-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.922 DEBUG:boto:path=/wafmoa-60/obj33 2023-10-12T01:16:47.923 DEBUG:boto:auth_path=/wafmoa-60/obj33 2023-10-12T01:16:47.923 DEBUG:boto:Method: GET 2023-10-12T01:16:47.923 DEBUG:boto:Path: /wafmoa-60/obj33 2023-10-12T01:16:47.924 DEBUG:boto:Data: 2023-10-12T01:16:47.924 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.925 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.926 DEBUG:boto:Params: {} 2023-10-12T01:16:47.926 DEBUG:boto:Token: None 2023-10-12T01:16:47.927 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj33 2023-10-12T01:16:47.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H/i0cHZO7zxyZA19BeNHgKaQCNg= 2023-10-12T01:16:47.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H/i0cHZO7zxyZA19BeNHgKaQCNg=', 'Content-Length': '0'} 2023-10-12T01:16:47.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae13fc4e8c19e89-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.930 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.931 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.931 DEBUG:boto:path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.932 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.932 DEBUG:boto:Method: GET 2023-10-12T01:16:47.933 DEBUG:boto:Path: /wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.933 DEBUG:boto:Data: 2023-10-12T01:16:47.933 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.934 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.935 DEBUG:boto:Params: {} 2023-10-12T01:16:47.935 DEBUG:boto:Token: None 2023-10-12T01:16:47.936 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f84c9cccfa700b5-00652748ff-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.939 DEBUG:boto:b'wafmoa-60obj331000falseobj332023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.940 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:47.940 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:47.941 DEBUG:boto:path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.941 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.942 DEBUG:boto:Method: GET 2023-10-12T01:16:47.942 DEBUG:boto:Path: /wafmoa-60/?prefix=obj33 2023-10-12T01:16:47.943 DEBUG:boto:Data: 2023-10-12T01:16:47.943 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.944 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.944 DEBUG:boto:Params: {} 2023-10-12T01:16:47.944 DEBUG:boto:Token: None 2023-10-12T01:16:47.945 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/ 2023-10-12T01:16:47.945 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss= 2023-10-12T01:16:47.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EBpfo0OzR0PimwvZVUYdp8SQyss=', 'Content-Length': '0'} 2023-10-12T01:16:47.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084fce0ee583522bf-00652748ff-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.949 DEBUG:boto:b'wafmoa-60obj331000falseobj332023-10-12T01:14:00.817Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:47.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:16:47.950 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:47.950 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:47.951 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.951 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.952 DEBUG:boto:Method: GET 2023-10-12T01:16:47.952 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.952 DEBUG:boto:Data: 2023-10-12T01:16:47.953 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.954 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.954 DEBUG:boto:Params: {} 2023-10-12T01:16:47.955 DEBUG:boto:Token: None 2023-10-12T01:16:47.955 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.955 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY= 2023-10-12T01:16:47.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY=', 'Content-Length': '0'} 2023-10-12T01:16:47.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f4b5ac1db743ad-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.958 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:47.959 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:47.959 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.960 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.960 DEBUG:boto:Method: GET 2023-10-12T01:16:47.960 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.961 DEBUG:boto:Data: 2023-10-12T01:16:47.961 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.962 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.963 DEBUG:boto:Params: {} 2023-10-12T01:16:47.963 DEBUG:boto:Token: None 2023-10-12T01:16:47.963 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.964 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY= 2023-10-12T01:16:47.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY=', 'Content-Length': '0'} 2023-10-12T01:16:47.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b05279572c576a-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.967 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:47.967 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:47.968 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.968 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.968 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.969 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.969 DEBUG:boto:Data: 2023-10-12T01:16:47.970 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.970 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.971 DEBUG:boto:Params: {} 2023-10-12T01:16:47.971 DEBUG:boto:Token: None 2023-10-12T01:16:47.972 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:70DxgTlhOsF2Wd0OhQOkxCVOcso= 2023-10-12T01:16:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:70DxgTlhOsF2Wd0OhQOkxCVOcso=', 'Content-Length': '0'} 2023-10-12T01:16:47.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e83317c343fc6e8-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.975 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:47.975 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:47.976 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.976 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.977 DEBUG:boto:Method: HEAD 2023-10-12T01:16:47.977 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.977 DEBUG:boto:Data: 2023-10-12T01:16:47.978 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:47.979 DEBUG:boto:Port: 8000 2023-10-12T01:16:47.979 DEBUG:boto:Params: {} 2023-10-12T01:16:47.979 DEBUG:boto:Token: None 2023-10-12T01:16:47.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:70DxgTlhOsF2Wd0OhQOkxCVOcso= 2023-10-12T01:16:47.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:70DxgTlhOsF2Wd0OhQOkxCVOcso=', 'Content-Length': '0'} 2023-10-12T01:16:47.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e22c4f0c0d503b2-00652748ff-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:47.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:16:47.983 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:47.984 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:47.984 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.984 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.985 DEBUG:boto:Method: GET 2023-10-12T01:16:47.985 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.986 DEBUG:boto:Data: 2023-10-12T01:16:47.986 DEBUG:boto:Headers: {} 2023-10-12T01:16:47.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:47.987 DEBUG:boto:Port: 8001 2023-10-12T01:16:47.987 DEBUG:boto:Params: {} 2023-10-12T01:16:47.988 DEBUG:boto:Token: None 2023-10-12T01:16:47.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:47 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:47.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY= 2023-10-12T01:16:47.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FANA/tWaEWad8N2ETpwjNJyAKMY=', 'Content-Length': '0'} 2023-10-12T01:16:47.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000806dcae54e91e2d-00652748ff-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.034 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:48.035 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:48.035 DEBUG:boto:path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:48.036 DEBUG:boto:auth_path=/wafmoa-60/obj44?versionId=null 2023-10-12T01:16:48.036 DEBUG:boto:Method: GET 2023-10-12T01:16:48.036 DEBUG:boto:Path: /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:48.037 DEBUG:boto:Data: 2023-10-12T01:16:48.037 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.038 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.038 DEBUG:boto:Params: {} 2023-10-12T01:16:48.039 DEBUG:boto:Token: None 2023-10-12T01:16:48.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj44?versionId=null 2023-10-12T01:16:48.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xW+jVQ5GOgNOO30TMZcneocoRxM= 2023-10-12T01:16:48.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xW+jVQ5GOgNOO30TMZcneocoRxM=', 'Content-Length': '0'} 2023-10-12T01:16:48.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a03d4b6baf5e0b4-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.043 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.043 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.044 DEBUG:boto:path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.044 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.044 DEBUG:boto:Method: GET 2023-10-12T01:16:48.045 DEBUG:boto:Path: /wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.045 DEBUG:boto:Data: 2023-10-12T01:16:48.046 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.046 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.047 DEBUG:boto:Params: {} 2023-10-12T01:16:48.047 DEBUG:boto:Token: None 2023-10-12T01:16:48.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.048 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a2404aa1cce0c41-0065274900-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.051 DEBUG:boto:b'wafmoa-60obj441000falseobj442023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.052 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.053 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.053 DEBUG:boto:path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.054 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.054 DEBUG:boto:Method: GET 2023-10-12T01:16:48.054 DEBUG:boto:Path: /wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.055 DEBUG:boto:Data: 2023-10-12T01:16:48.055 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.056 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.057 DEBUG:boto:Params: {} 2023-10-12T01:16:48.057 DEBUG:boto:Token: None 2023-10-12T01:16:48.058 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa871062689cc08-0065274900-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.102 DEBUG:boto:b'wafmoa-60obj441000falseobj442023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.103 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:48.103 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:48.104 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.104 DEBUG:boto:Path: /wafmoa-60/obj44 2023-10-12T01:16:48.105 DEBUG:boto:Data: 2023-10-12T01:16:48.105 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.106 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.107 DEBUG:boto:Params: {} 2023-10-12T01:16:48.107 DEBUG:boto:Token: None 2023-10-12T01:16:48.107 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj44 2023-10-12T01:16:48.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zjSFN5JhH/Rx0MGZBjuD0OmLW9k= 2023-10-12T01:16:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zjSFN5JhH/Rx0MGZBjuD0OmLW9k=', 'Content-Length': '0'} 2023-10-12T01:16:48.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ea0e88cf18b931-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.111 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:48.111 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:48.112 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.112 DEBUG:boto:Path: /wafmoa-60/obj44 2023-10-12T01:16:48.113 DEBUG:boto:Data: 2023-10-12T01:16:48.113 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.114 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.114 DEBUG:boto:Params: {} 2023-10-12T01:16:48.115 DEBUG:boto:Token: None 2023-10-12T01:16:48.115 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj44 2023-10-12T01:16:48.115 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zjSFN5JhH/Rx0MGZBjuD0OmLW9k= 2023-10-12T01:16:48.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zjSFN5JhH/Rx0MGZBjuD0OmLW9k=', 'Content-Length': '0'} 2023-10-12T01:16:48.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ae902d463d58d3-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:16:48.119 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:48.119 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:48.119 DEBUG:boto:Method: GET 2023-10-12T01:16:48.120 DEBUG:boto:Path: /wafmoa-60/obj44 2023-10-12T01:16:48.120 DEBUG:boto:Data: 2023-10-12T01:16:48.121 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.122 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.122 DEBUG:boto:Params: {} 2023-10-12T01:16:48.123 DEBUG:boto:Token: None 2023-10-12T01:16:48.123 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj44 2023-10-12T01:16:48.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cS4AkmqqdYrSMRfaadmyVTbzSLw= 2023-10-12T01:16:48.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cS4AkmqqdYrSMRfaadmyVTbzSLw=', 'Content-Length': '0'} 2023-10-12T01:16:48.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dccc58fc64be6f8-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.126 DEBUG:boto:path=/wafmoa-60/obj44 2023-10-12T01:16:48.127 DEBUG:boto:auth_path=/wafmoa-60/obj44 2023-10-12T01:16:48.127 DEBUG:boto:Method: GET 2023-10-12T01:16:48.128 DEBUG:boto:Path: /wafmoa-60/obj44 2023-10-12T01:16:48.128 DEBUG:boto:Data: 2023-10-12T01:16:48.128 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.129 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.130 DEBUG:boto:Params: {} 2023-10-12T01:16:48.130 DEBUG:boto:Token: None 2023-10-12T01:16:48.131 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj44 2023-10-12T01:16:48.131 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cS4AkmqqdYrSMRfaadmyVTbzSLw= 2023-10-12T01:16:48.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cS4AkmqqdYrSMRfaadmyVTbzSLw=', 'Content-Length': '0'} 2023-10-12T01:16:48.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f002fb94679d7ec0-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.134 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.135 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.135 DEBUG:boto:path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.136 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.136 DEBUG:boto:Method: GET 2023-10-12T01:16:48.136 DEBUG:boto:Path: /wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.137 DEBUG:boto:Data: 2023-10-12T01:16:48.137 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.138 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.138 DEBUG:boto:Params: {} 2023-10-12T01:16:48.139 DEBUG:boto:Token: None 2023-10-12T01:16:48.139 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f75cb01deeee1289-0065274900-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.143 DEBUG:boto:b'wafmoa-60obj441000falseobj442023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.143 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.144 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.144 DEBUG:boto:path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.145 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.145 DEBUG:boto:Method: GET 2023-10-12T01:16:48.145 DEBUG:boto:Path: /wafmoa-60/?prefix=obj44 2023-10-12T01:16:48.146 DEBUG:boto:Data: 2023-10-12T01:16:48.146 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.147 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.148 DEBUG:boto:Params: {} 2023-10-12T01:16:48.148 DEBUG:boto:Token: None 2023-10-12T01:16:48.149 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc86f9aeb0cbda38-0065274900-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.152 DEBUG:boto:b'wafmoa-60obj441000falseobj442023-10-12T01:14:00.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:16:48.153 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.154 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.154 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.157 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.157 DEBUG:boto:Method: GET 2023-10-12T01:16:48.158 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.158 DEBUG:boto:Data: 2023-10-12T01:16:48.159 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.160 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.160 DEBUG:boto:Params: {} 2023-10-12T01:16:48.161 DEBUG:boto:Token: None 2023-10-12T01:16:48.161 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o= 2023-10-12T01:16:48.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o=', 'Content-Length': '0'} 2023-10-12T01:16:48.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012095bf700c44e8c-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.206 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.207 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.207 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.208 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.208 DEBUG:boto:Method: GET 2023-10-12T01:16:48.209 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.209 DEBUG:boto:Data: 2023-10-12T01:16:48.210 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.211 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.211 DEBUG:boto:Params: {} 2023-10-12T01:16:48.211 DEBUG:boto:Token: None 2023-10-12T01:16:48.212 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o= 2023-10-12T01:16:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o=', 'Content-Length': '0'} 2023-10-12T01:16:48.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002225cb3eb5ee61e3-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.216 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.216 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.216 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.217 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.217 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.218 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.218 DEBUG:boto:Data: 2023-10-12T01:16:48.218 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.219 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.220 DEBUG:boto:Params: {} 2023-10-12T01:16:48.220 DEBUG:boto:Token: None 2023-10-12T01:16:48.220 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i4AyQjhGlq1audLpkGg5UU8SdDY= 2023-10-12T01:16:48.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i4AyQjhGlq1audLpkGg5UU8SdDY=', 'Content-Length': '0'} 2023-10-12T01:16:48.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ddbb1bf4d782932-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.224 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.224 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.224 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.225 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.225 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.226 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.226 DEBUG:boto:Data: 2023-10-12T01:16:48.226 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.227 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.227 DEBUG:boto:Params: {} 2023-10-12T01:16:48.228 DEBUG:boto:Token: None 2023-10-12T01:16:48.228 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:i4AyQjhGlq1audLpkGg5UU8SdDY= 2023-10-12T01:16:48.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:i4AyQjhGlq1audLpkGg5UU8SdDY=', 'Content-Length': '0'} 2023-10-12T01:16:48.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af594b0c6139d82a-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:16:48.232 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.232 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.233 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.233 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.233 DEBUG:boto:Method: GET 2023-10-12T01:16:48.234 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.234 DEBUG:boto:Data: 2023-10-12T01:16:48.235 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.236 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.236 DEBUG:boto:Params: {} 2023-10-12T01:16:48.236 DEBUG:boto:Token: None 2023-10-12T01:16:48.237 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o= 2023-10-12T01:16:48.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o=', 'Content-Length': '0'} 2023-10-12T01:16:48.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b32357e4de3f215-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.240 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.241 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.241 DEBUG:boto:path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.242 DEBUG:boto:auth_path=/wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.242 DEBUG:boto:Method: GET 2023-10-12T01:16:48.243 DEBUG:boto:Path: /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.243 DEBUG:boto:Data: 2023-10-12T01:16:48.243 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.244 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.245 DEBUG:boto:Params: {} 2023-10-12T01:16:48.245 DEBUG:boto:Token: None 2023-10-12T01:16:48.246 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55?versionId=null 2023-10-12T01:16:48.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o= 2023-10-12T01:16:48.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T27VzZECOzwgih5KyEj5SCfad5o=', 'Content-Length': '0'} 2023-10-12T01:16:48.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f410be179e20c5c6-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.290 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.291 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.291 DEBUG:boto:path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.292 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.292 DEBUG:boto:Method: GET 2023-10-12T01:16:48.293 DEBUG:boto:Path: /wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.293 DEBUG:boto:Data: 2023-10-12T01:16:48.293 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.294 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.295 DEBUG:boto:Params: {} 2023-10-12T01:16:48.295 DEBUG:boto:Token: None 2023-10-12T01:16:48.296 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ba58838d2b98b8-0065274900-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.299 DEBUG:boto:b'wafmoa-60obj551000falseobj552023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.300 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.300 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.301 DEBUG:boto:path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.301 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.302 DEBUG:boto:Method: GET 2023-10-12T01:16:48.302 DEBUG:boto:Path: /wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.302 DEBUG:boto:Data: 2023-10-12T01:16:48.303 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.304 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.304 DEBUG:boto:Params: {} 2023-10-12T01:16:48.304 DEBUG:boto:Token: None 2023-10-12T01:16:48.305 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f774e623d992c25e-0065274900-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.309 DEBUG:boto:b'wafmoa-60obj551000falseobj552023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.309 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.310 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.310 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.311 DEBUG:boto:Path: /wafmoa-60/obj55 2023-10-12T01:16:48.311 DEBUG:boto:Data: 2023-10-12T01:16:48.311 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.312 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.313 DEBUG:boto:Params: {} 2023-10-12T01:16:48.313 DEBUG:boto:Token: None 2023-10-12T01:16:48.314 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55 2023-10-12T01:16:48.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ir2w7kfUgdBBqYwDf7QJPVDTbZU= 2023-10-12T01:16:48.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ir2w7kfUgdBBqYwDf7QJPVDTbZU=', 'Content-Length': '0'} 2023-10-12T01:16:48.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae81acbccd1f678-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.317 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.317 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.318 DEBUG:boto:Method: HEAD 2023-10-12T01:16:48.318 DEBUG:boto:Path: /wafmoa-60/obj55 2023-10-12T01:16:48.318 DEBUG:boto:Data: 2023-10-12T01:16:48.319 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.320 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.320 DEBUG:boto:Params: {} 2023-10-12T01:16:48.320 DEBUG:boto:Token: None 2023-10-12T01:16:48.321 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55 2023-10-12T01:16:48.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ir2w7kfUgdBBqYwDf7QJPVDTbZU= 2023-10-12T01:16:48.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ir2w7kfUgdBBqYwDf7QJPVDTbZU=', 'Content-Length': '0'} 2023-10-12T01:16:48.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b6849f0ea37e0e-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:16:48.325 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.325 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.325 DEBUG:boto:Method: GET 2023-10-12T01:16:48.326 DEBUG:boto:Path: /wafmoa-60/obj55 2023-10-12T01:16:48.326 DEBUG:boto:Data: 2023-10-12T01:16:48.327 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.327 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.328 DEBUG:boto:Params: {} 2023-10-12T01:16:48.328 DEBUG:boto:Token: None 2023-10-12T01:16:48.329 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55 2023-10-12T01:16:48.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZcHE+8WUY4mQvuQbBCNymE6O8j8= 2023-10-12T01:16:48.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZcHE+8WUY4mQvuQbBCNymE6O8j8=', 'Content-Length': '0'} 2023-10-12T01:16:48.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-60:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0720a83fc5baa65-0065274900-1ef8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.375 DEBUG:boto:path=/wafmoa-60/obj55 2023-10-12T01:16:48.376 DEBUG:boto:auth_path=/wafmoa-60/obj55 2023-10-12T01:16:48.376 DEBUG:boto:Method: GET 2023-10-12T01:16:48.377 DEBUG:boto:Path: /wafmoa-60/obj55 2023-10-12T01:16:48.377 DEBUG:boto:Data: 2023-10-12T01:16:48.378 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.379 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.379 DEBUG:boto:Params: {} 2023-10-12T01:16:48.380 DEBUG:boto:Token: None 2023-10-12T01:16:48.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/obj55 2023-10-12T01:16:48.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZcHE+8WUY4mQvuQbBCNymE6O8j8= 2023-10-12T01:16:48.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZcHE+8WUY4mQvuQbBCNymE6O8j8=', 'Content-Length': '0'} 2023-10-12T01:16:48.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed86532b656712d9-0065274900-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.384 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.384 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.385 DEBUG:boto:path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.385 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.385 DEBUG:boto:Method: GET 2023-10-12T01:16:48.386 DEBUG:boto:Path: /wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.386 DEBUG:boto:Data: 2023-10-12T01:16:48.387 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:48.388 DEBUG:boto:Port: 8001 2023-10-12T01:16:48.388 DEBUG:boto:Params: {} 2023-10-12T01:16:48.389 DEBUG:boto:Token: None 2023-10-12T01:16:48.389 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.390 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26af5496e729a02-0065274900-1ef8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.393 DEBUG:boto:b'wafmoa-60obj551000falseobj552023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.393 DEBUG:boto:path=/wafmoa-60/ 2023-10-12T01:16:48.394 DEBUG:boto:auth_path=/wafmoa-60/ 2023-10-12T01:16:48.394 DEBUG:boto:path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.395 DEBUG:boto:auth_path=/wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.395 DEBUG:boto:Method: GET 2023-10-12T01:16:48.396 DEBUG:boto:Path: /wafmoa-60/?prefix=obj55 2023-10-12T01:16:48.396 DEBUG:boto:Data: 2023-10-12T01:16:48.396 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.397 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.398 DEBUG:boto:Params: {} 2023-10-12T01:16:48.398 DEBUG:boto:Token: None 2023-10-12T01:16:48.399 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-60/ 2023-10-12T01:16:48.399 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk= 2023-10-12T01:16:48.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CYPbuD61P09MD3EV1j4csqfZNEk=', 'Content-Length': '0'} 2023-10-12T01:16:48.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb856d685b91dd6d-0065274900-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.402 DEBUG:boto:b'wafmoa-60obj551000falseobj552023-10-12T01:14:00.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:16:48.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-60 zones={a2, a1} 2023-10-12T01:16:48.403 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T01:16:48.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.405 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:48.453 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:48.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:48.501 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:16:48.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.502 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000019174", 2023-10-12T01:16:48.502 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:09:29.908137Z" 2023-10-12T01:16:48.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.503 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000030738", 2023-10-12T01:16:48.503 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:00.879285Z" 2023-10-12T01:16:48.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.503 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000026927", 2023-10-12T01:16:48.504 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:08.506204Z" 2023-10-12T01:16:48.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.504 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000026117", 2023-10-12T01:16:48.504 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:08.507956Z" 2023-10-12T01:16:48.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.505 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:16:48.506 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000019174', 1: '00000000000000000000:00000000000000030738', 2: '00000000000000000000:00000000000000026927', 3: '00000000000000000000:00000000000000026117'} 2023-10-12T01:16:48.506 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:16:48.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.508 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:48.558 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:48.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:48.602 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:16:48.602 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:16:48.603 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:16:48.603 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:16:48.603 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:16:48.603 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 1307313653611130125 2023-10-12T01:16:48.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.604 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:16:48.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:16:48.604 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.604 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000019174", 2023-10-12T01:16:48.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:16:48.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.606 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:09:29.908137Z" 2023-10-12T01:16:48.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:16:48.607 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.607 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000030738", 2023-10-12T01:16:48.607 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 51, 2023-10-12T01:16:48.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:00.879285Z" 2023-10-12T01:16:48.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:16:48.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.609 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000026927", 2023-10-12T01:16:48.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.610 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.610 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:08.506204Z" 2023-10-12T01:16:48.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.611 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:16:48.611 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.612 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.612 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000026117", 2023-10-12T01:16:48.612 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.612 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.613 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:08.507956Z" 2023-10-12T01:16:48.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:16:48.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.614 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:16:48.614 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 60, 2023-10-12T01:16:48.614 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 60 2023-10-12T01:16:48.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.615 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:16:48Z" 2023-10-12T01:16:48.615 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:16:48.616 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1307313653611130125 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:09:29.908137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030738", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-10-12T01:14:00.879285Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026927", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:14:08.506204Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026117", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:14:08.507956Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-10-12T01:16:48Z" } 2023-10-12T01:16:48.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:09:29.908137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030738', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-10-12T01:14:00.879285Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026927', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:14:08.506204Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026117', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:14:08.507956Z'}}] 2023-10-12T01:16:48.617 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000019174', 1: '00000000000000000000:00000000000000030738', 2: '00000000000000000000:00000000000000026927', 3: '00000000000000000000:00000000000000026117'} 2023-10-12T01:16:48.617 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000019174', 1: '00000000000000000000:00000000000000030738', 2: '00000000000000000000:00000000000000026927', 3: '00000000000000000000:00000000000000026117'} 2023-10-12T01:16:48.618 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:16:48.618 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T01:16:48.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.619 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:48.668 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:48.668 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:48.716 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:16:48.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.717 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005300", 2023-10-12T01:16:48.717 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:00.464534Z" 2023-10-12T01:16:48.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.717 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014601", 2023-10-12T01:16:48.718 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:04.475498Z" 2023-10-12T01:16:48.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000012775", 2023-10-12T01:16:48.719 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:23.171955Z" 2023-10-12T01:16:48.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.719 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010687", 2023-10-12T01:16:48.719 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:14:45.173905Z" 2023-10-12T01:16:48.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.720 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:16:48.721 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010687'} 2023-10-12T01:16:48.721 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:16:48.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.723 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:48.820 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:16:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:16:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:16:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:16:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:16:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T01:16:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:16:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005300", 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:00.464534Z" 2023-10-12T01:16:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:16:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014601", 2023-10-12T01:16:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:04.475498Z" 2023-10-12T01:16:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:16:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000012775", 2023-10-12T01:16:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:23.171955Z" 2023-10-12T01:16:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010687", 2023-10-12T01:16:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:14:45.173905Z" 2023-10-12T01:16:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:16:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:16:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T01:16:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T01:16:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:16:48Z" 2023-10-12T01:16:48.832 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:16:48.833 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005300", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:00.464534Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014601", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:14:04.475498Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:14:23.171955Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010687", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:14:45.173905Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T01:16:48Z" } 2023-10-12T01:16:48.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005300', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:00.464534Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:14:04.475498Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:14:23.171955Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010687', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:14:45.173905Z'}}] 2023-10-12T01:16:48.834 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010687'} 2023-10-12T01:16:48.834 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005300', 1: '00000000000000000000:00000000000000014601', 2: '00000000000000000000:00000000000000012775', 3: '00000000000000000000:00000000000000010687'} 2023-10-12T01:16:48.835 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:16:48.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-10-12T01:16:48.836 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:48.836 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:48.837 DEBUG:boto:Method: PUT 2023-10-12T01:16:48.837 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:48.837 DEBUG:boto:Data: 2023-10-12T01:16:48.838 DEBUG:boto:Headers: {} 2023-10-12T01:16:48.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:48.839 DEBUG:boto:Port: 8000 2023-10-12T01:16:48.839 DEBUG:boto:Params: {} 2023-10-12T01:16:48.840 DEBUG:boto:Token: None 2023-10-12T01:16:48.840 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:16:48 GMT /wafmoa-61/ 2023-10-12T01:16:48.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wzzyxwQOiFvMkM2WFcvbt3J/p5Q= 2023-10-12T01:16:48.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wzzyxwQOiFvMkM2WFcvbt3J/p5Q=', 'Content-Length': '0'} 2023-10-12T01:16:48.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e74476446252bb02-0065274900-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:48.856 DEBUG:rgw_multi.tests:created bucket=wafmoa-61 2023-10-12T01:16:48.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.858 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:48.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:48.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:48.936 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:16:48.951 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:16:48.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:16:48.952 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:16:48.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:16:48.953 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:16:48.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073408.854754_662.1", 2023-10-12T01:16:48.954 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:16:48.854754Z" 2023-10-12T01:16:48.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:48.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:48.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:16:48.954 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:16:48.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:48.955 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:16:48.956 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073408.854754_662.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:16:48.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:16:48.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:16:48.958 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:16:49.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:49.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:49.038+0000 7f1101a85780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:16:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:49.039+0000 7f1101a85780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:16:49.051 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:16:49.051 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:16:49.052 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:16:49.052 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:16:49.052 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:16:49.052 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:16:49.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:16:49.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.053 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:16:49.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:49.053 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:16:49.054 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:49.054 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:16:49.054 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:16:49.054 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:49.054 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:16:49.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:49.055 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:16:49.055 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:16:49.055 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:49.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:49.056 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:16:49.056 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:49.056 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:16:49.056 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:16:49.057 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:49.057 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:49.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:49.057 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:16:49.057 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:16:49.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:49.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:49.058 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:16:49.059 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:49.059 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:16:49.059 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073408.854754_662.1", 2023-10-12T01:16:49.059 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:49.059 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:49.060 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:49.060 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:16:48.854754Z", 2023-10-12T01:16:49.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:16:49.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:49.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:16:49.061 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:16:49.061 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:16:49.061 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:16:49.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:16:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:16:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:16:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:16:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:16:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:16:49.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:49.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:16:49.063 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:16:49.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.063 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:16:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:16:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:16:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:16:49.064 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:16:49Z" 2023-10-12T01:16:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:16:49.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073408.854754_662.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:16:48.854754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:16:49Z" } 2023-10-12T01:16:49.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073408.854754_662.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:16:48.854754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:16:49.066 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073408.854754_662.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:16:49.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073408.854754_662.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:16:49.067 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:16:49.067 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:16:49.068 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:16:49.069 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:16:49.069+0000 7f657048e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 174334) UID: 0 2023-10-12T01:16:49.069 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:16:49.069+0000 7f65746f77c0 -1 shutting down 2023-10-12T01:16:49.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:16:55.169 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:16:55.170 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.171 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.171 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.171 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.172 DEBUG:boto:Data: 2023-10-12T01:16:55.172 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.173 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.174 DEBUG:boto:Params: {} 2023-10-12T01:16:55.174 DEBUG:boto:Token: None 2023-10-12T01:16:55.175 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029f2b16bd0f1f9e5-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.179 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:16:55.179 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:16:55.180 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.180 DEBUG:boto:Path: /wafmoa-61/obj11 2023-10-12T01:16:55.181 DEBUG:boto:Data: 2023-10-12T01:16:55.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.182 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.182 DEBUG:boto:Params: {} 2023-10-12T01:16:55.183 DEBUG:boto:Token: None 2023-10-12T01:16:55.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj11 2023-10-12T01:16:55.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mrNV6mfCwLrXBiVbHCj3VK+BTeg= 2023-10-12T01:16:55.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mrNV6mfCwLrXBiVbHCj3VK+BTeg='} 2023-10-12T01:16:55.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be8e73d97a789a05-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.227 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.228 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.228 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.229 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.229 DEBUG:boto:Data: 2023-10-12T01:16:55.229 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.230 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.230 DEBUG:boto:Params: {} 2023-10-12T01:16:55.231 DEBUG:boto:Token: None 2023-10-12T01:16:55.231 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070ac935561491308-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.236 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:16:55.236 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:16:55.237 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.237 DEBUG:boto:Path: /wafmoa-61/obj22 2023-10-12T01:16:55.237 DEBUG:boto:Data: 2023-10-12T01:16:55.238 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.239 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.239 DEBUG:boto:Params: {} 2023-10-12T01:16:55.239 DEBUG:boto:Token: None 2023-10-12T01:16:55.240 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj22 2023-10-12T01:16:55.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sju7FxFyDEjrcH6hpeRv65B6kWI= 2023-10-12T01:16:55.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sju7FxFyDEjrcH6hpeRv65B6kWI='} 2023-10-12T01:16:55.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000975c9b102420ab12-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.247 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.247 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.247 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.248 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.248 DEBUG:boto:Data: 2023-10-12T01:16:55.248 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.249 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.250 DEBUG:boto:Params: {} 2023-10-12T01:16:55.250 DEBUG:boto:Token: None 2023-10-12T01:16:55.251 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.251 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000072c1fb8a6fcddd6-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.255 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:16:55.255 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:16:55.255 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.256 DEBUG:boto:Path: /wafmoa-61/obj33 2023-10-12T01:16:55.256 DEBUG:boto:Data: 2023-10-12T01:16:55.257 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.257 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.258 DEBUG:boto:Params: {} 2023-10-12T01:16:55.258 DEBUG:boto:Token: None 2023-10-12T01:16:55.259 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj33 2023-10-12T01:16:55.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FNz86w6cb4i7Rn7HEwkF8ZaPgMU= 2023-10-12T01:16:55.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FNz86w6cb4i7Rn7HEwkF8ZaPgMU='} 2023-10-12T01:16:55.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000462d6b75871fc0b-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.303 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.303 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.304 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.304 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.305 DEBUG:boto:Data: 2023-10-12T01:16:55.305 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.306 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.306 DEBUG:boto:Params: {} 2023-10-12T01:16:55.307 DEBUG:boto:Token: None 2023-10-12T01:16:55.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:16:55.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000117cc42ce6740f98-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.311 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:16:55.311 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:16:55.312 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.312 DEBUG:boto:Path: /wafmoa-61/obj44 2023-10-12T01:16:55.313 DEBUG:boto:Data: 2023-10-12T01:16:55.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.314 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.314 DEBUG:boto:Params: {} 2023-10-12T01:16:55.315 DEBUG:boto:Token: None 2023-10-12T01:16:55.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj44 2023-10-12T01:16:55.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CSruz6w2H+M/L7k7v68SP8dgkR0= 2023-10-12T01:16:55.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CSruz6w2H+M/L7k7v68SP8dgkR0='} 2023-10-12T01:16:55.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e8b9358a0eb4d2-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.323 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.323 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.324 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.324 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.325 DEBUG:boto:Data: 2023-10-12T01:16:55.325 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.326 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.326 DEBUG:boto:Params: {} 2023-10-12T01:16:55.327 DEBUG:boto:Token: None 2023-10-12T01:16:55.327 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003076d755528bb439-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.331 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:16:55.332 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:16:55.332 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.333 DEBUG:boto:Path: /wafmoa-61/obj55 2023-10-12T01:16:55.333 DEBUG:boto:Data: 2023-10-12T01:16:55.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.334 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.335 DEBUG:boto:Params: {} 2023-10-12T01:16:55.335 DEBUG:boto:Token: None 2023-10-12T01:16:55.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj55 2023-10-12T01:16:55.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kdG+1m/Tascwma4PWK6gKqgfLTE= 2023-10-12T01:16:55.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kdG+1m/Tascwma4PWK6gKqgfLTE='} 2023-10-12T01:16:55.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009a5cb93a1199eae-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:55.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:55.380 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:55.440 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:55.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:55.486 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:55.486 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:55.489 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-10-12T01:16:55.527 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.528 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.528 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.529 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.529 DEBUG:boto:Data: 2023-10-12T01:16:55.530 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.530 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.531 DEBUG:boto:Params: {} 2023-10-12T01:16:55.531 DEBUG:boto:Token: None 2023-10-12T01:16:55.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2960f92e95c8475-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.536 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:16:55.537 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:16:55.537 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.538 DEBUG:boto:Path: /wafmoa-61/obj17 2023-10-12T01:16:55.538 DEBUG:boto:Data: 2023-10-12T01:16:55.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.540 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.540 DEBUG:boto:Params: {} 2023-10-12T01:16:55.541 DEBUG:boto:Token: None 2023-10-12T01:16:55.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj17 2023-10-12T01:16:55.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mzNsKGjKkM7JC6nObjksGw8ehY0= 2023-10-12T01:16:55.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mzNsKGjKkM7JC6nObjksGw8ehY0='} 2023-10-12T01:16:55.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b22ec9477292061-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.548 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.548 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.549 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.549 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.550 DEBUG:boto:Data: 2023-10-12T01:16:55.550 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.551 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.552 DEBUG:boto:Params: {} 2023-10-12T01:16:55.552 DEBUG:boto:Token: None 2023-10-12T01:16:55.553 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.553 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000380df49f583aea2-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.557 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:16:55.557 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:16:55.558 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.558 DEBUG:boto:Path: /wafmoa-61/obj34 2023-10-12T01:16:55.559 DEBUG:boto:Data: 2023-10-12T01:16:55.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.561 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.562 DEBUG:boto:Params: {} 2023-10-12T01:16:55.563 DEBUG:boto:Token: None 2023-10-12T01:16:55.564 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj34 2023-10-12T01:16:55.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EpZO9TCc1lwAZx+tQyZn90W+I3U= 2023-10-12T01:16:55.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EpZO9TCc1lwAZx+tQyZn90W+I3U='} 2023-10-12T01:16:55.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000340c0ee1371a6a5e-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.570 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.571 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.571 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.572 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.572 DEBUG:boto:Data: 2023-10-12T01:16:55.573 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.574 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.574 DEBUG:boto:Params: {} 2023-10-12T01:16:55.575 DEBUG:boto:Token: None 2023-10-12T01:16:55.575 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7ef75cdd4839378-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.579 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:16:55.580 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:16:55.580 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.580 DEBUG:boto:Path: /wafmoa-61/obj51 2023-10-12T01:16:55.581 DEBUG:boto:Data: 2023-10-12T01:16:55.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.582 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.583 DEBUG:boto:Params: {} 2023-10-12T01:16:55.583 DEBUG:boto:Token: None 2023-10-12T01:16:55.583 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj51 2023-10-12T01:16:55.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:c7NOMMxG7Ek0j4C/dlRNiUXKeNQ= 2023-10-12T01:16:55.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:c7NOMMxG7Ek0j4C/dlRNiUXKeNQ='} 2023-10-12T01:16:55.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fff8330dffe5139-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.627 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.627 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.628 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.628 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.629 DEBUG:boto:Data: 2023-10-12T01:16:55.629 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.630 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.631 DEBUG:boto:Params: {} 2023-10-12T01:16:55.631 DEBUG:boto:Token: None 2023-10-12T01:16:55.632 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085c88abecaf9eb55-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.636 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:16:55.636 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:16:55.637 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.637 DEBUG:boto:Path: /wafmoa-61/obj68 2023-10-12T01:16:55.637 DEBUG:boto:Data: 2023-10-12T01:16:55.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.639 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.639 DEBUG:boto:Params: {} 2023-10-12T01:16:55.640 DEBUG:boto:Token: None 2023-10-12T01:16:55.640 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj68 2023-10-12T01:16:55.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xmrrmebxrh4zopozZv3rZgsWJYc= 2023-10-12T01:16:55.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xmrrmebxrh4zopozZv3rZgsWJYc='} 2023-10-12T01:16:55.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009307ced2ff1a2459-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.647 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.647 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.648 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.648 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.649 DEBUG:boto:Data: 2023-10-12T01:16:55.649 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.650 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.651 DEBUG:boto:Params: {} 2023-10-12T01:16:55.651 DEBUG:boto:Token: None 2023-10-12T01:16:55.652 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.652 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006535222471d63c1b-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.656 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:16:55.656 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:16:55.657 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.657 DEBUG:boto:Path: /wafmoa-61/obj85 2023-10-12T01:16:55.658 DEBUG:boto:Data: 2023-10-12T01:16:55.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.659 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.659 DEBUG:boto:Params: {} 2023-10-12T01:16:55.660 DEBUG:boto:Token: None 2023-10-12T01:16:55.661 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj85 2023-10-12T01:16:55.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/96Wisbes4MslXORsk/z2AcJZPc= 2023-10-12T01:16:55.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/96Wisbes4MslXORsk/z2AcJZPc='} 2023-10-12T01:16:55.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fe1c6975fe3a2b4-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:55.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:55.708 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:55.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:55.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:55.803 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:55.803 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:55.810 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-10-12T01:16:55.853 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.853 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.854 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.854 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.854 DEBUG:boto:Data: 2023-10-12T01:16:55.855 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.856 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.856 DEBUG:boto:Params: {} 2023-10-12T01:16:55.856 DEBUG:boto:Token: None 2023-10-12T01:16:55.857 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027aeb370de531c04-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.861 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:16:55.862 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:16:55.862 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.863 DEBUG:boto:Path: /wafmoa-61/obj19 2023-10-12T01:16:55.863 DEBUG:boto:Data: 2023-10-12T01:16:55.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.864 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.865 DEBUG:boto:Params: {} 2023-10-12T01:16:55.865 DEBUG:boto:Token: None 2023-10-12T01:16:55.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj19 2023-10-12T01:16:55.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RGvJA/yhl753E+NDT5D1t3DjlO4= 2023-10-12T01:16:55.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RGvJA/yhl753E+NDT5D1t3DjlO4='} 2023-10-12T01:16:55.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e528498e5b4caa3c-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.872 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.873 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.873 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.873 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.874 DEBUG:boto:Data: 2023-10-12T01:16:55.874 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.875 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.875 DEBUG:boto:Params: {} 2023-10-12T01:16:55.876 DEBUG:boto:Token: None 2023-10-12T01:16:55.876 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093d29097faddf71b-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.880 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:16:55.881 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:16:55.881 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.882 DEBUG:boto:Path: /wafmoa-61/obj38 2023-10-12T01:16:55.882 DEBUG:boto:Data: 2023-10-12T01:16:55.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.883 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.884 DEBUG:boto:Params: {} 2023-10-12T01:16:55.884 DEBUG:boto:Token: None 2023-10-12T01:16:55.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj38 2023-10-12T01:16:55.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Nk5Etllj1VLaIbSEDj+59HkfPA0= 2023-10-12T01:16:55.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Nk5Etllj1VLaIbSEDj+59HkfPA0='} 2023-10-12T01:16:55.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d69467d8f80d7c51-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.889 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.890 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.890 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.890 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.891 DEBUG:boto:Data: 2023-10-12T01:16:55.891 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.892 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.892 DEBUG:boto:Params: {} 2023-10-12T01:16:55.893 DEBUG:boto:Token: None 2023-10-12T01:16:55.893 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c16eb7b39215732-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.897 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:16:55.898 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:16:55.898 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.898 DEBUG:boto:Path: /wafmoa-61/obj57 2023-10-12T01:16:55.899 DEBUG:boto:Data: 2023-10-12T01:16:55.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.900 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.900 DEBUG:boto:Params: {} 2023-10-12T01:16:55.901 DEBUG:boto:Token: None 2023-10-12T01:16:55.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj57 2023-10-12T01:16:55.902 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:71TuVa9LAt8qP9t9zRalgPyuBPg= 2023-10-12T01:16:55.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:71TuVa9LAt8qP9t9zRalgPyuBPg='} 2023-10-12T01:16:55.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000306f2ff4e5aef0cc-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.947 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.947 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.948 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.948 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.949 DEBUG:boto:Data: 2023-10-12T01:16:55.949 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.950 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.950 DEBUG:boto:Params: {} 2023-10-12T01:16:55.951 DEBUG:boto:Token: None 2023-10-12T01:16:55.951 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e9e5e69e442bcf-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.956 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:16:55.956 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:16:55.957 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.957 DEBUG:boto:Path: /wafmoa-61/obj76 2023-10-12T01:16:55.958 DEBUG:boto:Data: 2023-10-12T01:16:55.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.960 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.960 DEBUG:boto:Params: {} 2023-10-12T01:16:55.961 DEBUG:boto:Token: None 2023-10-12T01:16:55.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj76 2023-10-12T01:16:55.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5C7u/UgRWdLMzk6Gj1/Z2TJKQjM= 2023-10-12T01:16:55.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5C7u/UgRWdLMzk6Gj1/Z2TJKQjM='} 2023-10-12T01:16:55.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b420cd46da93301-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.970 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:55.970 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:55.970 DEBUG:boto:Method: HEAD 2023-10-12T01:16:55.971 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:55.971 DEBUG:boto:Data: 2023-10-12T01:16:55.972 DEBUG:boto:Headers: {} 2023-10-12T01:16:55.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.972 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.973 DEBUG:boto:Params: {} 2023-10-12T01:16:55.973 DEBUG:boto:Token: None 2023-10-12T01:16:55.974 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/ 2023-10-12T01:16:55.974 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8= 2023-10-12T01:16:55.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JnrxODNjMWIRe8cso2q3ic/Xtl8=', 'Content-Length': '0'} 2023-10-12T01:16:55.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034bf7be6bbc1bf2d-0065274907-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:55.978 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:16:55.978 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:16:55.978 DEBUG:boto:Method: PUT 2023-10-12T01:16:55.979 DEBUG:boto:Path: /wafmoa-61/obj95 2023-10-12T01:16:55.979 DEBUG:boto:Data: 2023-10-12T01:16:55.980 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:55.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:55.980 DEBUG:boto:Port: 8000 2023-10-12T01:16:55.981 DEBUG:boto:Params: {} 2023-10-12T01:16:55.981 DEBUG:boto:Token: None 2023-10-12T01:16:55.982 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:55 GMT /wafmoa-61/obj95 2023-10-12T01:16:55.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:H0QJJ6nrqtUx0pxwOPOzeFD2ReQ= 2023-10-12T01:16:55.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:55 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:H0QJJ6nrqtUx0pxwOPOzeFD2ReQ='} 2023-10-12T01:16:56.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001272f41a28243216-0065274907-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.028 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:56.080 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:56.080 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:56.123 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:56.124 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:56.130 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-10-12T01:16:56.175 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.175 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.176 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.176 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.176 DEBUG:boto:Data: 2023-10-12T01:16:56.177 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.177 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.178 DEBUG:boto:Params: {} 2023-10-12T01:16:56.178 DEBUG:boto:Token: None 2023-10-12T01:16:56.179 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.179 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8097b910cfcbb78-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.183 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:16:56.183 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:16:56.184 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.184 DEBUG:boto:Path: /wafmoa-61/obj23 2023-10-12T01:16:56.184 DEBUG:boto:Data: 2023-10-12T01:16:56.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.186 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.186 DEBUG:boto:Params: {} 2023-10-12T01:16:56.187 DEBUG:boto:Token: None 2023-10-12T01:16:56.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj23 2023-10-12T01:16:56.188 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PBkNZz0GZ0lIuDqCq/sk19adlAg= 2023-10-12T01:16:56.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PBkNZz0GZ0lIuDqCq/sk19adlAg='} 2023-10-12T01:16:56.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a68bcef7af6f188-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.194 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.195 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.195 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.196 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.196 DEBUG:boto:Data: 2023-10-12T01:16:56.196 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.197 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.197 DEBUG:boto:Params: {} 2023-10-12T01:16:56.198 DEBUG:boto:Token: None 2023-10-12T01:16:56.198 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.199 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7eb196297f61b86-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.203 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:16:56.203 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:16:56.204 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.204 DEBUG:boto:Path: /wafmoa-61/obj46 2023-10-12T01:16:56.204 DEBUG:boto:Data: 2023-10-12T01:16:56.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.205 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.206 DEBUG:boto:Params: {} 2023-10-12T01:16:56.206 DEBUG:boto:Token: None 2023-10-12T01:16:56.207 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj46 2023-10-12T01:16:56.207 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:akvyLx6F2KhvAU7NlZ8SD9Tn0HM= 2023-10-12T01:16:56.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:akvyLx6F2KhvAU7NlZ8SD9Tn0HM='} 2023-10-12T01:16:56.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060a6bf3ec411eef2-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.213 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.213 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.214 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.214 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.215 DEBUG:boto:Data: 2023-10-12T01:16:56.215 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.216 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.216 DEBUG:boto:Params: {} 2023-10-12T01:16:56.216 DEBUG:boto:Token: None 2023-10-12T01:16:56.217 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.217 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a51870a13abe601-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.221 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:16:56.222 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:16:56.222 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.222 DEBUG:boto:Path: /wafmoa-61/obj69 2023-10-12T01:16:56.223 DEBUG:boto:Data: 2023-10-12T01:16:56.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.224 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.224 DEBUG:boto:Params: {} 2023-10-12T01:16:56.225 DEBUG:boto:Token: None 2023-10-12T01:16:56.225 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj69 2023-10-12T01:16:56.226 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j5OzGZ7/hgMU8hdYsBvalD2PYKk= 2023-10-12T01:16:56.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j5OzGZ7/hgMU8hdYsBvalD2PYKk='} 2023-10-12T01:16:56.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a95bd088f0431d82-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.271 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.271 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.272 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.272 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.273 DEBUG:boto:Data: 2023-10-12T01:16:56.273 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.274 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.274 DEBUG:boto:Params: {} 2023-10-12T01:16:56.275 DEBUG:boto:Token: None 2023-10-12T01:16:56.275 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f949f3a0e710665-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.279 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:16:56.280 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:16:56.280 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.280 DEBUG:boto:Path: /wafmoa-61/obj92 2023-10-12T01:16:56.281 DEBUG:boto:Data: 2023-10-12T01:16:56.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.282 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.282 DEBUG:boto:Params: {} 2023-10-12T01:16:56.283 DEBUG:boto:Token: None 2023-10-12T01:16:56.283 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj92 2023-10-12T01:16:56.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PzvJDAHAsX1mi81p10d67MaJaXg= 2023-10-12T01:16:56.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PzvJDAHAsX1mi81p10d67MaJaXg='} 2023-10-12T01:16:56.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9054c8331b2337c-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.289 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.290 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.290 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.291 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.291 DEBUG:boto:Data: 2023-10-12T01:16:56.291 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.292 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.293 DEBUG:boto:Params: {} 2023-10-12T01:16:56.293 DEBUG:boto:Token: None 2023-10-12T01:16:56.294 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.294 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076c6cf431188f471-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.298 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:16:56.298 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:16:56.298 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.299 DEBUG:boto:Path: /wafmoa-61/obj115 2023-10-12T01:16:56.299 DEBUG:boto:Data: 2023-10-12T01:16:56.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.301 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.301 DEBUG:boto:Params: {} 2023-10-12T01:16:56.302 DEBUG:boto:Token: None 2023-10-12T01:16:56.302 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj115 2023-10-12T01:16:56.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ucOvLK0G6AojD9mW18+GB6PNNaY= 2023-10-12T01:16:56.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ucOvLK0G6AojD9mW18+GB6PNNaY='} 2023-10-12T01:16:56.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000157ef916b02b1d59-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:56.400 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:56.447 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:56.448 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:56.455 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-10-12T01:16:56.506 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.506 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.507 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.507 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.507 DEBUG:boto:Data: 2023-10-12T01:16:56.508 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.509 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.509 DEBUG:boto:Params: {} 2023-10-12T01:16:56.509 DEBUG:boto:Token: None 2023-10-12T01:16:56.510 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000356921ab3c9f3828-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.515 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:16:56.515 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:16:56.516 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.516 DEBUG:boto:Path: /wafmoa-61/obj29 2023-10-12T01:16:56.517 DEBUG:boto:Data: 2023-10-12T01:16:56.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.518 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.518 DEBUG:boto:Params: {} 2023-10-12T01:16:56.519 DEBUG:boto:Token: None 2023-10-12T01:16:56.519 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj29 2023-10-12T01:16:56.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BlTgVZ8oUyWJBdxxI669lAHiOaU= 2023-10-12T01:16:56.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BlTgVZ8oUyWJBdxxI669lAHiOaU='} 2023-10-12T01:16:56.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ee3bb0b36ae97d-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.526 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.526 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.526 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.527 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.527 DEBUG:boto:Data: 2023-10-12T01:16:56.528 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.528 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.529 DEBUG:boto:Params: {} 2023-10-12T01:16:56.529 DEBUG:boto:Token: None 2023-10-12T01:16:56.530 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.530 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa8385bd97775d5e-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.534 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:16:56.535 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:16:56.535 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.535 DEBUG:boto:Path: /wafmoa-61/obj58 2023-10-12T01:16:56.536 DEBUG:boto:Data: 2023-10-12T01:16:56.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.537 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.538 DEBUG:boto:Params: {} 2023-10-12T01:16:56.538 DEBUG:boto:Token: None 2023-10-12T01:16:56.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj58 2023-10-12T01:16:56.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:w4lTR26MDD3B2CZZU0cpmNsc91Y= 2023-10-12T01:16:56.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:w4lTR26MDD3B2CZZU0cpmNsc91Y='} 2023-10-12T01:16:56.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cc640a164620468-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.546 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.547 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.547 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.548 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.548 DEBUG:boto:Data: 2023-10-12T01:16:56.548 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.549 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.550 DEBUG:boto:Params: {} 2023-10-12T01:16:56.550 DEBUG:boto:Token: None 2023-10-12T01:16:56.550 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035a628e608f63ecf-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.555 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:16:56.556 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:16:56.556 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.556 DEBUG:boto:Path: /wafmoa-61/obj87 2023-10-12T01:16:56.557 DEBUG:boto:Data: 2023-10-12T01:16:56.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.558 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.558 DEBUG:boto:Params: {} 2023-10-12T01:16:56.559 DEBUG:boto:Token: None 2023-10-12T01:16:56.559 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj87 2023-10-12T01:16:56.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:emONYrCd7qMdfRJyQarEa3daCTI= 2023-10-12T01:16:56.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:emONYrCd7qMdfRJyQarEa3daCTI='} 2023-10-12T01:16:56.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d58e23adf75bc5d6-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.603 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.603 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.604 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.604 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.605 DEBUG:boto:Data: 2023-10-12T01:16:56.605 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.606 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.606 DEBUG:boto:Params: {} 2023-10-12T01:16:56.607 DEBUG:boto:Token: None 2023-10-12T01:16:56.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2a4b46c4bf21f3c-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.612 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:16:56.612 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:16:56.613 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.613 DEBUG:boto:Path: /wafmoa-61/obj116 2023-10-12T01:16:56.614 DEBUG:boto:Data: 2023-10-12T01:16:56.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.615 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.615 DEBUG:boto:Params: {} 2023-10-12T01:16:56.616 DEBUG:boto:Token: None 2023-10-12T01:16:56.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj116 2023-10-12T01:16:56.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9ABRFuCXIdvmCRyPdsiWE3G4+U0= 2023-10-12T01:16:56.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9ABRFuCXIdvmCRyPdsiWE3G4+U0='} 2023-10-12T01:16:56.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000168f597c603aec47-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.622 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.623 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.623 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.624 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.624 DEBUG:boto:Data: 2023-10-12T01:16:56.624 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.625 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.625 DEBUG:boto:Params: {} 2023-10-12T01:16:56.626 DEBUG:boto:Token: None 2023-10-12T01:16:56.626 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e5658f3547ba7d6-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.631 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:16:56.631 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:16:56.632 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.632 DEBUG:boto:Path: /wafmoa-61/obj145 2023-10-12T01:16:56.632 DEBUG:boto:Data: 2023-10-12T01:16:56.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.633 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.634 DEBUG:boto:Params: {} 2023-10-12T01:16:56.634 DEBUG:boto:Token: None 2023-10-12T01:16:56.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj145 2023-10-12T01:16:56.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MGFkNi6qSL/3lexRQR3twT/kvK4= 2023-10-12T01:16:56.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MGFkNi6qSL/3lexRQR3twT/kvK4='} 2023-10-12T01:16:56.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000079cd4e1f44578c9-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:56.680 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:56.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:56.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:56.778 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:56.779 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:56.789 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-10-12T01:16:56.846 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.846 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.847 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.847 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.848 DEBUG:boto:Data: 2023-10-12T01:16:56.848 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.849 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.849 DEBUG:boto:Params: {} 2023-10-12T01:16:56.850 DEBUG:boto:Token: None 2023-10-12T01:16:56.850 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a393e377f98266-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.855 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:16:56.856 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:16:56.856 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.857 DEBUG:boto:Path: /wafmoa-61/obj31 2023-10-12T01:16:56.857 DEBUG:boto:Data: 2023-10-12T01:16:56.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.858 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.859 DEBUG:boto:Params: {} 2023-10-12T01:16:56.859 DEBUG:boto:Token: None 2023-10-12T01:16:56.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj31 2023-10-12T01:16:56.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mUEW4VVvuKfYLczv82TZcVrECqc= 2023-10-12T01:16:56.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mUEW4VVvuKfYLczv82TZcVrECqc='} 2023-10-12T01:16:56.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebacf376209785fc-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.867 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.867 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.868 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.868 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.868 DEBUG:boto:Data: 2023-10-12T01:16:56.869 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.870 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.870 DEBUG:boto:Params: {} 2023-10-12T01:16:56.871 DEBUG:boto:Token: None 2023-10-12T01:16:56.871 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000078ddbbec67449f6-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.876 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:16:56.876 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:16:56.876 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.877 DEBUG:boto:Path: /wafmoa-61/obj62 2023-10-12T01:16:56.877 DEBUG:boto:Data: 2023-10-12T01:16:56.878 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.878 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.879 DEBUG:boto:Params: {} 2023-10-12T01:16:56.879 DEBUG:boto:Token: None 2023-10-12T01:16:56.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj62 2023-10-12T01:16:56.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Cv2v/nX7acWMQ17AJ5HQUAWrSHs= 2023-10-12T01:16:56.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Cv2v/nX7acWMQ17AJ5HQUAWrSHs='} 2023-10-12T01:16:56.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071bd33ebb244417b-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.927 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.927 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.928 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.928 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.929 DEBUG:boto:Data: 2023-10-12T01:16:56.929 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.930 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.930 DEBUG:boto:Params: {} 2023-10-12T01:16:56.931 DEBUG:boto:Token: None 2023-10-12T01:16:56.931 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4fdd041b2f3c683-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.937 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:16:56.938 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:16:56.938 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.938 DEBUG:boto:Path: /wafmoa-61/obj93 2023-10-12T01:16:56.939 DEBUG:boto:Data: 2023-10-12T01:16:56.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.940 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.940 DEBUG:boto:Params: {} 2023-10-12T01:16:56.941 DEBUG:boto:Token: None 2023-10-12T01:16:56.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj93 2023-10-12T01:16:56.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:csnqjDFp8Jz1EFIJQlT2KMNHFZ8= 2023-10-12T01:16:56.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:csnqjDFp8Jz1EFIJQlT2KMNHFZ8='} 2023-10-12T01:16:56.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001168c0bb22e7ba8f-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.948 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:56.948 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:56.949 DEBUG:boto:Method: HEAD 2023-10-12T01:16:56.949 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:56.950 DEBUG:boto:Data: 2023-10-12T01:16:56.950 DEBUG:boto:Headers: {} 2023-10-12T01:16:56.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.951 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.951 DEBUG:boto:Params: {} 2023-10-12T01:16:56.951 DEBUG:boto:Token: None 2023-10-12T01:16:56.952 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/ 2023-10-12T01:16:56.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo= 2023-10-12T01:16:56.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:x260Nu3dG+T1ZGGvusRHCjUhCDo=', 'Content-Length': '0'} 2023-10-12T01:16:56.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064bd71280fabdd23-0065274908-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:56.956 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:16:56.957 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:16:56.957 DEBUG:boto:Method: PUT 2023-10-12T01:16:56.958 DEBUG:boto:Path: /wafmoa-61/obj124 2023-10-12T01:16:56.958 DEBUG:boto:Data: 2023-10-12T01:16:56.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:56.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:56.959 DEBUG:boto:Port: 8000 2023-10-12T01:16:56.960 DEBUG:boto:Params: {} 2023-10-12T01:16:56.960 DEBUG:boto:Token: None 2023-10-12T01:16:56.960 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:56 GMT /wafmoa-61/obj124 2023-10-12T01:16:56.961 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zuQnmf/SWxpTf2Rhedqw5vRfB68= 2023-10-12T01:16:56.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:56 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zuQnmf/SWxpTf2Rhedqw5vRfB68='} 2023-10-12T01:16:57.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051f837b2b641cbc5-0065274908-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.011 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.011 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.012 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.012 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.013 DEBUG:boto:Data: 2023-10-12T01:16:57.013 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.014 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.014 DEBUG:boto:Params: {} 2023-10-12T01:16:57.015 DEBUG:boto:Token: None 2023-10-12T01:16:57.015 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edecc2b1ee1523a3-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.020 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:16:57.021 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:16:57.021 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.021 DEBUG:boto:Path: /wafmoa-61/obj155 2023-10-12T01:16:57.022 DEBUG:boto:Data: 2023-10-12T01:16:57.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.023 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.023 DEBUG:boto:Params: {} 2023-10-12T01:16:57.024 DEBUG:boto:Token: None 2023-10-12T01:16:57.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj155 2023-10-12T01:16:57.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:meqw3WWYN6IFDwrdnjr+15PdU/A= 2023-10-12T01:16:57.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:meqw3WWYN6IFDwrdnjr+15PdU/A='} 2023-10-12T01:16:57.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068cfa4a50d47ba72-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:57.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:57.032 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:57.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:57.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:57.134 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:16:57.135 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-61 2023-10-12T01:16:57.145 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-10-12T01:16:57.206 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.206 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.207 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.207 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.208 DEBUG:boto:Data: 2023-10-12T01:16:57.208 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.209 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.209 DEBUG:boto:Params: {} 2023-10-12T01:16:57.210 DEBUG:boto:Token: None 2023-10-12T01:16:57.210 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.211 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d5e340991dd3d33-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.219 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:16:57.220 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:16:57.220 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.221 DEBUG:boto:Path: /wafmoa-61/obj37 2023-10-12T01:16:57.221 DEBUG:boto:Data: 2023-10-12T01:16:57.221 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.222 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.223 DEBUG:boto:Params: {} 2023-10-12T01:16:57.223 DEBUG:boto:Token: None 2023-10-12T01:16:57.224 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj37 2023-10-12T01:16:57.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DxecS80E2CkYm+aasACZqVeaPIY= 2023-10-12T01:16:57.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DxecS80E2CkYm+aasACZqVeaPIY='} 2023-10-12T01:16:57.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072c7fbe483b67b87-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.267 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.267 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.268 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.268 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.269 DEBUG:boto:Data: 2023-10-12T01:16:57.269 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.270 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.271 DEBUG:boto:Params: {} 2023-10-12T01:16:57.271 DEBUG:boto:Token: None 2023-10-12T01:16:57.272 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.272 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000024dab7869323cb2-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.277 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:16:57.277 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:16:57.278 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.278 DEBUG:boto:Path: /wafmoa-61/obj74 2023-10-12T01:16:57.278 DEBUG:boto:Data: 2023-10-12T01:16:57.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.280 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.280 DEBUG:boto:Params: {} 2023-10-12T01:16:57.280 DEBUG:boto:Token: None 2023-10-12T01:16:57.281 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj74 2023-10-12T01:16:57.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ToP6zDSvALd+7ZQl72n8BWYnzeE= 2023-10-12T01:16:57.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ToP6zDSvALd+7ZQl72n8BWYnzeE='} 2023-10-12T01:16:57.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a4946c53866f935-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.288 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.288 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.288 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.289 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.289 DEBUG:boto:Data: 2023-10-12T01:16:57.290 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.290 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.291 DEBUG:boto:Params: {} 2023-10-12T01:16:57.291 DEBUG:boto:Token: None 2023-10-12T01:16:57.292 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f4b55fc18c7723d-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.297 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:16:57.297 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:16:57.297 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.298 DEBUG:boto:Path: /wafmoa-61/obj111 2023-10-12T01:16:57.298 DEBUG:boto:Data: 2023-10-12T01:16:57.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.299 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.300 DEBUG:boto:Params: {} 2023-10-12T01:16:57.301 DEBUG:boto:Token: None 2023-10-12T01:16:57.302 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj111 2023-10-12T01:16:57.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6u8ie/yBeDtktkoN6ECcTR7obT4= 2023-10-12T01:16:57.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6u8ie/yBeDtktkoN6ECcTR7obT4='} 2023-10-12T01:16:57.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f259c90cffd98f7b-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.347 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.347 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.348 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.348 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.349 DEBUG:boto:Data: 2023-10-12T01:16:57.349 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.350 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.350 DEBUG:boto:Params: {} 2023-10-12T01:16:57.350 DEBUG:boto:Token: None 2023-10-12T01:16:57.351 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.351 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a11ee6956305a64-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.356 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:16:57.356 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:16:57.356 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.357 DEBUG:boto:Path: /wafmoa-61/obj148 2023-10-12T01:16:57.357 DEBUG:boto:Data: 2023-10-12T01:16:57.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.358 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.359 DEBUG:boto:Params: {} 2023-10-12T01:16:57.359 DEBUG:boto:Token: None 2023-10-12T01:16:57.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj148 2023-10-12T01:16:57.360 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jUD5sWpFRxqPLQwzec4J53eTKC0= 2023-10-12T01:16:57.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jUD5sWpFRxqPLQwzec4J53eTKC0='} 2023-10-12T01:16:57.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000422f01a6af6ff81c-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.366 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:16:57.367 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:16:57.367 DEBUG:boto:Method: HEAD 2023-10-12T01:16:57.368 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:16:57.368 DEBUG:boto:Data: 2023-10-12T01:16:57.368 DEBUG:boto:Headers: {} 2023-10-12T01:16:57.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.369 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.369 DEBUG:boto:Params: {} 2023-10-12T01:16:57.370 DEBUG:boto:Token: None 2023-10-12T01:16:57.370 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/ 2023-10-12T01:16:57.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE= 2023-10-12T01:16:57.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:smMYwrlKulGsL3eM+0IAY4eYBUE=', 'Content-Length': '0'} 2023-10-12T01:16:57.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000895a0bffc82d8ec8-0065274909-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.375 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:16:57.376 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:16:57.376 DEBUG:boto:Method: PUT 2023-10-12T01:16:57.377 DEBUG:boto:Path: /wafmoa-61/obj185 2023-10-12T01:16:57.377 DEBUG:boto:Data: 2023-10-12T01:16:57.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:16:57.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:16:57.378 DEBUG:boto:Port: 8000 2023-10-12T01:16:57.379 DEBUG:boto:Params: {} 2023-10-12T01:16:57.379 DEBUG:boto:Token: None 2023-10-12T01:16:57.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:16:57 GMT /wafmoa-61/obj185 2023-10-12T01:16:57.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:05RF15v3QW0vW/qIXQ45w8WJ2/0= 2023-10-12T01:16:57.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:16:57 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:05RF15v3QW0vW/qIXQ45w8WJ2/0='} 2023-10-12T01:16:57.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003644510ba5390ac8-0065274909-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:16:57.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:57.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:16:57.424 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:16:57.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:57.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:57.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:57.507+0000 7f84437df780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-12T01:16:57.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:57.509+0000 7f84437df780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-12T01:16:57.530 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T01:16:57.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:16:57.532 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T01:16:57.533 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:57.553 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:16:57.553 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:16:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T01:16:57.554 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:16:58.556 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:16:58.578 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:16:58.578 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:16:58.579 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:16:58.580 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T01:16:58.580 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-61 2023-10-12T01:16:58.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:16:58.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:16:58.582 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:16:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:16:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:16:58.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:16:58.716+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:03.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:03.716+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:08.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:08.717+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:13.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:13.718+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:18.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:18.719+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:23.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:23.719+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:28.719+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:33.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:33.720+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:38.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:38.720+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:43.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:43.721+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:48.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:48.722+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:53.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:53.723+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:17:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:17:58.724+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:03.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:03.724+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:08.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:08.724+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:13.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:13.725+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:18.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:18.725+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:23.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:23.726+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:28.727+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:33.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:33.728+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:38.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:38.729+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:43.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:43.728+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:48.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:48.729+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:53.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:53.730+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:18:58.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:18:58.730+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:03.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:03.731+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:08.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:08.732+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:13.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:13.733+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:18.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:18.733+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:23.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:23.733+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:28.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:28.734+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:33.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:33.734+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:38.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:38.735+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:43.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:43.737+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:48.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:48.737+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:53.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:53.738+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:19:58.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:19:58.738+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:03.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:03.738+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:08.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:08.739+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:13.740+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:18.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:18.740+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:23.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:23.741+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:28.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:28.741+0000 7fdb7b45b780 1 waiting to reach incremental sync.. 2023-10-12T01:20:33.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:33.745+0000 7fdb7b45b780 1 bucket sync caught up with source: 2023-10-12T01:20:33.745 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.73.6, , , , , , 00000000002.65.6, , , , , 00000000002.80.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.88.6] 2023-10-12T01:20:33.746 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.73.6, , , , , , 00000000002.65.6, , , , , 00000000002.80.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.88.6] 2023-10-12T01:20:33.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:33.745+0000 7fdb7b45b780 0 bucket checkpoint complete 2023-10-12T01:20:33.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-61 2023-10-12T01:20:33.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:20:33.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:20:33.764 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:20:33.826 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:33.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:33.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:33.866+0000 7fbfc8a0a780 1 bucket sync caught up with source: 2023-10-12T01:20:33.866 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.232.6, 00000000002.207.6, 00000000002.191.6, 00000000005.172.6, 00000000004.371.6, 00000000001.205.6, 00000000001.158.6, 00000000004.190.6, 00000000007.146.6, 00000000006.385.6] 2023-10-12T01:20:33.866 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.232.6, 00000000002.207.6, 00000000002.191.6, 00000000005.172.6, 00000000004.371.6, 00000000001.205.6, 00000000001.158.6, 00000000004.190.6, 00000000007.146.6, 00000000006.385.6] 2023-10-12T01:20:33.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:33.866+0000 7fbfc8a0a780 0 bucket checkpoint complete 2023-10-12T01:20:33.881 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-61 zones={a2, a1} 2023-10-12T01:20:33.881 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.882 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.882 DEBUG:boto:Method: HEAD 2023-10-12T01:20:33.883 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:20:33.883 DEBUG:boto:Data: 2023-10-12T01:20:33.884 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:33.885 DEBUG:boto:Port: 8001 2023-10-12T01:20:33.885 DEBUG:boto:Params: {} 2023-10-12T01:20:33.886 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:20:33.886 DEBUG:boto:Token: None 2023-10-12T01:20:33.887 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/ 2023-10-12T01:20:33.887 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zO2huEx9WjurCdyRqvSyfhUB1rs= 2023-10-12T01:20:33.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zO2huEx9WjurCdyRqvSyfhUB1rs=', 'Content-Length': '0'} 2023-10-12T01:20:33.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c850bf60b949115a-00652749e1-1f78-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.894 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.895 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.896 DEBUG:boto:Method: HEAD 2023-10-12T01:20:33.896 DEBUG:boto:Path: /wafmoa-61/ 2023-10-12T01:20:33.896 DEBUG:boto:Data: 2023-10-12T01:20:33.897 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:33.897 DEBUG:boto:Port: 8000 2023-10-12T01:20:33.898 DEBUG:boto:Params: {} 2023-10-12T01:20:33.899 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:20:33.899 DEBUG:boto:Token: None 2023-10-12T01:20:33.900 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/ 2023-10-12T01:20:33.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zO2huEx9WjurCdyRqvSyfhUB1rs= 2023-10-12T01:20:33.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zO2huEx9WjurCdyRqvSyfhUB1rs=', 'Content-Length': '0'} 2023-10-12T01:20:33.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099eff34bc8dea169-00652749e1-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.907 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:20:33.907 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.908 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.908 DEBUG:boto:path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.909 DEBUG:boto:auth_path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.909 DEBUG:boto:Method: GET 2023-10-12T01:20:33.910 DEBUG:boto:Path: /wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.911 DEBUG:boto:Data: 2023-10-12T01:20:33.911 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:33.913 DEBUG:boto:Port: 8001 2023-10-12T01:20:33.914 DEBUG:boto:Params: {} 2023-10-12T01:20:33.914 DEBUG:boto:Token: None 2023-10-12T01:20:33.915 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/?versions 2023-10-12T01:20:33.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY= 2023-10-12T01:20:33.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY=', 'Content-Length': '0'} 2023-10-12T01:20:33.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2eff2c5a29a9608-00652749e1-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.919 DEBUG:boto:b'wafmoa-61999falseobj11nulltrue2023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:33.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:20:33.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:20:33.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:20:33.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:20:33.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:20:33.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:20:33.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:20:33.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:20:33.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:20:33.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:20:33.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:20:33.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:20:33.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:20:33.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:20:33.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:20:33.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:20:33.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:20:33.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:20:33.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:20:33.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:20:33.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:20:33.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:20:33.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:20:33.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:20:33.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:20:33.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:20:33.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:20:33.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:20:33.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:20:33.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:20:33.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:20:33.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:20:33.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:20:33.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:20:33.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:20:33.937 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:20:33.938 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.938 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.939 DEBUG:boto:path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.939 DEBUG:boto:auth_path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.939 DEBUG:boto:Method: GET 2023-10-12T01:20:33.940 DEBUG:boto:Path: /wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.940 DEBUG:boto:Data: 2023-10-12T01:20:33.941 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:33.942 DEBUG:boto:Port: 8000 2023-10-12T01:20:33.942 DEBUG:boto:Params: {} 2023-10-12T01:20:33.943 DEBUG:boto:Token: None 2023-10-12T01:20:33.944 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/?versions 2023-10-12T01:20:33.944 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY= 2023-10-12T01:20:33.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY=', 'Content-Length': '0'} 2023-10-12T01:20:33.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e124ace0d93b8e0-00652749e1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.950 DEBUG:boto:b'wafmoa-61999falseobj11nulltrue2023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:33.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:20:33.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-12T01:20:33.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-12T01:20:33.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-12T01:20:33.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-12T01:20:33.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-12T01:20:33.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-12T01:20:33.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-12T01:20:33.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-12T01:20:33.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-12T01:20:33.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-12T01:20:33.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:20:33.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-12T01:20:33.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-12T01:20:33.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-12T01:20:33.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:20:33.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-12T01:20:33.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-12T01:20:33.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-12T01:20:33.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:20:33.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-12T01:20:33.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-12T01:20:33.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:20:33.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-12T01:20:33.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-12T01:20:33.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-12T01:20:33.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-12T01:20:33.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-12T01:20:33.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-12T01:20:33.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-12T01:20:33.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-12T01:20:33.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-12T01:20:33.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-12T01:20:33.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-12T01:20:33.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-12T01:20:33.965 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.966 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.966 DEBUG:boto:path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.967 DEBUG:boto:auth_path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.967 DEBUG:boto:Method: GET 2023-10-12T01:20:33.967 DEBUG:boto:Path: /wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.968 DEBUG:boto:Data: 2023-10-12T01:20:33.968 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:33.969 DEBUG:boto:Port: 8001 2023-10-12T01:20:33.969 DEBUG:boto:Params: {} 2023-10-12T01:20:33.970 DEBUG:boto:Token: None 2023-10-12T01:20:33.970 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/?versions 2023-10-12T01:20:33.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY= 2023-10-12T01:20:33.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY=', 'Content-Length': '0'} 2023-10-12T01:20:33.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c223ce7271015a93-00652749e1-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.974 DEBUG:boto:b'wafmoa-61999falseobj11nulltrue2023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:33.976 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:33.977 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:33.977 DEBUG:boto:path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.977 DEBUG:boto:auth_path=/wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.978 DEBUG:boto:Method: GET 2023-10-12T01:20:33.978 DEBUG:boto:Path: /wafmoa-61/?versions&max-keys=999 2023-10-12T01:20:33.979 DEBUG:boto:Data: 2023-10-12T01:20:33.979 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:33.980 DEBUG:boto:Port: 8000 2023-10-12T01:20:33.980 DEBUG:boto:Params: {} 2023-10-12T01:20:33.981 DEBUG:boto:Token: None 2023-10-12T01:20:33.981 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/?versions 2023-10-12T01:20:33.982 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY= 2023-10-12T01:20:33.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1qo6YMjLgiTbpw9zh0RmibGduyY=', 'Content-Length': '0'} 2023-10-12T01:20:33.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b826c7d7137b933a-00652749e1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.987 DEBUG:boto:b'wafmoa-61999falseobj11nulltrue2023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:33.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:20:33.990 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:33.990 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:33.991 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:33.991 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:33.991 DEBUG:boto:Method: GET 2023-10-12T01:20:33.992 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:33.992 DEBUG:boto:Data: 2023-10-12T01:20:33.993 DEBUG:boto:Headers: {} 2023-10-12T01:20:33.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:33.994 DEBUG:boto:Port: 8001 2023-10-12T01:20:33.994 DEBUG:boto:Params: {} 2023-10-12T01:20:33.994 DEBUG:boto:Token: None 2023-10-12T01:20:33.995 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:33 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:33.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FSJMTBsXF6Z78Wj3K7bd/wBpvu0= 2023-10-12T01:20:33.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FSJMTBsXF6Z78Wj3K7bd/wBpvu0=', 'Content-Length': '0'} 2023-10-12T01:20:33.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da9fb072e97d728-00652749e1-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:33.999 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:33.999 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.000 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.000 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.000 DEBUG:boto:Method: GET 2023-10-12T01:20:34.001 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.001 DEBUG:boto:Data: 2023-10-12T01:20:34.002 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.003 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.003 DEBUG:boto:Params: {} 2023-10-12T01:20:34.004 DEBUG:boto:Token: None 2023-10-12T01:20:34.004 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.005 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M= 2023-10-12T01:20:34.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M=', 'Content-Length': '0'} 2023-10-12T01:20:34.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904c2136ab02dcce-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.050 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.051 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.051 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.052 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.052 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.052 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.053 DEBUG:boto:Data: 2023-10-12T01:20:34.053 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.054 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.055 DEBUG:boto:Params: {} 2023-10-12T01:20:34.055 DEBUG:boto:Token: None 2023-10-12T01:20:34.056 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ITIVLyFofnQ3T+aVNFgUj0uWfas= 2023-10-12T01:20:34.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ITIVLyFofnQ3T+aVNFgUj0uWfas=', 'Content-Length': '0'} 2023-10-12T01:20:34.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f403687a4792615-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.059 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.060 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.060 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.060 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.061 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.062 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.062 DEBUG:boto:Data: 2023-10-12T01:20:34.063 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.063 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.064 DEBUG:boto:Params: {} 2023-10-12T01:20:34.064 DEBUG:boto:Token: None 2023-10-12T01:20:34.065 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ITIVLyFofnQ3T+aVNFgUj0uWfas= 2023-10-12T01:20:34.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ITIVLyFofnQ3T+aVNFgUj0uWfas=', 'Content-Length': '0'} 2023-10-12T01:20:34.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000503956bb07f61bf0-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:20:34.069 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.069 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.070 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.070 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.071 DEBUG:boto:Method: GET 2023-10-12T01:20:34.071 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.072 DEBUG:boto:Data: 2023-10-12T01:20:34.072 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.073 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.073 DEBUG:boto:Params: {} 2023-10-12T01:20:34.074 DEBUG:boto:Token: None 2023-10-12T01:20:34.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M= 2023-10-12T01:20:34.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M=', 'Content-Length': '0'} 2023-10-12T01:20:34.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd51d13717b9b685-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.078 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.079 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.079 DEBUG:boto:path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.079 DEBUG:boto:auth_path=/wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.080 DEBUG:boto:Method: GET 2023-10-12T01:20:34.080 DEBUG:boto:Path: /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.081 DEBUG:boto:Data: 2023-10-12T01:20:34.081 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.082 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.083 DEBUG:boto:Params: {} 2023-10-12T01:20:34.083 DEBUG:boto:Token: None 2023-10-12T01:20:34.083 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11?versionId=null 2023-10-12T01:20:34.084 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M= 2023-10-12T01:20:34.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lxyMMqPKTsHrXiLV6X7r/aU1C5M=', 'Content-Length': '0'} 2023-10-12T01:20:34.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a23019a980a563-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.087 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.087 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.088 DEBUG:boto:path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.089 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.089 DEBUG:boto:Method: GET 2023-10-12T01:20:34.090 DEBUG:boto:Path: /wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.090 DEBUG:boto:Data: 2023-10-12T01:20:34.090 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.091 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.092 DEBUG:boto:Params: {} 2023-10-12T01:20:34.092 DEBUG:boto:Token: None 2023-10-12T01:20:34.093 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.093 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d97da53e29e42f25-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.100 DEBUG:boto:b'wafmoa-61obj111000falseobj112023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.101 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.101 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.102 DEBUG:boto:path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.102 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.103 DEBUG:boto:Method: GET 2023-10-12T01:20:34.103 DEBUG:boto:Path: /wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.104 DEBUG:boto:Data: 2023-10-12T01:20:34.104 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.105 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.106 DEBUG:boto:Params: {} 2023-10-12T01:20:34.106 DEBUG:boto:Token: None 2023-10-12T01:20:34.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000502c7e577184a4a0-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.114 DEBUG:boto:b'wafmoa-61obj111000falseobj112023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.115 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.115 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.116 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.116 DEBUG:boto:Path: /wafmoa-61/obj11 2023-10-12T01:20:34.117 DEBUG:boto:Data: 2023-10-12T01:20:34.117 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.118 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.118 DEBUG:boto:Params: {} 2023-10-12T01:20:34.119 DEBUG:boto:Token: None 2023-10-12T01:20:34.119 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11 2023-10-12T01:20:34.120 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gNifWaNHrRVAdURJz2OWBA+NhmA= 2023-10-12T01:20:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gNifWaNHrRVAdURJz2OWBA+NhmA=', 'Content-Length': '0'} 2023-10-12T01:20:34.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9a0a00b7a5d6b0-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.123 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.123 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.123 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.124 DEBUG:boto:Path: /wafmoa-61/obj11 2023-10-12T01:20:34.124 DEBUG:boto:Data: 2023-10-12T01:20:34.125 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.125 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.126 DEBUG:boto:Params: {} 2023-10-12T01:20:34.126 DEBUG:boto:Token: None 2023-10-12T01:20:34.127 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11 2023-10-12T01:20:34.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gNifWaNHrRVAdURJz2OWBA+NhmA= 2023-10-12T01:20:34.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gNifWaNHrRVAdURJz2OWBA+NhmA=', 'Content-Length': '0'} 2023-10-12T01:20:34.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cdcafff25dcb8bd-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:20:34.130 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.131 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.131 DEBUG:boto:Method: GET 2023-10-12T01:20:34.132 DEBUG:boto:Path: /wafmoa-61/obj11 2023-10-12T01:20:34.132 DEBUG:boto:Data: 2023-10-12T01:20:34.133 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.134 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.134 DEBUG:boto:Params: {} 2023-10-12T01:20:34.135 DEBUG:boto:Token: None 2023-10-12T01:20:34.135 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11 2023-10-12T01:20:34.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDqFUihEn1l6G9+AQ7NgvgYUTbo= 2023-10-12T01:20:34.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDqFUihEn1l6G9+AQ7NgvgYUTbo=', 'Content-Length': '0'} 2023-10-12T01:20:34.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000501d630bb22edc98-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.140 DEBUG:boto:path=/wafmoa-61/obj11 2023-10-12T01:20:34.140 DEBUG:boto:auth_path=/wafmoa-61/obj11 2023-10-12T01:20:34.141 DEBUG:boto:Method: GET 2023-10-12T01:20:34.142 DEBUG:boto:Path: /wafmoa-61/obj11 2023-10-12T01:20:34.143 DEBUG:boto:Data: 2023-10-12T01:20:34.143 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.145 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.146 DEBUG:boto:Params: {} 2023-10-12T01:20:34.146 DEBUG:boto:Token: None 2023-10-12T01:20:34.147 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj11 2023-10-12T01:20:34.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDqFUihEn1l6G9+AQ7NgvgYUTbo= 2023-10-12T01:20:34.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDqFUihEn1l6G9+AQ7NgvgYUTbo=', 'Content-Length': '0'} 2023-10-12T01:20:34.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c878deefcf6d83-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.151 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.151 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.152 DEBUG:boto:path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.152 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.153 DEBUG:boto:Method: GET 2023-10-12T01:20:34.153 DEBUG:boto:Path: /wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.154 DEBUG:boto:Data: 2023-10-12T01:20:34.154 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.155 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.156 DEBUG:boto:Params: {} 2023-10-12T01:20:34.156 DEBUG:boto:Token: None 2023-10-12T01:20:34.157 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c06f992c3ba72098-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.161 DEBUG:boto:b'wafmoa-61obj111000falseobj112023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.162 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.162 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.163 DEBUG:boto:path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.163 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.164 DEBUG:boto:Method: GET 2023-10-12T01:20:34.164 DEBUG:boto:Path: /wafmoa-61/?prefix=obj11 2023-10-12T01:20:34.165 DEBUG:boto:Data: 2023-10-12T01:20:34.165 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.167 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.167 DEBUG:boto:Params: {} 2023-10-12T01:20:34.168 DEBUG:boto:Token: None 2023-10-12T01:20:34.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.169 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b791260a568b9ef-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.173 DEBUG:boto:b'wafmoa-61obj111000falseobj112023-10-12T01:16:55.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:20:34.175 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.175 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.176 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.176 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.177 DEBUG:boto:Method: GET 2023-10-12T01:20:34.177 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.177 DEBUG:boto:Data: 2023-10-12T01:20:34.178 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.179 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.179 DEBUG:boto:Params: {} 2023-10-12T01:20:34.180 DEBUG:boto:Token: None 2023-10-12T01:20:34.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs= 2023-10-12T01:20:34.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs=', 'Content-Length': '0'} 2023-10-12T01:20:34.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066589874eba1a20b-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.226 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.227 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.227 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.228 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.228 DEBUG:boto:Method: GET 2023-10-12T01:20:34.229 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.229 DEBUG:boto:Data: 2023-10-12T01:20:34.230 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.231 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.231 DEBUG:boto:Params: {} 2023-10-12T01:20:34.231 DEBUG:boto:Token: None 2023-10-12T01:20:34.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs= 2023-10-12T01:20:34.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs=', 'Content-Length': '0'} 2023-10-12T01:20:34.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a8bc3229165c60-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.235 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.236 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.236 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.237 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.237 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.237 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.238 DEBUG:boto:Data: 2023-10-12T01:20:34.238 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.239 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.239 DEBUG:boto:Params: {} 2023-10-12T01:20:34.240 DEBUG:boto:Token: None 2023-10-12T01:20:34.240 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.241 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1o0yuBG7qgKhwu5+glw7/Zq9W40= 2023-10-12T01:20:34.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1o0yuBG7qgKhwu5+glw7/Zq9W40=', 'Content-Length': '0'} 2023-10-12T01:20:34.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4dd8a0e05b1c858-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.243 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.244 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.244 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.244 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.245 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.245 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.246 DEBUG:boto:Data: 2023-10-12T01:20:34.246 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.247 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.247 DEBUG:boto:Params: {} 2023-10-12T01:20:34.247 DEBUG:boto:Token: None 2023-10-12T01:20:34.248 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1o0yuBG7qgKhwu5+glw7/Zq9W40= 2023-10-12T01:20:34.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1o0yuBG7qgKhwu5+glw7/Zq9W40=', 'Content-Length': '0'} 2023-10-12T01:20:34.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e75b8a7ab8fca2-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:20:34.251 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.252 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.252 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.253 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.253 DEBUG:boto:Method: GET 2023-10-12T01:20:34.254 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.254 DEBUG:boto:Data: 2023-10-12T01:20:34.254 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.255 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.256 DEBUG:boto:Params: {} 2023-10-12T01:20:34.256 DEBUG:boto:Token: None 2023-10-12T01:20:34.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.257 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs= 2023-10-12T01:20:34.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs=', 'Content-Length': '0'} 2023-10-12T01:20:34.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49281f36adc956b-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.260 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.261 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.261 DEBUG:boto:path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.261 DEBUG:boto:auth_path=/wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.262 DEBUG:boto:Method: GET 2023-10-12T01:20:34.262 DEBUG:boto:Path: /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.263 DEBUG:boto:Data: 2023-10-12T01:20:34.263 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.264 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.264 DEBUG:boto:Params: {} 2023-10-12T01:20:34.265 DEBUG:boto:Token: None 2023-10-12T01:20:34.265 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111?versionId=null 2023-10-12T01:20:34.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs= 2023-10-12T01:20:34.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HQY6eDCLX8o8ihtoe2uZ+BwNXQs=', 'Content-Length': '0'} 2023-10-12T01:20:34.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1d4abe51b8cfc2-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.310 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.311 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.311 DEBUG:boto:path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.312 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.312 DEBUG:boto:Method: GET 2023-10-12T01:20:34.313 DEBUG:boto:Path: /wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.313 DEBUG:boto:Data: 2023-10-12T01:20:34.313 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.314 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.315 DEBUG:boto:Params: {} 2023-10-12T01:20:34.315 DEBUG:boto:Token: None 2023-10-12T01:20:34.316 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651835349c3aee9d-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.320 DEBUG:boto:b'wafmoa-61obj1111000falseobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.321 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.321 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.321 DEBUG:boto:path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.322 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.322 DEBUG:boto:Method: GET 2023-10-12T01:20:34.323 DEBUG:boto:Path: /wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.323 DEBUG:boto:Data: 2023-10-12T01:20:34.324 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.325 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.325 DEBUG:boto:Params: {} 2023-10-12T01:20:34.325 DEBUG:boto:Token: None 2023-10-12T01:20:34.326 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b443304a7d24db20-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.331 DEBUG:boto:b'wafmoa-61obj1111000falseobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.332 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.333 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.333 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.333 DEBUG:boto:Path: /wafmoa-61/obj111 2023-10-12T01:20:34.334 DEBUG:boto:Data: 2023-10-12T01:20:34.334 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.335 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.335 DEBUG:boto:Params: {} 2023-10-12T01:20:34.336 DEBUG:boto:Token: None 2023-10-12T01:20:34.336 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111 2023-10-12T01:20:34.337 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S0yr5GO+FOadX7CfL5JYIXzIU7w= 2023-10-12T01:20:34.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S0yr5GO+FOadX7CfL5JYIXzIU7w=', 'Content-Length': '0'} 2023-10-12T01:20:34.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fa729e2d0e51aa-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.340 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.340 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.340 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.341 DEBUG:boto:Path: /wafmoa-61/obj111 2023-10-12T01:20:34.341 DEBUG:boto:Data: 2023-10-12T01:20:34.342 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.343 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.343 DEBUG:boto:Params: {} 2023-10-12T01:20:34.343 DEBUG:boto:Token: None 2023-10-12T01:20:34.344 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111 2023-10-12T01:20:34.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S0yr5GO+FOadX7CfL5JYIXzIU7w= 2023-10-12T01:20:34.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S0yr5GO+FOadX7CfL5JYIXzIU7w=', 'Content-Length': '0'} 2023-10-12T01:20:34.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55c2af236a19ae1-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-12T01:20:34.348 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.348 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.349 DEBUG:boto:Method: GET 2023-10-12T01:20:34.349 DEBUG:boto:Path: /wafmoa-61/obj111 2023-10-12T01:20:34.349 DEBUG:boto:Data: 2023-10-12T01:20:34.350 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.351 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.351 DEBUG:boto:Params: {} 2023-10-12T01:20:34.352 DEBUG:boto:Token: None 2023-10-12T01:20:34.352 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111 2023-10-12T01:20:34.353 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RRtb+t2iFTbbufOmA/PsEy2PYQo= 2023-10-12T01:20:34.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RRtb+t2iFTbbufOmA/PsEy2PYQo=', 'Content-Length': '0'} 2023-10-12T01:20:34.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fb47525c2438ae-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.398 DEBUG:boto:path=/wafmoa-61/obj111 2023-10-12T01:20:34.399 DEBUG:boto:auth_path=/wafmoa-61/obj111 2023-10-12T01:20:34.400 DEBUG:boto:Method: GET 2023-10-12T01:20:34.400 DEBUG:boto:Path: /wafmoa-61/obj111 2023-10-12T01:20:34.400 DEBUG:boto:Data: 2023-10-12T01:20:34.401 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.402 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.402 DEBUG:boto:Params: {} 2023-10-12T01:20:34.403 DEBUG:boto:Token: None 2023-10-12T01:20:34.403 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj111 2023-10-12T01:20:34.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RRtb+t2iFTbbufOmA/PsEy2PYQo= 2023-10-12T01:20:34.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RRtb+t2iFTbbufOmA/PsEy2PYQo=', 'Content-Length': '0'} 2023-10-12T01:20:34.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ce77020895cd3d-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.407 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.407 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.408 DEBUG:boto:path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.408 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.409 DEBUG:boto:Method: GET 2023-10-12T01:20:34.409 DEBUG:boto:Path: /wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.409 DEBUG:boto:Data: 2023-10-12T01:20:34.410 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.411 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.411 DEBUG:boto:Params: {} 2023-10-12T01:20:34.412 DEBUG:boto:Token: None 2023-10-12T01:20:34.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfe9f0ded00f318-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.416 DEBUG:boto:b'wafmoa-61obj1111000falseobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.417 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.418 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.418 DEBUG:boto:path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.418 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.419 DEBUG:boto:Method: GET 2023-10-12T01:20:34.419 DEBUG:boto:Path: /wafmoa-61/?prefix=obj111 2023-10-12T01:20:34.420 DEBUG:boto:Data: 2023-10-12T01:20:34.420 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.421 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.422 DEBUG:boto:Params: {} 2023-10-12T01:20:34.422 DEBUG:boto:Token: None 2023-10-12T01:20:34.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.423 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4bfc5c701daa48-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.470 DEBUG:boto:b'wafmoa-61obj1111000falseobj1112023-10-12T01:16:57.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:20:34.472 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.472 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.473 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.473 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.474 DEBUG:boto:Method: GET 2023-10-12T01:20:34.474 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.474 DEBUG:boto:Data: 2023-10-12T01:20:34.475 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.476 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.476 DEBUG:boto:Params: {} 2023-10-12T01:20:34.477 DEBUG:boto:Token: None 2023-10-12T01:20:34.477 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg= 2023-10-12T01:20:34.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg=', 'Content-Length': '0'} 2023-10-12T01:20:34.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5071af536dcd677-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.481 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.482 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.482 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.483 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.483 DEBUG:boto:Method: GET 2023-10-12T01:20:34.484 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.484 DEBUG:boto:Data: 2023-10-12T01:20:34.484 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.485 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.486 DEBUG:boto:Params: {} 2023-10-12T01:20:34.486 DEBUG:boto:Token: None 2023-10-12T01:20:34.487 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.487 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg= 2023-10-12T01:20:34.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg=', 'Content-Length': '0'} 2023-10-12T01:20:34.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382e11c7506af717-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.490 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.490 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.491 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.491 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.492 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.492 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.492 DEBUG:boto:Data: 2023-10-12T01:20:34.493 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.494 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.494 DEBUG:boto:Params: {} 2023-10-12T01:20:34.495 DEBUG:boto:Token: None 2023-10-12T01:20:34.495 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.495 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pE+LkTs8oyAaZ1be4APQ+p9UyzY= 2023-10-12T01:20:34.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pE+LkTs8oyAaZ1be4APQ+p9UyzY=', 'Content-Length': '0'} 2023-10-12T01:20:34.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ef480bcb08fa7b-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.498 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.498 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.499 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.499 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.500 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.500 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.500 DEBUG:boto:Data: 2023-10-12T01:20:34.501 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.502 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.502 DEBUG:boto:Params: {} 2023-10-12T01:20:34.503 DEBUG:boto:Token: None 2023-10-12T01:20:34.503 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.504 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pE+LkTs8oyAaZ1be4APQ+p9UyzY= 2023-10-12T01:20:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pE+LkTs8oyAaZ1be4APQ+p9UyzY=', 'Content-Length': '0'} 2023-10-12T01:20:34.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebaec07a8d0616cb-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:20:34.507 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.509 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.510 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.510 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.510 DEBUG:boto:Method: GET 2023-10-12T01:20:34.511 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.511 DEBUG:boto:Data: 2023-10-12T01:20:34.512 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.513 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.513 DEBUG:boto:Params: {} 2023-10-12T01:20:34.514 DEBUG:boto:Token: None 2023-10-12T01:20:34.514 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg= 2023-10-12T01:20:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg=', 'Content-Length': '0'} 2023-10-12T01:20:34.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d461bfa96afee8-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.518 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.518 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.519 DEBUG:boto:path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.519 DEBUG:boto:auth_path=/wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.520 DEBUG:boto:Method: GET 2023-10-12T01:20:34.520 DEBUG:boto:Path: /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.520 DEBUG:boto:Data: 2023-10-12T01:20:34.521 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.522 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.522 DEBUG:boto:Params: {} 2023-10-12T01:20:34.523 DEBUG:boto:Token: None 2023-10-12T01:20:34.523 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115?versionId=null 2023-10-12T01:20:34.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg= 2023-10-12T01:20:34.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Rj8plB44SUhnvjDbmPmSFtThzHg=', 'Content-Length': '0'} 2023-10-12T01:20:34.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043759d5eaf88249b-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.527 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.527 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.528 DEBUG:boto:path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.528 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.528 DEBUG:boto:Method: GET 2023-10-12T01:20:34.529 DEBUG:boto:Path: /wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.529 DEBUG:boto:Data: 2023-10-12T01:20:34.530 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.530 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.531 DEBUG:boto:Params: {} 2023-10-12T01:20:34.531 DEBUG:boto:Token: None 2023-10-12T01:20:34.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bc56ba48d5fd8c-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.535 DEBUG:boto:b'wafmoa-61obj1151000falseobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.536 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.537 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.537 DEBUG:boto:path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.537 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.538 DEBUG:boto:Method: GET 2023-10-12T01:20:34.538 DEBUG:boto:Path: /wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.539 DEBUG:boto:Data: 2023-10-12T01:20:34.539 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.540 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.540 DEBUG:boto:Params: {} 2023-10-12T01:20:34.541 DEBUG:boto:Token: None 2023-10-12T01:20:34.541 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.542 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eac2f800a147b495-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.586 DEBUG:boto:b'wafmoa-61obj1151000falseobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.587 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.587 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.588 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.588 DEBUG:boto:Path: /wafmoa-61/obj115 2023-10-12T01:20:34.589 DEBUG:boto:Data: 2023-10-12T01:20:34.589 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.590 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.590 DEBUG:boto:Params: {} 2023-10-12T01:20:34.591 DEBUG:boto:Token: None 2023-10-12T01:20:34.591 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115 2023-10-12T01:20:34.592 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lbPt3QF06m3JOfXUoYOzHOBDJ1E= 2023-10-12T01:20:34.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lbPt3QF06m3JOfXUoYOzHOBDJ1E=', 'Content-Length': '0'} 2023-10-12T01:20:34.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf41f7350a94a756-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.595 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.595 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.596 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.596 DEBUG:boto:Path: /wafmoa-61/obj115 2023-10-12T01:20:34.597 DEBUG:boto:Data: 2023-10-12T01:20:34.597 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.598 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.598 DEBUG:boto:Params: {} 2023-10-12T01:20:34.599 DEBUG:boto:Token: None 2023-10-12T01:20:34.599 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115 2023-10-12T01:20:34.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lbPt3QF06m3JOfXUoYOzHOBDJ1E= 2023-10-12T01:20:34.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lbPt3QF06m3JOfXUoYOzHOBDJ1E=', 'Content-Length': '0'} 2023-10-12T01:20:34.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080761fb0145e882f-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-12T01:20:34.603 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.603 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.604 DEBUG:boto:Method: GET 2023-10-12T01:20:34.604 DEBUG:boto:Path: /wafmoa-61/obj115 2023-10-12T01:20:34.605 DEBUG:boto:Data: 2023-10-12T01:20:34.605 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.606 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.606 DEBUG:boto:Params: {} 2023-10-12T01:20:34.607 DEBUG:boto:Token: None 2023-10-12T01:20:34.607 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115 2023-10-12T01:20:34.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYsp7QE4GDqqySEBYnQEU9FpzjQ= 2023-10-12T01:20:34.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYsp7QE4GDqqySEBYnQEU9FpzjQ=', 'Content-Length': '0'} 2023-10-12T01:20:34.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a47e15e413743f-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.611 DEBUG:boto:path=/wafmoa-61/obj115 2023-10-12T01:20:34.611 DEBUG:boto:auth_path=/wafmoa-61/obj115 2023-10-12T01:20:34.612 DEBUG:boto:Method: GET 2023-10-12T01:20:34.612 DEBUG:boto:Path: /wafmoa-61/obj115 2023-10-12T01:20:34.612 DEBUG:boto:Data: 2023-10-12T01:20:34.613 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.614 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.614 DEBUG:boto:Params: {} 2023-10-12T01:20:34.615 DEBUG:boto:Token: None 2023-10-12T01:20:34.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj115 2023-10-12T01:20:34.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LYsp7QE4GDqqySEBYnQEU9FpzjQ= 2023-10-12T01:20:34.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LYsp7QE4GDqqySEBYnQEU9FpzjQ=', 'Content-Length': '0'} 2023-10-12T01:20:34.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e9c58cc3aaf73a-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.619 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.619 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.620 DEBUG:boto:path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.620 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.620 DEBUG:boto:Method: GET 2023-10-12T01:20:34.621 DEBUG:boto:Path: /wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.621 DEBUG:boto:Data: 2023-10-12T01:20:34.622 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.623 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.623 DEBUG:boto:Params: {} 2023-10-12T01:20:34.624 DEBUG:boto:Token: None 2023-10-12T01:20:34.624 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.625 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2082378e0ab8e7-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.670 DEBUG:boto:b'wafmoa-61obj1151000falseobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.671 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.672 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.672 DEBUG:boto:path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.672 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.673 DEBUG:boto:Method: GET 2023-10-12T01:20:34.673 DEBUG:boto:Path: /wafmoa-61/?prefix=obj115 2023-10-12T01:20:34.674 DEBUG:boto:Data: 2023-10-12T01:20:34.674 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.675 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.675 DEBUG:boto:Params: {} 2023-10-12T01:20:34.676 DEBUG:boto:Token: None 2023-10-12T01:20:34.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003741c1fc1f46c2d1-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.682 DEBUG:boto:b'wafmoa-61obj1151000falseobj1152023-10-12T01:16:56.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:20:34.683 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.684 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.684 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.685 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.685 DEBUG:boto:Method: GET 2023-10-12T01:20:34.685 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.686 DEBUG:boto:Data: 2023-10-12T01:20:34.686 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.687 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.688 DEBUG:boto:Params: {} 2023-10-12T01:20:34.688 DEBUG:boto:Token: None 2023-10-12T01:20:34.689 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo= 2023-10-12T01:20:34.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo=', 'Content-Length': '0'} 2023-10-12T01:20:34.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9cf54a5ee42c97a-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.692 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.693 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.693 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.694 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.694 DEBUG:boto:Method: GET 2023-10-12T01:20:34.695 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.695 DEBUG:boto:Data: 2023-10-12T01:20:34.696 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.696 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.697 DEBUG:boto:Params: {} 2023-10-12T01:20:34.697 DEBUG:boto:Token: None 2023-10-12T01:20:34.698 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.698 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo= 2023-10-12T01:20:34.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo=', 'Content-Length': '0'} 2023-10-12T01:20:34.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009551d7a90b06368e-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.701 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.702 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.702 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.703 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.703 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.704 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.704 DEBUG:boto:Data: 2023-10-12T01:20:34.704 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.705 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.706 DEBUG:boto:Params: {} 2023-10-12T01:20:34.706 DEBUG:boto:Token: None 2023-10-12T01:20:34.707 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.707 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pm55IOvdmN9IzGupt1vDmNUinhE= 2023-10-12T01:20:34.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pm55IOvdmN9IzGupt1vDmNUinhE=', 'Content-Length': '0'} 2023-10-12T01:20:34.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000facc7e7fa3dbe464-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.710 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.710 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.711 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.711 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.711 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.712 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.712 DEBUG:boto:Data: 2023-10-12T01:20:34.713 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.714 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.714 DEBUG:boto:Params: {} 2023-10-12T01:20:34.717 DEBUG:boto:Token: None 2023-10-12T01:20:34.718 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pm55IOvdmN9IzGupt1vDmNUinhE= 2023-10-12T01:20:34.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pm55IOvdmN9IzGupt1vDmNUinhE=', 'Content-Length': '0'} 2023-10-12T01:20:34.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613855826e0be98a-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:20:34.722 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.722 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.723 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.723 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.724 DEBUG:boto:Method: GET 2023-10-12T01:20:34.724 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.724 DEBUG:boto:Data: 2023-10-12T01:20:34.725 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.726 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.726 DEBUG:boto:Params: {} 2023-10-12T01:20:34.727 DEBUG:boto:Token: None 2023-10-12T01:20:34.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo= 2023-10-12T01:20:34.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo=', 'Content-Length': '0'} 2023-10-12T01:20:34.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067060d46e08c9bcb-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.730 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.731 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.731 DEBUG:boto:path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.732 DEBUG:boto:auth_path=/wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.732 DEBUG:boto:Method: GET 2023-10-12T01:20:34.733 DEBUG:boto:Path: /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.733 DEBUG:boto:Data: 2023-10-12T01:20:34.734 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.734 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.735 DEBUG:boto:Params: {} 2023-10-12T01:20:34.735 DEBUG:boto:Token: None 2023-10-12T01:20:34.736 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116?versionId=null 2023-10-12T01:20:34.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo= 2023-10-12T01:20:34.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MEdI6UhpmHFBMNzF91n/JHoeaKo=', 'Content-Length': '0'} 2023-10-12T01:20:34.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b850b376938964-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.739 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.740 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.740 DEBUG:boto:path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.741 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.741 DEBUG:boto:Method: GET 2023-10-12T01:20:34.742 DEBUG:boto:Path: /wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.742 DEBUG:boto:Data: 2023-10-12T01:20:34.743 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.743 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.744 DEBUG:boto:Params: {} 2023-10-12T01:20:34.744 DEBUG:boto:Token: None 2023-10-12T01:20:34.745 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168f929f7d885965-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.749 DEBUG:boto:b'wafmoa-61obj1161000falseobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.750 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.750 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.751 DEBUG:boto:path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.751 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.751 DEBUG:boto:Method: GET 2023-10-12T01:20:34.752 DEBUG:boto:Path: /wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.752 DEBUG:boto:Data: 2023-10-12T01:20:34.753 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.754 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.754 DEBUG:boto:Params: {} 2023-10-12T01:20:34.755 DEBUG:boto:Token: None 2023-10-12T01:20:34.755 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a1cc680461010b-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.760 DEBUG:boto:b'wafmoa-61obj1161000falseobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.761 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.761 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.762 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.762 DEBUG:boto:Path: /wafmoa-61/obj116 2023-10-12T01:20:34.763 DEBUG:boto:Data: 2023-10-12T01:20:34.763 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.764 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.764 DEBUG:boto:Params: {} 2023-10-12T01:20:34.765 DEBUG:boto:Token: None 2023-10-12T01:20:34.765 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116 2023-10-12T01:20:34.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NvVSQJQRX16iAk3BikC+mIiW8pQ= 2023-10-12T01:20:34.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NvVSQJQRX16iAk3BikC+mIiW8pQ=', 'Content-Length': '0'} 2023-10-12T01:20:34.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0b8dfbd1011656-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.769 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.769 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.769 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.770 DEBUG:boto:Path: /wafmoa-61/obj116 2023-10-12T01:20:34.770 DEBUG:boto:Data: 2023-10-12T01:20:34.771 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.772 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.772 DEBUG:boto:Params: {} 2023-10-12T01:20:34.772 DEBUG:boto:Token: None 2023-10-12T01:20:34.773 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116 2023-10-12T01:20:34.773 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NvVSQJQRX16iAk3BikC+mIiW8pQ= 2023-10-12T01:20:34.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NvVSQJQRX16iAk3BikC+mIiW8pQ=', 'Content-Length': '0'} 2023-10-12T01:20:34.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054d12b678db1357-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-12T01:20:34.776 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.777 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.777 DEBUG:boto:Method: GET 2023-10-12T01:20:34.777 DEBUG:boto:Path: /wafmoa-61/obj116 2023-10-12T01:20:34.778 DEBUG:boto:Data: 2023-10-12T01:20:34.778 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.779 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.780 DEBUG:boto:Params: {} 2023-10-12T01:20:34.780 DEBUG:boto:Token: None 2023-10-12T01:20:34.781 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116 2023-10-12T01:20:34.781 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PPBkDv58q5squ8awKn/u9OqXiR0= 2023-10-12T01:20:34.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PPBkDv58q5squ8awKn/u9OqXiR0=', 'Content-Length': '0'} 2023-10-12T01:20:34.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb82716cd3c35437-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.784 DEBUG:boto:path=/wafmoa-61/obj116 2023-10-12T01:20:34.785 DEBUG:boto:auth_path=/wafmoa-61/obj116 2023-10-12T01:20:34.785 DEBUG:boto:Method: GET 2023-10-12T01:20:34.786 DEBUG:boto:Path: /wafmoa-61/obj116 2023-10-12T01:20:34.786 DEBUG:boto:Data: 2023-10-12T01:20:34.786 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.787 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.788 DEBUG:boto:Params: {} 2023-10-12T01:20:34.788 DEBUG:boto:Token: None 2023-10-12T01:20:34.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj116 2023-10-12T01:20:34.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PPBkDv58q5squ8awKn/u9OqXiR0= 2023-10-12T01:20:34.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PPBkDv58q5squ8awKn/u9OqXiR0=', 'Content-Length': '0'} 2023-10-12T01:20:34.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c2eec138877248-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.834 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.835 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.835 DEBUG:boto:path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.836 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.836 DEBUG:boto:Method: GET 2023-10-12T01:20:34.837 DEBUG:boto:Path: /wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.837 DEBUG:boto:Data: 2023-10-12T01:20:34.837 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.838 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.839 DEBUG:boto:Params: {} 2023-10-12T01:20:34.839 DEBUG:boto:Token: None 2023-10-12T01:20:34.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e451d1e9de14e653-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.844 DEBUG:boto:b'wafmoa-61obj1161000falseobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.845 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.845 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.846 DEBUG:boto:path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.846 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.847 DEBUG:boto:Method: GET 2023-10-12T01:20:34.847 DEBUG:boto:Path: /wafmoa-61/?prefix=obj116 2023-10-12T01:20:34.847 DEBUG:boto:Data: 2023-10-12T01:20:34.848 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.849 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.849 DEBUG:boto:Params: {} 2023-10-12T01:20:34.850 DEBUG:boto:Token: None 2023-10-12T01:20:34.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094c5a45c243f3c6-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.855 DEBUG:boto:b'wafmoa-61obj1161000falseobj1162023-10-12T01:16:56.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:20:34.857 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.857 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.857 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.858 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.858 DEBUG:boto:Method: GET 2023-10-12T01:20:34.859 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.859 DEBUG:boto:Data: 2023-10-12T01:20:34.860 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.861 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.861 DEBUG:boto:Params: {} 2023-10-12T01:20:34.862 DEBUG:boto:Token: None 2023-10-12T01:20:34.862 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.863 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw= 2023-10-12T01:20:34.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw=', 'Content-Length': '0'} 2023-10-12T01:20:34.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777596f4343e82f4-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.866 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.866 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.866 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.867 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.867 DEBUG:boto:Method: GET 2023-10-12T01:20:34.868 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.868 DEBUG:boto:Data: 2023-10-12T01:20:34.869 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.870 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.870 DEBUG:boto:Params: {} 2023-10-12T01:20:34.870 DEBUG:boto:Token: None 2023-10-12T01:20:34.871 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw= 2023-10-12T01:20:34.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw=', 'Content-Length': '0'} 2023-10-12T01:20:34.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000995f38ce0f9a3e68-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.875 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.875 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.875 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.876 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.876 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.877 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.877 DEBUG:boto:Data: 2023-10-12T01:20:34.877 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.878 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.879 DEBUG:boto:Params: {} 2023-10-12T01:20:34.879 DEBUG:boto:Token: None 2023-10-12T01:20:34.880 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.880 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5cUL3FtuwG58RZpOE0z38tOkI3U= 2023-10-12T01:20:34.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5cUL3FtuwG58RZpOE0z38tOkI3U=', 'Content-Length': '0'} 2023-10-12T01:20:34.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6653a2fa60c7c38-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.883 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.883 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.884 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.884 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.885 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.885 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.885 DEBUG:boto:Data: 2023-10-12T01:20:34.886 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.887 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.887 DEBUG:boto:Params: {} 2023-10-12T01:20:34.888 DEBUG:boto:Token: None 2023-10-12T01:20:34.888 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.888 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5cUL3FtuwG58RZpOE0z38tOkI3U= 2023-10-12T01:20:34.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5cUL3FtuwG58RZpOE0z38tOkI3U=', 'Content-Length': '0'} 2023-10-12T01:20:34.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662e4587b6f620b8-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:20:34.892 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.892 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.893 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.893 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.894 DEBUG:boto:Method: GET 2023-10-12T01:20:34.894 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.895 DEBUG:boto:Data: 2023-10-12T01:20:34.895 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.896 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.896 DEBUG:boto:Params: {} 2023-10-12T01:20:34.897 DEBUG:boto:Token: None 2023-10-12T01:20:34.897 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw= 2023-10-12T01:20:34.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw=', 'Content-Length': '0'} 2023-10-12T01:20:34.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d568c5464ec590-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.942 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.943 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.943 DEBUG:boto:path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.944 DEBUG:boto:auth_path=/wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.944 DEBUG:boto:Method: GET 2023-10-12T01:20:34.945 DEBUG:boto:Path: /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.945 DEBUG:boto:Data: 2023-10-12T01:20:34.946 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.947 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.947 DEBUG:boto:Params: {} 2023-10-12T01:20:34.948 DEBUG:boto:Token: None 2023-10-12T01:20:34.948 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124?versionId=null 2023-10-12T01:20:34.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw= 2023-10-12T01:20:34.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WXpxGn2RdMU3iE6HaSTeP3Fmhvw=', 'Content-Length': '0'} 2023-10-12T01:20:34.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254e079b36732d5d-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.952 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.953 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.953 DEBUG:boto:path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.954 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.954 DEBUG:boto:Method: GET 2023-10-12T01:20:34.955 DEBUG:boto:Path: /wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.955 DEBUG:boto:Data: 2023-10-12T01:20:34.956 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.957 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.957 DEBUG:boto:Params: {} 2023-10-12T01:20:34.958 DEBUG:boto:Token: None 2023-10-12T01:20:34.958 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000169c7eb51e139829-00652749e2-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.962 DEBUG:boto:b'wafmoa-61obj1241000falseobj1242023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.963 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:34.964 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:34.964 DEBUG:boto:path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.964 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.965 DEBUG:boto:Method: GET 2023-10-12T01:20:34.965 DEBUG:boto:Path: /wafmoa-61/?prefix=obj124 2023-10-12T01:20:34.966 DEBUG:boto:Data: 2023-10-12T01:20:34.967 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.968 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.968 DEBUG:boto:Params: {} 2023-10-12T01:20:34.969 DEBUG:boto:Token: None 2023-10-12T01:20:34.969 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/ 2023-10-12T01:20:34.970 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs= 2023-10-12T01:20:34.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MwE9xRNfalIPrW6o5PjOr+u8TCs=', 'Content-Length': '0'} 2023-10-12T01:20:34.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513e7dfdf68abf5a-00652749e2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.974 DEBUG:boto:b'wafmoa-61obj1241000falseobj1242023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:34.975 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.975 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.976 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.976 DEBUG:boto:Path: /wafmoa-61/obj124 2023-10-12T01:20:34.977 DEBUG:boto:Data: 2023-10-12T01:20:34.977 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.978 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.978 DEBUG:boto:Params: {} 2023-10-12T01:20:34.979 DEBUG:boto:Token: None 2023-10-12T01:20:34.979 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124 2023-10-12T01:20:34.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eEwFwxzvdWDWjgkOKmsh2HCNbl4= 2023-10-12T01:20:34.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eEwFwxzvdWDWjgkOKmsh2HCNbl4=', 'Content-Length': '0'} 2023-10-12T01:20:34.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0a8cd34b1d4f13-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.983 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.983 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.983 DEBUG:boto:Method: HEAD 2023-10-12T01:20:34.984 DEBUG:boto:Path: /wafmoa-61/obj124 2023-10-12T01:20:34.984 DEBUG:boto:Data: 2023-10-12T01:20:34.985 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:34.985 DEBUG:boto:Port: 8000 2023-10-12T01:20:34.986 DEBUG:boto:Params: {} 2023-10-12T01:20:34.986 DEBUG:boto:Token: None 2023-10-12T01:20:34.987 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124 2023-10-12T01:20:34.987 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eEwFwxzvdWDWjgkOKmsh2HCNbl4= 2023-10-12T01:20:34.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eEwFwxzvdWDWjgkOKmsh2HCNbl4=', 'Content-Length': '0'} 2023-10-12T01:20:34.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d67042236e74678-00652749e2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-12T01:20:34.990 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.991 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.991 DEBUG:boto:Method: GET 2023-10-12T01:20:34.992 DEBUG:boto:Path: /wafmoa-61/obj124 2023-10-12T01:20:34.992 DEBUG:boto:Data: 2023-10-12T01:20:34.992 DEBUG:boto:Headers: {} 2023-10-12T01:20:34.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:34.993 DEBUG:boto:Port: 8001 2023-10-12T01:20:34.994 DEBUG:boto:Params: {} 2023-10-12T01:20:34.994 DEBUG:boto:Token: None 2023-10-12T01:20:34.994 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:34 GMT /wafmoa-61/obj124 2023-10-12T01:20:34.995 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5R6L5dIaHcSusfKEopNMZ6jA894= 2023-10-12T01:20:34.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:34 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5R6L5dIaHcSusfKEopNMZ6jA894=', 'Content-Length': '0'} 2023-10-12T01:20:34.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48da2646b0dc910-00652749e2-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:34.998 DEBUG:boto:path=/wafmoa-61/obj124 2023-10-12T01:20:34.999 DEBUG:boto:auth_path=/wafmoa-61/obj124 2023-10-12T01:20:34.999 DEBUG:boto:Method: GET 2023-10-12T01:20:34.999 DEBUG:boto:Path: /wafmoa-61/obj124 2023-10-12T01:20:35.000 DEBUG:boto:Data: 2023-10-12T01:20:35.000 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.001 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.001 DEBUG:boto:Params: {} 2023-10-12T01:20:35.002 DEBUG:boto:Token: None 2023-10-12T01:20:35.002 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj124 2023-10-12T01:20:35.003 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Obu5kqo3fuQeCCF7g0yNxO9QLfo= 2023-10-12T01:20:35.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Obu5kqo3fuQeCCF7g0yNxO9QLfo=', 'Content-Length': '0'} 2023-10-12T01:20:35.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7678c0860c3598-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.006 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.006 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.007 DEBUG:boto:path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.007 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.008 DEBUG:boto:Method: GET 2023-10-12T01:20:35.008 DEBUG:boto:Path: /wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.009 DEBUG:boto:Data: 2023-10-12T01:20:35.009 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.010 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.010 DEBUG:boto:Params: {} 2023-10-12T01:20:35.011 DEBUG:boto:Token: None 2023-10-12T01:20:35.011 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.012 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000919ad76152fb8794-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.015 DEBUG:boto:b'wafmoa-61obj1241000falseobj1242023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.016 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.016 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.017 DEBUG:boto:path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.017 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.018 DEBUG:boto:Method: GET 2023-10-12T01:20:35.018 DEBUG:boto:Path: /wafmoa-61/?prefix=obj124 2023-10-12T01:20:35.018 DEBUG:boto:Data: 2023-10-12T01:20:35.019 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.020 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.020 DEBUG:boto:Params: {} 2023-10-12T01:20:35.021 DEBUG:boto:Token: None 2023-10-12T01:20:35.021 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.021 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d9cdef035a3298-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.026 DEBUG:boto:b'wafmoa-61obj1241000falseobj1242023-10-12T01:16:56.964Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:20:35.027 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.028 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.028 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.029 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.029 DEBUG:boto:Method: GET 2023-10-12T01:20:35.030 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.030 DEBUG:boto:Data: 2023-10-12T01:20:35.031 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.031 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.032 DEBUG:boto:Params: {} 2023-10-12T01:20:35.032 DEBUG:boto:Token: None 2023-10-12T01:20:35.033 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.033 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo= 2023-10-12T01:20:35.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo=', 'Content-Length': '0'} 2023-10-12T01:20:35.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001212199bfc568f47-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.036 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.037 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.037 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.038 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.038 DEBUG:boto:Method: GET 2023-10-12T01:20:35.039 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.039 DEBUG:boto:Data: 2023-10-12T01:20:35.039 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.040 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.041 DEBUG:boto:Params: {} 2023-10-12T01:20:35.041 DEBUG:boto:Token: None 2023-10-12T01:20:35.041 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.042 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo= 2023-10-12T01:20:35.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo=', 'Content-Length': '0'} 2023-10-12T01:20:35.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b553c4fcc8072f51-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.045 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.046 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.046 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.047 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.047 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.047 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.048 DEBUG:boto:Data: 2023-10-12T01:20:35.048 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.049 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.049 DEBUG:boto:Params: {} 2023-10-12T01:20:35.050 DEBUG:boto:Token: None 2023-10-12T01:20:35.050 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ9Rcr0/HQY/k4cbEV0s7ZzFx3E= 2023-10-12T01:20:35.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ9Rcr0/HQY/k4cbEV0s7ZzFx3E=', 'Content-Length': '0'} 2023-10-12T01:20:35.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a260887455bd8146-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.054 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.054 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.055 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.055 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.055 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.056 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.056 DEBUG:boto:Data: 2023-10-12T01:20:35.057 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.057 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.058 DEBUG:boto:Params: {} 2023-10-12T01:20:35.058 DEBUG:boto:Token: None 2023-10-12T01:20:35.059 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.059 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FZ9Rcr0/HQY/k4cbEV0s7ZzFx3E= 2023-10-12T01:20:35.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FZ9Rcr0/HQY/k4cbEV0s7ZzFx3E=', 'Content-Length': '0'} 2023-10-12T01:20:35.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb22fb387eae5b91-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:20:35.062 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.063 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.063 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.064 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.064 DEBUG:boto:Method: GET 2023-10-12T01:20:35.064 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.065 DEBUG:boto:Data: 2023-10-12T01:20:35.065 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.066 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.067 DEBUG:boto:Params: {} 2023-10-12T01:20:35.067 DEBUG:boto:Token: None 2023-10-12T01:20:35.068 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo= 2023-10-12T01:20:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo=', 'Content-Length': '0'} 2023-10-12T01:20:35.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f675d71a8b268c19-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.071 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.071 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.072 DEBUG:boto:path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.072 DEBUG:boto:auth_path=/wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.073 DEBUG:boto:Method: GET 2023-10-12T01:20:35.073 DEBUG:boto:Path: /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.073 DEBUG:boto:Data: 2023-10-12T01:20:35.074 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.075 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.075 DEBUG:boto:Params: {} 2023-10-12T01:20:35.076 DEBUG:boto:Token: None 2023-10-12T01:20:35.076 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145?versionId=null 2023-10-12T01:20:35.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo= 2023-10-12T01:20:35.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:m4KGaOkGVyYjAQ0NRQBqIVLMgjo=', 'Content-Length': '0'} 2023-10-12T01:20:35.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007139dc9056996fa7-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.079 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.080 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.080 DEBUG:boto:path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.081 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.081 DEBUG:boto:Method: GET 2023-10-12T01:20:35.081 DEBUG:boto:Path: /wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.082 DEBUG:boto:Data: 2023-10-12T01:20:35.082 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.083 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.084 DEBUG:boto:Params: {} 2023-10-12T01:20:35.084 DEBUG:boto:Token: None 2023-10-12T01:20:35.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ccba42167dbeb8d-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.134 DEBUG:boto:b'wafmoa-61obj1451000falseobj1452023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.135 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.136 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.136 DEBUG:boto:path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.137 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.137 DEBUG:boto:Method: GET 2023-10-12T01:20:35.138 DEBUG:boto:Path: /wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.138 DEBUG:boto:Data: 2023-10-12T01:20:35.139 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.139 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.140 DEBUG:boto:Params: {} 2023-10-12T01:20:35.140 DEBUG:boto:Token: None 2023-10-12T01:20:35.141 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0718fdb1d746f12-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.146 DEBUG:boto:b'wafmoa-61obj1451000falseobj1452023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.147 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.148 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.148 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.148 DEBUG:boto:Path: /wafmoa-61/obj145 2023-10-12T01:20:35.149 DEBUG:boto:Data: 2023-10-12T01:20:35.149 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.150 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.150 DEBUG:boto:Params: {} 2023-10-12T01:20:35.151 DEBUG:boto:Token: None 2023-10-12T01:20:35.151 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145 2023-10-12T01:20:35.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bYIsP+Nb+x8GhfBPPl57HwYc/FE= 2023-10-12T01:20:35.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bYIsP+Nb+x8GhfBPPl57HwYc/FE=', 'Content-Length': '0'} 2023-10-12T01:20:35.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0703bd8ef3bd96-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.155 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.155 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.156 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.156 DEBUG:boto:Path: /wafmoa-61/obj145 2023-10-12T01:20:35.156 DEBUG:boto:Data: 2023-10-12T01:20:35.157 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.158 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.158 DEBUG:boto:Params: {} 2023-10-12T01:20:35.159 DEBUG:boto:Token: None 2023-10-12T01:20:35.159 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145 2023-10-12T01:20:35.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bYIsP+Nb+x8GhfBPPl57HwYc/FE= 2023-10-12T01:20:35.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bYIsP+Nb+x8GhfBPPl57HwYc/FE=', 'Content-Length': '0'} 2023-10-12T01:20:35.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805aa53e7cc4cd1e-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-12T01:20:35.163 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.163 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.164 DEBUG:boto:Method: GET 2023-10-12T01:20:35.164 DEBUG:boto:Path: /wafmoa-61/obj145 2023-10-12T01:20:35.165 DEBUG:boto:Data: 2023-10-12T01:20:35.165 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.166 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.166 DEBUG:boto:Params: {} 2023-10-12T01:20:35.167 DEBUG:boto:Token: None 2023-10-12T01:20:35.167 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145 2023-10-12T01:20:35.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezL2NmPDhED5quDwcMRZyif6L1Q= 2023-10-12T01:20:35.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezL2NmPDhED5quDwcMRZyif6L1Q=', 'Content-Length': '0'} 2023-10-12T01:20:35.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005087817b1e6eb7c0-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.171 DEBUG:boto:path=/wafmoa-61/obj145 2023-10-12T01:20:35.171 DEBUG:boto:auth_path=/wafmoa-61/obj145 2023-10-12T01:20:35.172 DEBUG:boto:Method: GET 2023-10-12T01:20:35.172 DEBUG:boto:Path: /wafmoa-61/obj145 2023-10-12T01:20:35.172 DEBUG:boto:Data: 2023-10-12T01:20:35.173 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.174 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.174 DEBUG:boto:Params: {} 2023-10-12T01:20:35.175 DEBUG:boto:Token: None 2023-10-12T01:20:35.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj145 2023-10-12T01:20:35.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ezL2NmPDhED5quDwcMRZyif6L1Q= 2023-10-12T01:20:35.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ezL2NmPDhED5quDwcMRZyif6L1Q=', 'Content-Length': '0'} 2023-10-12T01:20:35.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7dac5bd78a46c9-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.179 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.179 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.180 DEBUG:boto:path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.180 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.181 DEBUG:boto:Method: GET 2023-10-12T01:20:35.181 DEBUG:boto:Path: /wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.182 DEBUG:boto:Data: 2023-10-12T01:20:35.182 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.183 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.183 DEBUG:boto:Params: {} 2023-10-12T01:20:35.184 DEBUG:boto:Token: None 2023-10-12T01:20:35.184 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.185 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778b8e6af839aecb-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.188 DEBUG:boto:b'wafmoa-61obj1451000falseobj1452023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.189 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.189 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.189 DEBUG:boto:path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.190 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.190 DEBUG:boto:Method: GET 2023-10-12T01:20:35.191 DEBUG:boto:Path: /wafmoa-61/?prefix=obj145 2023-10-12T01:20:35.191 DEBUG:boto:Data: 2023-10-12T01:20:35.192 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.192 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.193 DEBUG:boto:Params: {} 2023-10-12T01:20:35.193 DEBUG:boto:Token: None 2023-10-12T01:20:35.194 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.194 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002682143fb1f703a6-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.199 DEBUG:boto:b'wafmoa-61obj1451000falseobj1452023-10-12T01:16:56.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:20:35.200 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.201 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.201 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.201 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.202 DEBUG:boto:Method: GET 2023-10-12T01:20:35.202 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.203 DEBUG:boto:Data: 2023-10-12T01:20:35.203 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.204 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.204 DEBUG:boto:Params: {} 2023-10-12T01:20:35.205 DEBUG:boto:Token: None 2023-10-12T01:20:35.205 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk= 2023-10-12T01:20:35.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk=', 'Content-Length': '0'} 2023-10-12T01:20:35.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331c0c282054b348-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.209 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.209 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.209 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.210 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.210 DEBUG:boto:Method: GET 2023-10-12T01:20:35.211 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.211 DEBUG:boto:Data: 2023-10-12T01:20:35.212 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.213 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.213 DEBUG:boto:Params: {} 2023-10-12T01:20:35.213 DEBUG:boto:Token: None 2023-10-12T01:20:35.214 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.214 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk= 2023-10-12T01:20:35.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk=', 'Content-Length': '0'} 2023-10-12T01:20:35.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181b0f92d0cb13de-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.258 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.259 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.259 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.260 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.260 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.260 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.261 DEBUG:boto:Data: 2023-10-12T01:20:35.261 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.262 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.262 DEBUG:boto:Params: {} 2023-10-12T01:20:35.263 DEBUG:boto:Token: None 2023-10-12T01:20:35.263 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HYa88g2Ac1duGNsZ4sgPwrEBy3U= 2023-10-12T01:20:35.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HYa88g2Ac1duGNsZ4sgPwrEBy3U=', 'Content-Length': '0'} 2023-10-12T01:20:35.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc52638098d6af9-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.266 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.267 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.267 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.268 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.268 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.268 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.269 DEBUG:boto:Data: 2023-10-12T01:20:35.269 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.270 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.270 DEBUG:boto:Params: {} 2023-10-12T01:20:35.271 DEBUG:boto:Token: None 2023-10-12T01:20:35.271 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.271 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HYa88g2Ac1duGNsZ4sgPwrEBy3U= 2023-10-12T01:20:35.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HYa88g2Ac1duGNsZ4sgPwrEBy3U=', 'Content-Length': '0'} 2023-10-12T01:20:35.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009371a81a47e289f1-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:20:35.275 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.275 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.275 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.276 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.276 DEBUG:boto:Method: GET 2023-10-12T01:20:35.277 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.277 DEBUG:boto:Data: 2023-10-12T01:20:35.277 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.278 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.278 DEBUG:boto:Params: {} 2023-10-12T01:20:35.279 DEBUG:boto:Token: None 2023-10-12T01:20:35.279 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.280 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk= 2023-10-12T01:20:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk=', 'Content-Length': '0'} 2023-10-12T01:20:35.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37098689c8ce0aa-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.283 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.283 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.283 DEBUG:boto:path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.284 DEBUG:boto:auth_path=/wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.284 DEBUG:boto:Method: GET 2023-10-12T01:20:35.285 DEBUG:boto:Path: /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.285 DEBUG:boto:Data: 2023-10-12T01:20:35.285 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.286 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.287 DEBUG:boto:Params: {} 2023-10-12T01:20:35.287 DEBUG:boto:Token: None 2023-10-12T01:20:35.287 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148?versionId=null 2023-10-12T01:20:35.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk= 2023-10-12T01:20:35.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jrQYuPk7IDyNbCl8qlarujtuIEk=', 'Content-Length': '0'} 2023-10-12T01:20:35.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ace31ee6329303d-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.291 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.291 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.292 DEBUG:boto:path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.292 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.292 DEBUG:boto:Method: GET 2023-10-12T01:20:35.293 DEBUG:boto:Path: /wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.293 DEBUG:boto:Data: 2023-10-12T01:20:35.294 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.294 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.295 DEBUG:boto:Params: {} 2023-10-12T01:20:35.295 DEBUG:boto:Token: None 2023-10-12T01:20:35.296 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006294a1310d745886-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.299 DEBUG:boto:b'wafmoa-61obj1481000falseobj1482023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.300 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.300 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.301 DEBUG:boto:path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.301 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.302 DEBUG:boto:Method: GET 2023-10-12T01:20:35.302 DEBUG:boto:Path: /wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.302 DEBUG:boto:Data: 2023-10-12T01:20:35.303 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.304 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.304 DEBUG:boto:Params: {} 2023-10-12T01:20:35.304 DEBUG:boto:Token: None 2023-10-12T01:20:35.305 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db81cc54df3cee9-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.310 DEBUG:boto:b'wafmoa-61obj1481000falseobj1482023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.310 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.311 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.311 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.311 DEBUG:boto:Path: /wafmoa-61/obj148 2023-10-12T01:20:35.312 DEBUG:boto:Data: 2023-10-12T01:20:35.312 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.313 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.313 DEBUG:boto:Params: {} 2023-10-12T01:20:35.314 DEBUG:boto:Token: None 2023-10-12T01:20:35.314 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148 2023-10-12T01:20:35.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BsvIkSyRDl1JaZ/uT7r2Z+FSwdM= 2023-10-12T01:20:35.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BsvIkSyRDl1JaZ/uT7r2Z+FSwdM=', 'Content-Length': '0'} 2023-10-12T01:20:35.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb83f1f2b9176d1-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.317 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.318 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.318 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.318 DEBUG:boto:Path: /wafmoa-61/obj148 2023-10-12T01:20:35.319 DEBUG:boto:Data: 2023-10-12T01:20:35.319 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.320 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.320 DEBUG:boto:Params: {} 2023-10-12T01:20:35.321 DEBUG:boto:Token: None 2023-10-12T01:20:35.321 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148 2023-10-12T01:20:35.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BsvIkSyRDl1JaZ/uT7r2Z+FSwdM= 2023-10-12T01:20:35.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BsvIkSyRDl1JaZ/uT7r2Z+FSwdM=', 'Content-Length': '0'} 2023-10-12T01:20:35.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000554453a49c8eb91b-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-12T01:20:35.325 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.325 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.325 DEBUG:boto:Method: GET 2023-10-12T01:20:35.326 DEBUG:boto:Path: /wafmoa-61/obj148 2023-10-12T01:20:35.326 DEBUG:boto:Data: 2023-10-12T01:20:35.327 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.328 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.328 DEBUG:boto:Params: {} 2023-10-12T01:20:35.328 DEBUG:boto:Token: None 2023-10-12T01:20:35.329 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148 2023-10-12T01:20:35.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0jL1mxN5OIiEKOpa8fOGuElyD4= 2023-10-12T01:20:35.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0jL1mxN5OIiEKOpa8fOGuElyD4=', 'Content-Length': '0'} 2023-10-12T01:20:35.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24ce6264402aca6-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.378 DEBUG:boto:path=/wafmoa-61/obj148 2023-10-12T01:20:35.379 DEBUG:boto:auth_path=/wafmoa-61/obj148 2023-10-12T01:20:35.379 DEBUG:boto:Method: GET 2023-10-12T01:20:35.380 DEBUG:boto:Path: /wafmoa-61/obj148 2023-10-12T01:20:35.380 DEBUG:boto:Data: 2023-10-12T01:20:35.381 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.381 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.382 DEBUG:boto:Params: {} 2023-10-12T01:20:35.382 DEBUG:boto:Token: None 2023-10-12T01:20:35.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj148 2023-10-12T01:20:35.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:u0jL1mxN5OIiEKOpa8fOGuElyD4= 2023-10-12T01:20:35.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:u0jL1mxN5OIiEKOpa8fOGuElyD4=', 'Content-Length': '0'} 2023-10-12T01:20:35.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f715e508208b1be4-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.386 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.387 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.387 DEBUG:boto:path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.388 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.388 DEBUG:boto:Method: GET 2023-10-12T01:20:35.389 DEBUG:boto:Path: /wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.389 DEBUG:boto:Data: 2023-10-12T01:20:35.390 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.390 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.391 DEBUG:boto:Params: {} 2023-10-12T01:20:35.391 DEBUG:boto:Token: None 2023-10-12T01:20:35.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306c420ee5197a4d-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.397 DEBUG:boto:b'wafmoa-61obj1481000falseobj1482023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.398 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.398 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.399 DEBUG:boto:path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.399 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.399 DEBUG:boto:Method: GET 2023-10-12T01:20:35.400 DEBUG:boto:Path: /wafmoa-61/?prefix=obj148 2023-10-12T01:20:35.400 DEBUG:boto:Data: 2023-10-12T01:20:35.401 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.402 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.402 DEBUG:boto:Params: {} 2023-10-12T01:20:35.403 DEBUG:boto:Token: None 2023-10-12T01:20:35.403 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc203cb6207a217-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.409 DEBUG:boto:b'wafmoa-61obj1481000falseobj1482023-10-12T01:16:57.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:20:35.410 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.411 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.411 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.412 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.412 DEBUG:boto:Method: GET 2023-10-12T01:20:35.413 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.413 DEBUG:boto:Data: 2023-10-12T01:20:35.413 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.414 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.415 DEBUG:boto:Params: {} 2023-10-12T01:20:35.415 DEBUG:boto:Token: None 2023-10-12T01:20:35.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0= 2023-10-12T01:20:35.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0=', 'Content-Length': '0'} 2023-10-12T01:20:35.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d5a623fbc9da21-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.419 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.420 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.420 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.421 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.421 DEBUG:boto:Method: GET 2023-10-12T01:20:35.422 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.422 DEBUG:boto:Data: 2023-10-12T01:20:35.423 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.423 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.424 DEBUG:boto:Params: {} 2023-10-12T01:20:35.424 DEBUG:boto:Token: None 2023-10-12T01:20:35.425 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.425 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0= 2023-10-12T01:20:35.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0=', 'Content-Length': '0'} 2023-10-12T01:20:35.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008baf342308561ae-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.470 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.471 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.471 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.472 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.472 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.472 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.473 DEBUG:boto:Data: 2023-10-12T01:20:35.473 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.474 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.474 DEBUG:boto:Params: {} 2023-10-12T01:20:35.475 DEBUG:boto:Token: None 2023-10-12T01:20:35.475 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBF7YHaMaU8z2oj4n5pr1leZphQ= 2023-10-12T01:20:35.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBF7YHaMaU8z2oj4n5pr1leZphQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea585618f1d2932-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.478 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.479 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.479 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.480 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.480 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.480 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.481 DEBUG:boto:Data: 2023-10-12T01:20:35.481 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.482 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.483 DEBUG:boto:Params: {} 2023-10-12T01:20:35.483 DEBUG:boto:Token: None 2023-10-12T01:20:35.484 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zBF7YHaMaU8z2oj4n5pr1leZphQ= 2023-10-12T01:20:35.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zBF7YHaMaU8z2oj4n5pr1leZphQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004896f7ad3c62700f-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:20:35.487 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.488 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.488 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.488 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.489 DEBUG:boto:Method: GET 2023-10-12T01:20:35.489 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.490 DEBUG:boto:Data: 2023-10-12T01:20:35.490 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.491 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.491 DEBUG:boto:Params: {} 2023-10-12T01:20:35.492 DEBUG:boto:Token: None 2023-10-12T01:20:35.492 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0= 2023-10-12T01:20:35.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0=', 'Content-Length': '0'} 2023-10-12T01:20:35.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde4b7792e03a9b7-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.495 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.496 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.496 DEBUG:boto:path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.497 DEBUG:boto:auth_path=/wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.497 DEBUG:boto:Method: GET 2023-10-12T01:20:35.497 DEBUG:boto:Path: /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.498 DEBUG:boto:Data: 2023-10-12T01:20:35.498 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.499 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.499 DEBUG:boto:Params: {} 2023-10-12T01:20:35.500 DEBUG:boto:Token: None 2023-10-12T01:20:35.500 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155?versionId=null 2023-10-12T01:20:35.501 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0= 2023-10-12T01:20:35.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjCu2RVKvC7k9D4OvcDjG34JgV0=', 'Content-Length': '0'} 2023-10-12T01:20:35.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064594ac5b3145cc3-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.503 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.504 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.504 DEBUG:boto:path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.505 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.505 DEBUG:boto:Method: GET 2023-10-12T01:20:35.505 DEBUG:boto:Path: /wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.506 DEBUG:boto:Data: 2023-10-12T01:20:35.506 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.507 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.507 DEBUG:boto:Params: {} 2023-10-12T01:20:35.508 DEBUG:boto:Token: None 2023-10-12T01:20:35.508 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094d591554f1d276c-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.554 DEBUG:boto:b'wafmoa-61obj1551000falseobj1552023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.555 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.555 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.556 DEBUG:boto:path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.556 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.557 DEBUG:boto:Method: GET 2023-10-12T01:20:35.557 DEBUG:boto:Path: /wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.557 DEBUG:boto:Data: 2023-10-12T01:20:35.558 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.559 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.559 DEBUG:boto:Params: {} 2023-10-12T01:20:35.560 DEBUG:boto:Token: None 2023-10-12T01:20:35.560 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.560 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c844d7852084a3-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.565 DEBUG:boto:b'wafmoa-61obj1551000falseobj1552023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.566 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.566 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.567 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.567 DEBUG:boto:Path: /wafmoa-61/obj155 2023-10-12T01:20:35.568 DEBUG:boto:Data: 2023-10-12T01:20:35.568 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.569 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.569 DEBUG:boto:Params: {} 2023-10-12T01:20:35.569 DEBUG:boto:Token: None 2023-10-12T01:20:35.570 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155 2023-10-12T01:20:35.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2XeH17p7XF+3owIqLH1IjWFpX/o= 2023-10-12T01:20:35.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2XeH17p7XF+3owIqLH1IjWFpX/o=', 'Content-Length': '0'} 2023-10-12T01:20:35.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e04471bfdc3332-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.573 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.574 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.574 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.574 DEBUG:boto:Path: /wafmoa-61/obj155 2023-10-12T01:20:35.575 DEBUG:boto:Data: 2023-10-12T01:20:35.575 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.576 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.576 DEBUG:boto:Params: {} 2023-10-12T01:20:35.577 DEBUG:boto:Token: None 2023-10-12T01:20:35.577 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155 2023-10-12T01:20:35.578 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2XeH17p7XF+3owIqLH1IjWFpX/o= 2023-10-12T01:20:35.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2XeH17p7XF+3owIqLH1IjWFpX/o=', 'Content-Length': '0'} 2023-10-12T01:20:35.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d732f3bdc7781c26-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-12T01:20:35.581 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.581 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.582 DEBUG:boto:Method: GET 2023-10-12T01:20:35.582 DEBUG:boto:Path: /wafmoa-61/obj155 2023-10-12T01:20:35.582 DEBUG:boto:Data: 2023-10-12T01:20:35.583 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.584 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.584 DEBUG:boto:Params: {} 2023-10-12T01:20:35.585 DEBUG:boto:Token: None 2023-10-12T01:20:35.585 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155 2023-10-12T01:20:35.586 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9imdDRPHWbGpM57CWNkYm/AGyOA= 2023-10-12T01:20:35.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9imdDRPHWbGpM57CWNkYm/AGyOA=', 'Content-Length': '0'} 2023-10-12T01:20:35.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ff55ff7ad0bc33f-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.588 DEBUG:boto:path=/wafmoa-61/obj155 2023-10-12T01:20:35.589 DEBUG:boto:auth_path=/wafmoa-61/obj155 2023-10-12T01:20:35.589 DEBUG:boto:Method: GET 2023-10-12T01:20:35.590 DEBUG:boto:Path: /wafmoa-61/obj155 2023-10-12T01:20:35.590 DEBUG:boto:Data: 2023-10-12T01:20:35.590 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.591 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.592 DEBUG:boto:Params: {} 2023-10-12T01:20:35.592 DEBUG:boto:Token: None 2023-10-12T01:20:35.593 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj155 2023-10-12T01:20:35.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9imdDRPHWbGpM57CWNkYm/AGyOA= 2023-10-12T01:20:35.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9imdDRPHWbGpM57CWNkYm/AGyOA=', 'Content-Length': '0'} 2023-10-12T01:20:35.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550a054c359d0b70-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.642 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.643 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.643 DEBUG:boto:path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.644 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.644 DEBUG:boto:Method: GET 2023-10-12T01:20:35.645 DEBUG:boto:Path: /wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.645 DEBUG:boto:Data: 2023-10-12T01:20:35.645 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.646 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.647 DEBUG:boto:Params: {} 2023-10-12T01:20:35.647 DEBUG:boto:Token: None 2023-10-12T01:20:35.648 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.648 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436fa0631d585162-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.652 DEBUG:boto:b'wafmoa-61obj1551000falseobj1552023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.652 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.653 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.653 DEBUG:boto:path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.653 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.654 DEBUG:boto:Method: GET 2023-10-12T01:20:35.654 DEBUG:boto:Path: /wafmoa-61/?prefix=obj155 2023-10-12T01:20:35.655 DEBUG:boto:Data: 2023-10-12T01:20:35.655 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.656 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.657 DEBUG:boto:Params: {} 2023-10-12T01:20:35.657 DEBUG:boto:Token: None 2023-10-12T01:20:35.658 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.658 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc63aaafb52ad96d-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.663 DEBUG:boto:b'wafmoa-61obj1551000falseobj1552023-10-12T01:16:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:20:35.664 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.665 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.665 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.666 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.666 DEBUG:boto:Method: GET 2023-10-12T01:20:35.667 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.667 DEBUG:boto:Data: 2023-10-12T01:20:35.668 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.669 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.669 DEBUG:boto:Params: {} 2023-10-12T01:20:35.670 DEBUG:boto:Token: None 2023-10-12T01:20:35.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ= 2023-10-12T01:20:35.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005204c3f3a5a7999c-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.674 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.674 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.675 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.675 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.676 DEBUG:boto:Method: GET 2023-10-12T01:20:35.676 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.677 DEBUG:boto:Data: 2023-10-12T01:20:35.677 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.678 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.678 DEBUG:boto:Params: {} 2023-10-12T01:20:35.679 DEBUG:boto:Token: None 2023-10-12T01:20:35.679 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.680 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ= 2023-10-12T01:20:35.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024673691d6c2e331-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.683 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.683 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.684 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.684 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.685 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.685 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.685 DEBUG:boto:Data: 2023-10-12T01:20:35.686 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.687 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.687 DEBUG:boto:Params: {} 2023-10-12T01:20:35.687 DEBUG:boto:Token: None 2023-10-12T01:20:35.688 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.688 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HcL1EiDK8/le/A4SNNDEjUs9YO8= 2023-10-12T01:20:35.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HcL1EiDK8/le/A4SNNDEjUs9YO8=', 'Content-Length': '0'} 2023-10-12T01:20:35.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46237c7a6f2a081-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.691 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.692 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.692 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.692 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.693 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.693 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.694 DEBUG:boto:Data: 2023-10-12T01:20:35.694 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.695 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.695 DEBUG:boto:Params: {} 2023-10-12T01:20:35.696 DEBUG:boto:Token: None 2023-10-12T01:20:35.696 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.697 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HcL1EiDK8/le/A4SNNDEjUs9YO8= 2023-10-12T01:20:35.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HcL1EiDK8/le/A4SNNDEjUs9YO8=', 'Content-Length': '0'} 2023-10-12T01:20:35.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db7ed9415bfc5b0-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:20:35.700 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.701 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.701 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.702 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.702 DEBUG:boto:Method: GET 2023-10-12T01:20:35.702 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.703 DEBUG:boto:Data: 2023-10-12T01:20:35.703 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.704 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.705 DEBUG:boto:Params: {} 2023-10-12T01:20:35.705 DEBUG:boto:Token: None 2023-10-12T01:20:35.705 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.706 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ= 2023-10-12T01:20:35.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c0b09782146395-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.750 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.751 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.751 DEBUG:boto:path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.752 DEBUG:boto:auth_path=/wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.752 DEBUG:boto:Method: GET 2023-10-12T01:20:35.753 DEBUG:boto:Path: /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.753 DEBUG:boto:Data: 2023-10-12T01:20:35.753 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.754 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.755 DEBUG:boto:Params: {} 2023-10-12T01:20:35.755 DEBUG:boto:Token: None 2023-10-12T01:20:35.756 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17?versionId=null 2023-10-12T01:20:35.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ= 2023-10-12T01:20:35.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Oa4RcrUZfORwRIsEJ+zs4yAzchQ=', 'Content-Length': '0'} 2023-10-12T01:20:35.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dda4861542330f4-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.759 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.760 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.760 DEBUG:boto:path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.761 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.761 DEBUG:boto:Method: GET 2023-10-12T01:20:35.762 DEBUG:boto:Path: /wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.762 DEBUG:boto:Data: 2023-10-12T01:20:35.762 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.763 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.764 DEBUG:boto:Params: {} 2023-10-12T01:20:35.764 DEBUG:boto:Token: None 2023-10-12T01:20:35.765 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180b82f75758dc5b-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.769 DEBUG:boto:b'wafmoa-61obj171000falseobj172023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.769 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.770 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.770 DEBUG:boto:path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.771 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.771 DEBUG:boto:Method: GET 2023-10-12T01:20:35.772 DEBUG:boto:Path: /wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.772 DEBUG:boto:Data: 2023-10-12T01:20:35.772 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.773 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.774 DEBUG:boto:Params: {} 2023-10-12T01:20:35.774 DEBUG:boto:Token: None 2023-10-12T01:20:35.775 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009792121c0f7ff71b-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.780 DEBUG:boto:b'wafmoa-61obj171000falseobj172023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.780 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.781 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.781 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.782 DEBUG:boto:Path: /wafmoa-61/obj17 2023-10-12T01:20:35.782 DEBUG:boto:Data: 2023-10-12T01:20:35.783 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.784 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.784 DEBUG:boto:Params: {} 2023-10-12T01:20:35.784 DEBUG:boto:Token: None 2023-10-12T01:20:35.785 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17 2023-10-12T01:20:35.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J3JVI3IhLM2d0bk43T/XkJvZWW4= 2023-10-12T01:20:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J3JVI3IhLM2d0bk43T/XkJvZWW4=', 'Content-Length': '0'} 2023-10-12T01:20:35.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d5a2ae4400a6e9-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.788 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.789 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.789 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.790 DEBUG:boto:Path: /wafmoa-61/obj17 2023-10-12T01:20:35.790 DEBUG:boto:Data: 2023-10-12T01:20:35.790 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.791 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.792 DEBUG:boto:Params: {} 2023-10-12T01:20:35.792 DEBUG:boto:Token: None 2023-10-12T01:20:35.793 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17 2023-10-12T01:20:35.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J3JVI3IhLM2d0bk43T/XkJvZWW4= 2023-10-12T01:20:35.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J3JVI3IhLM2d0bk43T/XkJvZWW4=', 'Content-Length': '0'} 2023-10-12T01:20:35.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc5a121bc976734-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-12T01:20:35.796 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.797 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.797 DEBUG:boto:Method: GET 2023-10-12T01:20:35.798 DEBUG:boto:Path: /wafmoa-61/obj17 2023-10-12T01:20:35.798 DEBUG:boto:Data: 2023-10-12T01:20:35.799 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.799 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.800 DEBUG:boto:Params: {} 2023-10-12T01:20:35.800 DEBUG:boto:Token: None 2023-10-12T01:20:35.801 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17 2023-10-12T01:20:35.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sX0eYggPtFpV6J7grLqfDumvRVk= 2023-10-12T01:20:35.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sX0eYggPtFpV6J7grLqfDumvRVk=', 'Content-Length': '0'} 2023-10-12T01:20:35.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd115e1b8a311c2a-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.804 DEBUG:boto:path=/wafmoa-61/obj17 2023-10-12T01:20:35.804 DEBUG:boto:auth_path=/wafmoa-61/obj17 2023-10-12T01:20:35.805 DEBUG:boto:Method: GET 2023-10-12T01:20:35.805 DEBUG:boto:Path: /wafmoa-61/obj17 2023-10-12T01:20:35.806 DEBUG:boto:Data: 2023-10-12T01:20:35.806 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.807 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.807 DEBUG:boto:Params: {} 2023-10-12T01:20:35.808 DEBUG:boto:Token: None 2023-10-12T01:20:35.808 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj17 2023-10-12T01:20:35.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sX0eYggPtFpV6J7grLqfDumvRVk= 2023-10-12T01:20:35.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sX0eYggPtFpV6J7grLqfDumvRVk=', 'Content-Length': '0'} 2023-10-12T01:20:35.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d77e0d69cc14fa-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.812 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.812 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.813 DEBUG:boto:path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.813 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.814 DEBUG:boto:Method: GET 2023-10-12T01:20:35.814 DEBUG:boto:Path: /wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.814 DEBUG:boto:Data: 2023-10-12T01:20:35.815 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.816 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.816 DEBUG:boto:Params: {} 2023-10-12T01:20:35.817 DEBUG:boto:Token: None 2023-10-12T01:20:35.817 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.818 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004929224ab2715aac-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.821 DEBUG:boto:b'wafmoa-61obj171000falseobj172023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.822 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.823 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.823 DEBUG:boto:path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.823 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.824 DEBUG:boto:Method: GET 2023-10-12T01:20:35.824 DEBUG:boto:Path: /wafmoa-61/?prefix=obj17 2023-10-12T01:20:35.825 DEBUG:boto:Data: 2023-10-12T01:20:35.825 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.826 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.826 DEBUG:boto:Params: {} 2023-10-12T01:20:35.827 DEBUG:boto:Token: None 2023-10-12T01:20:35.827 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.828 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440129fc40d36331-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.874 DEBUG:boto:b'wafmoa-61obj171000falseobj172023-10-12T01:16:55.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:20:35.876 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.876 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.877 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.877 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.878 DEBUG:boto:Method: GET 2023-10-12T01:20:35.878 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.879 DEBUG:boto:Data: 2023-10-12T01:20:35.879 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.880 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.880 DEBUG:boto:Params: {} 2023-10-12T01:20:35.881 DEBUG:boto:Token: None 2023-10-12T01:20:35.881 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.882 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE= 2023-10-12T01:20:35.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE=', 'Content-Length': '0'} 2023-10-12T01:20:35.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a758c8f24e80a4cb-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.885 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.885 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.886 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.886 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.887 DEBUG:boto:Method: GET 2023-10-12T01:20:35.887 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.887 DEBUG:boto:Data: 2023-10-12T01:20:35.888 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.889 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.889 DEBUG:boto:Params: {} 2023-10-12T01:20:35.890 DEBUG:boto:Token: None 2023-10-12T01:20:35.890 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE= 2023-10-12T01:20:35.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE=', 'Content-Length': '0'} 2023-10-12T01:20:35.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003890f93c09727ca9-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.894 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.894 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.895 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.895 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.896 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.896 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.897 DEBUG:boto:Data: 2023-10-12T01:20:35.897 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.898 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.898 DEBUG:boto:Params: {} 2023-10-12T01:20:35.899 DEBUG:boto:Token: None 2023-10-12T01:20:35.899 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.900 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YOMBPkz4egNseiWrXx1Nu+9ErjI= 2023-10-12T01:20:35.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YOMBPkz4egNseiWrXx1Nu+9ErjI=', 'Content-Length': '0'} 2023-10-12T01:20:35.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcc65b4e121c5f4-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.903 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.903 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.904 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.904 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.904 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.905 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.905 DEBUG:boto:Data: 2023-10-12T01:20:35.906 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.907 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.907 DEBUG:boto:Params: {} 2023-10-12T01:20:35.907 DEBUG:boto:Token: None 2023-10-12T01:20:35.908 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YOMBPkz4egNseiWrXx1Nu+9ErjI= 2023-10-12T01:20:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YOMBPkz4egNseiWrXx1Nu+9ErjI=', 'Content-Length': '0'} 2023-10-12T01:20:35.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049533d2a4a5d7df2-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:20:35.912 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.912 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.913 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.913 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.914 DEBUG:boto:Method: GET 2023-10-12T01:20:35.914 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.914 DEBUG:boto:Data: 2023-10-12T01:20:35.915 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.916 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.916 DEBUG:boto:Params: {} 2023-10-12T01:20:35.917 DEBUG:boto:Token: None 2023-10-12T01:20:35.917 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.918 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE= 2023-10-12T01:20:35.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE=', 'Content-Length': '0'} 2023-10-12T01:20:35.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000083f64ead2fa05e-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.921 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.921 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.922 DEBUG:boto:path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.922 DEBUG:boto:auth_path=/wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.923 DEBUG:boto:Method: GET 2023-10-12T01:20:35.923 DEBUG:boto:Path: /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.924 DEBUG:boto:Data: 2023-10-12T01:20:35.924 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.925 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.925 DEBUG:boto:Params: {} 2023-10-12T01:20:35.926 DEBUG:boto:Token: None 2023-10-12T01:20:35.926 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185?versionId=null 2023-10-12T01:20:35.927 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE= 2023-10-12T01:20:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AourOK+6bxS2UqkOupV6TxW3/AE=', 'Content-Length': '0'} 2023-10-12T01:20:35.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda44a1ec36ece35-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.930 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.930 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.931 DEBUG:boto:path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.931 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.932 DEBUG:boto:Method: GET 2023-10-12T01:20:35.932 DEBUG:boto:Path: /wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.932 DEBUG:boto:Data: 2023-10-12T01:20:35.933 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.934 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.934 DEBUG:boto:Params: {} 2023-10-12T01:20:35.935 DEBUG:boto:Token: None 2023-10-12T01:20:35.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889f70181475069e-00652749e3-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.939 DEBUG:boto:b'wafmoa-61obj1851000falseobj1852023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.940 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:35.940 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:35.941 DEBUG:boto:path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.941 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.942 DEBUG:boto:Method: GET 2023-10-12T01:20:35.942 DEBUG:boto:Path: /wafmoa-61/?prefix=obj185 2023-10-12T01:20:35.943 DEBUG:boto:Data: 2023-10-12T01:20:35.943 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.944 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.944 DEBUG:boto:Params: {} 2023-10-12T01:20:35.945 DEBUG:boto:Token: None 2023-10-12T01:20:35.945 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/ 2023-10-12T01:20:35.946 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU= 2023-10-12T01:20:35.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4SROHclf4CjXdBbBexxbx/P2JCU=', 'Content-Length': '0'} 2023-10-12T01:20:35.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081eb2b99e54c77ac-00652749e3-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.951 DEBUG:boto:b'wafmoa-61obj1851000falseobj1852023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:35.951 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.952 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.952 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.953 DEBUG:boto:Path: /wafmoa-61/obj185 2023-10-12T01:20:35.953 DEBUG:boto:Data: 2023-10-12T01:20:35.954 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.955 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.955 DEBUG:boto:Params: {} 2023-10-12T01:20:35.955 DEBUG:boto:Token: None 2023-10-12T01:20:35.956 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185 2023-10-12T01:20:35.956 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUWm/Rbbe8LdgTuGO7wzGyMJpY0= 2023-10-12T01:20:35.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUWm/Rbbe8LdgTuGO7wzGyMJpY0=', 'Content-Length': '0'} 2023-10-12T01:20:35.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000612573673ee2bca9-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.959 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.959 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.960 DEBUG:boto:Method: HEAD 2023-10-12T01:20:35.960 DEBUG:boto:Path: /wafmoa-61/obj185 2023-10-12T01:20:35.960 DEBUG:boto:Data: 2023-10-12T01:20:35.961 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:35.962 DEBUG:boto:Port: 8000 2023-10-12T01:20:35.962 DEBUG:boto:Params: {} 2023-10-12T01:20:35.963 DEBUG:boto:Token: None 2023-10-12T01:20:35.963 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185 2023-10-12T01:20:35.963 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PUWm/Rbbe8LdgTuGO7wzGyMJpY0= 2023-10-12T01:20:35.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PUWm/Rbbe8LdgTuGO7wzGyMJpY0=', 'Content-Length': '0'} 2023-10-12T01:20:35.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977c81790040bee7-00652749e3-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:35.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-12T01:20:35.967 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:35.967 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:35.967 DEBUG:boto:Method: GET 2023-10-12T01:20:35.968 DEBUG:boto:Path: /wafmoa-61/obj185 2023-10-12T01:20:35.968 DEBUG:boto:Data: 2023-10-12T01:20:35.969 DEBUG:boto:Headers: {} 2023-10-12T01:20:35.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:35.970 DEBUG:boto:Port: 8001 2023-10-12T01:20:35.970 DEBUG:boto:Params: {} 2023-10-12T01:20:35.971 DEBUG:boto:Token: None 2023-10-12T01:20:35.971 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:35 GMT /wafmoa-61/obj185 2023-10-12T01:20:35.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EMYWvZmm7gvkNxXSazHyCGNL8pI= 2023-10-12T01:20:35.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:35 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EMYWvZmm7gvkNxXSazHyCGNL8pI=', 'Content-Length': '0'} 2023-10-12T01:20:35.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009159dc25ffa3759e-00652749e3-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.014 DEBUG:boto:path=/wafmoa-61/obj185 2023-10-12T01:20:36.015 DEBUG:boto:auth_path=/wafmoa-61/obj185 2023-10-12T01:20:36.015 DEBUG:boto:Method: GET 2023-10-12T01:20:36.015 DEBUG:boto:Path: /wafmoa-61/obj185 2023-10-12T01:20:36.016 DEBUG:boto:Data: 2023-10-12T01:20:36.016 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.017 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.017 DEBUG:boto:Params: {} 2023-10-12T01:20:36.018 DEBUG:boto:Token: None 2023-10-12T01:20:36.018 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj185 2023-10-12T01:20:36.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q3YgkBPboUoEFr6y1+6uMb7F6Ho= 2023-10-12T01:20:36.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q3YgkBPboUoEFr6y1+6uMb7F6Ho=', 'Content-Length': '0'} 2023-10-12T01:20:36.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb9cbe01908d1246-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.022 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.022 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.022 DEBUG:boto:path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.023 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.023 DEBUG:boto:Method: GET 2023-10-12T01:20:36.024 DEBUG:boto:Path: /wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.024 DEBUG:boto:Data: 2023-10-12T01:20:36.024 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.025 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.026 DEBUG:boto:Params: {} 2023-10-12T01:20:36.026 DEBUG:boto:Token: None 2023-10-12T01:20:36.027 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042dba234b77f0bd9-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.030 DEBUG:boto:b'wafmoa-61obj1851000falseobj1852023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.031 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.031 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.032 DEBUG:boto:path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.032 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.033 DEBUG:boto:Method: GET 2023-10-12T01:20:36.033 DEBUG:boto:Path: /wafmoa-61/?prefix=obj185 2023-10-12T01:20:36.033 DEBUG:boto:Data: 2023-10-12T01:20:36.034 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.035 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.035 DEBUG:boto:Params: {} 2023-10-12T01:20:36.035 DEBUG:boto:Token: None 2023-10-12T01:20:36.036 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a76bf21aa3c8d3-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.041 DEBUG:boto:b'wafmoa-61obj1851000falseobj1852023-10-12T01:16:57.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:20:36.042 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.042 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.043 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.043 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.043 DEBUG:boto:Method: GET 2023-10-12T01:20:36.044 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.044 DEBUG:boto:Data: 2023-10-12T01:20:36.045 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.046 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.046 DEBUG:boto:Params: {} 2023-10-12T01:20:36.046 DEBUG:boto:Token: None 2023-10-12T01:20:36.047 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc= 2023-10-12T01:20:36.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc=', 'Content-Length': '0'} 2023-10-12T01:20:36.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002747334e49d66a21-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.050 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.051 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.051 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.052 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.052 DEBUG:boto:Method: GET 2023-10-12T01:20:36.052 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.053 DEBUG:boto:Data: 2023-10-12T01:20:36.053 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.054 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.054 DEBUG:boto:Params: {} 2023-10-12T01:20:36.055 DEBUG:boto:Token: None 2023-10-12T01:20:36.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc= 2023-10-12T01:20:36.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc=', 'Content-Length': '0'} 2023-10-12T01:20:36.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a19b8bd34f9c24b-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.058 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.059 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.059 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.060 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.060 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.060 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.061 DEBUG:boto:Data: 2023-10-12T01:20:36.061 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.062 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.062 DEBUG:boto:Params: {} 2023-10-12T01:20:36.063 DEBUG:boto:Token: None 2023-10-12T01:20:36.063 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xoXJyFgeqK6FfhSuefF6vHQ0EZ0= 2023-10-12T01:20:36.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xoXJyFgeqK6FfhSuefF6vHQ0EZ0=', 'Content-Length': '0'} 2023-10-12T01:20:36.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d816de4e2a6a52b-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.066 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.067 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.067 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.068 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.068 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.068 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.069 DEBUG:boto:Data: 2023-10-12T01:20:36.069 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.070 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.070 DEBUG:boto:Params: {} 2023-10-12T01:20:36.071 DEBUG:boto:Token: None 2023-10-12T01:20:36.071 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.071 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xoXJyFgeqK6FfhSuefF6vHQ0EZ0= 2023-10-12T01:20:36.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xoXJyFgeqK6FfhSuefF6vHQ0EZ0=', 'Content-Length': '0'} 2023-10-12T01:20:36.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009fcb405b32b25fb-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:20:36.075 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.075 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.075 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.076 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.076 DEBUG:boto:Method: GET 2023-10-12T01:20:36.077 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.077 DEBUG:boto:Data: 2023-10-12T01:20:36.078 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.078 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.079 DEBUG:boto:Params: {} 2023-10-12T01:20:36.079 DEBUG:boto:Token: None 2023-10-12T01:20:36.080 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.080 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc= 2023-10-12T01:20:36.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc=', 'Content-Length': '0'} 2023-10-12T01:20:36.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db0c7d3ab5cb664-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.083 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.084 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.084 DEBUG:boto:path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.085 DEBUG:boto:auth_path=/wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.085 DEBUG:boto:Method: GET 2023-10-12T01:20:36.086 DEBUG:boto:Path: /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.086 DEBUG:boto:Data: 2023-10-12T01:20:36.086 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.087 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.088 DEBUG:boto:Params: {} 2023-10-12T01:20:36.088 DEBUG:boto:Token: None 2023-10-12T01:20:36.089 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19?versionId=null 2023-10-12T01:20:36.089 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc= 2023-10-12T01:20:36.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s+hMWtBSJ25eJYXTHcNk2LMNcNc=', 'Content-Length': '0'} 2023-10-12T01:20:36.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391ccbfabeb5256f-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.092 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.093 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.093 DEBUG:boto:path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.094 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.094 DEBUG:boto:Method: GET 2023-10-12T01:20:36.094 DEBUG:boto:Path: /wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.095 DEBUG:boto:Data: 2023-10-12T01:20:36.095 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.096 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.096 DEBUG:boto:Params: {} 2023-10-12T01:20:36.097 DEBUG:boto:Token: None 2023-10-12T01:20:36.097 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.098 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001131c93a725d6db2-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.101 DEBUG:boto:b'wafmoa-61obj191000falseobj192023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.102 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.102 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.103 DEBUG:boto:path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.103 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.103 DEBUG:boto:Method: GET 2023-10-12T01:20:36.104 DEBUG:boto:Path: /wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.104 DEBUG:boto:Data: 2023-10-12T01:20:36.105 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.106 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.106 DEBUG:boto:Params: {} 2023-10-12T01:20:36.106 DEBUG:boto:Token: None 2023-10-12T01:20:36.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.107 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36225e813c77240-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.112 DEBUG:boto:b'wafmoa-61obj191000falseobj192023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.113 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.113 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.114 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.114 DEBUG:boto:Path: /wafmoa-61/obj19 2023-10-12T01:20:36.115 DEBUG:boto:Data: 2023-10-12T01:20:36.115 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.116 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.116 DEBUG:boto:Params: {} 2023-10-12T01:20:36.117 DEBUG:boto:Token: None 2023-10-12T01:20:36.117 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19 2023-10-12T01:20:36.118 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6fK7ur95QdkoPbN7A6A6fLwgtBA= 2023-10-12T01:20:36.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6fK7ur95QdkoPbN7A6A6fLwgtBA=', 'Content-Length': '0'} 2023-10-12T01:20:36.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008517c3f1a2258c5e-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.120 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.121 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.121 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.122 DEBUG:boto:Path: /wafmoa-61/obj19 2023-10-12T01:20:36.122 DEBUG:boto:Data: 2023-10-12T01:20:36.122 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.123 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.124 DEBUG:boto:Params: {} 2023-10-12T01:20:36.124 DEBUG:boto:Token: None 2023-10-12T01:20:36.124 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19 2023-10-12T01:20:36.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6fK7ur95QdkoPbN7A6A6fLwgtBA= 2023-10-12T01:20:36.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6fK7ur95QdkoPbN7A6A6fLwgtBA=', 'Content-Length': '0'} 2023-10-12T01:20:36.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef4a9b92594641e-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-12T01:20:36.128 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.128 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.129 DEBUG:boto:Method: GET 2023-10-12T01:20:36.129 DEBUG:boto:Path: /wafmoa-61/obj19 2023-10-12T01:20:36.130 DEBUG:boto:Data: 2023-10-12T01:20:36.130 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.131 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.131 DEBUG:boto:Params: {} 2023-10-12T01:20:36.132 DEBUG:boto:Token: None 2023-10-12T01:20:36.132 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19 2023-10-12T01:20:36.133 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v0q5SCcQe+8UhMTZkuZxyA2ZqDs= 2023-10-12T01:20:36.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v0q5SCcQe+8UhMTZkuZxyA2ZqDs=', 'Content-Length': '0'} 2023-10-12T01:20:36.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8236e455bd9913-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.136 DEBUG:boto:path=/wafmoa-61/obj19 2023-10-12T01:20:36.136 DEBUG:boto:auth_path=/wafmoa-61/obj19 2023-10-12T01:20:36.137 DEBUG:boto:Method: GET 2023-10-12T01:20:36.137 DEBUG:boto:Path: /wafmoa-61/obj19 2023-10-12T01:20:36.138 DEBUG:boto:Data: 2023-10-12T01:20:36.138 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.139 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.140 DEBUG:boto:Params: {} 2023-10-12T01:20:36.140 DEBUG:boto:Token: None 2023-10-12T01:20:36.141 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj19 2023-10-12T01:20:36.141 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:v0q5SCcQe+8UhMTZkuZxyA2ZqDs= 2023-10-12T01:20:36.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:v0q5SCcQe+8UhMTZkuZxyA2ZqDs=', 'Content-Length': '0'} 2023-10-12T01:20:36.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814e8d8b13255b5b-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.190 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.191 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.191 DEBUG:boto:path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.192 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.192 DEBUG:boto:Method: GET 2023-10-12T01:20:36.193 DEBUG:boto:Path: /wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.193 DEBUG:boto:Data: 2023-10-12T01:20:36.194 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.195 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.195 DEBUG:boto:Params: {} 2023-10-12T01:20:36.196 DEBUG:boto:Token: None 2023-10-12T01:20:36.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2f8b0265e0d2bb-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.200 DEBUG:boto:b'wafmoa-61obj191000falseobj192023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.200 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.201 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.201 DEBUG:boto:path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.202 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.202 DEBUG:boto:Method: GET 2023-10-12T01:20:36.203 DEBUG:boto:Path: /wafmoa-61/?prefix=obj19 2023-10-12T01:20:36.203 DEBUG:boto:Data: 2023-10-12T01:20:36.203 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.204 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.205 DEBUG:boto:Params: {} 2023-10-12T01:20:36.205 DEBUG:boto:Token: None 2023-10-12T01:20:36.206 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.206 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb9cfedb71eaf07-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.211 DEBUG:boto:b'wafmoa-61obj191000falseobj192023-10-12T01:16:55.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:20:36.212 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.213 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.213 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.214 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.214 DEBUG:boto:Method: GET 2023-10-12T01:20:36.215 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.215 DEBUG:boto:Data: 2023-10-12T01:20:36.215 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.216 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.217 DEBUG:boto:Params: {} 2023-10-12T01:20:36.217 DEBUG:boto:Token: None 2023-10-12T01:20:36.218 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.218 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys= 2023-10-12T01:20:36.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys=', 'Content-Length': '0'} 2023-10-12T01:20:36.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fccd09e201773f-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.262 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.263 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.263 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.264 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.264 DEBUG:boto:Method: GET 2023-10-12T01:20:36.264 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.265 DEBUG:boto:Data: 2023-10-12T01:20:36.265 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.266 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.266 DEBUG:boto:Params: {} 2023-10-12T01:20:36.267 DEBUG:boto:Token: None 2023-10-12T01:20:36.267 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys= 2023-10-12T01:20:36.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys=', 'Content-Length': '0'} 2023-10-12T01:20:36.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd970ba738a336a4-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.271 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.271 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.272 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.272 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.272 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.273 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.273 DEBUG:boto:Data: 2023-10-12T01:20:36.274 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.274 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.275 DEBUG:boto:Params: {} 2023-10-12T01:20:36.275 DEBUG:boto:Token: None 2023-10-12T01:20:36.276 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YAm4I7VM6j6iAj+RPZeCzxj9eL8= 2023-10-12T01:20:36.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YAm4I7VM6j6iAj+RPZeCzxj9eL8=', 'Content-Length': '0'} 2023-10-12T01:20:36.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6f3e8e832eb803-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.279 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.279 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.280 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.280 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.280 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.281 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.281 DEBUG:boto:Data: 2023-10-12T01:20:36.282 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.282 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.283 DEBUG:boto:Params: {} 2023-10-12T01:20:36.283 DEBUG:boto:Token: None 2023-10-12T01:20:36.284 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YAm4I7VM6j6iAj+RPZeCzxj9eL8= 2023-10-12T01:20:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YAm4I7VM6j6iAj+RPZeCzxj9eL8=', 'Content-Length': '0'} 2023-10-12T01:20:36.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b53eaf3d77e2ace-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:20:36.287 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.288 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.288 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.289 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.289 DEBUG:boto:Method: GET 2023-10-12T01:20:36.289 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.290 DEBUG:boto:Data: 2023-10-12T01:20:36.290 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.291 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.291 DEBUG:boto:Params: {} 2023-10-12T01:20:36.292 DEBUG:boto:Token: None 2023-10-12T01:20:36.292 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.293 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys= 2023-10-12T01:20:36.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys=', 'Content-Length': '0'} 2023-10-12T01:20:36.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbbccbb5e0ee7dc-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.296 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.296 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.297 DEBUG:boto:path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.297 DEBUG:boto:auth_path=/wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.297 DEBUG:boto:Method: GET 2023-10-12T01:20:36.298 DEBUG:boto:Path: /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.298 DEBUG:boto:Data: 2023-10-12T01:20:36.299 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.299 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.300 DEBUG:boto:Params: {} 2023-10-12T01:20:36.300 DEBUG:boto:Token: None 2023-10-12T01:20:36.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22?versionId=null 2023-10-12T01:20:36.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys= 2023-10-12T01:20:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YdB/VLKTRaVMS0xIIcZ/QqR44ys=', 'Content-Length': '0'} 2023-10-12T01:20:36.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd5ef320f40cafc-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.346 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.347 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.347 DEBUG:boto:path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.348 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.348 DEBUG:boto:Method: GET 2023-10-12T01:20:36.349 DEBUG:boto:Path: /wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.349 DEBUG:boto:Data: 2023-10-12T01:20:36.349 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.350 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.353 DEBUG:boto:Params: {} 2023-10-12T01:20:36.353 DEBUG:boto:Token: None 2023-10-12T01:20:36.354 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.354 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005017e62ce1b2e8c2-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.358 DEBUG:boto:b'wafmoa-61obj221000falseobj222023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.359 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.359 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.360 DEBUG:boto:path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.360 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.360 DEBUG:boto:Method: GET 2023-10-12T01:20:36.361 DEBUG:boto:Path: /wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.361 DEBUG:boto:Data: 2023-10-12T01:20:36.362 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.362 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.363 DEBUG:boto:Params: {} 2023-10-12T01:20:36.363 DEBUG:boto:Token: None 2023-10-12T01:20:36.364 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.364 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008526a1cbc1f83b71-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.369 DEBUG:boto:b'wafmoa-61obj221000falseobj222023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.370 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.370 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.371 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.371 DEBUG:boto:Path: /wafmoa-61/obj22 2023-10-12T01:20:36.371 DEBUG:boto:Data: 2023-10-12T01:20:36.372 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.373 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.373 DEBUG:boto:Params: {} 2023-10-12T01:20:36.374 DEBUG:boto:Token: None 2023-10-12T01:20:36.374 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22 2023-10-12T01:20:36.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CesQhAm4yrr70xwusQL4kOo5H0w= 2023-10-12T01:20:36.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CesQhAm4yrr70xwusQL4kOo5H0w=', 'Content-Length': '0'} 2023-10-12T01:20:36.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3090ea4431fe26b-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.377 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.378 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.378 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.379 DEBUG:boto:Path: /wafmoa-61/obj22 2023-10-12T01:20:36.379 DEBUG:boto:Data: 2023-10-12T01:20:36.380 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.380 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.381 DEBUG:boto:Params: {} 2023-10-12T01:20:36.381 DEBUG:boto:Token: None 2023-10-12T01:20:36.381 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22 2023-10-12T01:20:36.382 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:CesQhAm4yrr70xwusQL4kOo5H0w= 2023-10-12T01:20:36.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:CesQhAm4yrr70xwusQL4kOo5H0w=', 'Content-Length': '0'} 2023-10-12T01:20:36.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001331b1cefa5aa82c-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:20:36.386 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.386 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.387 DEBUG:boto:Method: GET 2023-10-12T01:20:36.387 DEBUG:boto:Path: /wafmoa-61/obj22 2023-10-12T01:20:36.388 DEBUG:boto:Data: 2023-10-12T01:20:36.388 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.389 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.389 DEBUG:boto:Params: {} 2023-10-12T01:20:36.390 DEBUG:boto:Token: None 2023-10-12T01:20:36.390 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22 2023-10-12T01:20:36.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6rdYJGnZdkqObfAnmjYFqrxXLk= 2023-10-12T01:20:36.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6rdYJGnZdkqObfAnmjYFqrxXLk=', 'Content-Length': '0'} 2023-10-12T01:20:36.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047540c2c2087587a-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.394 DEBUG:boto:path=/wafmoa-61/obj22 2023-10-12T01:20:36.394 DEBUG:boto:auth_path=/wafmoa-61/obj22 2023-10-12T01:20:36.395 DEBUG:boto:Method: GET 2023-10-12T01:20:36.395 DEBUG:boto:Path: /wafmoa-61/obj22 2023-10-12T01:20:36.395 DEBUG:boto:Data: 2023-10-12T01:20:36.396 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.397 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.397 DEBUG:boto:Params: {} 2023-10-12T01:20:36.398 DEBUG:boto:Token: None 2023-10-12T01:20:36.398 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj22 2023-10-12T01:20:36.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p6rdYJGnZdkqObfAnmjYFqrxXLk= 2023-10-12T01:20:36.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p6rdYJGnZdkqObfAnmjYFqrxXLk=', 'Content-Length': '0'} 2023-10-12T01:20:36.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2049394acda59cd-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.402 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.402 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.403 DEBUG:boto:path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.403 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.404 DEBUG:boto:Method: GET 2023-10-12T01:20:36.404 DEBUG:boto:Path: /wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.404 DEBUG:boto:Data: 2023-10-12T01:20:36.405 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.406 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.406 DEBUG:boto:Params: {} 2023-10-12T01:20:36.407 DEBUG:boto:Token: None 2023-10-12T01:20:36.407 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.408 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8aa8af4c66aa67e-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.411 DEBUG:boto:b'wafmoa-61obj221000falseobj222023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.412 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.412 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.412 DEBUG:boto:path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.413 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.413 DEBUG:boto:Method: GET 2023-10-12T01:20:36.414 DEBUG:boto:Path: /wafmoa-61/?prefix=obj22 2023-10-12T01:20:36.414 DEBUG:boto:Data: 2023-10-12T01:20:36.415 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.416 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.416 DEBUG:boto:Params: {} 2023-10-12T01:20:36.417 DEBUG:boto:Token: None 2023-10-12T01:20:36.417 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.418 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4f0d63f8b6c7d0-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.422 DEBUG:boto:b'wafmoa-61obj221000falseobj222023-10-12T01:16:55.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:20:36.423 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.424 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.424 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.424 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.425 DEBUG:boto:Method: GET 2023-10-12T01:20:36.425 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.426 DEBUG:boto:Data: 2023-10-12T01:20:36.426 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.427 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.428 DEBUG:boto:Params: {} 2023-10-12T01:20:36.428 DEBUG:boto:Token: None 2023-10-12T01:20:36.429 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8= 2023-10-12T01:20:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8=', 'Content-Length': '0'} 2023-10-12T01:20:36.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f4f0860fe36716-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.474 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.475 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.475 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.475 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.476 DEBUG:boto:Method: GET 2023-10-12T01:20:36.476 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.477 DEBUG:boto:Data: 2023-10-12T01:20:36.477 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.478 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.479 DEBUG:boto:Params: {} 2023-10-12T01:20:36.479 DEBUG:boto:Token: None 2023-10-12T01:20:36.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.480 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8= 2023-10-12T01:20:36.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8=', 'Content-Length': '0'} 2023-10-12T01:20:36.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f816ee5b729d898f-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.483 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.483 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.484 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.484 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.485 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.485 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.485 DEBUG:boto:Data: 2023-10-12T01:20:36.486 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.487 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.487 DEBUG:boto:Params: {} 2023-10-12T01:20:36.488 DEBUG:boto:Token: None 2023-10-12T01:20:36.488 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JWoWc+7UGSKLBebFNg/RVvnl19U= 2023-10-12T01:20:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JWoWc+7UGSKLBebFNg/RVvnl19U=', 'Content-Length': '0'} 2023-10-12T01:20:36.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929df001b37a86dd-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.491 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.492 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.492 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.493 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.493 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.493 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.494 DEBUG:boto:Data: 2023-10-12T01:20:36.494 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.495 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.495 DEBUG:boto:Params: {} 2023-10-12T01:20:36.496 DEBUG:boto:Token: None 2023-10-12T01:20:36.496 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JWoWc+7UGSKLBebFNg/RVvnl19U= 2023-10-12T01:20:36.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JWoWc+7UGSKLBebFNg/RVvnl19U=', 'Content-Length': '0'} 2023-10-12T01:20:36.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080881d5bce609e54-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:20:36.500 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.500 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.501 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.501 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.501 DEBUG:boto:Method: GET 2023-10-12T01:20:36.502 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.502 DEBUG:boto:Data: 2023-10-12T01:20:36.503 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.503 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.504 DEBUG:boto:Params: {} 2023-10-12T01:20:36.504 DEBUG:boto:Token: None 2023-10-12T01:20:36.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8= 2023-10-12T01:20:36.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8=', 'Content-Length': '0'} 2023-10-12T01:20:36.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac042583d86821ca-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.508 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.509 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.509 DEBUG:boto:path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.510 DEBUG:boto:auth_path=/wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.510 DEBUG:boto:Method: GET 2023-10-12T01:20:36.511 DEBUG:boto:Path: /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.511 DEBUG:boto:Data: 2023-10-12T01:20:36.511 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.512 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.512 DEBUG:boto:Params: {} 2023-10-12T01:20:36.513 DEBUG:boto:Token: None 2023-10-12T01:20:36.513 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23?versionId=null 2023-10-12T01:20:36.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8= 2023-10-12T01:20:36.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ypNqGPvsFB345jHdBp9oRebvC8=', 'Content-Length': '0'} 2023-10-12T01:20:36.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147724e0e25022ae-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.558 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.559 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.559 DEBUG:boto:path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.560 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.560 DEBUG:boto:Method: GET 2023-10-12T01:20:36.560 DEBUG:boto:Path: /wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.561 DEBUG:boto:Data: 2023-10-12T01:20:36.561 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.562 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.562 DEBUG:boto:Params: {} 2023-10-12T01:20:36.563 DEBUG:boto:Token: None 2023-10-12T01:20:36.563 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3843ecb9060153e-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.567 DEBUG:boto:b'wafmoa-61obj231000falseobj232023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.567 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.568 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.568 DEBUG:boto:path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.569 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.569 DEBUG:boto:Method: GET 2023-10-12T01:20:36.569 DEBUG:boto:Path: /wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.570 DEBUG:boto:Data: 2023-10-12T01:20:36.570 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.571 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.572 DEBUG:boto:Params: {} 2023-10-12T01:20:36.572 DEBUG:boto:Token: None 2023-10-12T01:20:36.573 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.573 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061384d2869c37186-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.578 DEBUG:boto:b'wafmoa-61obj231000falseobj232023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.579 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.579 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.579 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.580 DEBUG:boto:Path: /wafmoa-61/obj23 2023-10-12T01:20:36.580 DEBUG:boto:Data: 2023-10-12T01:20:36.580 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.581 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.582 DEBUG:boto:Params: {} 2023-10-12T01:20:36.582 DEBUG:boto:Token: None 2023-10-12T01:20:36.583 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23 2023-10-12T01:20:36.583 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zI4yWV+xFDM4FvU76ljvHP1z7mg= 2023-10-12T01:20:36.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zI4yWV+xFDM4FvU76ljvHP1z7mg=', 'Content-Length': '0'} 2023-10-12T01:20:36.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d26e491d46f9f9-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.586 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.586 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.587 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.587 DEBUG:boto:Path: /wafmoa-61/obj23 2023-10-12T01:20:36.588 DEBUG:boto:Data: 2023-10-12T01:20:36.588 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.589 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.589 DEBUG:boto:Params: {} 2023-10-12T01:20:36.590 DEBUG:boto:Token: None 2023-10-12T01:20:36.590 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23 2023-10-12T01:20:36.590 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zI4yWV+xFDM4FvU76ljvHP1z7mg= 2023-10-12T01:20:36.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zI4yWV+xFDM4FvU76ljvHP1z7mg=', 'Content-Length': '0'} 2023-10-12T01:20:36.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000484c6860ff811fcb-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-12T01:20:36.594 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.594 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.594 DEBUG:boto:Method: GET 2023-10-12T01:20:36.595 DEBUG:boto:Path: /wafmoa-61/obj23 2023-10-12T01:20:36.595 DEBUG:boto:Data: 2023-10-12T01:20:36.596 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.596 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.597 DEBUG:boto:Params: {} 2023-10-12T01:20:36.597 DEBUG:boto:Token: None 2023-10-12T01:20:36.598 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23 2023-10-12T01:20:36.598 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ANRN6GOxaeq01MoBKgLUbQ7GWsA= 2023-10-12T01:20:36.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ANRN6GOxaeq01MoBKgLUbQ7GWsA=', 'Content-Length': '0'} 2023-10-12T01:20:36.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069b1f1f51c42047a-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.646 DEBUG:boto:path=/wafmoa-61/obj23 2023-10-12T01:20:36.647 DEBUG:boto:auth_path=/wafmoa-61/obj23 2023-10-12T01:20:36.647 DEBUG:boto:Method: GET 2023-10-12T01:20:36.648 DEBUG:boto:Path: /wafmoa-61/obj23 2023-10-12T01:20:36.648 DEBUG:boto:Data: 2023-10-12T01:20:36.649 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.649 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.650 DEBUG:boto:Params: {} 2023-10-12T01:20:36.650 DEBUG:boto:Token: None 2023-10-12T01:20:36.651 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj23 2023-10-12T01:20:36.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ANRN6GOxaeq01MoBKgLUbQ7GWsA= 2023-10-12T01:20:36.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ANRN6GOxaeq01MoBKgLUbQ7GWsA=', 'Content-Length': '0'} 2023-10-12T01:20:36.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e5a439576e5103-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.654 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.655 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.655 DEBUG:boto:path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.656 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.656 DEBUG:boto:Method: GET 2023-10-12T01:20:36.657 DEBUG:boto:Path: /wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.657 DEBUG:boto:Data: 2023-10-12T01:20:36.658 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.658 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.659 DEBUG:boto:Params: {} 2023-10-12T01:20:36.659 DEBUG:boto:Token: None 2023-10-12T01:20:36.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093dc0db658a2bd76-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.664 DEBUG:boto:b'wafmoa-61obj231000falseobj232023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.664 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.665 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.665 DEBUG:boto:path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.666 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.666 DEBUG:boto:Method: GET 2023-10-12T01:20:36.667 DEBUG:boto:Path: /wafmoa-61/?prefix=obj23 2023-10-12T01:20:36.667 DEBUG:boto:Data: 2023-10-12T01:20:36.668 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.669 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.669 DEBUG:boto:Params: {} 2023-10-12T01:20:36.670 DEBUG:boto:Token: None 2023-10-12T01:20:36.670 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.671 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076797532e616bb82-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.676 DEBUG:boto:b'wafmoa-61obj231000falseobj232023-10-12T01:16:56.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:20:36.677 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.677 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.678 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.678 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.679 DEBUG:boto:Method: GET 2023-10-12T01:20:36.679 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.680 DEBUG:boto:Data: 2023-10-12T01:20:36.680 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.681 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.681 DEBUG:boto:Params: {} 2023-10-12T01:20:36.682 DEBUG:boto:Token: None 2023-10-12T01:20:36.682 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw= 2023-10-12T01:20:36.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw=', 'Content-Length': '0'} 2023-10-12T01:20:36.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009323d15a5c7e3621-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.686 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.686 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.686 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.687 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.687 DEBUG:boto:Method: GET 2023-10-12T01:20:36.688 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.688 DEBUG:boto:Data: 2023-10-12T01:20:36.688 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.689 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.690 DEBUG:boto:Params: {} 2023-10-12T01:20:36.690 DEBUG:boto:Token: None 2023-10-12T01:20:36.691 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.691 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw= 2023-10-12T01:20:36.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw=', 'Content-Length': '0'} 2023-10-12T01:20:36.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671ad4828881e739-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.694 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.695 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.695 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.696 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.696 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.697 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.697 DEBUG:boto:Data: 2023-10-12T01:20:36.697 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.698 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.699 DEBUG:boto:Params: {} 2023-10-12T01:20:36.699 DEBUG:boto:Token: None 2023-10-12T01:20:36.699 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.700 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/kVGSCiytCR/920MfLf2fb4kEjk= 2023-10-12T01:20:36.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/kVGSCiytCR/920MfLf2fb4kEjk=', 'Content-Length': '0'} 2023-10-12T01:20:36.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007641ddbb6382cc3e-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.703 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.703 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.704 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.704 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.705 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.705 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.705 DEBUG:boto:Data: 2023-10-12T01:20:36.706 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.707 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.707 DEBUG:boto:Params: {} 2023-10-12T01:20:36.708 DEBUG:boto:Token: None 2023-10-12T01:20:36.708 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/kVGSCiytCR/920MfLf2fb4kEjk= 2023-10-12T01:20:36.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/kVGSCiytCR/920MfLf2fb4kEjk=', 'Content-Length': '0'} 2023-10-12T01:20:36.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066217d7b28e5e78e-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:20:36.711 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.712 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.712 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.713 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.713 DEBUG:boto:Method: GET 2023-10-12T01:20:36.713 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.714 DEBUG:boto:Data: 2023-10-12T01:20:36.715 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.715 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.716 DEBUG:boto:Params: {} 2023-10-12T01:20:36.716 DEBUG:boto:Token: None 2023-10-12T01:20:36.717 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw= 2023-10-12T01:20:36.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw=', 'Content-Length': '0'} 2023-10-12T01:20:36.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e95e2e36a9cbb3-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.766 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.767 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.767 DEBUG:boto:path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.768 DEBUG:boto:auth_path=/wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.768 DEBUG:boto:Method: GET 2023-10-12T01:20:36.769 DEBUG:boto:Path: /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.769 DEBUG:boto:Data: 2023-10-12T01:20:36.769 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.770 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.771 DEBUG:boto:Params: {} 2023-10-12T01:20:36.771 DEBUG:boto:Token: None 2023-10-12T01:20:36.772 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29?versionId=null 2023-10-12T01:20:36.773 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw= 2023-10-12T01:20:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qJacyNnFHNU+JCJ/nZYJuV0zMLw=', 'Content-Length': '0'} 2023-10-12T01:20:36.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba5d851bba59d6f-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.776 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.777 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.777 DEBUG:boto:path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.777 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.778 DEBUG:boto:Method: GET 2023-10-12T01:20:36.778 DEBUG:boto:Path: /wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.779 DEBUG:boto:Data: 2023-10-12T01:20:36.779 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.780 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.780 DEBUG:boto:Params: {} 2023-10-12T01:20:36.781 DEBUG:boto:Token: None 2023-10-12T01:20:36.781 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81e533cdd5f62be-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.785 DEBUG:boto:b'wafmoa-61obj291000falseobj292023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.786 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.786 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.787 DEBUG:boto:path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.787 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.788 DEBUG:boto:Method: GET 2023-10-12T01:20:36.788 DEBUG:boto:Path: /wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.789 DEBUG:boto:Data: 2023-10-12T01:20:36.789 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.790 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.791 DEBUG:boto:Params: {} 2023-10-12T01:20:36.791 DEBUG:boto:Token: None 2023-10-12T01:20:36.792 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.792 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000634c89f65eba7134-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.838 DEBUG:boto:b'wafmoa-61obj291000falseobj292023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.839 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.840 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.840 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.841 DEBUG:boto:Path: /wafmoa-61/obj29 2023-10-12T01:20:36.841 DEBUG:boto:Data: 2023-10-12T01:20:36.841 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.842 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.843 DEBUG:boto:Params: {} 2023-10-12T01:20:36.843 DEBUG:boto:Token: None 2023-10-12T01:20:36.844 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29 2023-10-12T01:20:36.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgfToTvQUm5jJZ5YSsPbFea30V4= 2023-10-12T01:20:36.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgfToTvQUm5jJZ5YSsPbFea30V4=', 'Content-Length': '0'} 2023-10-12T01:20:36.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b490dbdcd2646b49-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.847 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.847 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.848 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.848 DEBUG:boto:Path: /wafmoa-61/obj29 2023-10-12T01:20:36.849 DEBUG:boto:Data: 2023-10-12T01:20:36.849 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.850 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.850 DEBUG:boto:Params: {} 2023-10-12T01:20:36.851 DEBUG:boto:Token: None 2023-10-12T01:20:36.851 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29 2023-10-12T01:20:36.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rgfToTvQUm5jJZ5YSsPbFea30V4= 2023-10-12T01:20:36.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rgfToTvQUm5jJZ5YSsPbFea30V4=', 'Content-Length': '0'} 2023-10-12T01:20:36.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b51f2c994ed837d8-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-12T01:20:36.855 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.855 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.856 DEBUG:boto:Method: GET 2023-10-12T01:20:36.856 DEBUG:boto:Path: /wafmoa-61/obj29 2023-10-12T01:20:36.857 DEBUG:boto:Data: 2023-10-12T01:20:36.857 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.858 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.858 DEBUG:boto:Params: {} 2023-10-12T01:20:36.859 DEBUG:boto:Token: None 2023-10-12T01:20:36.859 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29 2023-10-12T01:20:36.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HxI3X2d6k3NNS7it35jqMRyuoKY= 2023-10-12T01:20:36.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HxI3X2d6k3NNS7it35jqMRyuoKY=', 'Content-Length': '0'} 2023-10-12T01:20:36.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d9136c9bb38dad-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.863 DEBUG:boto:path=/wafmoa-61/obj29 2023-10-12T01:20:36.863 DEBUG:boto:auth_path=/wafmoa-61/obj29 2023-10-12T01:20:36.864 DEBUG:boto:Method: GET 2023-10-12T01:20:36.864 DEBUG:boto:Path: /wafmoa-61/obj29 2023-10-12T01:20:36.864 DEBUG:boto:Data: 2023-10-12T01:20:36.865 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.866 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.866 DEBUG:boto:Params: {} 2023-10-12T01:20:36.867 DEBUG:boto:Token: None 2023-10-12T01:20:36.867 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj29 2023-10-12T01:20:36.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HxI3X2d6k3NNS7it35jqMRyuoKY= 2023-10-12T01:20:36.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HxI3X2d6k3NNS7it35jqMRyuoKY=', 'Content-Length': '0'} 2023-10-12T01:20:36.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5712eb8bda2a1b8-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.871 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.871 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.872 DEBUG:boto:path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.872 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.873 DEBUG:boto:Method: GET 2023-10-12T01:20:36.873 DEBUG:boto:Path: /wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.874 DEBUG:boto:Data: 2023-10-12T01:20:36.874 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.875 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.875 DEBUG:boto:Params: {} 2023-10-12T01:20:36.876 DEBUG:boto:Token: None 2023-10-12T01:20:36.876 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.877 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000601ace497f6b94-00652749e4-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.880 DEBUG:boto:b'wafmoa-61obj291000falseobj292023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.881 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:36.881 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:36.882 DEBUG:boto:path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.882 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.882 DEBUG:boto:Method: GET 2023-10-12T01:20:36.883 DEBUG:boto:Path: /wafmoa-61/?prefix=obj29 2023-10-12T01:20:36.883 DEBUG:boto:Data: 2023-10-12T01:20:36.884 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.885 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.885 DEBUG:boto:Params: {} 2023-10-12T01:20:36.885 DEBUG:boto:Token: None 2023-10-12T01:20:36.886 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/ 2023-10-12T01:20:36.886 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4= 2023-10-12T01:20:36.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3SC/aIKMRBF3cVW8uLEhXPwFGj4=', 'Content-Length': '0'} 2023-10-12T01:20:36.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f42da281ef7e60d-00652749e4-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.891 DEBUG:boto:b'wafmoa-61obj291000falseobj292023-10-12T01:16:56.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:36.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:20:36.892 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.893 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.893 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.894 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.894 DEBUG:boto:Method: GET 2023-10-12T01:20:36.895 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.895 DEBUG:boto:Data: 2023-10-12T01:20:36.896 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.897 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.897 DEBUG:boto:Params: {} 2023-10-12T01:20:36.897 DEBUG:boto:Token: None 2023-10-12T01:20:36.898 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.898 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM= 2023-10-12T01:20:36.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM=', 'Content-Length': '0'} 2023-10-12T01:20:36.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc1ed9736a34268-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.946 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.947 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.947 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.948 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.948 DEBUG:boto:Method: GET 2023-10-12T01:20:36.948 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.949 DEBUG:boto:Data: 2023-10-12T01:20:36.949 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.950 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.950 DEBUG:boto:Params: {} 2023-10-12T01:20:36.951 DEBUG:boto:Token: None 2023-10-12T01:20:36.951 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.952 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM= 2023-10-12T01:20:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM=', 'Content-Length': '0'} 2023-10-12T01:20:36.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044701d23a752e38e-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.955 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.956 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.956 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.956 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.957 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.957 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.958 DEBUG:boto:Data: 2023-10-12T01:20:36.958 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.959 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.961 DEBUG:boto:Params: {} 2023-10-12T01:20:36.961 DEBUG:boto:Token: None 2023-10-12T01:20:36.962 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.962 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/AeVY7fqytiYk/IR3LXikZfIq2o= 2023-10-12T01:20:36.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/AeVY7fqytiYk/IR3LXikZfIq2o=', 'Content-Length': '0'} 2023-10-12T01:20:36.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d510bcfb27aa2e9-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.966 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.966 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.966 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.967 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.967 DEBUG:boto:Method: HEAD 2023-10-12T01:20:36.968 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.968 DEBUG:boto:Data: 2023-10-12T01:20:36.968 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.969 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.970 DEBUG:boto:Params: {} 2023-10-12T01:20:36.970 DEBUG:boto:Token: None 2023-10-12T01:20:36.970 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/AeVY7fqytiYk/IR3LXikZfIq2o= 2023-10-12T01:20:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/AeVY7fqytiYk/IR3LXikZfIq2o=', 'Content-Length': '0'} 2023-10-12T01:20:36.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074dadc44c5fcbb0d-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:20:36.975 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.975 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.975 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.976 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.976 DEBUG:boto:Method: GET 2023-10-12T01:20:36.977 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.977 DEBUG:boto:Data: 2023-10-12T01:20:36.977 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:36.978 DEBUG:boto:Port: 8001 2023-10-12T01:20:36.979 DEBUG:boto:Params: {} 2023-10-12T01:20:36.979 DEBUG:boto:Token: None 2023-10-12T01:20:36.980 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM= 2023-10-12T01:20:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM=', 'Content-Length': '0'} 2023-10-12T01:20:36.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310546af5eb2f2cf-00652749e4-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:36.983 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:36.983 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:36.984 DEBUG:boto:path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.984 DEBUG:boto:auth_path=/wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.985 DEBUG:boto:Method: GET 2023-10-12T01:20:36.985 DEBUG:boto:Path: /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.985 DEBUG:boto:Data: 2023-10-12T01:20:36.986 DEBUG:boto:Headers: {} 2023-10-12T01:20:36.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:36.987 DEBUG:boto:Port: 8000 2023-10-12T01:20:36.987 DEBUG:boto:Params: {} 2023-10-12T01:20:36.987 DEBUG:boto:Token: None 2023-10-12T01:20:36.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:36 GMT /wafmoa-61/obj31?versionId=null 2023-10-12T01:20:36.988 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM= 2023-10-12T01:20:36.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:36 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3gwv3AFxOQ4RXwsvLa4/231OwlM=', 'Content-Length': '0'} 2023-10-12T01:20:36.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008365bee00c181f12-00652749e4-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.034 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.035 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.035 DEBUG:boto:path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.036 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.036 DEBUG:boto:Method: GET 2023-10-12T01:20:37.036 DEBUG:boto:Path: /wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.037 DEBUG:boto:Data: 2023-10-12T01:20:37.037 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.038 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.038 DEBUG:boto:Params: {} 2023-10-12T01:20:37.039 DEBUG:boto:Token: None 2023-10-12T01:20:37.039 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.040 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a305a3f902a5a6e-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.043 DEBUG:boto:b'wafmoa-61obj311000falseobj312023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.044 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.044 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.045 DEBUG:boto:path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.045 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.045 DEBUG:boto:Method: GET 2023-10-12T01:20:37.046 DEBUG:boto:Path: /wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.046 DEBUG:boto:Data: 2023-10-12T01:20:37.047 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.048 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.048 DEBUG:boto:Params: {} 2023-10-12T01:20:37.049 DEBUG:boto:Token: None 2023-10-12T01:20:37.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2ce7d1df9d4051-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.055 DEBUG:boto:b'wafmoa-61obj311000falseobj312023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.055 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:37.056 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:37.056 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.057 DEBUG:boto:Path: /wafmoa-61/obj31 2023-10-12T01:20:37.057 DEBUG:boto:Data: 2023-10-12T01:20:37.057 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.058 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.059 DEBUG:boto:Params: {} 2023-10-12T01:20:37.059 DEBUG:boto:Token: None 2023-10-12T01:20:37.060 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj31 2023-10-12T01:20:37.060 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Djq2aQ1BjPmuHZv8sdDrjS5o5Qs= 2023-10-12T01:20:37.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Djq2aQ1BjPmuHZv8sdDrjS5o5Qs=', 'Content-Length': '0'} 2023-10-12T01:20:37.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009992076f1e069dcc-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.063 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:37.064 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:37.064 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.064 DEBUG:boto:Path: /wafmoa-61/obj31 2023-10-12T01:20:37.065 DEBUG:boto:Data: 2023-10-12T01:20:37.065 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.066 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.067 DEBUG:boto:Params: {} 2023-10-12T01:20:37.067 DEBUG:boto:Token: None 2023-10-12T01:20:37.067 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj31 2023-10-12T01:20:37.068 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Djq2aQ1BjPmuHZv8sdDrjS5o5Qs= 2023-10-12T01:20:37.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Djq2aQ1BjPmuHZv8sdDrjS5o5Qs=', 'Content-Length': '0'} 2023-10-12T01:20:37.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761f3e13a1cdad8a-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-12T01:20:37.071 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:37.072 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:37.072 DEBUG:boto:Method: GET 2023-10-12T01:20:37.073 DEBUG:boto:Path: /wafmoa-61/obj31 2023-10-12T01:20:37.073 DEBUG:boto:Data: 2023-10-12T01:20:37.074 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.074 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.075 DEBUG:boto:Params: {} 2023-10-12T01:20:37.075 DEBUG:boto:Token: None 2023-10-12T01:20:37.076 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj31 2023-10-12T01:20:37.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mn55+CldA7vQGnwgcLrPsU0zXx0= 2023-10-12T01:20:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mn55+CldA7vQGnwgcLrPsU0zXx0=', 'Content-Length': '0'} 2023-10-12T01:20:37.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423fe34e57c9e918-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.122 DEBUG:boto:path=/wafmoa-61/obj31 2023-10-12T01:20:37.123 DEBUG:boto:auth_path=/wafmoa-61/obj31 2023-10-12T01:20:37.123 DEBUG:boto:Method: GET 2023-10-12T01:20:37.124 DEBUG:boto:Path: /wafmoa-61/obj31 2023-10-12T01:20:37.124 DEBUG:boto:Data: 2023-10-12T01:20:37.125 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.125 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.126 DEBUG:boto:Params: {} 2023-10-12T01:20:37.127 DEBUG:boto:Token: None 2023-10-12T01:20:37.127 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj31 2023-10-12T01:20:37.127 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mn55+CldA7vQGnwgcLrPsU0zXx0= 2023-10-12T01:20:37.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mn55+CldA7vQGnwgcLrPsU0zXx0=', 'Content-Length': '0'} 2023-10-12T01:20:37.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046db700a45c4d02a-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.131 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.131 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.132 DEBUG:boto:path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.132 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.133 DEBUG:boto:Method: GET 2023-10-12T01:20:37.133 DEBUG:boto:Path: /wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.133 DEBUG:boto:Data: 2023-10-12T01:20:37.134 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.135 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.135 DEBUG:boto:Params: {} 2023-10-12T01:20:37.136 DEBUG:boto:Token: None 2023-10-12T01:20:37.136 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.137 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdeec5dfb649f4d3-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.140 DEBUG:boto:b'wafmoa-61obj311000falseobj312023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.141 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.141 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.142 DEBUG:boto:path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.142 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.143 DEBUG:boto:Method: GET 2023-10-12T01:20:37.143 DEBUG:boto:Path: /wafmoa-61/?prefix=obj31 2023-10-12T01:20:37.143 DEBUG:boto:Data: 2023-10-12T01:20:37.144 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.145 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.145 DEBUG:boto:Params: {} 2023-10-12T01:20:37.146 DEBUG:boto:Token: None 2023-10-12T01:20:37.146 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.147 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95e824551fdb1dc-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.152 DEBUG:boto:b'wafmoa-61obj311000falseobj312023-10-12T01:16:56.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:20:37.153 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.153 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.154 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.154 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.155 DEBUG:boto:Method: GET 2023-10-12T01:20:37.155 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.155 DEBUG:boto:Data: 2023-10-12T01:20:37.156 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.157 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.157 DEBUG:boto:Params: {} 2023-10-12T01:20:37.158 DEBUG:boto:Token: None 2023-10-12T01:20:37.158 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk= 2023-10-12T01:20:37.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk=', 'Content-Length': '0'} 2023-10-12T01:20:37.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ad4173b694a709-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.162 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.162 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.163 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.163 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.164 DEBUG:boto:Method: GET 2023-10-12T01:20:37.164 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.165 DEBUG:boto:Data: 2023-10-12T01:20:37.165 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.166 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.167 DEBUG:boto:Params: {} 2023-10-12T01:20:37.167 DEBUG:boto:Token: None 2023-10-12T01:20:37.169 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk= 2023-10-12T01:20:37.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk=', 'Content-Length': '0'} 2023-10-12T01:20:37.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087e6710234744fc-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.218 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.219 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.219 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.220 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.220 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.220 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.221 DEBUG:boto:Data: 2023-10-12T01:20:37.221 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.222 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.222 DEBUG:boto:Params: {} 2023-10-12T01:20:37.223 DEBUG:boto:Token: None 2023-10-12T01:20:37.223 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7nE9EavBhJ2kkT+C3Bjw6OP4zmM= 2023-10-12T01:20:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7nE9EavBhJ2kkT+C3Bjw6OP4zmM=', 'Content-Length': '0'} 2023-10-12T01:20:37.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa06bea9b99898e-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.226 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.227 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.227 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.228 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.228 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.228 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.229 DEBUG:boto:Data: 2023-10-12T01:20:37.229 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.230 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.230 DEBUG:boto:Params: {} 2023-10-12T01:20:37.231 DEBUG:boto:Token: None 2023-10-12T01:20:37.231 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7nE9EavBhJ2kkT+C3Bjw6OP4zmM= 2023-10-12T01:20:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7nE9EavBhJ2kkT+C3Bjw6OP4zmM=', 'Content-Length': '0'} 2023-10-12T01:20:37.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048408b39e9e83d79-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:20:37.235 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.235 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.236 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.236 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.237 DEBUG:boto:Method: GET 2023-10-12T01:20:37.237 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.237 DEBUG:boto:Data: 2023-10-12T01:20:37.238 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.239 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.239 DEBUG:boto:Params: {} 2023-10-12T01:20:37.239 DEBUG:boto:Token: None 2023-10-12T01:20:37.240 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.240 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk= 2023-10-12T01:20:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk=', 'Content-Length': '0'} 2023-10-12T01:20:37.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba39db99703dee5-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.244 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.244 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.244 DEBUG:boto:path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.245 DEBUG:boto:auth_path=/wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.245 DEBUG:boto:Method: GET 2023-10-12T01:20:37.246 DEBUG:boto:Path: /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.246 DEBUG:boto:Data: 2023-10-12T01:20:37.247 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.247 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.248 DEBUG:boto:Params: {} 2023-10-12T01:20:37.248 DEBUG:boto:Token: None 2023-10-12T01:20:37.249 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33?versionId=null 2023-10-12T01:20:37.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk= 2023-10-12T01:20:37.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ozyg/Ep2pfPSE3gckwfle9hyxsk=', 'Content-Length': '0'} 2023-10-12T01:20:37.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c262d5cec3fdd111-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.253 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.253 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.253 DEBUG:boto:path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.254 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.254 DEBUG:boto:Method: GET 2023-10-12T01:20:37.255 DEBUG:boto:Path: /wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.255 DEBUG:boto:Data: 2023-10-12T01:20:37.255 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.256 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.257 DEBUG:boto:Params: {} 2023-10-12T01:20:37.257 DEBUG:boto:Token: None 2023-10-12T01:20:37.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcb486f06d3fe85d-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.302 DEBUG:boto:b'wafmoa-61obj331000falseobj332023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.303 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.303 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.304 DEBUG:boto:path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.304 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.305 DEBUG:boto:Method: GET 2023-10-12T01:20:37.305 DEBUG:boto:Path: /wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.305 DEBUG:boto:Data: 2023-10-12T01:20:37.306 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.307 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.307 DEBUG:boto:Params: {} 2023-10-12T01:20:37.308 DEBUG:boto:Token: None 2023-10-12T01:20:37.308 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.309 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005510aa479df52d04-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.314 DEBUG:boto:b'wafmoa-61obj331000falseobj332023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.315 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.315 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.315 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.316 DEBUG:boto:Path: /wafmoa-61/obj33 2023-10-12T01:20:37.316 DEBUG:boto:Data: 2023-10-12T01:20:37.317 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.317 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.318 DEBUG:boto:Params: {} 2023-10-12T01:20:37.318 DEBUG:boto:Token: None 2023-10-12T01:20:37.319 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33 2023-10-12T01:20:37.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NKDI8TcwlZL91rTrUy1Zvxpwn/E= 2023-10-12T01:20:37.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NKDI8TcwlZL91rTrUy1Zvxpwn/E=', 'Content-Length': '0'} 2023-10-12T01:20:37.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c1186d27512957-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.322 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.323 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.323 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.323 DEBUG:boto:Path: /wafmoa-61/obj33 2023-10-12T01:20:37.324 DEBUG:boto:Data: 2023-10-12T01:20:37.324 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.325 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.325 DEBUG:boto:Params: {} 2023-10-12T01:20:37.326 DEBUG:boto:Token: None 2023-10-12T01:20:37.326 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33 2023-10-12T01:20:37.327 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NKDI8TcwlZL91rTrUy1Zvxpwn/E= 2023-10-12T01:20:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NKDI8TcwlZL91rTrUy1Zvxpwn/E=', 'Content-Length': '0'} 2023-10-12T01:20:37.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22cb646623587d4-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:20:37.330 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.330 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.331 DEBUG:boto:Method: GET 2023-10-12T01:20:37.331 DEBUG:boto:Path: /wafmoa-61/obj33 2023-10-12T01:20:37.332 DEBUG:boto:Data: 2023-10-12T01:20:37.332 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.333 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.333 DEBUG:boto:Params: {} 2023-10-12T01:20:37.334 DEBUG:boto:Token: None 2023-10-12T01:20:37.334 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33 2023-10-12T01:20:37.335 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rOmRyYNoCEzBKe02L+iUZTmXmvE= 2023-10-12T01:20:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rOmRyYNoCEzBKe02L+iUZTmXmvE=', 'Content-Length': '0'} 2023-10-12T01:20:37.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3faada12d5248ed-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.338 DEBUG:boto:path=/wafmoa-61/obj33 2023-10-12T01:20:37.339 DEBUG:boto:auth_path=/wafmoa-61/obj33 2023-10-12T01:20:37.339 DEBUG:boto:Method: GET 2023-10-12T01:20:37.340 DEBUG:boto:Path: /wafmoa-61/obj33 2023-10-12T01:20:37.340 DEBUG:boto:Data: 2023-10-12T01:20:37.340 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.341 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.342 DEBUG:boto:Params: {} 2023-10-12T01:20:37.342 DEBUG:boto:Token: None 2023-10-12T01:20:37.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj33 2023-10-12T01:20:37.343 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rOmRyYNoCEzBKe02L+iUZTmXmvE= 2023-10-12T01:20:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rOmRyYNoCEzBKe02L+iUZTmXmvE=', 'Content-Length': '0'} 2023-10-12T01:20:37.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e734a480c585c68-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.346 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.347 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.347 DEBUG:boto:path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.348 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.348 DEBUG:boto:Method: GET 2023-10-12T01:20:37.348 DEBUG:boto:Path: /wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.349 DEBUG:boto:Data: 2023-10-12T01:20:37.349 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.350 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.351 DEBUG:boto:Params: {} 2023-10-12T01:20:37.351 DEBUG:boto:Token: None 2023-10-12T01:20:37.351 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.352 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b050b735c51859a7-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.355 DEBUG:boto:b'wafmoa-61obj331000falseobj332023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.356 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.357 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.357 DEBUG:boto:path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.357 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.358 DEBUG:boto:Method: GET 2023-10-12T01:20:37.358 DEBUG:boto:Path: /wafmoa-61/?prefix=obj33 2023-10-12T01:20:37.359 DEBUG:boto:Data: 2023-10-12T01:20:37.359 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.360 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.360 DEBUG:boto:Params: {} 2023-10-12T01:20:37.360 DEBUG:boto:Token: None 2023-10-12T01:20:37.361 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c226e310057ae24-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.366 DEBUG:boto:b'wafmoa-61obj331000falseobj332023-10-12T01:16:55.262Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:20:37.367 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.368 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.368 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.369 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.369 DEBUG:boto:Method: GET 2023-10-12T01:20:37.369 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.370 DEBUG:boto:Data: 2023-10-12T01:20:37.370 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.371 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.372 DEBUG:boto:Params: {} 2023-10-12T01:20:37.374 DEBUG:boto:Token: None 2023-10-12T01:20:37.374 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04= 2023-10-12T01:20:37.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04=', 'Content-Length': '0'} 2023-10-12T01:20:37.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f2a628452e3a7c-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.378 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.379 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.379 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.379 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.380 DEBUG:boto:Method: GET 2023-10-12T01:20:37.380 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.381 DEBUG:boto:Data: 2023-10-12T01:20:37.381 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.382 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.382 DEBUG:boto:Params: {} 2023-10-12T01:20:37.383 DEBUG:boto:Token: None 2023-10-12T01:20:37.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04= 2023-10-12T01:20:37.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04=', 'Content-Length': '0'} 2023-10-12T01:20:37.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e59f850904eb72-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.387 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.388 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.388 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.388 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.389 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.389 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.390 DEBUG:boto:Data: 2023-10-12T01:20:37.390 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.391 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.391 DEBUG:boto:Params: {} 2023-10-12T01:20:37.392 DEBUG:boto:Token: None 2023-10-12T01:20:37.392 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O4yrFmNOcxUQdL/zZ1DsB8UesjQ= 2023-10-12T01:20:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O4yrFmNOcxUQdL/zZ1DsB8UesjQ=', 'Content-Length': '0'} 2023-10-12T01:20:37.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff341aec6edaa5c5-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.396 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.396 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.397 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.397 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.397 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.398 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.398 DEBUG:boto:Data: 2023-10-12T01:20:37.399 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.399 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.400 DEBUG:boto:Params: {} 2023-10-12T01:20:37.400 DEBUG:boto:Token: None 2023-10-12T01:20:37.401 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O4yrFmNOcxUQdL/zZ1DsB8UesjQ= 2023-10-12T01:20:37.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O4yrFmNOcxUQdL/zZ1DsB8UesjQ=', 'Content-Length': '0'} 2023-10-12T01:20:37.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf2e8502b18b81d-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:20:37.404 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.405 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.405 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.406 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.406 DEBUG:boto:Method: GET 2023-10-12T01:20:37.407 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.407 DEBUG:boto:Data: 2023-10-12T01:20:37.408 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.408 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.409 DEBUG:boto:Params: {} 2023-10-12T01:20:37.409 DEBUG:boto:Token: None 2023-10-12T01:20:37.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04= 2023-10-12T01:20:37.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04=', 'Content-Length': '0'} 2023-10-12T01:20:37.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ee611f8a9e8904-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.454 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.455 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.455 DEBUG:boto:path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.456 DEBUG:boto:auth_path=/wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.456 DEBUG:boto:Method: GET 2023-10-12T01:20:37.457 DEBUG:boto:Path: /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.457 DEBUG:boto:Data: 2023-10-12T01:20:37.458 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.459 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.459 DEBUG:boto:Params: {} 2023-10-12T01:20:37.460 DEBUG:boto:Token: None 2023-10-12T01:20:37.460 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34?versionId=null 2023-10-12T01:20:37.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04= 2023-10-12T01:20:37.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s5Vse+B/d2VIQfbQW9gYOumvI04=', 'Content-Length': '0'} 2023-10-12T01:20:37.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382d55ebf0135723-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.464 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.464 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.465 DEBUG:boto:path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.465 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.466 DEBUG:boto:Method: GET 2023-10-12T01:20:37.466 DEBUG:boto:Path: /wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.467 DEBUG:boto:Data: 2023-10-12T01:20:37.467 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.468 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.468 DEBUG:boto:Params: {} 2023-10-12T01:20:37.469 DEBUG:boto:Token: None 2023-10-12T01:20:37.469 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.470 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb64c7c6bd88ecf-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.473 DEBUG:boto:b'wafmoa-61obj341000falseobj342023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.474 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.474 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.475 DEBUG:boto:path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.475 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.475 DEBUG:boto:Method: GET 2023-10-12T01:20:37.476 DEBUG:boto:Path: /wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.476 DEBUG:boto:Data: 2023-10-12T01:20:37.477 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.478 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.478 DEBUG:boto:Params: {} 2023-10-12T01:20:37.479 DEBUG:boto:Token: None 2023-10-12T01:20:37.479 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.479 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803cf6ad43dc2416-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.484 DEBUG:boto:b'wafmoa-61obj341000falseobj342023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.485 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.485 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.486 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.486 DEBUG:boto:Path: /wafmoa-61/obj34 2023-10-12T01:20:37.486 DEBUG:boto:Data: 2023-10-12T01:20:37.487 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.488 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.488 DEBUG:boto:Params: {} 2023-10-12T01:20:37.489 DEBUG:boto:Token: None 2023-10-12T01:20:37.489 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34 2023-10-12T01:20:37.489 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bqgOtIJWyiqdPrg2+vQDrV6NdvQ= 2023-10-12T01:20:37.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bqgOtIJWyiqdPrg2+vQDrV6NdvQ=', 'Content-Length': '0'} 2023-10-12T01:20:37.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebf5e9050b5a875-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.493 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.493 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.493 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.494 DEBUG:boto:Path: /wafmoa-61/obj34 2023-10-12T01:20:37.494 DEBUG:boto:Data: 2023-10-12T01:20:37.495 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.495 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.496 DEBUG:boto:Params: {} 2023-10-12T01:20:37.496 DEBUG:boto:Token: None 2023-10-12T01:20:37.497 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34 2023-10-12T01:20:37.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bqgOtIJWyiqdPrg2+vQDrV6NdvQ= 2023-10-12T01:20:37.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bqgOtIJWyiqdPrg2+vQDrV6NdvQ=', 'Content-Length': '0'} 2023-10-12T01:20:37.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3c872578a8f794-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-12T01:20:37.500 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.501 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.501 DEBUG:boto:Method: GET 2023-10-12T01:20:37.502 DEBUG:boto:Path: /wafmoa-61/obj34 2023-10-12T01:20:37.502 DEBUG:boto:Data: 2023-10-12T01:20:37.503 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.504 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.504 DEBUG:boto:Params: {} 2023-10-12T01:20:37.505 DEBUG:boto:Token: None 2023-10-12T01:20:37.505 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34 2023-10-12T01:20:37.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ubsPBUyyhvFuozHaohOrYt7h5q4= 2023-10-12T01:20:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ubsPBUyyhvFuozHaohOrYt7h5q4=', 'Content-Length': '0'} 2023-10-12T01:20:37.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9bd5cdad77594a-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.509 DEBUG:boto:path=/wafmoa-61/obj34 2023-10-12T01:20:37.509 DEBUG:boto:auth_path=/wafmoa-61/obj34 2023-10-12T01:20:37.510 DEBUG:boto:Method: GET 2023-10-12T01:20:37.510 DEBUG:boto:Path: /wafmoa-61/obj34 2023-10-12T01:20:37.510 DEBUG:boto:Data: 2023-10-12T01:20:37.511 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.512 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.512 DEBUG:boto:Params: {} 2023-10-12T01:20:37.513 DEBUG:boto:Token: None 2023-10-12T01:20:37.513 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj34 2023-10-12T01:20:37.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ubsPBUyyhvFuozHaohOrYt7h5q4= 2023-10-12T01:20:37.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ubsPBUyyhvFuozHaohOrYt7h5q4=', 'Content-Length': '0'} 2023-10-12T01:20:37.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf3b54bb439c252-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.517 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.517 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.518 DEBUG:boto:path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.518 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.518 DEBUG:boto:Method: GET 2023-10-12T01:20:37.519 DEBUG:boto:Path: /wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.519 DEBUG:boto:Data: 2023-10-12T01:20:37.520 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.520 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.521 DEBUG:boto:Params: {} 2023-10-12T01:20:37.521 DEBUG:boto:Token: None 2023-10-12T01:20:37.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3dd626047599dac-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.526 DEBUG:boto:b'wafmoa-61obj341000falseobj342023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.526 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.527 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.527 DEBUG:boto:path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.528 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.528 DEBUG:boto:Method: GET 2023-10-12T01:20:37.529 DEBUG:boto:Path: /wafmoa-61/?prefix=obj34 2023-10-12T01:20:37.529 DEBUG:boto:Data: 2023-10-12T01:20:37.529 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.530 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.531 DEBUG:boto:Params: {} 2023-10-12T01:20:37.531 DEBUG:boto:Token: None 2023-10-12T01:20:37.532 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e7bb09abb29458-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.537 DEBUG:boto:b'wafmoa-61obj341000falseobj342023-10-12T01:16:55.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:20:37.538 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.539 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.539 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.540 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.540 DEBUG:boto:Method: GET 2023-10-12T01:20:37.540 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.541 DEBUG:boto:Data: 2023-10-12T01:20:37.541 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.542 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.543 DEBUG:boto:Params: {} 2023-10-12T01:20:37.543 DEBUG:boto:Token: None 2023-10-12T01:20:37.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014= 2023-10-12T01:20:37.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014=', 'Content-Length': '0'} 2023-10-12T01:20:37.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf08a1914346e36-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.547 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.547 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.548 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.548 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.548 DEBUG:boto:Method: GET 2023-10-12T01:20:37.549 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.549 DEBUG:boto:Data: 2023-10-12T01:20:37.550 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.550 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.551 DEBUG:boto:Params: {} 2023-10-12T01:20:37.551 DEBUG:boto:Token: None 2023-10-12T01:20:37.552 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.552 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014= 2023-10-12T01:20:37.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014=', 'Content-Length': '0'} 2023-10-12T01:20:37.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da67f57154a977d3-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.598 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.599 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.599 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.600 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.600 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.601 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.601 DEBUG:boto:Data: 2023-10-12T01:20:37.601 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.602 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.603 DEBUG:boto:Params: {} 2023-10-12T01:20:37.603 DEBUG:boto:Token: None 2023-10-12T01:20:37.604 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.604 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K2BtjKuAL9N3lAeoOwOUjjr/DYg= 2023-10-12T01:20:37.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K2BtjKuAL9N3lAeoOwOUjjr/DYg=', 'Content-Length': '0'} 2023-10-12T01:20:37.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a181855476ad3921-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.607 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.608 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.608 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.609 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.609 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.609 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.610 DEBUG:boto:Data: 2023-10-12T01:20:37.610 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.611 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.611 DEBUG:boto:Params: {} 2023-10-12T01:20:37.612 DEBUG:boto:Token: None 2023-10-12T01:20:37.612 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K2BtjKuAL9N3lAeoOwOUjjr/DYg= 2023-10-12T01:20:37.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K2BtjKuAL9N3lAeoOwOUjjr/DYg=', 'Content-Length': '0'} 2023-10-12T01:20:37.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315710c3d0bfda90-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:20:37.616 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.616 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.617 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.617 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.618 DEBUG:boto:Method: GET 2023-10-12T01:20:37.618 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.618 DEBUG:boto:Data: 2023-10-12T01:20:37.619 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.620 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.620 DEBUG:boto:Params: {} 2023-10-12T01:20:37.621 DEBUG:boto:Token: None 2023-10-12T01:20:37.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014= 2023-10-12T01:20:37.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014=', 'Content-Length': '0'} 2023-10-12T01:20:37.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfdcb5a366fec46-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.625 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.625 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.626 DEBUG:boto:path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.626 DEBUG:boto:auth_path=/wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.627 DEBUG:boto:Method: GET 2023-10-12T01:20:37.627 DEBUG:boto:Path: /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.627 DEBUG:boto:Data: 2023-10-12T01:20:37.628 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.629 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.629 DEBUG:boto:Params: {} 2023-10-12T01:20:37.630 DEBUG:boto:Token: None 2023-10-12T01:20:37.630 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37?versionId=null 2023-10-12T01:20:37.631 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014= 2023-10-12T01:20:37.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I1jPWrMUwu5FeuG8qh20QzfL014=', 'Content-Length': '0'} 2023-10-12T01:20:37.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace5c4140dc3a851-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.634 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.634 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.635 DEBUG:boto:path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.635 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.636 DEBUG:boto:Method: GET 2023-10-12T01:20:37.636 DEBUG:boto:Path: /wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.637 DEBUG:boto:Data: 2023-10-12T01:20:37.637 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.638 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.638 DEBUG:boto:Params: {} 2023-10-12T01:20:37.639 DEBUG:boto:Token: None 2023-10-12T01:20:37.639 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.640 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a058d98050fe10a9-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.643 DEBUG:boto:b'wafmoa-61obj371000falseobj372023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.644 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.645 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.645 DEBUG:boto:path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.645 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.646 DEBUG:boto:Method: GET 2023-10-12T01:20:37.646 DEBUG:boto:Path: /wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.647 DEBUG:boto:Data: 2023-10-12T01:20:37.647 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.648 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.648 DEBUG:boto:Params: {} 2023-10-12T01:20:37.649 DEBUG:boto:Token: None 2023-10-12T01:20:37.649 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.650 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97b604a9ff2b011-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.655 DEBUG:boto:b'wafmoa-61obj371000falseobj372023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.656 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.657 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.657 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.657 DEBUG:boto:Path: /wafmoa-61/obj37 2023-10-12T01:20:37.658 DEBUG:boto:Data: 2023-10-12T01:20:37.658 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.659 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.660 DEBUG:boto:Params: {} 2023-10-12T01:20:37.660 DEBUG:boto:Token: None 2023-10-12T01:20:37.660 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37 2023-10-12T01:20:37.661 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oP0Q4pejSsr6tIV3+7aakZdYbn4= 2023-10-12T01:20:37.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oP0Q4pejSsr6tIV3+7aakZdYbn4=', 'Content-Length': '0'} 2023-10-12T01:20:37.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b2b2cd1926bdf3a-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.664 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.665 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.665 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.665 DEBUG:boto:Path: /wafmoa-61/obj37 2023-10-12T01:20:37.666 DEBUG:boto:Data: 2023-10-12T01:20:37.666 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.667 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.668 DEBUG:boto:Params: {} 2023-10-12T01:20:37.668 DEBUG:boto:Token: None 2023-10-12T01:20:37.668 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37 2023-10-12T01:20:37.669 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oP0Q4pejSsr6tIV3+7aakZdYbn4= 2023-10-12T01:20:37.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oP0Q4pejSsr6tIV3+7aakZdYbn4=', 'Content-Length': '0'} 2023-10-12T01:20:37.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfde45c273f627d7-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-12T01:20:37.672 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.673 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.673 DEBUG:boto:Method: GET 2023-10-12T01:20:37.674 DEBUG:boto:Path: /wafmoa-61/obj37 2023-10-12T01:20:37.674 DEBUG:boto:Data: 2023-10-12T01:20:37.675 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.675 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.676 DEBUG:boto:Params: {} 2023-10-12T01:20:37.676 DEBUG:boto:Token: None 2023-10-12T01:20:37.677 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37 2023-10-12T01:20:37.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bl4OcQuAXfqgbTFPzoU18Bplphs= 2023-10-12T01:20:37.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bl4OcQuAXfqgbTFPzoU18Bplphs=', 'Content-Length': '0'} 2023-10-12T01:20:37.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004713dd2335e8249e-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.722 DEBUG:boto:path=/wafmoa-61/obj37 2023-10-12T01:20:37.723 DEBUG:boto:auth_path=/wafmoa-61/obj37 2023-10-12T01:20:37.724 DEBUG:boto:Method: GET 2023-10-12T01:20:37.724 DEBUG:boto:Path: /wafmoa-61/obj37 2023-10-12T01:20:37.724 DEBUG:boto:Data: 2023-10-12T01:20:37.725 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.726 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.726 DEBUG:boto:Params: {} 2023-10-12T01:20:37.727 DEBUG:boto:Token: None 2023-10-12T01:20:37.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj37 2023-10-12T01:20:37.728 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bl4OcQuAXfqgbTFPzoU18Bplphs= 2023-10-12T01:20:37.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bl4OcQuAXfqgbTFPzoU18Bplphs=', 'Content-Length': '0'} 2023-10-12T01:20:37.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fb7645b6376c4f-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.731 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.732 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.732 DEBUG:boto:path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.733 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.733 DEBUG:boto:Method: GET 2023-10-12T01:20:37.734 DEBUG:boto:Path: /wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.734 DEBUG:boto:Data: 2023-10-12T01:20:37.735 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.736 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.736 DEBUG:boto:Params: {} 2023-10-12T01:20:37.737 DEBUG:boto:Token: None 2023-10-12T01:20:37.737 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf4d0fbdbb6c1c3-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.741 DEBUG:boto:b'wafmoa-61obj371000falseobj372023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.742 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.743 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.744 DEBUG:boto:path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.744 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.745 DEBUG:boto:Method: GET 2023-10-12T01:20:37.746 DEBUG:boto:Path: /wafmoa-61/?prefix=obj37 2023-10-12T01:20:37.746 DEBUG:boto:Data: 2023-10-12T01:20:37.747 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.748 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.748 DEBUG:boto:Params: {} 2023-10-12T01:20:37.749 DEBUG:boto:Token: None 2023-10-12T01:20:37.749 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.750 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645b61cb4576d4dc-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.755 DEBUG:boto:b'wafmoa-61obj371000falseobj372023-10-12T01:16:57.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:20:37.756 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.757 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.757 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.758 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.758 DEBUG:boto:Method: GET 2023-10-12T01:20:37.759 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.759 DEBUG:boto:Data: 2023-10-12T01:20:37.759 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.760 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.761 DEBUG:boto:Params: {} 2023-10-12T01:20:37.761 DEBUG:boto:Token: None 2023-10-12T01:20:37.762 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.762 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo= 2023-10-12T01:20:37.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo=', 'Content-Length': '0'} 2023-10-12T01:20:37.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f682a9142093c7-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.765 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.766 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.767 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.767 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.768 DEBUG:boto:Method: GET 2023-10-12T01:20:37.768 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.768 DEBUG:boto:Data: 2023-10-12T01:20:37.769 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.770 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.770 DEBUG:boto:Params: {} 2023-10-12T01:20:37.771 DEBUG:boto:Token: None 2023-10-12T01:20:37.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo= 2023-10-12T01:20:37.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo=', 'Content-Length': '0'} 2023-10-12T01:20:37.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003079a217c11bd7f5-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.775 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.776 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.777 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.778 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.778 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.781 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.782 DEBUG:boto:Data: 2023-10-12T01:20:37.782 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.783 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.784 DEBUG:boto:Params: {} 2023-10-12T01:20:37.784 DEBUG:boto:Token: None 2023-10-12T01:20:37.785 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xSN9l2KLCV8JHs9z14GteU/HDpM= 2023-10-12T01:20:37.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xSN9l2KLCV8JHs9z14GteU/HDpM=', 'Content-Length': '0'} 2023-10-12T01:20:37.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3e6434355098d8-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.788 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.789 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.789 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.789 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.790 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.790 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.791 DEBUG:boto:Data: 2023-10-12T01:20:37.791 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.792 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.793 DEBUG:boto:Params: {} 2023-10-12T01:20:37.793 DEBUG:boto:Token: None 2023-10-12T01:20:37.794 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.794 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xSN9l2KLCV8JHs9z14GteU/HDpM= 2023-10-12T01:20:37.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xSN9l2KLCV8JHs9z14GteU/HDpM=', 'Content-Length': '0'} 2023-10-12T01:20:37.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007247f76bdc6b86af-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:20:37.798 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.798 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.799 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.799 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.799 DEBUG:boto:Method: GET 2023-10-12T01:20:37.800 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.800 DEBUG:boto:Data: 2023-10-12T01:20:37.801 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.802 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.802 DEBUG:boto:Params: {} 2023-10-12T01:20:37.803 DEBUG:boto:Token: None 2023-10-12T01:20:37.803 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo= 2023-10-12T01:20:37.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo=', 'Content-Length': '0'} 2023-10-12T01:20:37.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7f4e9a0e095998-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.807 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.807 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.808 DEBUG:boto:path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.808 DEBUG:boto:auth_path=/wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.809 DEBUG:boto:Method: GET 2023-10-12T01:20:37.809 DEBUG:boto:Path: /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.809 DEBUG:boto:Data: 2023-10-12T01:20:37.810 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.811 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.811 DEBUG:boto:Params: {} 2023-10-12T01:20:37.812 DEBUG:boto:Token: None 2023-10-12T01:20:37.812 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38?versionId=null 2023-10-12T01:20:37.813 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo= 2023-10-12T01:20:37.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y7Lgd6GITUmWZ36WcarCeAkUUOo=', 'Content-Length': '0'} 2023-10-12T01:20:37.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e632159b2b8c4e0f-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.862 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.863 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.863 DEBUG:boto:path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.864 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.864 DEBUG:boto:Method: GET 2023-10-12T01:20:37.865 DEBUG:boto:Path: /wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.865 DEBUG:boto:Data: 2023-10-12T01:20:37.866 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.866 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.867 DEBUG:boto:Params: {} 2023-10-12T01:20:37.867 DEBUG:boto:Token: None 2023-10-12T01:20:37.868 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca10d0be03b0f0ad-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.872 DEBUG:boto:b'wafmoa-61obj381000falseobj382023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.873 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.874 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.874 DEBUG:boto:path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.875 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.875 DEBUG:boto:Method: GET 2023-10-12T01:20:37.875 DEBUG:boto:Path: /wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.876 DEBUG:boto:Data: 2023-10-12T01:20:37.876 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.877 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.878 DEBUG:boto:Params: {} 2023-10-12T01:20:37.878 DEBUG:boto:Token: None 2023-10-12T01:20:37.879 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.879 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004232f94d384fc1c1-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.885 DEBUG:boto:b'wafmoa-61obj381000falseobj382023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.885 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.886 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.886 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.887 DEBUG:boto:Path: /wafmoa-61/obj38 2023-10-12T01:20:37.887 DEBUG:boto:Data: 2023-10-12T01:20:37.888 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.889 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.889 DEBUG:boto:Params: {} 2023-10-12T01:20:37.890 DEBUG:boto:Token: None 2023-10-12T01:20:37.890 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38 2023-10-12T01:20:37.891 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zvkyt2eVgPhAj/zHa0jzcqTzJ2I= 2023-10-12T01:20:37.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zvkyt2eVgPhAj/zHa0jzcqTzJ2I=', 'Content-Length': '0'} 2023-10-12T01:20:37.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d618e64f21106bee-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.894 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.894 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.895 DEBUG:boto:Method: HEAD 2023-10-12T01:20:37.895 DEBUG:boto:Path: /wafmoa-61/obj38 2023-10-12T01:20:37.896 DEBUG:boto:Data: 2023-10-12T01:20:37.896 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.897 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.897 DEBUG:boto:Params: {} 2023-10-12T01:20:37.897 DEBUG:boto:Token: None 2023-10-12T01:20:37.898 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38 2023-10-12T01:20:37.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zvkyt2eVgPhAj/zHa0jzcqTzJ2I= 2023-10-12T01:20:37.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zvkyt2eVgPhAj/zHa0jzcqTzJ2I=', 'Content-Length': '0'} 2023-10-12T01:20:37.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a77e8576290495e-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-12T01:20:37.902 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.902 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.903 DEBUG:boto:Method: GET 2023-10-12T01:20:37.903 DEBUG:boto:Path: /wafmoa-61/obj38 2023-10-12T01:20:37.904 DEBUG:boto:Data: 2023-10-12T01:20:37.904 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.905 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.905 DEBUG:boto:Params: {} 2023-10-12T01:20:37.906 DEBUG:boto:Token: None 2023-10-12T01:20:37.907 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38 2023-10-12T01:20:37.907 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nOXisnnZMGOoaW64ITOKsXxigEE= 2023-10-12T01:20:37.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nOXisnnZMGOoaW64ITOKsXxigEE=', 'Content-Length': '0'} 2023-10-12T01:20:37.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00abd97a222ba2d-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.910 DEBUG:boto:path=/wafmoa-61/obj38 2023-10-12T01:20:37.911 DEBUG:boto:auth_path=/wafmoa-61/obj38 2023-10-12T01:20:37.911 DEBUG:boto:Method: GET 2023-10-12T01:20:37.912 DEBUG:boto:Path: /wafmoa-61/obj38 2023-10-12T01:20:37.912 DEBUG:boto:Data: 2023-10-12T01:20:37.913 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.914 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.914 DEBUG:boto:Params: {} 2023-10-12T01:20:37.915 DEBUG:boto:Token: None 2023-10-12T01:20:37.915 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj38 2023-10-12T01:20:37.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nOXisnnZMGOoaW64ITOKsXxigEE= 2023-10-12T01:20:37.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nOXisnnZMGOoaW64ITOKsXxigEE=', 'Content-Length': '0'} 2023-10-12T01:20:37.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a717d51f7ef57230-00652749e5-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.919 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.919 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.920 DEBUG:boto:path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.920 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.921 DEBUG:boto:Method: GET 2023-10-12T01:20:37.921 DEBUG:boto:Path: /wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.922 DEBUG:boto:Data: 2023-10-12T01:20:37.922 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.923 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.923 DEBUG:boto:Params: {} 2023-10-12T01:20:37.924 DEBUG:boto:Token: None 2023-10-12T01:20:37.924 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000451f62020a3aa813-00652749e5-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.929 DEBUG:boto:b'wafmoa-61obj381000falseobj382023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.929 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:37.930 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:37.930 DEBUG:boto:path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.931 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.931 DEBUG:boto:Method: GET 2023-10-12T01:20:37.932 DEBUG:boto:Path: /wafmoa-61/?prefix=obj38 2023-10-12T01:20:37.932 DEBUG:boto:Data: 2023-10-12T01:20:37.933 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.934 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.934 DEBUG:boto:Params: {} 2023-10-12T01:20:37.935 DEBUG:boto:Token: None 2023-10-12T01:20:37.935 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/ 2023-10-12T01:20:37.936 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc= 2023-10-12T01:20:37.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IUkr/0ufL8flFb3Pi8onnNRe+qc=', 'Content-Length': '0'} 2023-10-12T01:20:37.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1db86319f22d205-00652749e5-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.982 DEBUG:boto:b'wafmoa-61obj381000falseobj382023-10-12T01:16:55.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:37.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:20:37.985 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:37.986 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:37.986 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.987 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.987 DEBUG:boto:Method: GET 2023-10-12T01:20:37.988 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.988 DEBUG:boto:Data: 2023-10-12T01:20:37.989 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:37.989 DEBUG:boto:Port: 8001 2023-10-12T01:20:37.990 DEBUG:boto:Params: {} 2023-10-12T01:20:37.990 DEBUG:boto:Token: None 2023-10-12T01:20:37.991 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:37 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.991 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LsmZeHz0v15qzm7ezxulvsn+gRQ= 2023-10-12T01:20:37.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:37 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LsmZeHz0v15qzm7ezxulvsn+gRQ=', 'Content-Length': '0'} 2023-10-12T01:20:37.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081885f763a4a9ff-00652749e5-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:37.995 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:37.995 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:37.996 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.996 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.997 DEBUG:boto:Method: GET 2023-10-12T01:20:37.997 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:37.997 DEBUG:boto:Data: 2023-10-12T01:20:37.998 DEBUG:boto:Headers: {} 2023-10-12T01:20:37.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:37.999 DEBUG:boto:Port: 8000 2023-10-12T01:20:37.999 DEBUG:boto:Params: {} 2023-10-12T01:20:37.999 DEBUG:boto:Token: None 2023-10-12T01:20:38.000 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.000 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao= 2023-10-12T01:20:38.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao=', 'Content-Length': '0'} 2023-10-12T01:20:38.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6628fa12811a33-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.003 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.004 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.004 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.005 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.005 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.005 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.006 DEBUG:boto:Data: 2023-10-12T01:20:38.006 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.007 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.008 DEBUG:boto:Params: {} 2023-10-12T01:20:38.008 DEBUG:boto:Token: None 2023-10-12T01:20:38.008 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ekUazAb/uCtX7kFPKO5n6sB5aFE= 2023-10-12T01:20:38.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ekUazAb/uCtX7kFPKO5n6sB5aFE=', 'Content-Length': '0'} 2023-10-12T01:20:38.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef82cb732a1a49e-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.012 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.012 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.013 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.013 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.013 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.014 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.014 DEBUG:boto:Data: 2023-10-12T01:20:38.015 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.015 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.016 DEBUG:boto:Params: {} 2023-10-12T01:20:38.016 DEBUG:boto:Token: None 2023-10-12T01:20:38.017 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.017 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ekUazAb/uCtX7kFPKO5n6sB5aFE= 2023-10-12T01:20:38.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ekUazAb/uCtX7kFPKO5n6sB5aFE=', 'Content-Length': '0'} 2023-10-12T01:20:38.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89a421bcc6e3aa3-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:20:38.021 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.021 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.022 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.022 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.023 DEBUG:boto:Method: GET 2023-10-12T01:20:38.023 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.023 DEBUG:boto:Data: 2023-10-12T01:20:38.024 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.024 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.025 DEBUG:boto:Params: {} 2023-10-12T01:20:38.025 DEBUG:boto:Token: None 2023-10-12T01:20:38.026 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.026 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao= 2023-10-12T01:20:38.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao=', 'Content-Length': '0'} 2023-10-12T01:20:38.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b502c56fbdd8293c-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.070 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.071 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.071 DEBUG:boto:path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.071 DEBUG:boto:auth_path=/wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.072 DEBUG:boto:Method: GET 2023-10-12T01:20:38.072 DEBUG:boto:Path: /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.073 DEBUG:boto:Data: 2023-10-12T01:20:38.073 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.074 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.074 DEBUG:boto:Params: {} 2023-10-12T01:20:38.075 DEBUG:boto:Token: None 2023-10-12T01:20:38.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44?versionId=null 2023-10-12T01:20:38.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao= 2023-10-12T01:20:38.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KoYWozd2rk8rvrfhrnQYROTdVao=', 'Content-Length': '0'} 2023-10-12T01:20:38.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36d456cc81f9dbe-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.079 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.079 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.080 DEBUG:boto:path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.080 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.081 DEBUG:boto:Method: GET 2023-10-12T01:20:38.081 DEBUG:boto:Path: /wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.081 DEBUG:boto:Data: 2023-10-12T01:20:38.082 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.083 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.083 DEBUG:boto:Params: {} 2023-10-12T01:20:38.084 DEBUG:boto:Token: None 2023-10-12T01:20:38.084 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001932402c7f5daddb-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.088 DEBUG:boto:b'wafmoa-61obj441000falseobj442023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.089 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.090 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.090 DEBUG:boto:path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.090 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.091 DEBUG:boto:Method: GET 2023-10-12T01:20:38.091 DEBUG:boto:Path: /wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.092 DEBUG:boto:Data: 2023-10-12T01:20:38.092 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.093 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.093 DEBUG:boto:Params: {} 2023-10-12T01:20:38.094 DEBUG:boto:Token: None 2023-10-12T01:20:38.094 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.095 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960ba405d44ee12f-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.142 DEBUG:boto:b'wafmoa-61obj441000falseobj442023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.143 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.144 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.144 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.145 DEBUG:boto:Path: /wafmoa-61/obj44 2023-10-12T01:20:38.145 DEBUG:boto:Data: 2023-10-12T01:20:38.146 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.146 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.147 DEBUG:boto:Params: {} 2023-10-12T01:20:38.147 DEBUG:boto:Token: None 2023-10-12T01:20:38.148 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44 2023-10-12T01:20:38.148 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LCd4RC9miwYsJyGDfQRVt2c6QOQ= 2023-10-12T01:20:38.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LCd4RC9miwYsJyGDfQRVt2c6QOQ=', 'Content-Length': '0'} 2023-10-12T01:20:38.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f07f02868328376-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.152 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.152 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.153 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.153 DEBUG:boto:Path: /wafmoa-61/obj44 2023-10-12T01:20:38.154 DEBUG:boto:Data: 2023-10-12T01:20:38.154 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.155 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.155 DEBUG:boto:Params: {} 2023-10-12T01:20:38.156 DEBUG:boto:Token: None 2023-10-12T01:20:38.156 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44 2023-10-12T01:20:38.157 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LCd4RC9miwYsJyGDfQRVt2c6QOQ= 2023-10-12T01:20:38.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LCd4RC9miwYsJyGDfQRVt2c6QOQ=', 'Content-Length': '0'} 2023-10-12T01:20:38.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3fd11ce674aa0f-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:20:38.161 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.162 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.162 DEBUG:boto:Method: GET 2023-10-12T01:20:38.163 DEBUG:boto:Path: /wafmoa-61/obj44 2023-10-12T01:20:38.163 DEBUG:boto:Data: 2023-10-12T01:20:38.164 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.164 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.165 DEBUG:boto:Params: {} 2023-10-12T01:20:38.165 DEBUG:boto:Token: None 2023-10-12T01:20:38.166 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44 2023-10-12T01:20:38.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Afia4FKrFZixx8vql7znh3qU7Y8= 2023-10-12T01:20:38.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Afia4FKrFZixx8vql7znh3qU7Y8=', 'Content-Length': '0'} 2023-10-12T01:20:38.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3fa16f64031528-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.170 DEBUG:boto:path=/wafmoa-61/obj44 2023-10-12T01:20:38.170 DEBUG:boto:auth_path=/wafmoa-61/obj44 2023-10-12T01:20:38.171 DEBUG:boto:Method: GET 2023-10-12T01:20:38.171 DEBUG:boto:Path: /wafmoa-61/obj44 2023-10-12T01:20:38.172 DEBUG:boto:Data: 2023-10-12T01:20:38.172 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.173 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.173 DEBUG:boto:Params: {} 2023-10-12T01:20:38.174 DEBUG:boto:Token: None 2023-10-12T01:20:38.174 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj44 2023-10-12T01:20:38.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Afia4FKrFZixx8vql7znh3qU7Y8= 2023-10-12T01:20:38.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Afia4FKrFZixx8vql7znh3qU7Y8=', 'Content-Length': '0'} 2023-10-12T01:20:38.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59ec8fb60c1e4bf-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.178 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.178 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.179 DEBUG:boto:path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.179 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.180 DEBUG:boto:Method: GET 2023-10-12T01:20:38.180 DEBUG:boto:Path: /wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.180 DEBUG:boto:Data: 2023-10-12T01:20:38.181 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.182 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.182 DEBUG:boto:Params: {} 2023-10-12T01:20:38.183 DEBUG:boto:Token: None 2023-10-12T01:20:38.183 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080604a87e1fb30e3-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.190 DEBUG:boto:b'wafmoa-61obj441000falseobj442023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.190 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.191 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.191 DEBUG:boto:path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.192 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.192 DEBUG:boto:Method: GET 2023-10-12T01:20:38.193 DEBUG:boto:Path: /wafmoa-61/?prefix=obj44 2023-10-12T01:20:38.193 DEBUG:boto:Data: 2023-10-12T01:20:38.193 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.194 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.195 DEBUG:boto:Params: {} 2023-10-12T01:20:38.195 DEBUG:boto:Token: None 2023-10-12T01:20:38.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.196 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3be0e901667a15d-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.202 DEBUG:boto:b'wafmoa-61obj441000falseobj442023-10-12T01:16:55.318Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:20:38.203 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.203 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.204 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.204 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.205 DEBUG:boto:Method: GET 2023-10-12T01:20:38.205 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.206 DEBUG:boto:Data: 2023-10-12T01:20:38.206 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.207 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.207 DEBUG:boto:Params: {} 2023-10-12T01:20:38.208 DEBUG:boto:Token: None 2023-10-12T01:20:38.208 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I= 2023-10-12T01:20:38.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I=', 'Content-Length': '0'} 2023-10-12T01:20:38.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86e09e7fee4559c-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.254 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.255 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.256 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.256 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.256 DEBUG:boto:Method: GET 2023-10-12T01:20:38.257 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.257 DEBUG:boto:Data: 2023-10-12T01:20:38.258 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.259 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.259 DEBUG:boto:Params: {} 2023-10-12T01:20:38.260 DEBUG:boto:Token: None 2023-10-12T01:20:38.260 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.261 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I= 2023-10-12T01:20:38.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I=', 'Content-Length': '0'} 2023-10-12T01:20:38.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a849978e507e46-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.264 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.264 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.265 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.265 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.266 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.266 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.266 DEBUG:boto:Data: 2023-10-12T01:20:38.267 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.268 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.268 DEBUG:boto:Params: {} 2023-10-12T01:20:38.269 DEBUG:boto:Token: None 2023-10-12T01:20:38.269 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.270 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:40oRMVUefZrAdkWtuYzauivq85s= 2023-10-12T01:20:38.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:40oRMVUefZrAdkWtuYzauivq85s=', 'Content-Length': '0'} 2023-10-12T01:20:38.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096af8bbbe341fe73-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.273 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.273 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.274 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.274 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.274 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.275 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.275 DEBUG:boto:Data: 2023-10-12T01:20:38.276 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.276 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.277 DEBUG:boto:Params: {} 2023-10-12T01:20:38.277 DEBUG:boto:Token: None 2023-10-12T01:20:38.278 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.278 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:40oRMVUefZrAdkWtuYzauivq85s= 2023-10-12T01:20:38.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:40oRMVUefZrAdkWtuYzauivq85s=', 'Content-Length': '0'} 2023-10-12T01:20:38.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004140be407e1349ac-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:20:38.281 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.282 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.282 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.283 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.283 DEBUG:boto:Method: GET 2023-10-12T01:20:38.284 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.284 DEBUG:boto:Data: 2023-10-12T01:20:38.284 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.285 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.285 DEBUG:boto:Params: {} 2023-10-12T01:20:38.286 DEBUG:boto:Token: None 2023-10-12T01:20:38.286 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I= 2023-10-12T01:20:38.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I=', 'Content-Length': '0'} 2023-10-12T01:20:38.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7a56143b696b338-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.290 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.291 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.291 DEBUG:boto:path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.292 DEBUG:boto:auth_path=/wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.292 DEBUG:boto:Method: GET 2023-10-12T01:20:38.292 DEBUG:boto:Path: /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.293 DEBUG:boto:Data: 2023-10-12T01:20:38.293 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.294 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.295 DEBUG:boto:Params: {} 2023-10-12T01:20:38.295 DEBUG:boto:Token: None 2023-10-12T01:20:38.296 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46?versionId=null 2023-10-12T01:20:38.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I= 2023-10-12T01:20:38.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OdAAl33PbN7ZCRZahvcGRXv0z3I=', 'Content-Length': '0'} 2023-10-12T01:20:38.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762db1220adb5b4a-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.342 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.343 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.343 DEBUG:boto:path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.344 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.344 DEBUG:boto:Method: GET 2023-10-12T01:20:38.345 DEBUG:boto:Path: /wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.345 DEBUG:boto:Data: 2023-10-12T01:20:38.346 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.347 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.347 DEBUG:boto:Params: {} 2023-10-12T01:20:38.348 DEBUG:boto:Token: None 2023-10-12T01:20:38.350 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7f43421034ea11-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.355 DEBUG:boto:b'wafmoa-61obj461000falseobj462023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.355 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.356 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.356 DEBUG:boto:path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.356 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.357 DEBUG:boto:Method: GET 2023-10-12T01:20:38.357 DEBUG:boto:Path: /wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.358 DEBUG:boto:Data: 2023-10-12T01:20:38.358 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.359 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.360 DEBUG:boto:Params: {} 2023-10-12T01:20:38.360 DEBUG:boto:Token: None 2023-10-12T01:20:38.360 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.361 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024598935531870e6-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.366 DEBUG:boto:b'wafmoa-61obj461000falseobj462023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.367 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.367 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.368 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.368 DEBUG:boto:Path: /wafmoa-61/obj46 2023-10-12T01:20:38.369 DEBUG:boto:Data: 2023-10-12T01:20:38.369 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.370 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.370 DEBUG:boto:Params: {} 2023-10-12T01:20:38.371 DEBUG:boto:Token: None 2023-10-12T01:20:38.371 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46 2023-10-12T01:20:38.371 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q4xqvU71dOS2vdPwZrN3NkIJx0Q= 2023-10-12T01:20:38.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q4xqvU71dOS2vdPwZrN3NkIJx0Q=', 'Content-Length': '0'} 2023-10-12T01:20:38.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15ecd054853d737-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.374 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.375 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.375 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.376 DEBUG:boto:Path: /wafmoa-61/obj46 2023-10-12T01:20:38.376 DEBUG:boto:Data: 2023-10-12T01:20:38.377 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.378 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.378 DEBUG:boto:Params: {} 2023-10-12T01:20:38.379 DEBUG:boto:Token: None 2023-10-12T01:20:38.379 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46 2023-10-12T01:20:38.380 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q4xqvU71dOS2vdPwZrN3NkIJx0Q= 2023-10-12T01:20:38.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q4xqvU71dOS2vdPwZrN3NkIJx0Q=', 'Content-Length': '0'} 2023-10-12T01:20:38.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e190e0d85705f84-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-12T01:20:38.383 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.384 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.384 DEBUG:boto:Method: GET 2023-10-12T01:20:38.384 DEBUG:boto:Path: /wafmoa-61/obj46 2023-10-12T01:20:38.385 DEBUG:boto:Data: 2023-10-12T01:20:38.385 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.386 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.387 DEBUG:boto:Params: {} 2023-10-12T01:20:38.387 DEBUG:boto:Token: None 2023-10-12T01:20:38.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46 2023-10-12T01:20:38.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K0XVa09axQYfOlCA7Jn6MgZzyro= 2023-10-12T01:20:38.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K0XVa09axQYfOlCA7Jn6MgZzyro=', 'Content-Length': '0'} 2023-10-12T01:20:38.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e097d3829fd1e5-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.393 DEBUG:boto:path=/wafmoa-61/obj46 2023-10-12T01:20:38.393 DEBUG:boto:auth_path=/wafmoa-61/obj46 2023-10-12T01:20:38.394 DEBUG:boto:Method: GET 2023-10-12T01:20:38.395 DEBUG:boto:Path: /wafmoa-61/obj46 2023-10-12T01:20:38.395 DEBUG:boto:Data: 2023-10-12T01:20:38.395 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.396 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.397 DEBUG:boto:Params: {} 2023-10-12T01:20:38.397 DEBUG:boto:Token: None 2023-10-12T01:20:38.398 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj46 2023-10-12T01:20:38.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K0XVa09axQYfOlCA7Jn6MgZzyro= 2023-10-12T01:20:38.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K0XVa09axQYfOlCA7Jn6MgZzyro=', 'Content-Length': '0'} 2023-10-12T01:20:38.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566b36ecd81a86bf-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.401 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.402 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.402 DEBUG:boto:path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.402 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.403 DEBUG:boto:Method: GET 2023-10-12T01:20:38.403 DEBUG:boto:Path: /wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.404 DEBUG:boto:Data: 2023-10-12T01:20:38.404 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.405 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.406 DEBUG:boto:Params: {} 2023-10-12T01:20:38.406 DEBUG:boto:Token: None 2023-10-12T01:20:38.406 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.407 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030f7f22b8898f899-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.410 DEBUG:boto:b'wafmoa-61obj461000falseobj462023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.411 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.411 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.412 DEBUG:boto:path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.412 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.413 DEBUG:boto:Method: GET 2023-10-12T01:20:38.413 DEBUG:boto:Path: /wafmoa-61/?prefix=obj46 2023-10-12T01:20:38.413 DEBUG:boto:Data: 2023-10-12T01:20:38.414 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.415 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.415 DEBUG:boto:Params: {} 2023-10-12T01:20:38.415 DEBUG:boto:Token: None 2023-10-12T01:20:38.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ecdb5314b2f681d-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.421 DEBUG:boto:b'wafmoa-61obj461000falseobj462023-10-12T01:16:56.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:20:38.422 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.423 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.423 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.424 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.424 DEBUG:boto:Method: GET 2023-10-12T01:20:38.424 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.425 DEBUG:boto:Data: 2023-10-12T01:20:38.425 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.426 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.427 DEBUG:boto:Params: {} 2023-10-12T01:20:38.427 DEBUG:boto:Token: None 2023-10-12T01:20:38.427 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.428 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4= 2023-10-12T01:20:38.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4=', 'Content-Length': '0'} 2023-10-12T01:20:38.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1a68ff2d5dda54-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.431 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.432 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.433 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.433 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.433 DEBUG:boto:Method: GET 2023-10-12T01:20:38.434 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.434 DEBUG:boto:Data: 2023-10-12T01:20:38.435 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.436 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.436 DEBUG:boto:Params: {} 2023-10-12T01:20:38.437 DEBUG:boto:Token: None 2023-10-12T01:20:38.437 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.438 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4= 2023-10-12T01:20:38.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4=', 'Content-Length': '0'} 2023-10-12T01:20:38.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ebfca6a680659a8-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.441 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.442 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.442 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.443 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.443 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.443 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.444 DEBUG:boto:Data: 2023-10-12T01:20:38.444 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.445 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.445 DEBUG:boto:Params: {} 2023-10-12T01:20:38.446 DEBUG:boto:Token: None 2023-10-12T01:20:38.446 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.447 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Py8RJGc4gTt7nxsLKLYEfQDfFVo= 2023-10-12T01:20:38.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Py8RJGc4gTt7nxsLKLYEfQDfFVo=', 'Content-Length': '0'} 2023-10-12T01:20:38.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085206a1511a9bd2a-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.450 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.450 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.451 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.451 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.451 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.452 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.452 DEBUG:boto:Data: 2023-10-12T01:20:38.453 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.453 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.454 DEBUG:boto:Params: {} 2023-10-12T01:20:38.454 DEBUG:boto:Token: None 2023-10-12T01:20:38.455 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.455 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Py8RJGc4gTt7nxsLKLYEfQDfFVo= 2023-10-12T01:20:38.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Py8RJGc4gTt7nxsLKLYEfQDfFVo=', 'Content-Length': '0'} 2023-10-12T01:20:38.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff0c1abeeb675e9-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:20:38.459 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.459 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.460 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.460 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.461 DEBUG:boto:Method: GET 2023-10-12T01:20:38.461 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.461 DEBUG:boto:Data: 2023-10-12T01:20:38.462 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.463 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.463 DEBUG:boto:Params: {} 2023-10-12T01:20:38.464 DEBUG:boto:Token: None 2023-10-12T01:20:38.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4= 2023-10-12T01:20:38.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4=', 'Content-Length': '0'} 2023-10-12T01:20:38.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd95bc430cd97bc-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.510 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.511 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.511 DEBUG:boto:path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.512 DEBUG:boto:auth_path=/wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.512 DEBUG:boto:Method: GET 2023-10-12T01:20:38.512 DEBUG:boto:Path: /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.513 DEBUG:boto:Data: 2023-10-12T01:20:38.513 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.514 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.514 DEBUG:boto:Params: {} 2023-10-12T01:20:38.515 DEBUG:boto:Token: None 2023-10-12T01:20:38.515 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51?versionId=null 2023-10-12T01:20:38.516 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4= 2023-10-12T01:20:38.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LcMxYu3xbit3hFSSAUGGeX+WbK4=', 'Content-Length': '0'} 2023-10-12T01:20:38.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eb9bac8cc4529db-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.519 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.519 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.520 DEBUG:boto:path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.520 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.520 DEBUG:boto:Method: GET 2023-10-12T01:20:38.521 DEBUG:boto:Path: /wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.521 DEBUG:boto:Data: 2023-10-12T01:20:38.522 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.523 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.523 DEBUG:boto:Params: {} 2023-10-12T01:20:38.524 DEBUG:boto:Token: None 2023-10-12T01:20:38.524 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.524 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c61cfeeb9ad78e46-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.528 DEBUG:boto:b'wafmoa-61obj511000falseobj512023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.528 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.529 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.529 DEBUG:boto:path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.530 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.530 DEBUG:boto:Method: GET 2023-10-12T01:20:38.531 DEBUG:boto:Path: /wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.531 DEBUG:boto:Data: 2023-10-12T01:20:38.531 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.532 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.533 DEBUG:boto:Params: {} 2023-10-12T01:20:38.533 DEBUG:boto:Token: None 2023-10-12T01:20:38.534 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.534 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa4b8ed563f14615-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.582 DEBUG:boto:b'wafmoa-61obj511000falseobj512023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.583 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.583 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.584 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.584 DEBUG:boto:Path: /wafmoa-61/obj51 2023-10-12T01:20:38.585 DEBUG:boto:Data: 2023-10-12T01:20:38.585 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.586 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.586 DEBUG:boto:Params: {} 2023-10-12T01:20:38.587 DEBUG:boto:Token: None 2023-10-12T01:20:38.587 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51 2023-10-12T01:20:38.588 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uLUTkhyZ8fYnq22mq7s2s4Ahk8Q= 2023-10-12T01:20:38.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uLUTkhyZ8fYnq22mq7s2s4Ahk8Q=', 'Content-Length': '0'} 2023-10-12T01:20:38.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003801f8d5b2e82c9b-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.591 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.591 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.592 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.592 DEBUG:boto:Path: /wafmoa-61/obj51 2023-10-12T01:20:38.592 DEBUG:boto:Data: 2023-10-12T01:20:38.593 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.593 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.594 DEBUG:boto:Params: {} 2023-10-12T01:20:38.594 DEBUG:boto:Token: None 2023-10-12T01:20:38.595 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51 2023-10-12T01:20:38.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uLUTkhyZ8fYnq22mq7s2s4Ahk8Q= 2023-10-12T01:20:38.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uLUTkhyZ8fYnq22mq7s2s4Ahk8Q=', 'Content-Length': '0'} 2023-10-12T01:20:38.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000117bb92f5d8729d-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-12T01:20:38.600 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.601 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.601 DEBUG:boto:Method: GET 2023-10-12T01:20:38.602 DEBUG:boto:Path: /wafmoa-61/obj51 2023-10-12T01:20:38.602 DEBUG:boto:Data: 2023-10-12T01:20:38.603 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.603 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.604 DEBUG:boto:Params: {} 2023-10-12T01:20:38.604 DEBUG:boto:Token: None 2023-10-12T01:20:38.605 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51 2023-10-12T01:20:38.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5+f43SI9uTEYnQwWuSH+WuJop98= 2023-10-12T01:20:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5+f43SI9uTEYnQwWuSH+WuJop98=', 'Content-Length': '0'} 2023-10-12T01:20:38.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aceeec023fde24b-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.608 DEBUG:boto:path=/wafmoa-61/obj51 2023-10-12T01:20:38.609 DEBUG:boto:auth_path=/wafmoa-61/obj51 2023-10-12T01:20:38.609 DEBUG:boto:Method: GET 2023-10-12T01:20:38.610 DEBUG:boto:Path: /wafmoa-61/obj51 2023-10-12T01:20:38.610 DEBUG:boto:Data: 2023-10-12T01:20:38.611 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.611 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.612 DEBUG:boto:Params: {} 2023-10-12T01:20:38.612 DEBUG:boto:Token: None 2023-10-12T01:20:38.613 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj51 2023-10-12T01:20:38.613 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5+f43SI9uTEYnQwWuSH+WuJop98= 2023-10-12T01:20:38.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5+f43SI9uTEYnQwWuSH+WuJop98=', 'Content-Length': '0'} 2023-10-12T01:20:38.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d684b9a3ed8051a-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.616 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.617 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.617 DEBUG:boto:path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.618 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.618 DEBUG:boto:Method: GET 2023-10-12T01:20:38.618 DEBUG:boto:Path: /wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.619 DEBUG:boto:Data: 2023-10-12T01:20:38.619 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.620 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.620 DEBUG:boto:Params: {} 2023-10-12T01:20:38.621 DEBUG:boto:Token: None 2023-10-12T01:20:38.621 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.622 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e902e593d66aa60-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.626 DEBUG:boto:b'wafmoa-61obj511000falseobj512023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.626 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.627 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.627 DEBUG:boto:path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.628 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.628 DEBUG:boto:Method: GET 2023-10-12T01:20:38.629 DEBUG:boto:Path: /wafmoa-61/?prefix=obj51 2023-10-12T01:20:38.629 DEBUG:boto:Data: 2023-10-12T01:20:38.630 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.630 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.631 DEBUG:boto:Params: {} 2023-10-12T01:20:38.631 DEBUG:boto:Token: None 2023-10-12T01:20:38.632 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b516a0f3ecea2978-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.638 DEBUG:boto:b'wafmoa-61obj511000falseobj512023-10-12T01:16:55.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:20:38.639 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.639 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.640 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.640 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.640 DEBUG:boto:Method: GET 2023-10-12T01:20:38.641 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.641 DEBUG:boto:Data: 2023-10-12T01:20:38.642 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.643 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.643 DEBUG:boto:Params: {} 2023-10-12T01:20:38.644 DEBUG:boto:Token: None 2023-10-12T01:20:38.644 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.644 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk= 2023-10-12T01:20:38.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk=', 'Content-Length': '0'} 2023-10-12T01:20:38.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8c303151321373-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.648 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.648 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.648 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.649 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.649 DEBUG:boto:Method: GET 2023-10-12T01:20:38.650 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.650 DEBUG:boto:Data: 2023-10-12T01:20:38.651 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.651 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.652 DEBUG:boto:Params: {} 2023-10-12T01:20:38.652 DEBUG:boto:Token: None 2023-10-12T01:20:38.653 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.653 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk= 2023-10-12T01:20:38.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk=', 'Content-Length': '0'} 2023-10-12T01:20:38.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c78f175dea7876-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.698 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.699 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.699 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.699 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.700 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.700 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.701 DEBUG:boto:Data: 2023-10-12T01:20:38.701 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.702 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.702 DEBUG:boto:Params: {} 2023-10-12T01:20:38.703 DEBUG:boto:Token: None 2023-10-12T01:20:38.703 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.704 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y++STVBUuhFcc7/hAgvl9fZ6NSo= 2023-10-12T01:20:38.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y++STVBUuhFcc7/hAgvl9fZ6NSo=', 'Content-Length': '0'} 2023-10-12T01:20:38.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5efe12ad7b9982-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.707 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.707 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.708 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.708 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.708 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.709 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.709 DEBUG:boto:Data: 2023-10-12T01:20:38.710 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.710 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.711 DEBUG:boto:Params: {} 2023-10-12T01:20:38.711 DEBUG:boto:Token: None 2023-10-12T01:20:38.712 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.712 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Y++STVBUuhFcc7/hAgvl9fZ6NSo= 2023-10-12T01:20:38.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Y++STVBUuhFcc7/hAgvl9fZ6NSo=', 'Content-Length': '0'} 2023-10-12T01:20:38.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d86a9187174ca84-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:20:38.716 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.717 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.717 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.717 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.718 DEBUG:boto:Method: GET 2023-10-12T01:20:38.718 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.719 DEBUG:boto:Data: 2023-10-12T01:20:38.719 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.720 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.720 DEBUG:boto:Params: {} 2023-10-12T01:20:38.721 DEBUG:boto:Token: None 2023-10-12T01:20:38.721 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.722 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk= 2023-10-12T01:20:38.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk=', 'Content-Length': '0'} 2023-10-12T01:20:38.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a2d74f6941ae70-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.725 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.725 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.726 DEBUG:boto:path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.726 DEBUG:boto:auth_path=/wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.727 DEBUG:boto:Method: GET 2023-10-12T01:20:38.727 DEBUG:boto:Path: /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.727 DEBUG:boto:Data: 2023-10-12T01:20:38.728 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.729 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.729 DEBUG:boto:Params: {} 2023-10-12T01:20:38.730 DEBUG:boto:Token: None 2023-10-12T01:20:38.730 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55?versionId=null 2023-10-12T01:20:38.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk= 2023-10-12T01:20:38.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4TtUE8Zr24anWEyTN169bdKbtWk=', 'Content-Length': '0'} 2023-10-12T01:20:38.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750226aaec6a6eb1-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.734 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.734 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.735 DEBUG:boto:path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.735 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.736 DEBUG:boto:Method: GET 2023-10-12T01:20:38.736 DEBUG:boto:Path: /wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.737 DEBUG:boto:Data: 2023-10-12T01:20:38.737 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.738 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.738 DEBUG:boto:Params: {} 2023-10-12T01:20:38.739 DEBUG:boto:Token: None 2023-10-12T01:20:38.739 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.740 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe43761e0518777-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.743 DEBUG:boto:b'wafmoa-61obj551000falseobj552023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.744 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.744 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.745 DEBUG:boto:path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.745 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.745 DEBUG:boto:Method: GET 2023-10-12T01:20:38.746 DEBUG:boto:Path: /wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.746 DEBUG:boto:Data: 2023-10-12T01:20:38.747 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.747 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.748 DEBUG:boto:Params: {} 2023-10-12T01:20:38.748 DEBUG:boto:Token: None 2023-10-12T01:20:38.749 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.749 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e27ed3125ed5294-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.754 DEBUG:boto:b'wafmoa-61obj551000falseobj552023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.755 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.755 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.756 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.756 DEBUG:boto:Path: /wafmoa-61/obj55 2023-10-12T01:20:38.757 DEBUG:boto:Data: 2023-10-12T01:20:38.757 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.758 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.758 DEBUG:boto:Params: {} 2023-10-12T01:20:38.758 DEBUG:boto:Token: None 2023-10-12T01:20:38.759 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55 2023-10-12T01:20:38.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X0ZSGPCsu/kPMJVCVUD5Aobgcyc= 2023-10-12T01:20:38.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X0ZSGPCsu/kPMJVCVUD5Aobgcyc=', 'Content-Length': '0'} 2023-10-12T01:20:38.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006345c27364acbc4f-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.762 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.763 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.763 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.764 DEBUG:boto:Path: /wafmoa-61/obj55 2023-10-12T01:20:38.764 DEBUG:boto:Data: 2023-10-12T01:20:38.765 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.765 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.766 DEBUG:boto:Params: {} 2023-10-12T01:20:38.766 DEBUG:boto:Token: None 2023-10-12T01:20:38.767 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55 2023-10-12T01:20:38.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:X0ZSGPCsu/kPMJVCVUD5Aobgcyc= 2023-10-12T01:20:38.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:X0ZSGPCsu/kPMJVCVUD5Aobgcyc=', 'Content-Length': '0'} 2023-10-12T01:20:38.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106703bbf4b09ad1-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:20:38.770 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.771 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.771 DEBUG:boto:Method: GET 2023-10-12T01:20:38.772 DEBUG:boto:Path: /wafmoa-61/obj55 2023-10-12T01:20:38.772 DEBUG:boto:Data: 2023-10-12T01:20:38.772 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.773 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.774 DEBUG:boto:Params: {} 2023-10-12T01:20:38.774 DEBUG:boto:Token: None 2023-10-12T01:20:38.774 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55 2023-10-12T01:20:38.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SKRs8MCc6csBT87clgHsGG3tDRI= 2023-10-12T01:20:38.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SKRs8MCc6csBT87clgHsGG3tDRI=', 'Content-Length': '0'} 2023-10-12T01:20:38.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185540ff28e7e2f7-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.818 DEBUG:boto:path=/wafmoa-61/obj55 2023-10-12T01:20:38.819 DEBUG:boto:auth_path=/wafmoa-61/obj55 2023-10-12T01:20:38.820 DEBUG:boto:Method: GET 2023-10-12T01:20:38.820 DEBUG:boto:Path: /wafmoa-61/obj55 2023-10-12T01:20:38.820 DEBUG:boto:Data: 2023-10-12T01:20:38.821 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.821 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.822 DEBUG:boto:Params: {} 2023-10-12T01:20:38.822 DEBUG:boto:Token: None 2023-10-12T01:20:38.823 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj55 2023-10-12T01:20:38.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SKRs8MCc6csBT87clgHsGG3tDRI= 2023-10-12T01:20:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SKRs8MCc6csBT87clgHsGG3tDRI=', 'Content-Length': '0'} 2023-10-12T01:20:38.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c9e83f86365f6d-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.827 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.827 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.828 DEBUG:boto:path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.828 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.828 DEBUG:boto:Method: GET 2023-10-12T01:20:38.829 DEBUG:boto:Path: /wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.829 DEBUG:boto:Data: 2023-10-12T01:20:38.830 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.831 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.831 DEBUG:boto:Params: {} 2023-10-12T01:20:38.831 DEBUG:boto:Token: None 2023-10-12T01:20:38.832 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033930ab9a6febe51-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.836 DEBUG:boto:b'wafmoa-61obj551000falseobj552023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.837 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.837 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.837 DEBUG:boto:path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.838 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.838 DEBUG:boto:Method: GET 2023-10-12T01:20:38.839 DEBUG:boto:Path: /wafmoa-61/?prefix=obj55 2023-10-12T01:20:38.839 DEBUG:boto:Data: 2023-10-12T01:20:38.839 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.840 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.841 DEBUG:boto:Params: {} 2023-10-12T01:20:38.841 DEBUG:boto:Token: None 2023-10-12T01:20:38.842 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005abe6c1eac6081a3-00652749e6-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.890 DEBUG:boto:b'wafmoa-61obj551000falseobj552023-10-12T01:16:55.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:38.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:20:38.891 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.892 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.892 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.893 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.893 DEBUG:boto:Method: GET 2023-10-12T01:20:38.893 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.894 DEBUG:boto:Data: 2023-10-12T01:20:38.894 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.895 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.895 DEBUG:boto:Params: {} 2023-10-12T01:20:38.896 DEBUG:boto:Token: None 2023-10-12T01:20:38.896 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.897 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo= 2023-10-12T01:20:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo=', 'Content-Length': '0'} 2023-10-12T01:20:38.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004550e655ef41e48e-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.900 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.901 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.901 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.902 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.902 DEBUG:boto:Method: GET 2023-10-12T01:20:38.903 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.903 DEBUG:boto:Data: 2023-10-12T01:20:38.903 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.904 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.905 DEBUG:boto:Params: {} 2023-10-12T01:20:38.905 DEBUG:boto:Token: None 2023-10-12T01:20:38.906 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo= 2023-10-12T01:20:38.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo=', 'Content-Length': '0'} 2023-10-12T01:20:38.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc37ec75c62e2e7c-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.909 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.910 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.910 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.911 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.911 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.911 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.912 DEBUG:boto:Data: 2023-10-12T01:20:38.912 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.913 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.913 DEBUG:boto:Params: {} 2023-10-12T01:20:38.914 DEBUG:boto:Token: None 2023-10-12T01:20:38.914 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACO1WVnFLZuvjLQVURifOiFt6nQ= 2023-10-12T01:20:38.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACO1WVnFLZuvjLQVURifOiFt6nQ=', 'Content-Length': '0'} 2023-10-12T01:20:38.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb5820744027f46-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.918 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.918 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.919 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.919 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.919 DEBUG:boto:Method: HEAD 2023-10-12T01:20:38.920 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.920 DEBUG:boto:Data: 2023-10-12T01:20:38.921 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.921 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.922 DEBUG:boto:Params: {} 2023-10-12T01:20:38.922 DEBUG:boto:Token: None 2023-10-12T01:20:38.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.923 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ACO1WVnFLZuvjLQVURifOiFt6nQ= 2023-10-12T01:20:38.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ACO1WVnFLZuvjLQVURifOiFt6nQ=', 'Content-Length': '0'} 2023-10-12T01:20:38.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041464e41d85f6cfb-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:20:38.927 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.927 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.928 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.928 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.929 DEBUG:boto:Method: GET 2023-10-12T01:20:38.929 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.929 DEBUG:boto:Data: 2023-10-12T01:20:38.930 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.931 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.931 DEBUG:boto:Params: {} 2023-10-12T01:20:38.932 DEBUG:boto:Token: None 2023-10-12T01:20:38.932 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.933 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo= 2023-10-12T01:20:38.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo=', 'Content-Length': '0'} 2023-10-12T01:20:38.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0332a259d5f27db-00652749e6-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.936 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:38.936 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:38.937 DEBUG:boto:path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.937 DEBUG:boto:auth_path=/wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.938 DEBUG:boto:Method: GET 2023-10-12T01:20:38.938 DEBUG:boto:Path: /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.939 DEBUG:boto:Data: 2023-10-12T01:20:38.939 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:38.940 DEBUG:boto:Port: 8000 2023-10-12T01:20:38.940 DEBUG:boto:Params: {} 2023-10-12T01:20:38.941 DEBUG:boto:Token: None 2023-10-12T01:20:38.941 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/obj57?versionId=null 2023-10-12T01:20:38.942 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo= 2023-10-12T01:20:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7UN7hbIZ2o4BV+zRpDkpUt2+AHo=', 'Content-Length': '0'} 2023-10-12T01:20:38.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d7bf911764da8e-00652749e6-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:38.945 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:38.946 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:38.946 DEBUG:boto:path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:38.946 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:38.947 DEBUG:boto:Method: GET 2023-10-12T01:20:38.947 DEBUG:boto:Path: /wafmoa-61/?prefix=obj57 2023-10-12T01:20:38.948 DEBUG:boto:Data: 2023-10-12T01:20:38.948 DEBUG:boto:Headers: {} 2023-10-12T01:20:38.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:38.949 DEBUG:boto:Port: 8001 2023-10-12T01:20:38.949 DEBUG:boto:Params: {} 2023-10-12T01:20:38.950 DEBUG:boto:Token: None 2023-10-12T01:20:38.950 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:38 GMT /wafmoa-61/ 2023-10-12T01:20:38.951 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws= 2023-10-12T01:20:38.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:38 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0WobY03wOj4Xx/4rT+ao8A9X9Ws=', 'Content-Length': '0'} 2023-10-12T01:20:38.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc892a50b062706-00652749e6-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.002 DEBUG:boto:b'wafmoa-61obj571000falseobj572023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.003 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.003 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.004 DEBUG:boto:path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.004 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.005 DEBUG:boto:Method: GET 2023-10-12T01:20:39.005 DEBUG:boto:Path: /wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.006 DEBUG:boto:Data: 2023-10-12T01:20:39.006 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.007 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.007 DEBUG:boto:Params: {} 2023-10-12T01:20:39.008 DEBUG:boto:Token: None 2023-10-12T01:20:39.008 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.009 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf3c5b8d2a530cc-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.014 DEBUG:boto:b'wafmoa-61obj571000falseobj572023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.015 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:39.015 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:39.016 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.016 DEBUG:boto:Path: /wafmoa-61/obj57 2023-10-12T01:20:39.017 DEBUG:boto:Data: 2023-10-12T01:20:39.017 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.018 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.018 DEBUG:boto:Params: {} 2023-10-12T01:20:39.019 DEBUG:boto:Token: None 2023-10-12T01:20:39.019 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj57 2023-10-12T01:20:39.019 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jDqOuycoNvD++d0G4JNYRzGjGgI= 2023-10-12T01:20:39.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jDqOuycoNvD++d0G4JNYRzGjGgI=', 'Content-Length': '0'} 2023-10-12T01:20:39.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcefe91b61a64a08-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.022 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:39.023 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:39.023 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.024 DEBUG:boto:Path: /wafmoa-61/obj57 2023-10-12T01:20:39.024 DEBUG:boto:Data: 2023-10-12T01:20:39.024 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.025 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.025 DEBUG:boto:Params: {} 2023-10-12T01:20:39.026 DEBUG:boto:Token: None 2023-10-12T01:20:39.026 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj57 2023-10-12T01:20:39.027 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jDqOuycoNvD++d0G4JNYRzGjGgI= 2023-10-12T01:20:39.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jDqOuycoNvD++d0G4JNYRzGjGgI=', 'Content-Length': '0'} 2023-10-12T01:20:39.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff3a46c09938ce3-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-12T01:20:39.030 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:39.031 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:39.031 DEBUG:boto:Method: GET 2023-10-12T01:20:39.032 DEBUG:boto:Path: /wafmoa-61/obj57 2023-10-12T01:20:39.032 DEBUG:boto:Data: 2023-10-12T01:20:39.033 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.034 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.034 DEBUG:boto:Params: {} 2023-10-12T01:20:39.034 DEBUG:boto:Token: None 2023-10-12T01:20:39.035 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj57 2023-10-12T01:20:39.035 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MxbStUez1dv6iX8bAv6r8exBzhc= 2023-10-12T01:20:39.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MxbStUez1dv6iX8bAv6r8exBzhc=', 'Content-Length': '0'} 2023-10-12T01:20:39.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e772813c80c8f4-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.039 DEBUG:boto:path=/wafmoa-61/obj57 2023-10-12T01:20:39.039 DEBUG:boto:auth_path=/wafmoa-61/obj57 2023-10-12T01:20:39.039 DEBUG:boto:Method: GET 2023-10-12T01:20:39.040 DEBUG:boto:Path: /wafmoa-61/obj57 2023-10-12T01:20:39.040 DEBUG:boto:Data: 2023-10-12T01:20:39.041 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.041 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.042 DEBUG:boto:Params: {} 2023-10-12T01:20:39.042 DEBUG:boto:Token: None 2023-10-12T01:20:39.043 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj57 2023-10-12T01:20:39.043 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MxbStUez1dv6iX8bAv6r8exBzhc= 2023-10-12T01:20:39.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MxbStUez1dv6iX8bAv6r8exBzhc=', 'Content-Length': '0'} 2023-10-12T01:20:39.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55167a31193254a-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.047 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.047 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.048 DEBUG:boto:path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.048 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.048 DEBUG:boto:Method: GET 2023-10-12T01:20:39.049 DEBUG:boto:Path: /wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.049 DEBUG:boto:Data: 2023-10-12T01:20:39.050 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.050 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.051 DEBUG:boto:Params: {} 2023-10-12T01:20:39.051 DEBUG:boto:Token: None 2023-10-12T01:20:39.052 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.052 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001743a4d456e6bc70-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.056 DEBUG:boto:b'wafmoa-61obj571000falseobj572023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.056 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.057 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.057 DEBUG:boto:path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.058 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.058 DEBUG:boto:Method: GET 2023-10-12T01:20:39.059 DEBUG:boto:Path: /wafmoa-61/?prefix=obj57 2023-10-12T01:20:39.059 DEBUG:boto:Data: 2023-10-12T01:20:39.059 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.060 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.061 DEBUG:boto:Params: {} 2023-10-12T01:20:39.061 DEBUG:boto:Token: None 2023-10-12T01:20:39.062 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.062 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341ea3dfbeb7b782-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.067 DEBUG:boto:b'wafmoa-61obj571000falseobj572023-10-12T01:16:55.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:20:39.068 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.069 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.069 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.069 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.070 DEBUG:boto:Method: GET 2023-10-12T01:20:39.070 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.071 DEBUG:boto:Data: 2023-10-12T01:20:39.071 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.072 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.072 DEBUG:boto:Params: {} 2023-10-12T01:20:39.073 DEBUG:boto:Token: None 2023-10-12T01:20:39.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY= 2023-10-12T01:20:39.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY=', 'Content-Length': '0'} 2023-10-12T01:20:39.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391cc379ba58ad25-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.077 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.077 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.078 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.078 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.079 DEBUG:boto:Method: GET 2023-10-12T01:20:39.079 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.079 DEBUG:boto:Data: 2023-10-12T01:20:39.080 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.081 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.081 DEBUG:boto:Params: {} 2023-10-12T01:20:39.081 DEBUG:boto:Token: None 2023-10-12T01:20:39.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY= 2023-10-12T01:20:39.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY=', 'Content-Length': '0'} 2023-10-12T01:20:39.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff991541382481bf-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.085 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.086 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.086 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.087 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.087 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.087 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.088 DEBUG:boto:Data: 2023-10-12T01:20:39.088 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.089 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.089 DEBUG:boto:Params: {} 2023-10-12T01:20:39.090 DEBUG:boto:Token: None 2023-10-12T01:20:39.090 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tfSmROeIUKsVSpbw4qqOEIF8Hag= 2023-10-12T01:20:39.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tfSmROeIUKsVSpbw4qqOEIF8Hag=', 'Content-Length': '0'} 2023-10-12T01:20:39.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fdb556a3240589-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.093 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.094 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.094 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.095 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.095 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.095 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.096 DEBUG:boto:Data: 2023-10-12T01:20:39.096 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.097 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.097 DEBUG:boto:Params: {} 2023-10-12T01:20:39.098 DEBUG:boto:Token: None 2023-10-12T01:20:39.098 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.099 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tfSmROeIUKsVSpbw4qqOEIF8Hag= 2023-10-12T01:20:39.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tfSmROeIUKsVSpbw4qqOEIF8Hag=', 'Content-Length': '0'} 2023-10-12T01:20:39.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db0df8ed7889581-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:20:39.102 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.102 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.103 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.103 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.104 DEBUG:boto:Method: GET 2023-10-12T01:20:39.104 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.104 DEBUG:boto:Data: 2023-10-12T01:20:39.105 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.106 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.106 DEBUG:boto:Params: {} 2023-10-12T01:20:39.107 DEBUG:boto:Token: None 2023-10-12T01:20:39.107 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.108 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY= 2023-10-12T01:20:39.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY=', 'Content-Length': '0'} 2023-10-12T01:20:39.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f983d68716e6ea18-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.111 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.111 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.112 DEBUG:boto:path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.112 DEBUG:boto:auth_path=/wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.112 DEBUG:boto:Method: GET 2023-10-12T01:20:39.113 DEBUG:boto:Path: /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.113 DEBUG:boto:Data: 2023-10-12T01:20:39.114 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.115 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.115 DEBUG:boto:Params: {} 2023-10-12T01:20:39.115 DEBUG:boto:Token: None 2023-10-12T01:20:39.116 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58?versionId=null 2023-10-12T01:20:39.116 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY= 2023-10-12T01:20:39.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q7NeFarmTL9bljkF6lkwphGA6eY=', 'Content-Length': '0'} 2023-10-12T01:20:39.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd80f065c2672a7-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.119 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.120 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.120 DEBUG:boto:path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.121 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.121 DEBUG:boto:Method: GET 2023-10-12T01:20:39.121 DEBUG:boto:Path: /wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.122 DEBUG:boto:Data: 2023-10-12T01:20:39.122 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.123 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.123 DEBUG:boto:Params: {} 2023-10-12T01:20:39.124 DEBUG:boto:Token: None 2023-10-12T01:20:39.124 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.125 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb10ee873cf716bd-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.174 DEBUG:boto:b'wafmoa-61obj581000falseobj582023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.175 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.175 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.176 DEBUG:boto:path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.176 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.177 DEBUG:boto:Method: GET 2023-10-12T01:20:39.177 DEBUG:boto:Path: /wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.178 DEBUG:boto:Data: 2023-10-12T01:20:39.178 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.179 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.179 DEBUG:boto:Params: {} 2023-10-12T01:20:39.180 DEBUG:boto:Token: None 2023-10-12T01:20:39.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014cae7e457514c87-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.187 DEBUG:boto:b'wafmoa-61obj581000falseobj582023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.187 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.188 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.188 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.189 DEBUG:boto:Path: /wafmoa-61/obj58 2023-10-12T01:20:39.189 DEBUG:boto:Data: 2023-10-12T01:20:39.190 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.191 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.191 DEBUG:boto:Params: {} 2023-10-12T01:20:39.191 DEBUG:boto:Token: None 2023-10-12T01:20:39.192 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58 2023-10-12T01:20:39.192 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jb8qh5itpyh5Bltv1RbOl4Sj9pI= 2023-10-12T01:20:39.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jb8qh5itpyh5Bltv1RbOl4Sj9pI=', 'Content-Length': '0'} 2023-10-12T01:20:39.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efaeff1dd525e0a9-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.195 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.196 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.196 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.197 DEBUG:boto:Path: /wafmoa-61/obj58 2023-10-12T01:20:39.197 DEBUG:boto:Data: 2023-10-12T01:20:39.197 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.198 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.199 DEBUG:boto:Params: {} 2023-10-12T01:20:39.199 DEBUG:boto:Token: None 2023-10-12T01:20:39.200 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58 2023-10-12T01:20:39.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jb8qh5itpyh5Bltv1RbOl4Sj9pI= 2023-10-12T01:20:39.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jb8qh5itpyh5Bltv1RbOl4Sj9pI=', 'Content-Length': '0'} 2023-10-12T01:20:39.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89dcebc1e6b028c-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-12T01:20:39.203 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.204 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.204 DEBUG:boto:Method: GET 2023-10-12T01:20:39.204 DEBUG:boto:Path: /wafmoa-61/obj58 2023-10-12T01:20:39.205 DEBUG:boto:Data: 2023-10-12T01:20:39.205 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.206 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.206 DEBUG:boto:Params: {} 2023-10-12T01:20:39.209 DEBUG:boto:Token: None 2023-10-12T01:20:39.210 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58 2023-10-12T01:20:39.210 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8g6TpKhFRYZXrqrJc2XTPjoxo3E= 2023-10-12T01:20:39.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8g6TpKhFRYZXrqrJc2XTPjoxo3E=', 'Content-Length': '0'} 2023-10-12T01:20:39.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000922cfd27469155e-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.214 DEBUG:boto:path=/wafmoa-61/obj58 2023-10-12T01:20:39.214 DEBUG:boto:auth_path=/wafmoa-61/obj58 2023-10-12T01:20:39.214 DEBUG:boto:Method: GET 2023-10-12T01:20:39.215 DEBUG:boto:Path: /wafmoa-61/obj58 2023-10-12T01:20:39.215 DEBUG:boto:Data: 2023-10-12T01:20:39.216 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.217 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.217 DEBUG:boto:Params: {} 2023-10-12T01:20:39.218 DEBUG:boto:Token: None 2023-10-12T01:20:39.218 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj58 2023-10-12T01:20:39.219 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8g6TpKhFRYZXrqrJc2XTPjoxo3E= 2023-10-12T01:20:39.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8g6TpKhFRYZXrqrJc2XTPjoxo3E=', 'Content-Length': '0'} 2023-10-12T01:20:39.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58df8264108881c-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.262 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.263 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.263 DEBUG:boto:path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.264 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.264 DEBUG:boto:Method: GET 2023-10-12T01:20:39.265 DEBUG:boto:Path: /wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.265 DEBUG:boto:Data: 2023-10-12T01:20:39.266 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.267 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.267 DEBUG:boto:Params: {} 2023-10-12T01:20:39.267 DEBUG:boto:Token: None 2023-10-12T01:20:39.268 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.268 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d467127e24f006-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.272 DEBUG:boto:b'wafmoa-61obj581000falseobj582023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.273 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.273 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.274 DEBUG:boto:path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.274 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.275 DEBUG:boto:Method: GET 2023-10-12T01:20:39.275 DEBUG:boto:Path: /wafmoa-61/?prefix=obj58 2023-10-12T01:20:39.275 DEBUG:boto:Data: 2023-10-12T01:20:39.276 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.277 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.277 DEBUG:boto:Params: {} 2023-10-12T01:20:39.278 DEBUG:boto:Token: None 2023-10-12T01:20:39.278 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.279 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017015d2dd913d28b-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.284 DEBUG:boto:b'wafmoa-61obj581000falseobj582023-10-12T01:16:56.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:20:39.285 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.285 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.285 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.286 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.286 DEBUG:boto:Method: GET 2023-10-12T01:20:39.287 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.287 DEBUG:boto:Data: 2023-10-12T01:20:39.287 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.288 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.289 DEBUG:boto:Params: {} 2023-10-12T01:20:39.289 DEBUG:boto:Token: None 2023-10-12T01:20:39.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0= 2023-10-12T01:20:39.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0=', 'Content-Length': '0'} 2023-10-12T01:20:39.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af69e223e361d40-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.334 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.335 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.335 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.336 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.336 DEBUG:boto:Method: GET 2023-10-12T01:20:39.336 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.337 DEBUG:boto:Data: 2023-10-12T01:20:39.337 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.338 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.339 DEBUG:boto:Params: {} 2023-10-12T01:20:39.339 DEBUG:boto:Token: None 2023-10-12T01:20:39.339 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.340 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0= 2023-10-12T01:20:39.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0=', 'Content-Length': '0'} 2023-10-12T01:20:39.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037fe7cdee976f231-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.343 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.344 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.344 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.344 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.345 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.345 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.346 DEBUG:boto:Data: 2023-10-12T01:20:39.346 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.347 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.347 DEBUG:boto:Params: {} 2023-10-12T01:20:39.348 DEBUG:boto:Token: None 2023-10-12T01:20:39.348 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.349 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/+WbNgnI3qWq280nnPW8ym6Q0SU= 2023-10-12T01:20:39.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/+WbNgnI3qWq280nnPW8ym6Q0SU=', 'Content-Length': '0'} 2023-10-12T01:20:39.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035afeca00af1009d-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.352 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.352 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.353 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.353 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.353 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.354 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.354 DEBUG:boto:Data: 2023-10-12T01:20:39.355 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.356 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.356 DEBUG:boto:Params: {} 2023-10-12T01:20:39.356 DEBUG:boto:Token: None 2023-10-12T01:20:39.357 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/+WbNgnI3qWq280nnPW8ym6Q0SU= 2023-10-12T01:20:39.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/+WbNgnI3qWq280nnPW8ym6Q0SU=', 'Content-Length': '0'} 2023-10-12T01:20:39.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b64fedcd99d831a-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:20:39.361 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.361 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.361 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.362 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.362 DEBUG:boto:Method: GET 2023-10-12T01:20:39.363 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.363 DEBUG:boto:Data: 2023-10-12T01:20:39.364 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.364 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.365 DEBUG:boto:Params: {} 2023-10-12T01:20:39.365 DEBUG:boto:Token: None 2023-10-12T01:20:39.366 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0= 2023-10-12T01:20:39.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0=', 'Content-Length': '0'} 2023-10-12T01:20:39.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000414851756097b460-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.370 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.370 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.371 DEBUG:boto:path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.371 DEBUG:boto:auth_path=/wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.372 DEBUG:boto:Method: GET 2023-10-12T01:20:39.372 DEBUG:boto:Path: /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.372 DEBUG:boto:Data: 2023-10-12T01:20:39.373 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.374 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.374 DEBUG:boto:Params: {} 2023-10-12T01:20:39.374 DEBUG:boto:Token: None 2023-10-12T01:20:39.375 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62?versionId=null 2023-10-12T01:20:39.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0= 2023-10-12T01:20:39.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mS55FhRWCiXnt2NqOhmgPal+Rw0=', 'Content-Length': '0'} 2023-10-12T01:20:39.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257dddfabefa03a9-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.378 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.379 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.379 DEBUG:boto:path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.380 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.380 DEBUG:boto:Method: GET 2023-10-12T01:20:39.380 DEBUG:boto:Path: /wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.381 DEBUG:boto:Data: 2023-10-12T01:20:39.381 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.382 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.382 DEBUG:boto:Params: {} 2023-10-12T01:20:39.383 DEBUG:boto:Token: None 2023-10-12T01:20:39.383 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.384 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faef43eff8f146e7-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.387 DEBUG:boto:b'wafmoa-61obj621000falseobj622023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.388 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.388 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.389 DEBUG:boto:path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.389 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.389 DEBUG:boto:Method: GET 2023-10-12T01:20:39.390 DEBUG:boto:Path: /wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.390 DEBUG:boto:Data: 2023-10-12T01:20:39.391 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.392 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.392 DEBUG:boto:Params: {} 2023-10-12T01:20:39.392 DEBUG:boto:Token: None 2023-10-12T01:20:39.393 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.393 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf27fbbd5766f46-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.399 DEBUG:boto:b'wafmoa-61obj621000falseobj622023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.399 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.400 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.400 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.401 DEBUG:boto:Path: /wafmoa-61/obj62 2023-10-12T01:20:39.401 DEBUG:boto:Data: 2023-10-12T01:20:39.401 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.402 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.403 DEBUG:boto:Params: {} 2023-10-12T01:20:39.403 DEBUG:boto:Token: None 2023-10-12T01:20:39.403 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62 2023-10-12T01:20:39.404 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JqNtyah7H1ejW5cLuQhMe98TNII= 2023-10-12T01:20:39.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JqNtyah7H1ejW5cLuQhMe98TNII=', 'Content-Length': '0'} 2023-10-12T01:20:39.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b831d53ebe8c3bd-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.407 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.407 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.408 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.408 DEBUG:boto:Path: /wafmoa-61/obj62 2023-10-12T01:20:39.408 DEBUG:boto:Data: 2023-10-12T01:20:39.409 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.410 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.410 DEBUG:boto:Params: {} 2023-10-12T01:20:39.410 DEBUG:boto:Token: None 2023-10-12T01:20:39.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62 2023-10-12T01:20:39.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JqNtyah7H1ejW5cLuQhMe98TNII= 2023-10-12T01:20:39.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JqNtyah7H1ejW5cLuQhMe98TNII=', 'Content-Length': '0'} 2023-10-12T01:20:39.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26c732ff24e549d-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-12T01:20:39.416 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.417 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.417 DEBUG:boto:Method: GET 2023-10-12T01:20:39.417 DEBUG:boto:Path: /wafmoa-61/obj62 2023-10-12T01:20:39.418 DEBUG:boto:Data: 2023-10-12T01:20:39.418 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.419 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.420 DEBUG:boto:Params: {} 2023-10-12T01:20:39.420 DEBUG:boto:Token: None 2023-10-12T01:20:39.420 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62 2023-10-12T01:20:39.421 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zn6BxUqttuFVlUEzOjnv3QpY9jg= 2023-10-12T01:20:39.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zn6BxUqttuFVlUEzOjnv3QpY9jg=', 'Content-Length': '0'} 2023-10-12T01:20:39.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd98eda60fd3b54c-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.466 DEBUG:boto:path=/wafmoa-61/obj62 2023-10-12T01:20:39.467 DEBUG:boto:auth_path=/wafmoa-61/obj62 2023-10-12T01:20:39.467 DEBUG:boto:Method: GET 2023-10-12T01:20:39.468 DEBUG:boto:Path: /wafmoa-61/obj62 2023-10-12T01:20:39.468 DEBUG:boto:Data: 2023-10-12T01:20:39.469 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.469 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.470 DEBUG:boto:Params: {} 2023-10-12T01:20:39.470 DEBUG:boto:Token: None 2023-10-12T01:20:39.471 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj62 2023-10-12T01:20:39.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Zn6BxUqttuFVlUEzOjnv3QpY9jg= 2023-10-12T01:20:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Zn6BxUqttuFVlUEzOjnv3QpY9jg=', 'Content-Length': '0'} 2023-10-12T01:20:39.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f153c22da1e3e9-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.475 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.475 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.476 DEBUG:boto:path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.476 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.477 DEBUG:boto:Method: GET 2023-10-12T01:20:39.477 DEBUG:boto:Path: /wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.477 DEBUG:boto:Data: 2023-10-12T01:20:39.478 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.479 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.479 DEBUG:boto:Params: {} 2023-10-12T01:20:39.480 DEBUG:boto:Token: None 2023-10-12T01:20:39.480 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.480 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250d6423101d846b-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.484 DEBUG:boto:b'wafmoa-61obj621000falseobj622023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.484 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.485 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.485 DEBUG:boto:path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.486 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.486 DEBUG:boto:Method: GET 2023-10-12T01:20:39.487 DEBUG:boto:Path: /wafmoa-61/?prefix=obj62 2023-10-12T01:20:39.487 DEBUG:boto:Data: 2023-10-12T01:20:39.487 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.488 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.489 DEBUG:boto:Params: {} 2023-10-12T01:20:39.489 DEBUG:boto:Token: None 2023-10-12T01:20:39.490 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ece6feddaf34c9a9-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.496 DEBUG:boto:b'wafmoa-61obj621000falseobj622023-10-12T01:16:56.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:20:39.497 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.498 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.498 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.499 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.499 DEBUG:boto:Method: GET 2023-10-12T01:20:39.500 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.500 DEBUG:boto:Data: 2023-10-12T01:20:39.500 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.501 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.501 DEBUG:boto:Params: {} 2023-10-12T01:20:39.502 DEBUG:boto:Token: None 2023-10-12T01:20:39.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.503 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco= 2023-10-12T01:20:39.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco=', 'Content-Length': '0'} 2023-10-12T01:20:39.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fc4cfcb8552443-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.506 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.507 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.507 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.507 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.508 DEBUG:boto:Method: GET 2023-10-12T01:20:39.508 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.509 DEBUG:boto:Data: 2023-10-12T01:20:39.509 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.510 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.510 DEBUG:boto:Params: {} 2023-10-12T01:20:39.511 DEBUG:boto:Token: None 2023-10-12T01:20:39.511 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.512 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco= 2023-10-12T01:20:39.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco=', 'Content-Length': '0'} 2023-10-12T01:20:39.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006543b2de6a1afe48-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.515 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.515 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.516 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.516 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.516 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.517 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.517 DEBUG:boto:Data: 2023-10-12T01:20:39.518 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.518 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.519 DEBUG:boto:Params: {} 2023-10-12T01:20:39.519 DEBUG:boto:Token: None 2023-10-12T01:20:39.520 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.520 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:su1cKo6/I1CN016TCyUCHwrlBC0= 2023-10-12T01:20:39.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:su1cKo6/I1CN016TCyUCHwrlBC0=', 'Content-Length': '0'} 2023-10-12T01:20:39.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24b567ef70cb15f-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.523 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.523 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.523 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.524 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.524 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.525 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.525 DEBUG:boto:Data: 2023-10-12T01:20:39.525 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.526 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.526 DEBUG:boto:Params: {} 2023-10-12T01:20:39.527 DEBUG:boto:Token: None 2023-10-12T01:20:39.527 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:su1cKo6/I1CN016TCyUCHwrlBC0= 2023-10-12T01:20:39.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:su1cKo6/I1CN016TCyUCHwrlBC0=', 'Content-Length': '0'} 2023-10-12T01:20:39.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af8c2486753623a2-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:20:39.531 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.532 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.532 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.533 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.533 DEBUG:boto:Method: GET 2023-10-12T01:20:39.533 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.534 DEBUG:boto:Data: 2023-10-12T01:20:39.534 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.535 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.535 DEBUG:boto:Params: {} 2023-10-12T01:20:39.536 DEBUG:boto:Token: None 2023-10-12T01:20:39.536 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.537 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco= 2023-10-12T01:20:39.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco=', 'Content-Length': '0'} 2023-10-12T01:20:39.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016546258a2b80fa2-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.582 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.583 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.583 DEBUG:boto:path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.584 DEBUG:boto:auth_path=/wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.584 DEBUG:boto:Method: GET 2023-10-12T01:20:39.584 DEBUG:boto:Path: /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.585 DEBUG:boto:Data: 2023-10-12T01:20:39.585 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.586 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.586 DEBUG:boto:Params: {} 2023-10-12T01:20:39.587 DEBUG:boto:Token: None 2023-10-12T01:20:39.587 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68?versionId=null 2023-10-12T01:20:39.588 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco= 2023-10-12T01:20:39.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KyFKYgOxmRMC7WhZAhgJCfpzKco=', 'Content-Length': '0'} 2023-10-12T01:20:39.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d53f6aaf18034c1-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.591 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.591 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.592 DEBUG:boto:path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.592 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.592 DEBUG:boto:Method: GET 2023-10-12T01:20:39.593 DEBUG:boto:Path: /wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.593 DEBUG:boto:Data: 2023-10-12T01:20:39.594 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.595 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.595 DEBUG:boto:Params: {} 2023-10-12T01:20:39.595 DEBUG:boto:Token: None 2023-10-12T01:20:39.596 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.596 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000022b2be137ef15e9-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.600 DEBUG:boto:b'wafmoa-61obj681000falseobj682023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.600 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.601 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.601 DEBUG:boto:path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.602 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.602 DEBUG:boto:Method: GET 2023-10-12T01:20:39.603 DEBUG:boto:Path: /wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.603 DEBUG:boto:Data: 2023-10-12T01:20:39.603 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.604 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.605 DEBUG:boto:Params: {} 2023-10-12T01:20:39.605 DEBUG:boto:Token: None 2023-10-12T01:20:39.606 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.606 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007936f766aa7efd5a-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.611 DEBUG:boto:b'wafmoa-61obj681000falseobj682023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.612 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.612 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.613 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.613 DEBUG:boto:Path: /wafmoa-61/obj68 2023-10-12T01:20:39.614 DEBUG:boto:Data: 2023-10-12T01:20:39.614 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.615 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.615 DEBUG:boto:Params: {} 2023-10-12T01:20:39.616 DEBUG:boto:Token: None 2023-10-12T01:20:39.616 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68 2023-10-12T01:20:39.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3/WmD2DOKODSLuZDxVeX5+Gy2VI= 2023-10-12T01:20:39.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3/WmD2DOKODSLuZDxVeX5+Gy2VI=', 'Content-Length': '0'} 2023-10-12T01:20:39.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a936e345802cfd59-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.623 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.623 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.624 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.624 DEBUG:boto:Path: /wafmoa-61/obj68 2023-10-12T01:20:39.624 DEBUG:boto:Data: 2023-10-12T01:20:39.625 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.626 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.626 DEBUG:boto:Params: {} 2023-10-12T01:20:39.626 DEBUG:boto:Token: None 2023-10-12T01:20:39.627 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68 2023-10-12T01:20:39.627 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3/WmD2DOKODSLuZDxVeX5+Gy2VI= 2023-10-12T01:20:39.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3/WmD2DOKODSLuZDxVeX5+Gy2VI=', 'Content-Length': '0'} 2023-10-12T01:20:39.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000133e8bbc1195edbb-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-12T01:20:39.631 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.631 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.631 DEBUG:boto:Method: GET 2023-10-12T01:20:39.632 DEBUG:boto:Path: /wafmoa-61/obj68 2023-10-12T01:20:39.632 DEBUG:boto:Data: 2023-10-12T01:20:39.633 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.633 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.634 DEBUG:boto:Params: {} 2023-10-12T01:20:39.634 DEBUG:boto:Token: None 2023-10-12T01:20:39.635 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68 2023-10-12T01:20:39.635 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xmP0OOdZYDXmyS1stoakeOXawb8= 2023-10-12T01:20:39.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xmP0OOdZYDXmyS1stoakeOXawb8=', 'Content-Length': '0'} 2023-10-12T01:20:39.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3dec4b57d1a414-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.678 DEBUG:boto:path=/wafmoa-61/obj68 2023-10-12T01:20:39.679 DEBUG:boto:auth_path=/wafmoa-61/obj68 2023-10-12T01:20:39.679 DEBUG:boto:Method: GET 2023-10-12T01:20:39.680 DEBUG:boto:Path: /wafmoa-61/obj68 2023-10-12T01:20:39.680 DEBUG:boto:Data: 2023-10-12T01:20:39.680 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.681 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.682 DEBUG:boto:Params: {} 2023-10-12T01:20:39.682 DEBUG:boto:Token: None 2023-10-12T01:20:39.683 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj68 2023-10-12T01:20:39.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xmP0OOdZYDXmyS1stoakeOXawb8= 2023-10-12T01:20:39.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xmP0OOdZYDXmyS1stoakeOXawb8=', 'Content-Length': '0'} 2023-10-12T01:20:39.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa77e5dc414c1a4a-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.686 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.687 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.687 DEBUG:boto:path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.688 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.688 DEBUG:boto:Method: GET 2023-10-12T01:20:39.688 DEBUG:boto:Path: /wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.689 DEBUG:boto:Data: 2023-10-12T01:20:39.689 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.690 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.691 DEBUG:boto:Params: {} 2023-10-12T01:20:39.691 DEBUG:boto:Token: None 2023-10-12T01:20:39.692 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.692 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e712a02d53660cd-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.696 DEBUG:boto:b'wafmoa-61obj681000falseobj682023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.697 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.698 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.698 DEBUG:boto:path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.699 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.699 DEBUG:boto:Method: GET 2023-10-12T01:20:39.700 DEBUG:boto:Path: /wafmoa-61/?prefix=obj68 2023-10-12T01:20:39.700 DEBUG:boto:Data: 2023-10-12T01:20:39.700 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.701 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.702 DEBUG:boto:Params: {} 2023-10-12T01:20:39.702 DEBUG:boto:Token: None 2023-10-12T01:20:39.703 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.703 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51ab484041b87b4-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.708 DEBUG:boto:b'wafmoa-61obj681000falseobj682023-10-12T01:16:55.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:20:39.709 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.710 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.710 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.711 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.711 DEBUG:boto:Method: GET 2023-10-12T01:20:39.712 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.712 DEBUG:boto:Data: 2023-10-12T01:20:39.712 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.713 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.714 DEBUG:boto:Params: {} 2023-10-12T01:20:39.714 DEBUG:boto:Token: None 2023-10-12T01:20:39.715 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.715 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4= 2023-10-12T01:20:39.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4=', 'Content-Length': '0'} 2023-10-12T01:20:39.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b53bfa90152dd5-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.719 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.719 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.719 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.720 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.720 DEBUG:boto:Method: GET 2023-10-12T01:20:39.721 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.721 DEBUG:boto:Data: 2023-10-12T01:20:39.722 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.722 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.723 DEBUG:boto:Params: {} 2023-10-12T01:20:39.723 DEBUG:boto:Token: None 2023-10-12T01:20:39.724 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.724 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4= 2023-10-12T01:20:39.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4=', 'Content-Length': '0'} 2023-10-12T01:20:39.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f986150bbecc5d-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.770 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.771 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.771 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.772 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.772 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.773 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.773 DEBUG:boto:Data: 2023-10-12T01:20:39.773 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.774 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.774 DEBUG:boto:Params: {} 2023-10-12T01:20:39.775 DEBUG:boto:Token: None 2023-10-12T01:20:39.775 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+gl8YUtqmffAPHFvIHc2wj6rkxk= 2023-10-12T01:20:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+gl8YUtqmffAPHFvIHc2wj6rkxk=', 'Content-Length': '0'} 2023-10-12T01:20:39.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000960a1b4f8ae7f8ec-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.779 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.779 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.780 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.780 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.780 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.781 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.781 DEBUG:boto:Data: 2023-10-12T01:20:39.781 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.782 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.783 DEBUG:boto:Params: {} 2023-10-12T01:20:39.783 DEBUG:boto:Token: None 2023-10-12T01:20:39.784 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+gl8YUtqmffAPHFvIHc2wj6rkxk= 2023-10-12T01:20:39.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+gl8YUtqmffAPHFvIHc2wj6rkxk=', 'Content-Length': '0'} 2023-10-12T01:20:39.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136f45b98ed869b4-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:20:39.787 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.788 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.788 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.789 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.789 DEBUG:boto:Method: GET 2023-10-12T01:20:39.789 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.790 DEBUG:boto:Data: 2023-10-12T01:20:39.790 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.791 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.792 DEBUG:boto:Params: {} 2023-10-12T01:20:39.792 DEBUG:boto:Token: None 2023-10-12T01:20:39.793 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4= 2023-10-12T01:20:39.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4=', 'Content-Length': '0'} 2023-10-12T01:20:39.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f108783201f81dd3-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.838 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.839 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.839 DEBUG:boto:path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.840 DEBUG:boto:auth_path=/wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.840 DEBUG:boto:Method: GET 2023-10-12T01:20:39.841 DEBUG:boto:Path: /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.841 DEBUG:boto:Data: 2023-10-12T01:20:39.841 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.842 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.843 DEBUG:boto:Params: {} 2023-10-12T01:20:39.843 DEBUG:boto:Token: None 2023-10-12T01:20:39.844 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69?versionId=null 2023-10-12T01:20:39.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4= 2023-10-12T01:20:39.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y5CNK0zzNR/ShRFYvdETO0emBG4=', 'Content-Length': '0'} 2023-10-12T01:20:39.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003918b038a350d957-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.848 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.849 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.849 DEBUG:boto:path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.850 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.850 DEBUG:boto:Method: GET 2023-10-12T01:20:39.850 DEBUG:boto:Path: /wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.851 DEBUG:boto:Data: 2023-10-12T01:20:39.851 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.852 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.853 DEBUG:boto:Params: {} 2023-10-12T01:20:39.853 DEBUG:boto:Token: None 2023-10-12T01:20:39.854 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.854 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e4d5a9017b4e279-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.858 DEBUG:boto:b'wafmoa-61obj691000falseobj692023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.859 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.859 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.860 DEBUG:boto:path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.860 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.861 DEBUG:boto:Method: GET 2023-10-12T01:20:39.861 DEBUG:boto:Path: /wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.862 DEBUG:boto:Data: 2023-10-12T01:20:39.862 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.863 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.864 DEBUG:boto:Params: {} 2023-10-12T01:20:39.864 DEBUG:boto:Token: None 2023-10-12T01:20:39.864 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6a711662a35f835-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.870 DEBUG:boto:b'wafmoa-61obj691000falseobj692023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.871 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.871 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.872 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.872 DEBUG:boto:Path: /wafmoa-61/obj69 2023-10-12T01:20:39.872 DEBUG:boto:Data: 2023-10-12T01:20:39.873 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.874 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.874 DEBUG:boto:Params: {} 2023-10-12T01:20:39.875 DEBUG:boto:Token: None 2023-10-12T01:20:39.875 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69 2023-10-12T01:20:39.876 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:afwgwvPwvHo/zwKhB+opT/3lLho= 2023-10-12T01:20:39.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:afwgwvPwvHo/zwKhB+opT/3lLho=', 'Content-Length': '0'} 2023-10-12T01:20:39.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059e8b4cf11cb93a7-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.879 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.880 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.881 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.881 DEBUG:boto:Path: /wafmoa-61/obj69 2023-10-12T01:20:39.882 DEBUG:boto:Data: 2023-10-12T01:20:39.882 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.883 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.883 DEBUG:boto:Params: {} 2023-10-12T01:20:39.884 DEBUG:boto:Token: None 2023-10-12T01:20:39.884 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69 2023-10-12T01:20:39.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:afwgwvPwvHo/zwKhB+opT/3lLho= 2023-10-12T01:20:39.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:afwgwvPwvHo/zwKhB+opT/3lLho=', 'Content-Length': '0'} 2023-10-12T01:20:39.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc5098324d576dc-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-12T01:20:39.888 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.889 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.889 DEBUG:boto:Method: GET 2023-10-12T01:20:39.890 DEBUG:boto:Path: /wafmoa-61/obj69 2023-10-12T01:20:39.891 DEBUG:boto:Data: 2023-10-12T01:20:39.891 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.892 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.892 DEBUG:boto:Params: {} 2023-10-12T01:20:39.893 DEBUG:boto:Token: None 2023-10-12T01:20:39.893 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69 2023-10-12T01:20:39.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Akn7JueMZrV7gIX5XIhzPMFqar0= 2023-10-12T01:20:39.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Akn7JueMZrV7gIX5XIhzPMFqar0=', 'Content-Length': '0'} 2023-10-12T01:20:39.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eba7381ab0305e0-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.898 DEBUG:boto:path=/wafmoa-61/obj69 2023-10-12T01:20:39.898 DEBUG:boto:auth_path=/wafmoa-61/obj69 2023-10-12T01:20:39.899 DEBUG:boto:Method: GET 2023-10-12T01:20:39.899 DEBUG:boto:Path: /wafmoa-61/obj69 2023-10-12T01:20:39.899 DEBUG:boto:Data: 2023-10-12T01:20:39.900 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.901 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.902 DEBUG:boto:Params: {} 2023-10-12T01:20:39.902 DEBUG:boto:Token: None 2023-10-12T01:20:39.903 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj69 2023-10-12T01:20:39.903 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Akn7JueMZrV7gIX5XIhzPMFqar0= 2023-10-12T01:20:39.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Akn7JueMZrV7gIX5XIhzPMFqar0=', 'Content-Length': '0'} 2023-10-12T01:20:39.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c67404e4bc7eca-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.950 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.952 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.952 DEBUG:boto:path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.953 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.953 DEBUG:boto:Method: GET 2023-10-12T01:20:39.954 DEBUG:boto:Path: /wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.954 DEBUG:boto:Data: 2023-10-12T01:20:39.955 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.955 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.956 DEBUG:boto:Params: {} 2023-10-12T01:20:39.956 DEBUG:boto:Token: None 2023-10-12T01:20:39.957 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.957 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f820241babc894be-00652749e7-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.961 DEBUG:boto:b'wafmoa-61obj691000falseobj692023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.962 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:39.963 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:39.963 DEBUG:boto:path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.963 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.964 DEBUG:boto:Method: GET 2023-10-12T01:20:39.964 DEBUG:boto:Path: /wafmoa-61/?prefix=obj69 2023-10-12T01:20:39.965 DEBUG:boto:Data: 2023-10-12T01:20:39.965 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.966 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.967 DEBUG:boto:Params: {} 2023-10-12T01:20:39.967 DEBUG:boto:Token: None 2023-10-12T01:20:39.968 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/ 2023-10-12T01:20:39.968 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw= 2023-10-12T01:20:39.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8aCjNdAowZIQzm79wMzR9CYHrEw=', 'Content-Length': '0'} 2023-10-12T01:20:39.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007777690d23480930-00652749e7-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.973 DEBUG:boto:b'wafmoa-61obj691000falseobj692023-10-12T01:16:56.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:39.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:20:39.974 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:39.974 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:39.975 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.975 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.976 DEBUG:boto:Method: GET 2023-10-12T01:20:39.976 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.977 DEBUG:boto:Data: 2023-10-12T01:20:39.977 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.978 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.978 DEBUG:boto:Params: {} 2023-10-12T01:20:39.979 DEBUG:boto:Token: None 2023-10-12T01:20:39.979 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p93tY91KyTUFHApicZ+USI/mykI= 2023-10-12T01:20:39.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p93tY91KyTUFHApicZ+USI/mykI=', 'Content-Length': '0'} 2023-10-12T01:20:39.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f62e4346d5d98fd-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.983 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:39.983 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:39.984 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.984 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.984 DEBUG:boto:Method: GET 2023-10-12T01:20:39.985 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.985 DEBUG:boto:Data: 2023-10-12T01:20:39.986 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:39.987 DEBUG:boto:Port: 8000 2023-10-12T01:20:39.987 DEBUG:boto:Params: {} 2023-10-12T01:20:39.988 DEBUG:boto:Token: None 2023-10-12T01:20:39.988 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p93tY91KyTUFHApicZ+USI/mykI= 2023-10-12T01:20:39.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p93tY91KyTUFHApicZ+USI/mykI=', 'Content-Length': '0'} 2023-10-12T01:20:39.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2e95f44633d5c2-00652749e7-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:39.992 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:39.993 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:39.993 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.993 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.994 DEBUG:boto:Method: HEAD 2023-10-12T01:20:39.994 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.995 DEBUG:boto:Data: 2023-10-12T01:20:39.995 DEBUG:boto:Headers: {} 2023-10-12T01:20:39.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:39.996 DEBUG:boto:Port: 8001 2023-10-12T01:20:39.996 DEBUG:boto:Params: {} 2023-10-12T01:20:39.997 DEBUG:boto:Token: None 2023-10-12T01:20:39.997 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:39 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:39.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SZ5kz9IcSkgtcCR9pl1Nldqj1+I= 2023-10-12T01:20:39.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SZ5kz9IcSkgtcCR9pl1Nldqj1+I=', 'Content-Length': '0'} 2023-10-12T01:20:40.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ef344f86e9f87ea-00652749e7-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.000 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.001 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.001 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.002 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.002 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.002 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.003 DEBUG:boto:Data: 2023-10-12T01:20:40.003 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.004 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.004 DEBUG:boto:Params: {} 2023-10-12T01:20:40.005 DEBUG:boto:Token: None 2023-10-12T01:20:40.005 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DRNKVhmiPDKsJov3nLMj9bD1DPQ= 2023-10-12T01:20:40.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DRNKVhmiPDKsJov3nLMj9bD1DPQ=', 'Content-Length': '0'} 2023-10-12T01:20:40.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e718e74d0b930270-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:20:40.009 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.009 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.010 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.010 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.011 DEBUG:boto:Method: GET 2023-10-12T01:20:40.011 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.011 DEBUG:boto:Data: 2023-10-12T01:20:40.012 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.013 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.013 DEBUG:boto:Params: {} 2023-10-12T01:20:40.014 DEBUG:boto:Token: None 2023-10-12T01:20:40.014 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.015 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mo9+uq1huGfAAIHUSY1lGKODRvI= 2023-10-12T01:20:40.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mo9+uq1huGfAAIHUSY1lGKODRvI=', 'Content-Length': '0'} 2023-10-12T01:20:40.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac7ff1115571346-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.018 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.019 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.019 DEBUG:boto:path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.020 DEBUG:boto:auth_path=/wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.020 DEBUG:boto:Method: GET 2023-10-12T01:20:40.020 DEBUG:boto:Path: /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.021 DEBUG:boto:Data: 2023-10-12T01:20:40.021 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.022 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.022 DEBUG:boto:Params: {} 2023-10-12T01:20:40.023 DEBUG:boto:Token: None 2023-10-12T01:20:40.023 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74?versionId=null 2023-10-12T01:20:40.024 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mo9+uq1huGfAAIHUSY1lGKODRvI= 2023-10-12T01:20:40.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mo9+uq1huGfAAIHUSY1lGKODRvI=', 'Content-Length': '0'} 2023-10-12T01:20:40.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620eb0f0dccbe3d7-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.070 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.071 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.071 DEBUG:boto:path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.072 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.072 DEBUG:boto:Method: GET 2023-10-12T01:20:40.072 DEBUG:boto:Path: /wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.073 DEBUG:boto:Data: 2023-10-12T01:20:40.073 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.074 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.074 DEBUG:boto:Params: {} 2023-10-12T01:20:40.075 DEBUG:boto:Token: None 2023-10-12T01:20:40.075 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b71e13a952e64fc-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.079 DEBUG:boto:b'wafmoa-61obj741000falseobj742023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.080 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.080 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.081 DEBUG:boto:path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.081 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.081 DEBUG:boto:Method: GET 2023-10-12T01:20:40.082 DEBUG:boto:Path: /wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.082 DEBUG:boto:Data: 2023-10-12T01:20:40.083 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.084 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.084 DEBUG:boto:Params: {} 2023-10-12T01:20:40.084 DEBUG:boto:Token: None 2023-10-12T01:20:40.085 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea51bddb8fcbf66f-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.090 DEBUG:boto:b'wafmoa-61obj741000falseobj742023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.091 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.091 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.091 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.092 DEBUG:boto:Path: /wafmoa-61/obj74 2023-10-12T01:20:40.092 DEBUG:boto:Data: 2023-10-12T01:20:40.093 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.093 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.094 DEBUG:boto:Params: {} 2023-10-12T01:20:40.094 DEBUG:boto:Token: None 2023-10-12T01:20:40.095 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74 2023-10-12T01:20:40.095 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4/csofnoPKTEHaGW6P4QkXlQy+E= 2023-10-12T01:20:40.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4/csofnoPKTEHaGW6P4QkXlQy+E=', 'Content-Length': '0'} 2023-10-12T01:20:40.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2ad1e843ce1801-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.098 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.099 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.099 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.099 DEBUG:boto:Path: /wafmoa-61/obj74 2023-10-12T01:20:40.100 DEBUG:boto:Data: 2023-10-12T01:20:40.100 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.101 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.101 DEBUG:boto:Params: {} 2023-10-12T01:20:40.102 DEBUG:boto:Token: None 2023-10-12T01:20:40.102 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74 2023-10-12T01:20:40.103 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4/csofnoPKTEHaGW6P4QkXlQy+E= 2023-10-12T01:20:40.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4/csofnoPKTEHaGW6P4QkXlQy+E=', 'Content-Length': '0'} 2023-10-12T01:20:40.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d3c1d45565cbee-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-12T01:20:40.107 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.107 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.107 DEBUG:boto:Method: GET 2023-10-12T01:20:40.108 DEBUG:boto:Path: /wafmoa-61/obj74 2023-10-12T01:20:40.108 DEBUG:boto:Data: 2023-10-12T01:20:40.109 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.109 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.110 DEBUG:boto:Params: {} 2023-10-12T01:20:40.111 DEBUG:boto:Token: None 2023-10-12T01:20:40.111 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74 2023-10-12T01:20:40.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:13/La3L/TD8F9rTLkOpjLen3m3M= 2023-10-12T01:20:40.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:13/La3L/TD8F9rTLkOpjLen3m3M=', 'Content-Length': '0'} 2023-10-12T01:20:40.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3a13336ca0a9c7-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.154 DEBUG:boto:path=/wafmoa-61/obj74 2023-10-12T01:20:40.155 DEBUG:boto:auth_path=/wafmoa-61/obj74 2023-10-12T01:20:40.155 DEBUG:boto:Method: GET 2023-10-12T01:20:40.156 DEBUG:boto:Path: /wafmoa-61/obj74 2023-10-12T01:20:40.156 DEBUG:boto:Data: 2023-10-12T01:20:40.157 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.157 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.158 DEBUG:boto:Params: {} 2023-10-12T01:20:40.158 DEBUG:boto:Token: None 2023-10-12T01:20:40.159 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj74 2023-10-12T01:20:40.159 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:13/La3L/TD8F9rTLkOpjLen3m3M= 2023-10-12T01:20:40.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:13/La3L/TD8F9rTLkOpjLen3m3M=', 'Content-Length': '0'} 2023-10-12T01:20:40.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ae396769bf48a1-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.163 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.163 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.163 DEBUG:boto:path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.164 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.164 DEBUG:boto:Method: GET 2023-10-12T01:20:40.165 DEBUG:boto:Path: /wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.165 DEBUG:boto:Data: 2023-10-12T01:20:40.166 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.167 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.167 DEBUG:boto:Params: {} 2023-10-12T01:20:40.167 DEBUG:boto:Token: None 2023-10-12T01:20:40.168 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf12395c2ea6ab2-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.173 DEBUG:boto:b'wafmoa-61obj741000falseobj742023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.174 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.174 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.175 DEBUG:boto:path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.176 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.176 DEBUG:boto:Method: GET 2023-10-12T01:20:40.177 DEBUG:boto:Path: /wafmoa-61/?prefix=obj74 2023-10-12T01:20:40.177 DEBUG:boto:Data: 2023-10-12T01:20:40.177 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.178 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.179 DEBUG:boto:Params: {} 2023-10-12T01:20:40.179 DEBUG:boto:Token: None 2023-10-12T01:20:40.180 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.180 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ab8d43d6e0fa8f-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.185 DEBUG:boto:b'wafmoa-61obj741000falseobj742023-10-12T01:16:57.284Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:20:40.187 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.187 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.187 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.188 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.189 DEBUG:boto:Method: GET 2023-10-12T01:20:40.189 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.190 DEBUG:boto:Data: 2023-10-12T01:20:40.191 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.192 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.193 DEBUG:boto:Params: {} 2023-10-12T01:20:40.194 DEBUG:boto:Token: None 2023-10-12T01:20:40.194 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU= 2023-10-12T01:20:40.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU=', 'Content-Length': '0'} 2023-10-12T01:20:40.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af306296667c442-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.199 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.200 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.201 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.201 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.202 DEBUG:boto:Method: GET 2023-10-12T01:20:40.203 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.203 DEBUG:boto:Data: 2023-10-12T01:20:40.204 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.205 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.206 DEBUG:boto:Params: {} 2023-10-12T01:20:40.206 DEBUG:boto:Token: None 2023-10-12T01:20:40.207 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.207 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU= 2023-10-12T01:20:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU=', 'Content-Length': '0'} 2023-10-12T01:20:40.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8cc5e495f559814-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.210 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.211 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.211 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.212 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.212 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.212 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.213 DEBUG:boto:Data: 2023-10-12T01:20:40.213 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.214 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.215 DEBUG:boto:Params: {} 2023-10-12T01:20:40.215 DEBUG:boto:Token: None 2023-10-12T01:20:40.215 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.216 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GACKXzQquV+t+0r8CWVJK+u9Fk= 2023-10-12T01:20:40.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GACKXzQquV+t+0r8CWVJK+u9Fk=', 'Content-Length': '0'} 2023-10-12T01:20:40.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff8dc68d6bd4a0c-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.219 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.219 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.220 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.220 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.220 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.221 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.221 DEBUG:boto:Data: 2023-10-12T01:20:40.222 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.222 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.223 DEBUG:boto:Params: {} 2023-10-12T01:20:40.223 DEBUG:boto:Token: None 2023-10-12T01:20:40.224 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.224 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6GACKXzQquV+t+0r8CWVJK+u9Fk= 2023-10-12T01:20:40.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6GACKXzQquV+t+0r8CWVJK+u9Fk=', 'Content-Length': '0'} 2023-10-12T01:20:40.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750506f2e6041b84-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:20:40.227 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.228 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.228 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.229 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.229 DEBUG:boto:Method: GET 2023-10-12T01:20:40.229 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.230 DEBUG:boto:Data: 2023-10-12T01:20:40.230 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.231 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.231 DEBUG:boto:Params: {} 2023-10-12T01:20:40.232 DEBUG:boto:Token: None 2023-10-12T01:20:40.232 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU= 2023-10-12T01:20:40.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU=', 'Content-Length': '0'} 2023-10-12T01:20:40.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042964df7a9597af0-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.278 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.279 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.279 DEBUG:boto:path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.279 DEBUG:boto:auth_path=/wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.280 DEBUG:boto:Method: GET 2023-10-12T01:20:40.280 DEBUG:boto:Path: /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.281 DEBUG:boto:Data: 2023-10-12T01:20:40.281 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.282 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.282 DEBUG:boto:Params: {} 2023-10-12T01:20:40.283 DEBUG:boto:Token: None 2023-10-12T01:20:40.283 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76?versionId=null 2023-10-12T01:20:40.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU= 2023-10-12T01:20:40.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QhGusXViaUWKl+SEnNHYl0rr4jU=', 'Content-Length': '0'} 2023-10-12T01:20:40.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4d6ec60413c213-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.287 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.287 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.287 DEBUG:boto:path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.288 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.288 DEBUG:boto:Method: GET 2023-10-12T01:20:40.289 DEBUG:boto:Path: /wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.289 DEBUG:boto:Data: 2023-10-12T01:20:40.289 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.290 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.291 DEBUG:boto:Params: {} 2023-10-12T01:20:40.291 DEBUG:boto:Token: None 2023-10-12T01:20:40.291 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.292 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c923a50761e6880-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.295 DEBUG:boto:b'wafmoa-61obj761000falseobj762023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.296 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.297 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.297 DEBUG:boto:path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.297 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.298 DEBUG:boto:Method: GET 2023-10-12T01:20:40.298 DEBUG:boto:Path: /wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.298 DEBUG:boto:Data: 2023-10-12T01:20:40.299 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.300 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.300 DEBUG:boto:Params: {} 2023-10-12T01:20:40.300 DEBUG:boto:Token: None 2023-10-12T01:20:40.301 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040358e8bf76ddf47-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.307 DEBUG:boto:b'wafmoa-61obj761000falseobj762023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.307 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.308 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.308 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.308 DEBUG:boto:Path: /wafmoa-61/obj76 2023-10-12T01:20:40.309 DEBUG:boto:Data: 2023-10-12T01:20:40.309 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.310 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.310 DEBUG:boto:Params: {} 2023-10-12T01:20:40.311 DEBUG:boto:Token: None 2023-10-12T01:20:40.311 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76 2023-10-12T01:20:40.311 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PojPsPEuv9qvVtkWCAoRz9VDEZA= 2023-10-12T01:20:40.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PojPsPEuv9qvVtkWCAoRz9VDEZA=', 'Content-Length': '0'} 2023-10-12T01:20:40.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb122c6ce014b46-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.314 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.315 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.315 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.316 DEBUG:boto:Path: /wafmoa-61/obj76 2023-10-12T01:20:40.316 DEBUG:boto:Data: 2023-10-12T01:20:40.316 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.317 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.317 DEBUG:boto:Params: {} 2023-10-12T01:20:40.318 DEBUG:boto:Token: None 2023-10-12T01:20:40.318 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76 2023-10-12T01:20:40.319 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PojPsPEuv9qvVtkWCAoRz9VDEZA= 2023-10-12T01:20:40.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PojPsPEuv9qvVtkWCAoRz9VDEZA=', 'Content-Length': '0'} 2023-10-12T01:20:40.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e4d63c8dc893bf-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-12T01:20:40.322 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.322 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.323 DEBUG:boto:Method: GET 2023-10-12T01:20:40.323 DEBUG:boto:Path: /wafmoa-61/obj76 2023-10-12T01:20:40.324 DEBUG:boto:Data: 2023-10-12T01:20:40.324 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.325 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.325 DEBUG:boto:Params: {} 2023-10-12T01:20:40.326 DEBUG:boto:Token: None 2023-10-12T01:20:40.326 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76 2023-10-12T01:20:40.326 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zXSrZd3w4+WlDBo/SUhrdYXZ9A= 2023-10-12T01:20:40.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zXSrZd3w4+WlDBo/SUhrdYXZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:40.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1182ea110dca061-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.329 DEBUG:boto:path=/wafmoa-61/obj76 2023-10-12T01:20:40.330 DEBUG:boto:auth_path=/wafmoa-61/obj76 2023-10-12T01:20:40.330 DEBUG:boto:Method: GET 2023-10-12T01:20:40.331 DEBUG:boto:Path: /wafmoa-61/obj76 2023-10-12T01:20:40.331 DEBUG:boto:Data: 2023-10-12T01:20:40.331 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.332 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.333 DEBUG:boto:Params: {} 2023-10-12T01:20:40.333 DEBUG:boto:Token: None 2023-10-12T01:20:40.334 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj76 2023-10-12T01:20:40.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/zXSrZd3w4+WlDBo/SUhrdYXZ9A= 2023-10-12T01:20:40.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/zXSrZd3w4+WlDBo/SUhrdYXZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:40.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986116233a66f89b-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.382 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.383 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.383 DEBUG:boto:path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.384 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.384 DEBUG:boto:Method: GET 2023-10-12T01:20:40.385 DEBUG:boto:Path: /wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.385 DEBUG:boto:Data: 2023-10-12T01:20:40.385 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.386 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.387 DEBUG:boto:Params: {} 2023-10-12T01:20:40.387 DEBUG:boto:Token: None 2023-10-12T01:20:40.388 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.388 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c86f6cb8f4f82f2-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.392 DEBUG:boto:b'wafmoa-61obj761000falseobj762023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.393 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.393 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.394 DEBUG:boto:path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.394 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.395 DEBUG:boto:Method: GET 2023-10-12T01:20:40.395 DEBUG:boto:Path: /wafmoa-61/?prefix=obj76 2023-10-12T01:20:40.396 DEBUG:boto:Data: 2023-10-12T01:20:40.396 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.397 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.397 DEBUG:boto:Params: {} 2023-10-12T01:20:40.398 DEBUG:boto:Token: None 2023-10-12T01:20:40.398 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.399 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009745576735c7a120-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.404 DEBUG:boto:b'wafmoa-61obj761000falseobj762023-10-12T01:16:55.966Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:20:40.405 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.406 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.406 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.406 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.407 DEBUG:boto:Method: GET 2023-10-12T01:20:40.407 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.408 DEBUG:boto:Data: 2023-10-12T01:20:40.408 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.409 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.409 DEBUG:boto:Params: {} 2023-10-12T01:20:40.410 DEBUG:boto:Token: None 2023-10-12T01:20:40.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw= 2023-10-12T01:20:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw=', 'Content-Length': '0'} 2023-10-12T01:20:40.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae30b09f44772ad2-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.458 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.459 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.460 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.460 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.460 DEBUG:boto:Method: GET 2023-10-12T01:20:40.461 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.461 DEBUG:boto:Data: 2023-10-12T01:20:40.462 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.462 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.463 DEBUG:boto:Params: {} 2023-10-12T01:20:40.463 DEBUG:boto:Token: None 2023-10-12T01:20:40.464 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.464 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw= 2023-10-12T01:20:40.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw=', 'Content-Length': '0'} 2023-10-12T01:20:40.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed81d6f618aeecb-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.468 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.468 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.468 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.469 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.469 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.470 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.470 DEBUG:boto:Data: 2023-10-12T01:20:40.470 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.471 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.472 DEBUG:boto:Params: {} 2023-10-12T01:20:40.472 DEBUG:boto:Token: None 2023-10-12T01:20:40.473 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.473 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DSvZzC4bLl3pdr7YjD072ft5xck= 2023-10-12T01:20:40.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DSvZzC4bLl3pdr7YjD072ft5xck=', 'Content-Length': '0'} 2023-10-12T01:20:40.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d97754fced09b99-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.476 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.477 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.477 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.478 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.478 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.478 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.479 DEBUG:boto:Data: 2023-10-12T01:20:40.479 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.480 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.480 DEBUG:boto:Params: {} 2023-10-12T01:20:40.481 DEBUG:boto:Token: None 2023-10-12T01:20:40.481 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.482 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DSvZzC4bLl3pdr7YjD072ft5xck= 2023-10-12T01:20:40.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DSvZzC4bLl3pdr7YjD072ft5xck=', 'Content-Length': '0'} 2023-10-12T01:20:40.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9cb295a0af20c7-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:20:40.485 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.485 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.486 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.486 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.487 DEBUG:boto:Method: GET 2023-10-12T01:20:40.487 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.487 DEBUG:boto:Data: 2023-10-12T01:20:40.488 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.489 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.489 DEBUG:boto:Params: {} 2023-10-12T01:20:40.490 DEBUG:boto:Token: None 2023-10-12T01:20:40.490 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.490 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw= 2023-10-12T01:20:40.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw=', 'Content-Length': '0'} 2023-10-12T01:20:40.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9d49e7d64b834d-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.494 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.494 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.495 DEBUG:boto:path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.495 DEBUG:boto:auth_path=/wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.495 DEBUG:boto:Method: GET 2023-10-12T01:20:40.496 DEBUG:boto:Path: /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.496 DEBUG:boto:Data: 2023-10-12T01:20:40.497 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.497 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.498 DEBUG:boto:Params: {} 2023-10-12T01:20:40.498 DEBUG:boto:Token: None 2023-10-12T01:20:40.499 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85?versionId=null 2023-10-12T01:20:40.499 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw= 2023-10-12T01:20:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MYGSsit4lf91BM4UvYpDnR1BTsw=', 'Content-Length': '0'} 2023-10-12T01:20:40.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ab1a42ffe998f4-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.502 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.502 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.503 DEBUG:boto:path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.503 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.504 DEBUG:boto:Method: GET 2023-10-12T01:20:40.504 DEBUG:boto:Path: /wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.505 DEBUG:boto:Data: 2023-10-12T01:20:40.505 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.506 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.506 DEBUG:boto:Params: {} 2023-10-12T01:20:40.507 DEBUG:boto:Token: None 2023-10-12T01:20:40.507 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.508 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033bf2130433c21e5-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.511 DEBUG:boto:b'wafmoa-61obj851000falseobj852023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.512 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.512 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.513 DEBUG:boto:path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.513 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.514 DEBUG:boto:Method: GET 2023-10-12T01:20:40.514 DEBUG:boto:Path: /wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.515 DEBUG:boto:Data: 2023-10-12T01:20:40.515 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.516 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.516 DEBUG:boto:Params: {} 2023-10-12T01:20:40.517 DEBUG:boto:Token: None 2023-10-12T01:20:40.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.517 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2754efcdd57b47f-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.523 DEBUG:boto:b'wafmoa-61obj851000falseobj852023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.523 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.524 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.524 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.525 DEBUG:boto:Path: /wafmoa-61/obj85 2023-10-12T01:20:40.525 DEBUG:boto:Data: 2023-10-12T01:20:40.525 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.526 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.527 DEBUG:boto:Params: {} 2023-10-12T01:20:40.527 DEBUG:boto:Token: None 2023-10-12T01:20:40.527 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85 2023-10-12T01:20:40.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+kvYwRRMmh/jMkspVOz/L0SNVsY= 2023-10-12T01:20:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+kvYwRRMmh/jMkspVOz/L0SNVsY=', 'Content-Length': '0'} 2023-10-12T01:20:40.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca4d6a0497a452cc-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.531 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.531 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.532 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.532 DEBUG:boto:Path: /wafmoa-61/obj85 2023-10-12T01:20:40.532 DEBUG:boto:Data: 2023-10-12T01:20:40.533 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.534 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.534 DEBUG:boto:Params: {} 2023-10-12T01:20:40.535 DEBUG:boto:Token: None 2023-10-12T01:20:40.535 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85 2023-10-12T01:20:40.535 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+kvYwRRMmh/jMkspVOz/L0SNVsY= 2023-10-12T01:20:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+kvYwRRMmh/jMkspVOz/L0SNVsY=', 'Content-Length': '0'} 2023-10-12T01:20:40.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001367c287b48a28de-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-12T01:20:40.539 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.540 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.540 DEBUG:boto:Method: GET 2023-10-12T01:20:40.540 DEBUG:boto:Path: /wafmoa-61/obj85 2023-10-12T01:20:40.541 DEBUG:boto:Data: 2023-10-12T01:20:40.541 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.542 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.542 DEBUG:boto:Params: {} 2023-10-12T01:20:40.543 DEBUG:boto:Token: None 2023-10-12T01:20:40.543 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85 2023-10-12T01:20:40.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9XZ5oBnHdn9i8vGlf3+omyrPurM= 2023-10-12T01:20:40.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9XZ5oBnHdn9i8vGlf3+omyrPurM=', 'Content-Length': '0'} 2023-10-12T01:20:40.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c72ee5ec97005d-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.590 DEBUG:boto:path=/wafmoa-61/obj85 2023-10-12T01:20:40.591 DEBUG:boto:auth_path=/wafmoa-61/obj85 2023-10-12T01:20:40.591 DEBUG:boto:Method: GET 2023-10-12T01:20:40.591 DEBUG:boto:Path: /wafmoa-61/obj85 2023-10-12T01:20:40.592 DEBUG:boto:Data: 2023-10-12T01:20:40.592 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.593 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.593 DEBUG:boto:Params: {} 2023-10-12T01:20:40.594 DEBUG:boto:Token: None 2023-10-12T01:20:40.594 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj85 2023-10-12T01:20:40.595 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9XZ5oBnHdn9i8vGlf3+omyrPurM= 2023-10-12T01:20:40.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9XZ5oBnHdn9i8vGlf3+omyrPurM=', 'Content-Length': '0'} 2023-10-12T01:20:40.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e652efd004b48083-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.598 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.599 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.599 DEBUG:boto:path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.599 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.600 DEBUG:boto:Method: GET 2023-10-12T01:20:40.600 DEBUG:boto:Path: /wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.601 DEBUG:boto:Data: 2023-10-12T01:20:40.601 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.602 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.603 DEBUG:boto:Params: {} 2023-10-12T01:20:40.604 DEBUG:boto:Token: None 2023-10-12T01:20:40.604 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.605 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004f8dea849e09b4f-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.610 DEBUG:boto:b'wafmoa-61obj851000falseobj852023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.610 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.611 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.611 DEBUG:boto:path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.612 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.612 DEBUG:boto:Method: GET 2023-10-12T01:20:40.613 DEBUG:boto:Path: /wafmoa-61/?prefix=obj85 2023-10-12T01:20:40.613 DEBUG:boto:Data: 2023-10-12T01:20:40.613 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.614 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.615 DEBUG:boto:Params: {} 2023-10-12T01:20:40.615 DEBUG:boto:Token: None 2023-10-12T01:20:40.616 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.616 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3348b4390b388e-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.621 DEBUG:boto:b'wafmoa-61obj851000falseobj852023-10-12T01:16:55.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:20:40.622 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.623 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.623 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.623 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.624 DEBUG:boto:Method: GET 2023-10-12T01:20:40.624 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.625 DEBUG:boto:Data: 2023-10-12T01:20:40.625 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.626 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.626 DEBUG:boto:Params: {} 2023-10-12T01:20:40.627 DEBUG:boto:Token: None 2023-10-12T01:20:40.627 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.628 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4= 2023-10-12T01:20:40.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4=', 'Content-Length': '0'} 2023-10-12T01:20:40.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4033c43f0c9a19d-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.631 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.632 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.632 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.633 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.633 DEBUG:boto:Method: GET 2023-10-12T01:20:40.633 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.634 DEBUG:boto:Data: 2023-10-12T01:20:40.634 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.635 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.636 DEBUG:boto:Params: {} 2023-10-12T01:20:40.636 DEBUG:boto:Token: None 2023-10-12T01:20:40.637 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.637 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4= 2023-10-12T01:20:40.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4=', 'Content-Length': '0'} 2023-10-12T01:20:40.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b636806c8f7b6cf4-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.641 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.641 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.642 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.642 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.642 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.643 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.643 DEBUG:boto:Data: 2023-10-12T01:20:40.644 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.644 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.645 DEBUG:boto:Params: {} 2023-10-12T01:20:40.645 DEBUG:boto:Token: None 2023-10-12T01:20:40.646 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.646 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dErUnESQ9kDh3ohQoJO66lu00Pw= 2023-10-12T01:20:40.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dErUnESQ9kDh3ohQoJO66lu00Pw=', 'Content-Length': '0'} 2023-10-12T01:20:40.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751f4f1b6089b0d4-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.650 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.650 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.650 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.651 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.651 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.652 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.652 DEBUG:boto:Data: 2023-10-12T01:20:40.653 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.653 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.654 DEBUG:boto:Params: {} 2023-10-12T01:20:40.654 DEBUG:boto:Token: None 2023-10-12T01:20:40.655 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.655 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dErUnESQ9kDh3ohQoJO66lu00Pw= 2023-10-12T01:20:40.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dErUnESQ9kDh3ohQoJO66lu00Pw=', 'Content-Length': '0'} 2023-10-12T01:20:40.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007074674e12848223-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:20:40.659 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.659 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.660 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.660 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.661 DEBUG:boto:Method: GET 2023-10-12T01:20:40.661 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.662 DEBUG:boto:Data: 2023-10-12T01:20:40.662 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.663 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.663 DEBUG:boto:Params: {} 2023-10-12T01:20:40.664 DEBUG:boto:Token: None 2023-10-12T01:20:40.664 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.665 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4= 2023-10-12T01:20:40.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4=', 'Content-Length': '0'} 2023-10-12T01:20:40.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010904ff4a2a18cd0-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.668 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.668 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.668 DEBUG:boto:path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.669 DEBUG:boto:auth_path=/wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.669 DEBUG:boto:Method: GET 2023-10-12T01:20:40.670 DEBUG:boto:Path: /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.670 DEBUG:boto:Data: 2023-10-12T01:20:40.671 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.672 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.672 DEBUG:boto:Params: {} 2023-10-12T01:20:40.673 DEBUG:boto:Token: None 2023-10-12T01:20:40.673 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87?versionId=null 2023-10-12T01:20:40.674 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4= 2023-10-12T01:20:40.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:MhxfQxeFIYfZih1xE+i9gOsm1T4=', 'Content-Length': '0'} 2023-10-12T01:20:40.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003478162d1e10291f-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.677 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.677 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.678 DEBUG:boto:path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.678 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.679 DEBUG:boto:Method: GET 2023-10-12T01:20:40.679 DEBUG:boto:Path: /wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.679 DEBUG:boto:Data: 2023-10-12T01:20:40.680 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.681 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.681 DEBUG:boto:Params: {} 2023-10-12T01:20:40.682 DEBUG:boto:Token: None 2023-10-12T01:20:40.682 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7fe8ab2d090289a-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.726 DEBUG:boto:b'wafmoa-61obj871000falseobj872023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.727 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.728 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.728 DEBUG:boto:path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.728 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.729 DEBUG:boto:Method: GET 2023-10-12T01:20:40.729 DEBUG:boto:Path: /wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.730 DEBUG:boto:Data: 2023-10-12T01:20:40.730 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.731 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.731 DEBUG:boto:Params: {} 2023-10-12T01:20:40.732 DEBUG:boto:Token: None 2023-10-12T01:20:40.732 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.733 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b2b27c3500ccaa-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.738 DEBUG:boto:b'wafmoa-61obj871000falseobj872023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.739 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.739 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.740 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.740 DEBUG:boto:Path: /wafmoa-61/obj87 2023-10-12T01:20:40.741 DEBUG:boto:Data: 2023-10-12T01:20:40.741 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.742 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.742 DEBUG:boto:Params: {} 2023-10-12T01:20:40.743 DEBUG:boto:Token: None 2023-10-12T01:20:40.743 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87 2023-10-12T01:20:40.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vju/bL5tse3OnAcICJh+EXZGeUw= 2023-10-12T01:20:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vju/bL5tse3OnAcICJh+EXZGeUw=', 'Content-Length': '0'} 2023-10-12T01:20:40.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034d4dfae33ffad6e-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.747 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.747 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.748 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.748 DEBUG:boto:Path: /wafmoa-61/obj87 2023-10-12T01:20:40.748 DEBUG:boto:Data: 2023-10-12T01:20:40.749 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.749 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.750 DEBUG:boto:Params: {} 2023-10-12T01:20:40.750 DEBUG:boto:Token: None 2023-10-12T01:20:40.751 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87 2023-10-12T01:20:40.751 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vju/bL5tse3OnAcICJh+EXZGeUw= 2023-10-12T01:20:40.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vju/bL5tse3OnAcICJh+EXZGeUw=', 'Content-Length': '0'} 2023-10-12T01:20:40.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b969e420c9f463-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-12T01:20:40.754 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.755 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.755 DEBUG:boto:Method: GET 2023-10-12T01:20:40.756 DEBUG:boto:Path: /wafmoa-61/obj87 2023-10-12T01:20:40.756 DEBUG:boto:Data: 2023-10-12T01:20:40.756 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.757 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.758 DEBUG:boto:Params: {} 2023-10-12T01:20:40.758 DEBUG:boto:Token: None 2023-10-12T01:20:40.759 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87 2023-10-12T01:20:40.759 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9VQQdTzpN01MUNq3vi67a+eVkE= 2023-10-12T01:20:40.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9VQQdTzpN01MUNq3vi67a+eVkE=', 'Content-Length': '0'} 2023-10-12T01:20:40.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a2443de9d7319a-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.762 DEBUG:boto:path=/wafmoa-61/obj87 2023-10-12T01:20:40.762 DEBUG:boto:auth_path=/wafmoa-61/obj87 2023-10-12T01:20:40.763 DEBUG:boto:Method: GET 2023-10-12T01:20:40.763 DEBUG:boto:Path: /wafmoa-61/obj87 2023-10-12T01:20:40.764 DEBUG:boto:Data: 2023-10-12T01:20:40.764 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.765 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.765 DEBUG:boto:Params: {} 2023-10-12T01:20:40.766 DEBUG:boto:Token: None 2023-10-12T01:20:40.766 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj87 2023-10-12T01:20:40.767 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r9VQQdTzpN01MUNq3vi67a+eVkE= 2023-10-12T01:20:40.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r9VQQdTzpN01MUNq3vi67a+eVkE=', 'Content-Length': '0'} 2023-10-12T01:20:40.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679f8757166056ae-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.770 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.770 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.771 DEBUG:boto:path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.771 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.771 DEBUG:boto:Method: GET 2023-10-12T01:20:40.772 DEBUG:boto:Path: /wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.772 DEBUG:boto:Data: 2023-10-12T01:20:40.773 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.774 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.774 DEBUG:boto:Params: {} 2023-10-12T01:20:40.775 DEBUG:boto:Token: None 2023-10-12T01:20:40.775 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.776 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a83488eb835c91-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.779 DEBUG:boto:b'wafmoa-61obj871000falseobj872023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.780 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.780 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.780 DEBUG:boto:path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.781 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.781 DEBUG:boto:Method: GET 2023-10-12T01:20:40.781 DEBUG:boto:Path: /wafmoa-61/?prefix=obj87 2023-10-12T01:20:40.782 DEBUG:boto:Data: 2023-10-12T01:20:40.782 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.783 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.784 DEBUG:boto:Params: {} 2023-10-12T01:20:40.784 DEBUG:boto:Token: None 2023-10-12T01:20:40.785 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a381ea8ac25e75-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.791 DEBUG:boto:b'wafmoa-61obj871000falseobj872023-10-12T01:16:56.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:20:40.792 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.792 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.792 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.793 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.793 DEBUG:boto:Method: GET 2023-10-12T01:20:40.794 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.794 DEBUG:boto:Data: 2023-10-12T01:20:40.794 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.795 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.796 DEBUG:boto:Params: {} 2023-10-12T01:20:40.796 DEBUG:boto:Token: None 2023-10-12T01:20:40.797 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.797 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II= 2023-10-12T01:20:40.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II=', 'Content-Length': '0'} 2023-10-12T01:20:40.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2143783b279eab-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.800 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.801 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.801 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.802 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.802 DEBUG:boto:Method: GET 2023-10-12T01:20:40.802 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.803 DEBUG:boto:Data: 2023-10-12T01:20:40.803 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.804 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.805 DEBUG:boto:Params: {} 2023-10-12T01:20:40.805 DEBUG:boto:Token: None 2023-10-12T01:20:40.805 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.806 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II= 2023-10-12T01:20:40.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II=', 'Content-Length': '0'} 2023-10-12T01:20:40.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f215940708c210-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.809 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.810 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.810 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.810 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.811 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.811 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.812 DEBUG:boto:Data: 2023-10-12T01:20:40.812 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.813 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.813 DEBUG:boto:Params: {} 2023-10-12T01:20:40.813 DEBUG:boto:Token: None 2023-10-12T01:20:40.814 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.814 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XR68oKJ0QDppShyo7qdvH5IPV4A= 2023-10-12T01:20:40.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XR68oKJ0QDppShyo7qdvH5IPV4A=', 'Content-Length': '0'} 2023-10-12T01:20:40.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62d6d41480c8f6e-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.818 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.818 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.819 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.819 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.819 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.820 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.820 DEBUG:boto:Data: 2023-10-12T01:20:40.820 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.821 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.822 DEBUG:boto:Params: {} 2023-10-12T01:20:40.822 DEBUG:boto:Token: None 2023-10-12T01:20:40.822 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.823 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XR68oKJ0QDppShyo7qdvH5IPV4A= 2023-10-12T01:20:40.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XR68oKJ0QDppShyo7qdvH5IPV4A=', 'Content-Length': '0'} 2023-10-12T01:20:40.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c75307d11b15027-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:20:40.827 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.827 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.827 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.828 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.828 DEBUG:boto:Method: GET 2023-10-12T01:20:40.828 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.829 DEBUG:boto:Data: 2023-10-12T01:20:40.829 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.830 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.830 DEBUG:boto:Params: {} 2023-10-12T01:20:40.831 DEBUG:boto:Token: None 2023-10-12T01:20:40.831 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II= 2023-10-12T01:20:40.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II=', 'Content-Length': '0'} 2023-10-12T01:20:40.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead81bb8f8ab7180-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.835 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.835 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.836 DEBUG:boto:path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.836 DEBUG:boto:auth_path=/wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.837 DEBUG:boto:Method: GET 2023-10-12T01:20:40.837 DEBUG:boto:Path: /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.837 DEBUG:boto:Data: 2023-10-12T01:20:40.838 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.839 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.839 DEBUG:boto:Params: {} 2023-10-12T01:20:40.840 DEBUG:boto:Token: None 2023-10-12T01:20:40.840 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92?versionId=null 2023-10-12T01:20:40.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II= 2023-10-12T01:20:40.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TEB3ZDePGtm1qOC/jq66PpBR5II=', 'Content-Length': '0'} 2023-10-12T01:20:40.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dd18a895876c87b-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.844 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.844 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.845 DEBUG:boto:path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.845 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.845 DEBUG:boto:Method: GET 2023-10-12T01:20:40.846 DEBUG:boto:Path: /wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.846 DEBUG:boto:Data: 2023-10-12T01:20:40.847 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.848 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.848 DEBUG:boto:Params: {} 2023-10-12T01:20:40.849 DEBUG:boto:Token: None 2023-10-12T01:20:40.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.851 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074b1e940c5cdbe9-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.898 DEBUG:boto:b'wafmoa-61obj921000falseobj922023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.899 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.900 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.900 DEBUG:boto:path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.901 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.901 DEBUG:boto:Method: GET 2023-10-12T01:20:40.901 DEBUG:boto:Path: /wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.902 DEBUG:boto:Data: 2023-10-12T01:20:40.902 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.903 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.904 DEBUG:boto:Params: {} 2023-10-12T01:20:40.904 DEBUG:boto:Token: None 2023-10-12T01:20:40.905 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.905 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092fe5d22a731f5ca-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.910 DEBUG:boto:b'wafmoa-61obj921000falseobj922023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.911 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.911 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.912 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.912 DEBUG:boto:Path: /wafmoa-61/obj92 2023-10-12T01:20:40.913 DEBUG:boto:Data: 2023-10-12T01:20:40.913 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.914 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.914 DEBUG:boto:Params: {} 2023-10-12T01:20:40.915 DEBUG:boto:Token: None 2023-10-12T01:20:40.915 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92 2023-10-12T01:20:40.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Hn+muCxuMzLt6IXnmHNceb8BH4= 2023-10-12T01:20:40.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Hn+muCxuMzLt6IXnmHNceb8BH4=', 'Content-Length': '0'} 2023-10-12T01:20:40.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056383b7a32b23ff2-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.919 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.919 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.920 DEBUG:boto:Method: HEAD 2023-10-12T01:20:40.920 DEBUG:boto:Path: /wafmoa-61/obj92 2023-10-12T01:20:40.921 DEBUG:boto:Data: 2023-10-12T01:20:40.921 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.922 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.922 DEBUG:boto:Params: {} 2023-10-12T01:20:40.923 DEBUG:boto:Token: None 2023-10-12T01:20:40.923 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92 2023-10-12T01:20:40.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4Hn+muCxuMzLt6IXnmHNceb8BH4= 2023-10-12T01:20:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4Hn+muCxuMzLt6IXnmHNceb8BH4=', 'Content-Length': '0'} 2023-10-12T01:20:40.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ff36d854d6aa3c-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-12T01:20:40.927 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.928 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.928 DEBUG:boto:Method: GET 2023-10-12T01:20:40.928 DEBUG:boto:Path: /wafmoa-61/obj92 2023-10-12T01:20:40.929 DEBUG:boto:Data: 2023-10-12T01:20:40.929 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.930 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.931 DEBUG:boto:Params: {} 2023-10-12T01:20:40.931 DEBUG:boto:Token: None 2023-10-12T01:20:40.931 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92 2023-10-12T01:20:40.932 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lfwpeuDv4Na1t43WMntqKMXB+Yc= 2023-10-12T01:20:40.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lfwpeuDv4Na1t43WMntqKMXB+Yc=', 'Content-Length': '0'} 2023-10-12T01:20:40.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83fe9c9571e28de-00652749e8-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.935 DEBUG:boto:path=/wafmoa-61/obj92 2023-10-12T01:20:40.936 DEBUG:boto:auth_path=/wafmoa-61/obj92 2023-10-12T01:20:40.936 DEBUG:boto:Method: GET 2023-10-12T01:20:40.936 DEBUG:boto:Path: /wafmoa-61/obj92 2023-10-12T01:20:40.937 DEBUG:boto:Data: 2023-10-12T01:20:40.937 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.938 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.939 DEBUG:boto:Params: {} 2023-10-12T01:20:40.939 DEBUG:boto:Token: None 2023-10-12T01:20:40.940 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/obj92 2023-10-12T01:20:40.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lfwpeuDv4Na1t43WMntqKMXB+Yc= 2023-10-12T01:20:40.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lfwpeuDv4Na1t43WMntqKMXB+Yc=', 'Content-Length': '0'} 2023-10-12T01:20:40.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680506cf23f197c6-00652749e8-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.944 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.944 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.945 DEBUG:boto:path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.945 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.946 DEBUG:boto:Method: GET 2023-10-12T01:20:40.946 DEBUG:boto:Path: /wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.947 DEBUG:boto:Data: 2023-10-12T01:20:40.947 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:40.948 DEBUG:boto:Port: 8001 2023-10-12T01:20:40.948 DEBUG:boto:Params: {} 2023-10-12T01:20:40.949 DEBUG:boto:Token: None 2023-10-12T01:20:40.949 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.950 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fccdbb605cd431-00652749e8-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:40.953 DEBUG:boto:b'wafmoa-61obj921000falseobj922023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:40.954 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:40.955 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:40.955 DEBUG:boto:path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.956 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.956 DEBUG:boto:Method: GET 2023-10-12T01:20:40.956 DEBUG:boto:Path: /wafmoa-61/?prefix=obj92 2023-10-12T01:20:40.957 DEBUG:boto:Data: 2023-10-12T01:20:40.957 DEBUG:boto:Headers: {} 2023-10-12T01:20:40.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:40.958 DEBUG:boto:Port: 8000 2023-10-12T01:20:40.958 DEBUG:boto:Params: {} 2023-10-12T01:20:40.959 DEBUG:boto:Token: None 2023-10-12T01:20:40.959 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:40 GMT /wafmoa-61/ 2023-10-12T01:20:40.960 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw= 2023-10-12T01:20:40.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:40 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/TuosfaVOJhIfPcS7cGePfhRZgw=', 'Content-Length': '0'} 2023-10-12T01:20:40.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff66c32169cf6c9-00652749e8-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.006 DEBUG:boto:b'wafmoa-61obj921000falseobj922023-10-12T01:16:56.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:20:41.008 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.008 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.008 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.009 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.009 DEBUG:boto:Method: GET 2023-10-12T01:20:41.010 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.010 DEBUG:boto:Data: 2023-10-12T01:20:41.010 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.011 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.012 DEBUG:boto:Params: {} 2023-10-12T01:20:41.012 DEBUG:boto:Token: None 2023-10-12T01:20:41.013 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.013 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A= 2023-10-12T01:20:41.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:41.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc7656ca48a78b4-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.017 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.017 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.017 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.018 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.018 DEBUG:boto:Method: GET 2023-10-12T01:20:41.019 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.019 DEBUG:boto:Data: 2023-10-12T01:20:41.019 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.020 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.021 DEBUG:boto:Params: {} 2023-10-12T01:20:41.021 DEBUG:boto:Token: None 2023-10-12T01:20:41.022 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.022 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A= 2023-10-12T01:20:41.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:41.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa614d002a8c5d3-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.025 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.026 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.026 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.026 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.027 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.027 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.028 DEBUG:boto:Data: 2023-10-12T01:20:41.028 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.029 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.029 DEBUG:boto:Params: {} 2023-10-12T01:20:41.029 DEBUG:boto:Token: None 2023-10-12T01:20:41.030 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.030 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nyWvamVtY6cPcbSlTyEeymzLr/M= 2023-10-12T01:20:41.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nyWvamVtY6cPcbSlTyEeymzLr/M=', 'Content-Length': '0'} 2023-10-12T01:20:41.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019404aab32cdc0d-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.033 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.034 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.034 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.034 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.035 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.035 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.035 DEBUG:boto:Data: 2023-10-12T01:20:41.036 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.037 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.037 DEBUG:boto:Params: {} 2023-10-12T01:20:41.037 DEBUG:boto:Token: None 2023-10-12T01:20:41.038 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.038 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nyWvamVtY6cPcbSlTyEeymzLr/M= 2023-10-12T01:20:41.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nyWvamVtY6cPcbSlTyEeymzLr/M=', 'Content-Length': '0'} 2023-10-12T01:20:41.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0534b5a37c7821-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:20:41.042 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.042 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.042 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.043 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.043 DEBUG:boto:Method: GET 2023-10-12T01:20:41.043 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.044 DEBUG:boto:Data: 2023-10-12T01:20:41.044 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.045 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.045 DEBUG:boto:Params: {} 2023-10-12T01:20:41.046 DEBUG:boto:Token: None 2023-10-12T01:20:41.046 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A= 2023-10-12T01:20:41.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:41.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc4513a9c522131-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.050 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.050 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.051 DEBUG:boto:path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.051 DEBUG:boto:auth_path=/wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.052 DEBUG:boto:Method: GET 2023-10-12T01:20:41.052 DEBUG:boto:Path: /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.052 DEBUG:boto:Data: 2023-10-12T01:20:41.053 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.054 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.054 DEBUG:boto:Params: {} 2023-10-12T01:20:41.055 DEBUG:boto:Token: None 2023-10-12T01:20:41.055 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93?versionId=null 2023-10-12T01:20:41.055 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A= 2023-10-12T01:20:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nFfkSbraydlVfQqe2CE9GdPbZ9A=', 'Content-Length': '0'} 2023-10-12T01:20:41.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7028bc27470b7e-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.058 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.059 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.059 DEBUG:boto:path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.060 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.060 DEBUG:boto:Method: GET 2023-10-12T01:20:41.060 DEBUG:boto:Path: /wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.061 DEBUG:boto:Data: 2023-10-12T01:20:41.061 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.062 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.062 DEBUG:boto:Params: {} 2023-10-12T01:20:41.063 DEBUG:boto:Token: None 2023-10-12T01:20:41.063 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.064 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810eb10fd2dc6ce6-00652749e9-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.067 DEBUG:boto:b'wafmoa-61obj931000falseobj932023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.068 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.068 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.069 DEBUG:boto:path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.069 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.070 DEBUG:boto:Method: GET 2023-10-12T01:20:41.070 DEBUG:boto:Path: /wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.070 DEBUG:boto:Data: 2023-10-12T01:20:41.071 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.072 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.072 DEBUG:boto:Params: {} 2023-10-12T01:20:41.073 DEBUG:boto:Token: None 2023-10-12T01:20:41.073 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.073 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c879f4a35b737e3d-00652749e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.079 DEBUG:boto:b'wafmoa-61obj931000falseobj932023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.079 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.080 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.080 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.080 DEBUG:boto:Path: /wafmoa-61/obj93 2023-10-12T01:20:41.081 DEBUG:boto:Data: 2023-10-12T01:20:41.081 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.082 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.082 DEBUG:boto:Params: {} 2023-10-12T01:20:41.083 DEBUG:boto:Token: None 2023-10-12T01:20:41.083 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93 2023-10-12T01:20:41.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ngL78WGisZzOTW9Zob/qFWl+IwY= 2023-10-12T01:20:41.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ngL78WGisZzOTW9Zob/qFWl+IwY=', 'Content-Length': '0'} 2023-10-12T01:20:41.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bdcee9e9bfac261-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.087 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.087 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.088 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.088 DEBUG:boto:Path: /wafmoa-61/obj93 2023-10-12T01:20:41.088 DEBUG:boto:Data: 2023-10-12T01:20:41.089 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.089 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.090 DEBUG:boto:Params: {} 2023-10-12T01:20:41.090 DEBUG:boto:Token: None 2023-10-12T01:20:41.091 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93 2023-10-12T01:20:41.091 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ngL78WGisZzOTW9Zob/qFWl+IwY= 2023-10-12T01:20:41.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ngL78WGisZzOTW9Zob/qFWl+IwY=', 'Content-Length': '0'} 2023-10-12T01:20:41.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d910bfc65105e86-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-12T01:20:41.095 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.095 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.096 DEBUG:boto:Method: GET 2023-10-12T01:20:41.096 DEBUG:boto:Path: /wafmoa-61/obj93 2023-10-12T01:20:41.097 DEBUG:boto:Data: 2023-10-12T01:20:41.097 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.098 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.098 DEBUG:boto:Params: {} 2023-10-12T01:20:41.099 DEBUG:boto:Token: None 2023-10-12T01:20:41.099 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93 2023-10-12T01:20:41.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4YX0kXK/aYBzAkfgWn47IAAX7Kw= 2023-10-12T01:20:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4YX0kXK/aYBzAkfgWn47IAAX7Kw=', 'Content-Length': '0'} 2023-10-12T01:20:41.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085df6c2886f05211-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.150 DEBUG:boto:path=/wafmoa-61/obj93 2023-10-12T01:20:41.151 DEBUG:boto:auth_path=/wafmoa-61/obj93 2023-10-12T01:20:41.151 DEBUG:boto:Method: GET 2023-10-12T01:20:41.152 DEBUG:boto:Path: /wafmoa-61/obj93 2023-10-12T01:20:41.152 DEBUG:boto:Data: 2023-10-12T01:20:41.153 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.153 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.154 DEBUG:boto:Params: {} 2023-10-12T01:20:41.154 DEBUG:boto:Token: None 2023-10-12T01:20:41.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj93 2023-10-12T01:20:41.155 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4YX0kXK/aYBzAkfgWn47IAAX7Kw= 2023-10-12T01:20:41.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4YX0kXK/aYBzAkfgWn47IAAX7Kw=', 'Content-Length': '0'} 2023-10-12T01:20:41.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1aea26a1999e3ac-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.159 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.159 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.160 DEBUG:boto:path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.160 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.161 DEBUG:boto:Method: GET 2023-10-12T01:20:41.161 DEBUG:boto:Path: /wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.162 DEBUG:boto:Data: 2023-10-12T01:20:41.162 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.163 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.163 DEBUG:boto:Params: {} 2023-10-12T01:20:41.164 DEBUG:boto:Token: None 2023-10-12T01:20:41.164 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ac34ed82e092df-00652749e9-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.168 DEBUG:boto:b'wafmoa-61obj931000falseobj932023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.169 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.170 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.170 DEBUG:boto:path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.170 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.171 DEBUG:boto:Method: GET 2023-10-12T01:20:41.171 DEBUG:boto:Path: /wafmoa-61/?prefix=obj93 2023-10-12T01:20:41.172 DEBUG:boto:Data: 2023-10-12T01:20:41.172 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.173 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.174 DEBUG:boto:Params: {} 2023-10-12T01:20:41.174 DEBUG:boto:Token: None 2023-10-12T01:20:41.175 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.175 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051369dca6c8995ef-00652749e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.222 DEBUG:boto:b'wafmoa-61obj931000falseobj932023-10-12T01:16:56.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:20:41.224 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.224 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.224 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.225 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.225 DEBUG:boto:Method: GET 2023-10-12T01:20:41.226 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.226 DEBUG:boto:Data: 2023-10-12T01:20:41.227 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.227 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.228 DEBUG:boto:Params: {} 2023-10-12T01:20:41.228 DEBUG:boto:Token: None 2023-10-12T01:20:41.229 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk= 2023-10-12T01:20:41.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk=', 'Content-Length': '0'} 2023-10-12T01:20:41.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c62121423b18d0-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.233 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.233 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.234 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.234 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.235 DEBUG:boto:Method: GET 2023-10-12T01:20:41.235 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.235 DEBUG:boto:Data: 2023-10-12T01:20:41.236 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.237 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.237 DEBUG:boto:Params: {} 2023-10-12T01:20:41.237 DEBUG:boto:Token: None 2023-10-12T01:20:41.238 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.238 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk= 2023-10-12T01:20:41.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk=', 'Content-Length': '0'} 2023-10-12T01:20:41.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc241645f9d7a41e-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.242 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.242 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.243 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.243 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.244 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.244 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.244 DEBUG:boto:Data: 2023-10-12T01:20:41.245 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.245 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.246 DEBUG:boto:Params: {} 2023-10-12T01:20:41.246 DEBUG:boto:Token: None 2023-10-12T01:20:41.247 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.247 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A78bSIBJ1HxbhMR+TscLqFTZ57Y= 2023-10-12T01:20:41.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A78bSIBJ1HxbhMR+TscLqFTZ57Y=', 'Content-Length': '0'} 2023-10-12T01:20:41.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002feccad4863e1361-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.250 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.250 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.251 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.251 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.252 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.252 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.252 DEBUG:boto:Data: 2023-10-12T01:20:41.253 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.254 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.254 DEBUG:boto:Params: {} 2023-10-12T01:20:41.255 DEBUG:boto:Token: None 2023-10-12T01:20:41.255 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.255 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:A78bSIBJ1HxbhMR+TscLqFTZ57Y= 2023-10-12T01:20:41.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:A78bSIBJ1HxbhMR+TscLqFTZ57Y=', 'Content-Length': '0'} 2023-10-12T01:20:41.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c15c3614149d78-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:20:41.259 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.259 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.260 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.260 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.261 DEBUG:boto:Method: GET 2023-10-12T01:20:41.261 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.261 DEBUG:boto:Data: 2023-10-12T01:20:41.262 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.263 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.263 DEBUG:boto:Params: {} 2023-10-12T01:20:41.263 DEBUG:boto:Token: None 2023-10-12T01:20:41.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk= 2023-10-12T01:20:41.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk=', 'Content-Length': '0'} 2023-10-12T01:20:41.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e0ce25fe7c71c7-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.267 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.267 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.268 DEBUG:boto:path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.268 DEBUG:boto:auth_path=/wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.269 DEBUG:boto:Method: GET 2023-10-12T01:20:41.269 DEBUG:boto:Path: /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.270 DEBUG:boto:Data: 2023-10-12T01:20:41.270 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.271 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.271 DEBUG:boto:Params: {} 2023-10-12T01:20:41.272 DEBUG:boto:Token: None 2023-10-12T01:20:41.272 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95?versionId=null 2023-10-12T01:20:41.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk= 2023-10-12T01:20:41.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mBtwb4n3/pguA5JWnRXSNsLAiQk=', 'Content-Length': '0'} 2023-10-12T01:20:41.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac8fd6297c344be-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.276 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.276 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.277 DEBUG:boto:path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.277 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.278 DEBUG:boto:Method: GET 2023-10-12T01:20:41.278 DEBUG:boto:Path: /wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.279 DEBUG:boto:Data: 2023-10-12T01:20:41.279 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.280 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.280 DEBUG:boto:Params: {} 2023-10-12T01:20:41.281 DEBUG:boto:Token: None 2023-10-12T01:20:41.281 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000220ab8b5d39eeb2d-00652749e9-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.285 DEBUG:boto:b'wafmoa-61obj951000falseobj952023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.285 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.286 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.286 DEBUG:boto:path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.287 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.287 DEBUG:boto:Method: GET 2023-10-12T01:20:41.287 DEBUG:boto:Path: /wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.288 DEBUG:boto:Data: 2023-10-12T01:20:41.288 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.289 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.289 DEBUG:boto:Params: {} 2023-10-12T01:20:41.290 DEBUG:boto:Token: None 2023-10-12T01:20:41.290 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.291 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a8c5836bdd7374-00652749e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.296 DEBUG:boto:b'wafmoa-61obj951000falseobj952023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.296 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.297 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.297 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.297 DEBUG:boto:Path: /wafmoa-61/obj95 2023-10-12T01:20:41.298 DEBUG:boto:Data: 2023-10-12T01:20:41.298 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.299 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.299 DEBUG:boto:Params: {} 2023-10-12T01:20:41.300 DEBUG:boto:Token: None 2023-10-12T01:20:41.300 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95 2023-10-12T01:20:41.301 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NmGbc6JPdu733diypPG5smZ4GaM= 2023-10-12T01:20:41.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NmGbc6JPdu733diypPG5smZ4GaM=', 'Content-Length': '0'} 2023-10-12T01:20:41.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005215ae9b529f663e-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.303 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.304 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.304 DEBUG:boto:Method: HEAD 2023-10-12T01:20:41.304 DEBUG:boto:Path: /wafmoa-61/obj95 2023-10-12T01:20:41.305 DEBUG:boto:Data: 2023-10-12T01:20:41.305 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.306 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.306 DEBUG:boto:Params: {} 2023-10-12T01:20:41.307 DEBUG:boto:Token: None 2023-10-12T01:20:41.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95 2023-10-12T01:20:41.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NmGbc6JPdu733diypPG5smZ4GaM= 2023-10-12T01:20:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NmGbc6JPdu733diypPG5smZ4GaM=', 'Content-Length': '0'} 2023-10-12T01:20:41.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c7e4a2e3c7749c2-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-12T01:20:41.311 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.312 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.312 DEBUG:boto:Method: GET 2023-10-12T01:20:41.312 DEBUG:boto:Path: /wafmoa-61/obj95 2023-10-12T01:20:41.313 DEBUG:boto:Data: 2023-10-12T01:20:41.313 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.314 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.314 DEBUG:boto:Params: {} 2023-10-12T01:20:41.315 DEBUG:boto:Token: None 2023-10-12T01:20:41.315 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95 2023-10-12T01:20:41.316 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1GzrervARTeJZfIIee1pgPj1PY8= 2023-10-12T01:20:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1GzrervARTeJZfIIee1pgPj1PY8=', 'Content-Length': '0'} 2023-10-12T01:20:41.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-61:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591e22f7ae9c65b2-00652749e9-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.358 DEBUG:boto:path=/wafmoa-61/obj95 2023-10-12T01:20:41.359 DEBUG:boto:auth_path=/wafmoa-61/obj95 2023-10-12T01:20:41.359 DEBUG:boto:Method: GET 2023-10-12T01:20:41.360 DEBUG:boto:Path: /wafmoa-61/obj95 2023-10-12T01:20:41.360 DEBUG:boto:Data: 2023-10-12T01:20:41.361 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.362 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.362 DEBUG:boto:Params: {} 2023-10-12T01:20:41.362 DEBUG:boto:Token: None 2023-10-12T01:20:41.363 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/obj95 2023-10-12T01:20:41.363 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:1GzrervARTeJZfIIee1pgPj1PY8= 2023-10-12T01:20:41.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:1GzrervARTeJZfIIee1pgPj1PY8=', 'Content-Length': '0'} 2023-10-12T01:20:41.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:16:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bace5ad79828ab0-00652749e9-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.367 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.367 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.368 DEBUG:boto:path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.368 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.368 DEBUG:boto:Method: GET 2023-10-12T01:20:41.369 DEBUG:boto:Path: /wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.369 DEBUG:boto:Data: 2023-10-12T01:20:41.370 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:20:41.370 DEBUG:boto:Port: 8001 2023-10-12T01:20:41.371 DEBUG:boto:Params: {} 2023-10-12T01:20:41.371 DEBUG:boto:Token: None 2023-10-12T01:20:41.372 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.373 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005081aa461f9e6368-00652749e9-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.376 DEBUG:boto:b'wafmoa-61obj951000falseobj952023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.377 DEBUG:boto:path=/wafmoa-61/ 2023-10-12T01:20:41.377 DEBUG:boto:auth_path=/wafmoa-61/ 2023-10-12T01:20:41.378 DEBUG:boto:path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.378 DEBUG:boto:auth_path=/wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.379 DEBUG:boto:Method: GET 2023-10-12T01:20:41.379 DEBUG:boto:Path: /wafmoa-61/?prefix=obj95 2023-10-12T01:20:41.379 DEBUG:boto:Data: 2023-10-12T01:20:41.380 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.381 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.381 DEBUG:boto:Params: {} 2023-10-12T01:20:41.382 DEBUG:boto:Token: None 2023-10-12T01:20:41.382 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-61/ 2023-10-12T01:20:41.383 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU= 2023-10-12T01:20:41.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p/JgnQi+LURFB34v1hBDxuDfmDU=', 'Content-Length': '0'} 2023-10-12T01:20:41.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2870bf01631329-00652749e9-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.387 DEBUG:boto:b'wafmoa-61obj951000falseobj952023-10-12T01:16:55.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:20:41.388 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-61 zones={a2, a1} 2023-10-12T01:20:41.389 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T01:20:41.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.390 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:41.444 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:41.445 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:41.496 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:20:41.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.496 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000023563", 2023-10-12T01:20:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:17:04.510533Z" 2023-10-12T01:20:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037387", 2023-10-12T01:20:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:16:57.385919Z" 2023-10-12T01:20:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032585", 2023-10-12T01:20:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:17:04.512195Z" 2023-10-12T01:20:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032489", 2023-10-12T01:20:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:17:04.513747Z" 2023-10-12T01:20:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.500 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:20:41.500 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037387', 2: '00000000000000000000:00000000000000032585', 3: '00000000000000000000:00000000000000032489'} 2023-10-12T01:20:41.501 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:20:41.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.502 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:41.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:41.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:41.597 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:20:41.597 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:20:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:20:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:20:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:20:41.598 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12333732815478984382 2023-10-12T01:20:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:20:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:20:41.599 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000023563", 2023-10-12T01:20:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:20:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:17:04.510533Z" 2023-10-12T01:20:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:20:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037387", 2023-10-12T01:20:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 52, 2023-10-12T01:20:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:16:57.385919Z" 2023-10-12T01:20:41.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:20:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.604 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032585", 2023-10-12T01:20:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.605 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:17:04.512195Z" 2023-10-12T01:20:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:20:41.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032489", 2023-10-12T01:20:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:17:04.513747Z" 2023-10-12T01:20:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:20:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:20:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 61, 2023-10-12T01:20:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 61 2023-10-12T01:20:41.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.610 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:20:41Z" 2023-10-12T01:20:41.610 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:20:41.611 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12333732815478984382 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023563", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:17:04.510533Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037387", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-10-12T01:16:57.385919Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032585", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:17:04.512195Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032489", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:17:04.513747Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-10-12T01:20:41Z" } 2023-10-12T01:20:41.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023563', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:17:04.510533Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037387', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-10-12T01:16:57.385919Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032585', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:17:04.512195Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032489', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:17:04.513747Z'}}] 2023-10-12T01:20:41.612 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037387', 2: '00000000000000000000:00000000000000032585', 3: '00000000000000000000:00000000000000032489'} 2023-10-12T01:20:41.612 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037387', 2: '00000000000000000000:00000000000000032585', 3: '00000000000000000000:00000000000000032489'} 2023-10-12T01:20:41.613 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:20:41.613 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T01:20:41.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.614 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:41.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:41.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:41.712 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:20:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-10-12T01:20:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:18:47.639595Z" 2023-10-12T01:20:41.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.714 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015165", 2023-10-12T01:20:41.714 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:18:47.641299Z" 2023-10-12T01:20:41.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.715 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013621", 2023-10-12T01:20:41.715 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:18:47.643543Z" 2023-10-12T01:20:41.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.715 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-10-12T01:20:41.716 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:18:47.645080Z" 2023-10-12T01:20:41.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.716 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:20:41.717 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011394'} 2023-10-12T01:20:41.718 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:20:41.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.719 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:41.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:41.772 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:41.817 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:20:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:20:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:20:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:20:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:20:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T01:20:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:20:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:20:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-10-12T01:20:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:18:47.639595Z" 2023-10-12T01:20:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:20:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015165", 2023-10-12T01:20:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:18:47.641299Z" 2023-10-12T01:20:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:20:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013621", 2023-10-12T01:20:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:18:47.643543Z" 2023-10-12T01:20:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:20:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:20:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-10-12T01:20:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:18:47.645080Z" 2023-10-12T01:20:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:20:41Z" 2023-10-12T01:20:41.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:20:41.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:18:47.639595Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015165", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:18:47.641299Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:18:47.643543Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011394", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:18:47.645080Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T01:20:41Z" } 2023-10-12T01:20:41.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:18:47.639595Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015165', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:18:47.641299Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:18:47.643543Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011394', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:18:47.645080Z'}}] 2023-10-12T01:20:41.832 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011394'} 2023-10-12T01:20:41.832 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015165', 2: '00000000000000000000:00000000000000013621', 3: '00000000000000000000:00000000000000011394'} 2023-10-12T01:20:41.832 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:20:41.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-10-12T01:20:41.834 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:20:41.835 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:20:41.835 DEBUG:boto:Method: PUT 2023-10-12T01:20:41.836 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:20:41.836 DEBUG:boto:Data: 2023-10-12T01:20:41.836 DEBUG:boto:Headers: {} 2023-10-12T01:20:41.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:41.837 DEBUG:boto:Port: 8000 2023-10-12T01:20:41.837 DEBUG:boto:Params: {} 2023-10-12T01:20:41.838 DEBUG:boto:Token: None 2023-10-12T01:20:41.838 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:20:41 GMT /wafmoa-62/ 2023-10-12T01:20:41.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dlbfoDysbdga0Il63QT+XktgI2M= 2023-10-12T01:20:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:41 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dlbfoDysbdga0Il63QT+XktgI2M=', 'Content-Length': '0'} 2023-10-12T01:20:41.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000098a65d3a41ce4a88-00652749e9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:20:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:41.854 DEBUG:rgw_multi.tests:created bucket=wafmoa-62 2023-10-12T01:20:41.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.855 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:41.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:41.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:41.938 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:20:41.951 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:20:41.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.951 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:20:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:20:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:20:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:20:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073641.852711_690.1", 2023-10-12T01:20:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:20:41.852711Z" 2023-10-12T01:20:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:20:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:20:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:41.955 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:20:41.956 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073641.852711_690.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:20:41.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:20:41.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:20:41.958 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:20:42.007 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:42.007 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:42.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:42.034+0000 7f7191926780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:20:42.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:42.035+0000 7f7191926780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:20:42.048 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:20:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:20:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:20:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:20:42.049 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:20:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:20:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:20:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:20:42.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:20:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:20:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:20:42.051 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:20:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:20:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:20:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:20:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:42.053 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:20:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:20:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:42.054 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:20:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:20:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:20:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:20:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073641.852711_690.1", 2023-10-12T01:20:42.056 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:20:41.852711Z", 2023-10-12T01:20:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:20:42.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:20:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:20:42.058 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:20:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:20:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:20:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:20:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:20:42.059 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:20:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:20:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:20:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:20:42.060 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:20:42.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.061 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:20:42.061 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:20:42.061 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:20:42.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:20:42.062 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:20:42Z" 2023-10-12T01:20:42.062 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:20:42.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073641.852711_690.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:20:41.852711Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:20:42Z" } 2023-10-12T01:20:42.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073641.852711_690.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:20:41.852711Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:20:42.064 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073641.852711_690.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:20:42.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073641.852711_690.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:20:42.065 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:20:42.065 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:20:42.066 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:20:42.066 DEBUG:boto:Method: HEAD 2023-10-12T01:20:42.066 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:20:42.067 DEBUG:boto:Data: 2023-10-12T01:20:42.067 DEBUG:boto:Headers: {} 2023-10-12T01:20:42.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:42.068 DEBUG:boto:Port: 8000 2023-10-12T01:20:42.068 DEBUG:boto:Params: {} 2023-10-12T01:20:42.069 DEBUG:boto:Token: None 2023-10-12T01:20:42.069 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:20:42 GMT /wafmoa-62/ 2023-10-12T01:20:42.070 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VKrvwIeTkW2Xo501FLJ5osm0/w4= 2023-10-12T01:20:42.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:20:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VKrvwIeTkW2Xo501FLJ5osm0/w4=', 'Content-Length': '0'} 2023-10-12T01:20:42.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006130e51a1d603373-00652749ea-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:42.077 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:20:42.078 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:20:42.078 DEBUG:boto:Method: PUT 2023-10-12T01:20:42.079 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:20:42.079 DEBUG:boto:Data: 2023-10-12T01:20:42.080 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:20:42.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:20:42.080 DEBUG:boto:Port: 8000 2023-10-12T01:20:42.081 DEBUG:boto:Params: {} 2023-10-12T01:20:42.081 DEBUG:boto:Token: None 2023-10-12T01:20:42.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:20:42 GMT /wafmoa-62/dummy 2023-10-12T01:20:42.082 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FoGhukXBjwaGQAFHGykPz9kxgx0= 2023-10-12T01:20:42.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:20:42 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FoGhukXBjwaGQAFHGykPz9kxgx0='} 2023-10-12T01:20:42.089 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fef8fc471d1d0c4-00652749ea-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:20:42.089 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-62 2023-10-12T01:20:42.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:20:42.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:20:42.090 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:20:42.147 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:20:42.147 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:20:42.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:42.180+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:20:47.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:47.181+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:20:52.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:52.181+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:20:57.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:20:57.182+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:02.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:02.183+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:07.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:07.184+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:12.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:12.185+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:17.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:17.184+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:22.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:22.185+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:27.186+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:32.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:32.186+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:37.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:37.187+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:42.188+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:47.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:47.189+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:52.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:52.190+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:21:57.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:21:57.189+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:02.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:02.190+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:07.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:07.191+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:12.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:12.191+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:17.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:17.193+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:22.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:22.194+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:27.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:27.194+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:32.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:32.194+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:37.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:37.195+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:42.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:42.195+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:47.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:47.196+0000 7fee953cd780 1 waiting to reach incremental sync.. 2023-10-12T01:22:52.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:52.197+0000 7fee953cd780 1 bucket sync caught up with source: 2023-10-12T01:22:52.199 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:22:52.199 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:22:52.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:52.197+0000 7fee953cd780 0 bucket checkpoint complete 2023-10-12T01:22:52.216 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-62 2023-10-12T01:22:52.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:22:52.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:22:52.218 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:22:52.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:22:52.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:22:52.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:52.320+0000 7f3fc256e780 1 waiting to reach incremental sync.. 2023-10-12T01:22:57.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:22:57.321+0000 7f3fc256e780 1 waiting to reach incremental sync.. 2023-10-12T01:23:02.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:02.322+0000 7f3fc256e780 1 waiting to reach incremental sync.. 2023-10-12T01:23:04.365 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:23:04.365+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:23:07.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:07.323+0000 7f3fc256e780 1 bucket sync caught up with source: 2023-10-12T01:23:07.324 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.176.6, , , , , ] 2023-10-12T01:23:07.324 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.176.6, , , , , ] 2023-10-12T01:23:07.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:07.323+0000 7f3fc256e780 0 bucket checkpoint complete 2023-10-12T01:23:07.340 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-62 zones={a2, a1} 2023-10-12T01:23:07.340 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.341 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.341 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.342 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:07.342 DEBUG:boto:Data: 2023-10-12T01:23:07.343 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.344 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.344 DEBUG:boto:Params: {} 2023-10-12T01:23:07.345 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:23:07.345 DEBUG:boto:Token: None 2023-10-12T01:23:07.346 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ry+xEaTfk8v1s/43qFykmUCnMHg= 2023-10-12T01:23:07.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ry+xEaTfk8v1s/43qFykmUCnMHg=', 'Content-Length': '0'} 2023-10-12T01:23:07.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b6118cbf9c2e2e2-0065274a7b-1f78-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.352 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.352 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.353 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.353 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:07.354 DEBUG:boto:Data: 2023-10-12T01:23:07.354 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.355 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.355 DEBUG:boto:Params: {} 2023-10-12T01:23:07.356 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:23:07.356 DEBUG:boto:Token: None 2023-10-12T01:23:07.357 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.357 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ry+xEaTfk8v1s/43qFykmUCnMHg= 2023-10-12T01:23:07.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ry+xEaTfk8v1s/43qFykmUCnMHg=', 'Content-Length': '0'} 2023-10-12T01:23:07.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017761fe2ba76ba55-0065274a7b-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:23:07.363 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.363 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.364 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.364 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.365 DEBUG:boto:Method: GET 2023-10-12T01:23:07.365 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.366 DEBUG:boto:Data: 2023-10-12T01:23:07.366 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.367 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.367 DEBUG:boto:Params: {} 2023-10-12T01:23:07.368 DEBUG:boto:Token: None 2023-10-12T01:23:07.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/?versions 2023-10-12T01:23:07.369 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY= 2023-10-12T01:23:07.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY=', 'Content-Length': '0'} 2023-10-12T01:23:07.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a98ecb471562d33-0065274a7b-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.372 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.373 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T01:23:07.374 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:23:07.374 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.375 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.375 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.375 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.376 DEBUG:boto:Method: GET 2023-10-12T01:23:07.376 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.377 DEBUG:boto:Data: 2023-10-12T01:23:07.377 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.378 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.379 DEBUG:boto:Params: {} 2023-10-12T01:23:07.379 DEBUG:boto:Token: None 2023-10-12T01:23:07.380 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/?versions 2023-10-12T01:23:07.381 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY= 2023-10-12T01:23:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY=', 'Content-Length': '0'} 2023-10-12T01:23:07.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e22a1a68ebb2d5d3-0065274a7b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.384 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.384 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T01:23:07.385 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.385 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.386 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.386 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.387 DEBUG:boto:Method: GET 2023-10-12T01:23:07.387 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.389 DEBUG:boto:Data: 2023-10-12T01:23:07.389 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.390 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.391 DEBUG:boto:Params: {} 2023-10-12T01:23:07.391 DEBUG:boto:Token: None 2023-10-12T01:23:07.392 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/?versions 2023-10-12T01:23:07.392 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY= 2023-10-12T01:23:07.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY=', 'Content-Length': '0'} 2023-10-12T01:23:07.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061bb8ebf3a44acb3-0065274a7b-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.395 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.396 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.397 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.397 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.397 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.398 DEBUG:boto:Method: GET 2023-10-12T01:23:07.398 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:23:07.399 DEBUG:boto:Data: 2023-10-12T01:23:07.399 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.400 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.400 DEBUG:boto:Params: {} 2023-10-12T01:23:07.401 DEBUG:boto:Token: None 2023-10-12T01:23:07.401 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/?versions 2023-10-12T01:23:07.402 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY= 2023-10-12T01:23:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YFvs3WkjUD9kYHnphOIaj7lHeMY=', 'Content-Length': '0'} 2023-10-12T01:23:07.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b0f9f78911f8235-0065274a7b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.405 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:23:07.407 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.407 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.408 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.408 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.408 DEBUG:boto:Method: GET 2023-10-12T01:23:07.409 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.409 DEBUG:boto:Data: 2023-10-12T01:23:07.410 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.411 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.411 DEBUG:boto:Params: {} 2023-10-12T01:23:07.411 DEBUG:boto:Token: None 2023-10-12T01:23:07.412 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.413 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c= 2023-10-12T01:23:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c=', 'Content-Length': '0'} 2023-10-12T01:23:07.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007699ce2afcf8a39-0065274a7b-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.416 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.416 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.417 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.417 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.418 DEBUG:boto:Method: GET 2023-10-12T01:23:07.418 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.418 DEBUG:boto:Data: 2023-10-12T01:23:07.419 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.420 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.420 DEBUG:boto:Params: {} 2023-10-12T01:23:07.421 DEBUG:boto:Token: None 2023-10-12T01:23:07.421 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.422 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c= 2023-10-12T01:23:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c=', 'Content-Length': '0'} 2023-10-12T01:23:07.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b22aa8cd54d3a4e-0065274a7b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.425 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.425 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.426 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.426 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.427 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.427 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.428 DEBUG:boto:Data: 2023-10-12T01:23:07.428 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.429 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.429 DEBUG:boto:Params: {} 2023-10-12T01:23:07.430 DEBUG:boto:Token: None 2023-10-12T01:23:07.430 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gLEUGH7uILZ1liveR1hhtnKndrw= 2023-10-12T01:23:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gLEUGH7uILZ1liveR1hhtnKndrw=', 'Content-Length': '0'} 2023-10-12T01:23:07.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b8bb343733bfc6-0065274a7b-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.433 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.434 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.434 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.435 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.435 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.435 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.436 DEBUG:boto:Data: 2023-10-12T01:23:07.436 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.437 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.438 DEBUG:boto:Params: {} 2023-10-12T01:23:07.438 DEBUG:boto:Token: None 2023-10-12T01:23:07.439 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gLEUGH7uILZ1liveR1hhtnKndrw= 2023-10-12T01:23:07.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gLEUGH7uILZ1liveR1hhtnKndrw=', 'Content-Length': '0'} 2023-10-12T01:23:07.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eccb9541203d2cd0-0065274a7b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:23:07.443 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.443 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.443 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.444 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.444 DEBUG:boto:Method: GET 2023-10-12T01:23:07.445 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.445 DEBUG:boto:Data: 2023-10-12T01:23:07.446 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.447 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.447 DEBUG:boto:Params: {} 2023-10-12T01:23:07.447 DEBUG:boto:Token: None 2023-10-12T01:23:07.448 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.448 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c= 2023-10-12T01:23:07.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c=', 'Content-Length': '0'} 2023-10-12T01:23:07.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b8754853576315-0065274a7b-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.494 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.495 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.495 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.496 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.496 DEBUG:boto:Method: GET 2023-10-12T01:23:07.497 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.497 DEBUG:boto:Data: 2023-10-12T01:23:07.498 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.498 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.499 DEBUG:boto:Params: {} 2023-10-12T01:23:07.499 DEBUG:boto:Token: None 2023-10-12T01:23:07.500 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:23:07.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c= 2023-10-12T01:23:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:njt5C80zYJWf9b7JR7mYZKaY47c=', 'Content-Length': '0'} 2023-10-12T01:23:07.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd07c8291fb0ee2d-0065274a7b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.504 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.504 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.504 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.505 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.505 DEBUG:boto:Method: GET 2023-10-12T01:23:07.506 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.506 DEBUG:boto:Data: 2023-10-12T01:23:07.507 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.507 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.508 DEBUG:boto:Params: {} 2023-10-12T01:23:07.508 DEBUG:boto:Token: None 2023-10-12T01:23:07.509 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA= 2023-10-12T01:23:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA=', 'Content-Length': '0'} 2023-10-12T01:23:07.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a2d56a9c477bd1-0065274a7b-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.513 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.514 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.514 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.515 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.515 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.515 DEBUG:boto:Method: GET 2023-10-12T01:23:07.516 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.516 DEBUG:boto:Data: 2023-10-12T01:23:07.517 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.518 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.521 DEBUG:boto:Params: {} 2023-10-12T01:23:07.521 DEBUG:boto:Token: None 2023-10-12T01:23:07.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA= 2023-10-12T01:23:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA=', 'Content-Length': '0'} 2023-10-12T01:23:07.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e8fd0416d48828-0065274a7b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.526 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.527 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.527 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.528 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.528 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:23:07.529 DEBUG:boto:Data: 2023-10-12T01:23:07.529 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.530 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.531 DEBUG:boto:Params: {} 2023-10-12T01:23:07.531 DEBUG:boto:Token: None 2023-10-12T01:23:07.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy 2023-10-12T01:23:07.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjpGQiWdS2+aGPjT41uofxAL7UY= 2023-10-12T01:23:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjpGQiWdS2+aGPjT41uofxAL7UY=', 'Content-Length': '0'} 2023-10-12T01:23:07.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189c8f533a998388-0065274a7b-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.535 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.535 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.536 DEBUG:boto:Method: HEAD 2023-10-12T01:23:07.536 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:23:07.537 DEBUG:boto:Data: 2023-10-12T01:23:07.537 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.538 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.538 DEBUG:boto:Params: {} 2023-10-12T01:23:07.539 DEBUG:boto:Token: None 2023-10-12T01:23:07.539 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy 2023-10-12T01:23:07.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LjpGQiWdS2+aGPjT41uofxAL7UY= 2023-10-12T01:23:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LjpGQiWdS2+aGPjT41uofxAL7UY=', 'Content-Length': '0'} 2023-10-12T01:23:07.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004066e8d746e69bcd-0065274a7b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:23:07.543 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.544 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.544 DEBUG:boto:Method: GET 2023-10-12T01:23:07.545 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:23:07.545 DEBUG:boto:Data: 2023-10-12T01:23:07.545 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.546 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.547 DEBUG:boto:Params: {} 2023-10-12T01:23:07.547 DEBUG:boto:Token: None 2023-10-12T01:23:07.548 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy 2023-10-12T01:23:07.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V1mguYiyauxz2BDaaPzooIK3MwE= 2023-10-12T01:23:07.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V1mguYiyauxz2BDaaPzooIK3MwE=', 'Content-Length': '0'} 2023-10-12T01:23:07.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6677a56ed874e8f-0065274a7b-1f78-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.551 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:23:07.552 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:23:07.552 DEBUG:boto:Method: GET 2023-10-12T01:23:07.553 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:23:07.553 DEBUG:boto:Data: 2023-10-12T01:23:07.553 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.554 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.555 DEBUG:boto:Params: {} 2023-10-12T01:23:07.555 DEBUG:boto:Token: None 2023-10-12T01:23:07.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/dummy 2023-10-12T01:23:07.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V1mguYiyauxz2BDaaPzooIK3MwE= 2023-10-12T01:23:07.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V1mguYiyauxz2BDaaPzooIK3MwE=', 'Content-Length': '0'} 2023-10-12T01:23:07.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7d178a5d3885bc-0065274a7b-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.606 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.607 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.607 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.608 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.608 DEBUG:boto:Method: GET 2023-10-12T01:23:07.609 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.609 DEBUG:boto:Data: 2023-10-12T01:23:07.610 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:07.611 DEBUG:boto:Port: 8001 2023-10-12T01:23:07.611 DEBUG:boto:Params: {} 2023-10-12T01:23:07.611 DEBUG:boto:Token: None 2023-10-12T01:23:07.612 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.614 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA= 2023-10-12T01:23:07.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA=', 'Content-Length': '0'} 2023-10-12T01:23:07.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f45ed0cc28448bc3-0065274a7b-1f78-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.618 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.618 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:07.619 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:07.619 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.620 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.620 DEBUG:boto:Method: GET 2023-10-12T01:23:07.621 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:23:07.621 DEBUG:boto:Data: 2023-10-12T01:23:07.621 DEBUG:boto:Headers: {} 2023-10-12T01:23:07.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:07.623 DEBUG:boto:Port: 8000 2023-10-12T01:23:07.623 DEBUG:boto:Params: {} 2023-10-12T01:23:07.624 DEBUG:boto:Token: None 2023-10-12T01:23:07.625 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:23:07 GMT /wafmoa-62/ 2023-10-12T01:23:07.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA= 2023-10-12T01:23:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7W2igfrFeJoOrQOraEM03Q+i2WA=', 'Content-Length': '0'} 2023-10-12T01:23:07.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049610473db53f765-0065274a7b-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:23:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:07.629 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:23:07.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-62 zones={a2, a1} 2023-10-12T01:23:07.631 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:23:07.631 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:23:07.632 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:23:07.632+0000 7f5b0169c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 176659) UID: 0 2023-10-12T01:23:07.632 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:23:07.632+0000 7f5b059057c0 -1 shutting down 2023-10-12T01:23:11.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:23:13.730 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:23:13.731 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:13.731 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:13.732 DEBUG:boto:Method: HEAD 2023-10-12T01:23:13.732 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:13.732 DEBUG:boto:Data: 2023-10-12T01:23:13.733 DEBUG:boto:Headers: {} 2023-10-12T01:23:13.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.734 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.734 DEBUG:boto:Params: {} 2023-10-12T01:23:13.735 DEBUG:boto:Token: None 2023-10-12T01:23:13.735 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/ 2023-10-12T01:23:13.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY= 2023-10-12T01:23:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY=', 'Content-Length': '0'} 2023-10-12T01:23:13.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0f00146d60a2425-0065274a81-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.740 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:23:13.740 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:23:13.740 DEBUG:boto:Method: PUT 2023-10-12T01:23:13.741 DEBUG:boto:Path: /wafmoa-62/obj11 2023-10-12T01:23:13.741 DEBUG:boto:Data: 2023-10-12T01:23:13.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:23:13.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.742 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.743 DEBUG:boto:Params: {} 2023-10-12T01:23:13.743 DEBUG:boto:Token: None 2023-10-12T01:23:13.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/obj11 2023-10-12T01:23:13.744 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hY/EvEX9sUzJzaURTO5XZyNxSAY= 2023-10-12T01:23:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hY/EvEX9sUzJzaURTO5XZyNxSAY='} 2023-10-12T01:23:13.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095d34dc46fe26d86-0065274a81-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.752 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:13.752 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:13.753 DEBUG:boto:Method: HEAD 2023-10-12T01:23:13.753 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:13.754 DEBUG:boto:Data: 2023-10-12T01:23:13.754 DEBUG:boto:Headers: {} 2023-10-12T01:23:13.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.755 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.755 DEBUG:boto:Params: {} 2023-10-12T01:23:13.756 DEBUG:boto:Token: None 2023-10-12T01:23:13.756 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/ 2023-10-12T01:23:13.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY= 2023-10-12T01:23:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY=', 'Content-Length': '0'} 2023-10-12T01:23:13.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005761631d97e01f40-0065274a81-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.760 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:23:13.761 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:23:13.761 DEBUG:boto:Method: PUT 2023-10-12T01:23:13.761 DEBUG:boto:Path: /wafmoa-62/obj22 2023-10-12T01:23:13.762 DEBUG:boto:Data: 2023-10-12T01:23:13.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:23:13.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.763 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.763 DEBUG:boto:Params: {} 2023-10-12T01:23:13.764 DEBUG:boto:Token: None 2023-10-12T01:23:13.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/obj22 2023-10-12T01:23:13.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EyO+Z/K9kmRLJp/4eoB1sZ3DnxM= 2023-10-12T01:23:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EyO+Z/K9kmRLJp/4eoB1sZ3DnxM='} 2023-10-12T01:23:13.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bd8bc3e54fbb9f6-0065274a81-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.812 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:13.813 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:13.813 DEBUG:boto:Method: HEAD 2023-10-12T01:23:13.814 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:13.814 DEBUG:boto:Data: 2023-10-12T01:23:13.814 DEBUG:boto:Headers: {} 2023-10-12T01:23:13.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.815 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.816 DEBUG:boto:Params: {} 2023-10-12T01:23:13.816 DEBUG:boto:Token: None 2023-10-12T01:23:13.816 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/ 2023-10-12T01:23:13.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY= 2023-10-12T01:23:13.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY=', 'Content-Length': '0'} 2023-10-12T01:23:13.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017cb255b54d6f901-0065274a81-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.821 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:23:13.821 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:23:13.821 DEBUG:boto:Method: PUT 2023-10-12T01:23:13.822 DEBUG:boto:Path: /wafmoa-62/obj33 2023-10-12T01:23:13.822 DEBUG:boto:Data: 2023-10-12T01:23:13.823 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:23:13.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.824 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.824 DEBUG:boto:Params: {} 2023-10-12T01:23:13.824 DEBUG:boto:Token: None 2023-10-12T01:23:13.825 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/obj33 2023-10-12T01:23:13.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6ohAqX4hzvz5k9OijjnU+YS37mk= 2023-10-12T01:23:13.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6ohAqX4hzvz5k9OijjnU+YS37mk='} 2023-10-12T01:23:13.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2e736a6c7d0aa18-0065274a81-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.833 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:13.833 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:13.833 DEBUG:boto:Method: HEAD 2023-10-12T01:23:13.834 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:13.834 DEBUG:boto:Data: 2023-10-12T01:23:13.835 DEBUG:boto:Headers: {} 2023-10-12T01:23:13.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.835 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.836 DEBUG:boto:Params: {} 2023-10-12T01:23:13.836 DEBUG:boto:Token: None 2023-10-12T01:23:13.837 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/ 2023-10-12T01:23:13.837 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY= 2023-10-12T01:23:13.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY=', 'Content-Length': '0'} 2023-10-12T01:23:13.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a55d63d59717d306-0065274a81-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.840 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:23:13.841 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:23:13.841 DEBUG:boto:Method: PUT 2023-10-12T01:23:13.842 DEBUG:boto:Path: /wafmoa-62/obj44 2023-10-12T01:23:13.842 DEBUG:boto:Data: 2023-10-12T01:23:13.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:23:13.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.843 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.844 DEBUG:boto:Params: {} 2023-10-12T01:23:13.844 DEBUG:boto:Token: None 2023-10-12T01:23:13.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/obj44 2023-10-12T01:23:13.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xl2m1CR6PkmV33YRY1CvKt+ROcI= 2023-10-12T01:23:13.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xl2m1CR6PkmV33YRY1CvKt+ROcI='} 2023-10-12T01:23:13.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037daa5b5080041e8-0065274a81-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.852 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:23:13.853 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:23:13.853 DEBUG:boto:Method: HEAD 2023-10-12T01:23:13.854 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:23:13.854 DEBUG:boto:Data: 2023-10-12T01:23:13.855 DEBUG:boto:Headers: {} 2023-10-12T01:23:13.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.855 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.856 DEBUG:boto:Params: {} 2023-10-12T01:23:13.856 DEBUG:boto:Token: None 2023-10-12T01:23:13.857 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/ 2023-10-12T01:23:13.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY= 2023-10-12T01:23:13.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F3YO/TiDLBiBekFpLSHQr/0RDNY=', 'Content-Length': '0'} 2023-10-12T01:23:13.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209e3e7167b2846a-0065274a81-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.861 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:23:13.861 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:23:13.862 DEBUG:boto:Method: PUT 2023-10-12T01:23:13.862 DEBUG:boto:Path: /wafmoa-62/obj55 2023-10-12T01:23:13.863 DEBUG:boto:Data: 2023-10-12T01:23:13.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:23:13.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:23:13.864 DEBUG:boto:Port: 8000 2023-10-12T01:23:13.864 DEBUG:boto:Params: {} 2023-10-12T01:23:13.865 DEBUG:boto:Token: None 2023-10-12T01:23:13.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:23:13 GMT /wafmoa-62/obj55 2023-10-12T01:23:13.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9mQPpS67ExYnIbm5jlXZ3Eq21IQ= 2023-10-12T01:23:13.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:23:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9mQPpS67ExYnIbm5jlXZ3Eq21IQ='} 2023-10-12T01:23:13.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001147a881a9f14d4e-0065274a81-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:23:13.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:23:13.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:23:13.912 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:23:13.971 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:23:13.972 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:23:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:14.005+0000 7f5127c8d780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-12T01:23:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:14.006+0000 7f5127c8d780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-12T01:23:14.029 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T01:23:14.030 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:23:14.032 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T01:23:14.032 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:14.054 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:23:14.054 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:23:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T01:23:14.056 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:23:15.057 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:23:15.080 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:23:15.080 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:23:15.081 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:23:15.082 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T01:23:15.082 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-62 2023-10-12T01:23:15.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:23:15.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:23:15.084 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:23:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:23:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:23:15.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:15.213+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:15.214 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:15.214 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:17.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:23:20.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:20.214+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:20.215 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:20.216 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:25.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:25.216+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:25.217 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:25.217 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:30.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:30.217+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:30.218 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:30.219 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:35.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:35.219+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:35.219 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:35.220 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:40.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:40.220+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:40.221 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:40.221 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:45.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:45.221+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:45.222 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:45.222 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:50.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:50.222+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:50.223 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:50.223 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:23:55.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:23:55.223+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:23:55.224 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:23:55.224 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:00.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:00.224+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:00.225 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:00.226 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:05.225+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:05.227 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:05.227 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:10.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:10.227+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:10.228 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:10.228 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:15.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:15.229+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:15.229 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:15.230 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:20.230+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:20.231 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:20.231 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:25.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:25.230+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:25.232 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:25.232 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:30.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:30.231+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:30.233 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:30.233 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:35.233+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:35.234 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:35.235 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:40.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:40.234+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:40.235 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:40.236 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:45.235+0000 7f8cb35c4780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:45.237 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , , , ] 2023-10-12T01:24:45.237 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:50.238+0000 7f8cb35c4780 1 bucket sync caught up with source: 2023-10-12T01:24:50.238 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:50.238 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.79.6, , , 00000000001.79.6, 00000000002.74.6, 00000000002.109.6] 2023-10-12T01:24:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:50.238+0000 7f8cb35c4780 0 bucket checkpoint complete 2023-10-12T01:24:50.255 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-62 2023-10-12T01:24:50.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:24:50.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:24:50.257 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:24:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:24:50.320 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:24:50.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:50.357+0000 7f793817e780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:50.357 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.176.6, , , , , ] 2023-10-12T01:24:50.358 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.176.6, , , 00000000001.372.6, 00000000002.285.6, 00000000002.330.6] 2023-10-12T01:24:55.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:24:55.358+0000 7f793817e780 1 waiting for incremental sync to catch up: 2023-10-12T01:24:55.359 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.176.6, , , , , ] 2023-10-12T01:24:55.359 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.176.6, , , 00000000001.372.6, 00000000002.285.6, 00000000002.330.6] 2023-10-12T01:25:00.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:00.360+0000 7f793817e780 1 waiting for incremental sync to catch up: 2023-10-12T01:25:00.360 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.176.6, , , , , ] 2023-10-12T01:25:00.360 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.176.6, , , 00000000001.372.6, 00000000002.285.6, 00000000002.330.6] 2023-10-12T01:25:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:05.360+0000 7f793817e780 1 bucket sync caught up with source: 2023-10-12T01:25:05.361 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.176.6, , , 00000000001.372.6, 00000000002.285.6, 00000000002.330.6] 2023-10-12T01:25:05.361 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.176.6, , , 00000000001.372.6, 00000000002.285.6, 00000000002.330.6] 2023-10-12T01:25:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:05.360+0000 7f793817e780 0 bucket checkpoint complete 2023-10-12T01:25:05.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-62 zones={a2, a1} 2023-10-12T01:25:05.379 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.379 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.380 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.380 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:25:05.381 DEBUG:boto:Data: 2023-10-12T01:25:05.381 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.382 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.383 DEBUG:boto:Params: {} 2023-10-12T01:25:05.383 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:25:05.384 DEBUG:boto:Token: None 2023-10-12T01:25:05.384 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.385 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RWhwe8EEdTvpISw10kWIZslresk= 2023-10-12T01:25:05.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RWhwe8EEdTvpISw10kWIZslresk=', 'Content-Length': '0'} 2023-10-12T01:25:05.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000148e9f9aece9196-0065274af1-1fd8-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.392 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.393 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.393 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.394 DEBUG:boto:Path: /wafmoa-62/ 2023-10-12T01:25:05.394 DEBUG:boto:Data: 2023-10-12T01:25:05.395 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.395 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.396 DEBUG:boto:Params: {} 2023-10-12T01:25:05.396 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:25:05.397 DEBUG:boto:Token: None 2023-10-12T01:25:05.397 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.398 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RWhwe8EEdTvpISw10kWIZslresk= 2023-10-12T01:25:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RWhwe8EEdTvpISw10kWIZslresk=', 'Content-Length': '0'} 2023-10-12T01:25:05.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eacc89740979da82-0065274af1-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.404 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:25:05.404 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.405 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.405 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.406 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.406 DEBUG:boto:Method: GET 2023-10-12T01:25:05.407 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.407 DEBUG:boto:Data: 2023-10-12T01:25:05.408 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.408 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.409 DEBUG:boto:Params: {} 2023-10-12T01:25:05.409 DEBUG:boto:Token: None 2023-10-12T01:25:05.410 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/?versions 2023-10-12T01:25:05.410 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM= 2023-10-12T01:25:05.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM=', 'Content-Length': '0'} 2023-10-12T01:25:05.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127e0c6bd8cb2fd6-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.414 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.415 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T01:25:05.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:25:05.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:25:05.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:25:05.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:25:05.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:25:05.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:25:05.418 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.418 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.419 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.419 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.420 DEBUG:boto:Method: GET 2023-10-12T01:25:05.420 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.421 DEBUG:boto:Data: 2023-10-12T01:25:05.421 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.422 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.422 DEBUG:boto:Params: {} 2023-10-12T01:25:05.423 DEBUG:boto:Token: None 2023-10-12T01:25:05.423 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/?versions 2023-10-12T01:25:05.424 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM= 2023-10-12T01:25:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM=', 'Content-Length': '0'} 2023-10-12T01:25:05.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f4b7a39eb97bb8-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.427 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.428 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-12T01:25:05.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:25:05.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:25:05.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:25:05.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:25:05.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:25:05.431 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.431 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.432 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.432 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.432 DEBUG:boto:Method: GET 2023-10-12T01:25:05.433 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.433 DEBUG:boto:Data: 2023-10-12T01:25:05.434 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.434 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.435 DEBUG:boto:Params: {} 2023-10-12T01:25:05.435 DEBUG:boto:Token: None 2023-10-12T01:25:05.436 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/?versions 2023-10-12T01:25:05.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM= 2023-10-12T01:25:05.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM=', 'Content-Length': '0'} 2023-10-12T01:25:05.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040dc9ead24f9ae22-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.440 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.441 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.441 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.441 DEBUG:boto:path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.442 DEBUG:boto:auth_path=/wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.442 DEBUG:boto:Method: GET 2023-10-12T01:25:05.443 DEBUG:boto:Path: /wafmoa-62/?versions&max-keys=999 2023-10-12T01:25:05.443 DEBUG:boto:Data: 2023-10-12T01:25:05.444 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.445 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.445 DEBUG:boto:Params: {} 2023-10-12T01:25:05.445 DEBUG:boto:Token: None 2023-10-12T01:25:05.446 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/?versions 2023-10-12T01:25:05.446 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM= 2023-10-12T01:25:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8ku4KDg3FbJ8nr2IcCNgeS1ccTM=', 'Content-Length': '0'} 2023-10-12T01:25:05.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c39441e168b11a-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.450 DEBUG:boto:b'wafmoa-62999falsedummynulltrue2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:25:05.451 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.452 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.452 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.453 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.453 DEBUG:boto:Method: GET 2023-10-12T01:25:05.454 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.454 DEBUG:boto:Data: 2023-10-12T01:25:05.454 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.455 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.456 DEBUG:boto:Params: {} 2023-10-12T01:25:05.456 DEBUG:boto:Token: None 2023-10-12T01:25:05.457 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.457 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU= 2023-10-12T01:25:05.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU=', 'Content-Length': '0'} 2023-10-12T01:25:05.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b419bd1d3b3cde6-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.461 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.461 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.462 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.462 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.462 DEBUG:boto:Method: GET 2023-10-12T01:25:05.463 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.463 DEBUG:boto:Data: 2023-10-12T01:25:05.464 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.465 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.465 DEBUG:boto:Params: {} 2023-10-12T01:25:05.465 DEBUG:boto:Token: None 2023-10-12T01:25:05.466 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.466 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU= 2023-10-12T01:25:05.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU=', 'Content-Length': '0'} 2023-10-12T01:25:05.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af1994b1b47e909a-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.470 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.470 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.471 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.471 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.472 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.472 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.472 DEBUG:boto:Data: 2023-10-12T01:25:05.473 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.474 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.474 DEBUG:boto:Params: {} 2023-10-12T01:25:05.475 DEBUG:boto:Token: None 2023-10-12T01:25:05.475 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.476 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EYaZIizyx1Y7M3VQ2EapOb1K0G8= 2023-10-12T01:25:05.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EYaZIizyx1Y7M3VQ2EapOb1K0G8=', 'Content-Length': '0'} 2023-10-12T01:25:05.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33f10ae95c6e2fd-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.478 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.479 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.479 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.480 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.480 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.481 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.481 DEBUG:boto:Data: 2023-10-12T01:25:05.481 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.482 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.483 DEBUG:boto:Params: {} 2023-10-12T01:25:05.483 DEBUG:boto:Token: None 2023-10-12T01:25:05.484 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.484 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EYaZIizyx1Y7M3VQ2EapOb1K0G8= 2023-10-12T01:25:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EYaZIizyx1Y7M3VQ2EapOb1K0G8=', 'Content-Length': '0'} 2023-10-12T01:25:05.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f437b62f1be018-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:25:05.487 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.488 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.488 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.489 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.489 DEBUG:boto:Method: GET 2023-10-12T01:25:05.489 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.490 DEBUG:boto:Data: 2023-10-12T01:25:05.490 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.491 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.492 DEBUG:boto:Params: {} 2023-10-12T01:25:05.492 DEBUG:boto:Token: None 2023-10-12T01:25:05.493 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU= 2023-10-12T01:25:05.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU=', 'Content-Length': '0'} 2023-10-12T01:25:05.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513dcbc31b23094b-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.497 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.497 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.498 DEBUG:boto:path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.498 DEBUG:boto:auth_path=/wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.499 DEBUG:boto:Method: GET 2023-10-12T01:25:05.499 DEBUG:boto:Path: /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.500 DEBUG:boto:Data: 2023-10-12T01:25:05.500 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.501 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.501 DEBUG:boto:Params: {} 2023-10-12T01:25:05.502 DEBUG:boto:Token: None 2023-10-12T01:25:05.502 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy?versionId=null 2023-10-12T01:25:05.503 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU= 2023-10-12T01:25:05.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J5whhwKtScS5yAHLsVIGMaAOXCU=', 'Content-Length': '0'} 2023-10-12T01:25:05.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389fb488178c84b0-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.550 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.551 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.552 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.552 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.553 DEBUG:boto:Method: GET 2023-10-12T01:25:05.553 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.554 DEBUG:boto:Data: 2023-10-12T01:25:05.554 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.555 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.555 DEBUG:boto:Params: {} 2023-10-12T01:25:05.556 DEBUG:boto:Token: None 2023-10-12T01:25:05.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.557 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271dfceaa717c6b4-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.560 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.561 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.562 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.562 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.563 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.563 DEBUG:boto:Method: GET 2023-10-12T01:25:05.563 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.564 DEBUG:boto:Data: 2023-10-12T01:25:05.564 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.565 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.566 DEBUG:boto:Params: {} 2023-10-12T01:25:05.566 DEBUG:boto:Token: None 2023-10-12T01:25:05.567 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.567 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed56c9600427639-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.571 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.571 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.572 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.572 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.573 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:25:05.573 DEBUG:boto:Data: 2023-10-12T01:25:05.574 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.574 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.575 DEBUG:boto:Params: {} 2023-10-12T01:25:05.575 DEBUG:boto:Token: None 2023-10-12T01:25:05.576 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy 2023-10-12T01:25:05.576 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V74xRdZ+vdUH8XyItCedbmAZOVo= 2023-10-12T01:25:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V74xRdZ+vdUH8XyItCedbmAZOVo=', 'Content-Length': '0'} 2023-10-12T01:25:05.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002595be1efd937bcf-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.579 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.580 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.580 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.581 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:25:05.581 DEBUG:boto:Data: 2023-10-12T01:25:05.581 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.582 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.583 DEBUG:boto:Params: {} 2023-10-12T01:25:05.583 DEBUG:boto:Token: None 2023-10-12T01:25:05.584 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy 2023-10-12T01:25:05.584 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V74xRdZ+vdUH8XyItCedbmAZOVo= 2023-10-12T01:25:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V74xRdZ+vdUH8XyItCedbmAZOVo=', 'Content-Length': '0'} 2023-10-12T01:25:05.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be2642e90b4fa43b-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-12T01:25:05.588 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.588 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.589 DEBUG:boto:Method: GET 2023-10-12T01:25:05.589 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:25:05.590 DEBUG:boto:Data: 2023-10-12T01:25:05.590 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.591 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.591 DEBUG:boto:Params: {} 2023-10-12T01:25:05.592 DEBUG:boto:Token: None 2023-10-12T01:25:05.592 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy 2023-10-12T01:25:05.593 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vh+qzyUML69PuHP/dS3zXR1z3ic= 2023-10-12T01:25:05.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vh+qzyUML69PuHP/dS3zXR1z3ic=', 'Content-Length': '0'} 2023-10-12T01:25:05.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009616abe8fc8a1d6-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.638 DEBUG:boto:path=/wafmoa-62/dummy 2023-10-12T01:25:05.639 DEBUG:boto:auth_path=/wafmoa-62/dummy 2023-10-12T01:25:05.639 DEBUG:boto:Method: GET 2023-10-12T01:25:05.640 DEBUG:boto:Path: /wafmoa-62/dummy 2023-10-12T01:25:05.640 DEBUG:boto:Data: 2023-10-12T01:25:05.641 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.641 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.642 DEBUG:boto:Params: {} 2023-10-12T01:25:05.642 DEBUG:boto:Token: None 2023-10-12T01:25:05.643 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/dummy 2023-10-12T01:25:05.643 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vh+qzyUML69PuHP/dS3zXR1z3ic= 2023-10-12T01:25:05.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vh+qzyUML69PuHP/dS3zXR1z3ic=', 'Content-Length': '0'} 2023-10-12T01:25:05.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009360692fefb45a96-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.647 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.647 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.648 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.648 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.649 DEBUG:boto:Method: GET 2023-10-12T01:25:05.649 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.649 DEBUG:boto:Data: 2023-10-12T01:25:05.650 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.651 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.651 DEBUG:boto:Params: {} 2023-10-12T01:25:05.652 DEBUG:boto:Token: None 2023-10-12T01:25:05.652 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.653 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097dac761d9472fa2-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.656 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.657 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.657 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.658 DEBUG:boto:path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.658 DEBUG:boto:auth_path=/wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.659 DEBUG:boto:Method: GET 2023-10-12T01:25:05.659 DEBUG:boto:Path: /wafmoa-62/?prefix=dummy 2023-10-12T01:25:05.660 DEBUG:boto:Data: 2023-10-12T01:25:05.660 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.661 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.661 DEBUG:boto:Params: {} 2023-10-12T01:25:05.662 DEBUG:boto:Token: None 2023-10-12T01:25:05.662 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.663 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31239e2f6e79fab-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.710 DEBUG:boto:b'wafmoa-62dummy1000falsedummy2023-10-12T01:20:42.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:25:05.712 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.712 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.713 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.713 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.713 DEBUG:boto:Method: GET 2023-10-12T01:25:05.714 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.714 DEBUG:boto:Data: 2023-10-12T01:25:05.715 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.716 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.716 DEBUG:boto:Params: {} 2023-10-12T01:25:05.717 DEBUG:boto:Token: None 2023-10-12T01:25:05.717 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU= 2023-10-12T01:25:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU=', 'Content-Length': '0'} 2023-10-12T01:25:05.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752bab0fb72a630e-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.721 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.722 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.722 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.723 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.723 DEBUG:boto:Method: GET 2023-10-12T01:25:05.724 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.724 DEBUG:boto:Data: 2023-10-12T01:25:05.724 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.725 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.726 DEBUG:boto:Params: {} 2023-10-12T01:25:05.726 DEBUG:boto:Token: None 2023-10-12T01:25:05.727 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU= 2023-10-12T01:25:05.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU=', 'Content-Length': '0'} 2023-10-12T01:25:05.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e12978064e6b7cd3-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.731 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.731 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.732 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.732 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.733 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.733 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.734 DEBUG:boto:Data: 2023-10-12T01:25:05.734 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.735 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.735 DEBUG:boto:Params: {} 2023-10-12T01:25:05.736 DEBUG:boto:Token: None 2023-10-12T01:25:05.736 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.737 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vGbiTvpSJ6keCaPqfYJ+oR0Vo3g= 2023-10-12T01:25:05.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vGbiTvpSJ6keCaPqfYJ+oR0Vo3g=', 'Content-Length': '0'} 2023-10-12T01:25:05.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128299082e0fa8f2-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.740 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.740 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.741 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.741 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.741 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.742 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.742 DEBUG:boto:Data: 2023-10-12T01:25:05.743 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.743 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.744 DEBUG:boto:Params: {} 2023-10-12T01:25:05.745 DEBUG:boto:Token: None 2023-10-12T01:25:05.745 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vGbiTvpSJ6keCaPqfYJ+oR0Vo3g= 2023-10-12T01:25:05.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vGbiTvpSJ6keCaPqfYJ+oR0Vo3g=', 'Content-Length': '0'} 2023-10-12T01:25:05.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f11fbc70f0af439-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:25:05.749 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.749 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.750 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.750 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.751 DEBUG:boto:Method: GET 2023-10-12T01:25:05.751 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.751 DEBUG:boto:Data: 2023-10-12T01:25:05.752 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.753 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.753 DEBUG:boto:Params: {} 2023-10-12T01:25:05.754 DEBUG:boto:Token: None 2023-10-12T01:25:05.754 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.755 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU= 2023-10-12T01:25:05.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU=', 'Content-Length': '0'} 2023-10-12T01:25:05.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f457dfa998e3d48-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.758 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.759 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.759 DEBUG:boto:path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.760 DEBUG:boto:auth_path=/wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.760 DEBUG:boto:Method: GET 2023-10-12T01:25:05.760 DEBUG:boto:Path: /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.761 DEBUG:boto:Data: 2023-10-12T01:25:05.761 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.762 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.763 DEBUG:boto:Params: {} 2023-10-12T01:25:05.763 DEBUG:boto:Token: None 2023-10-12T01:25:05.764 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11?versionId=null 2023-10-12T01:25:05.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU= 2023-10-12T01:25:05.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bC1M1/UgQnFY1/Z2F/ZQQThsxgU=', 'Content-Length': '0'} 2023-10-12T01:25:05.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587614ce7ec91086-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.767 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.768 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.768 DEBUG:boto:path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.769 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.769 DEBUG:boto:Method: GET 2023-10-12T01:25:05.769 DEBUG:boto:Path: /wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.770 DEBUG:boto:Data: 2023-10-12T01:25:05.770 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.771 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.772 DEBUG:boto:Params: {} 2023-10-12T01:25:05.772 DEBUG:boto:Token: None 2023-10-12T01:25:05.773 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.774 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006233704c355706f0-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.777 DEBUG:boto:b'wafmoa-62obj111000falseobj112023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.778 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.779 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.779 DEBUG:boto:path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.779 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.780 DEBUG:boto:Method: GET 2023-10-12T01:25:05.780 DEBUG:boto:Path: /wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.780 DEBUG:boto:Data: 2023-10-12T01:25:05.781 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.782 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.782 DEBUG:boto:Params: {} 2023-10-12T01:25:05.783 DEBUG:boto:Token: None 2023-10-12T01:25:05.783 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.784 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05968a85a698e59-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.834 DEBUG:boto:b'wafmoa-62obj111000falseobj112023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.835 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.836 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.836 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.837 DEBUG:boto:Path: /wafmoa-62/obj11 2023-10-12T01:25:05.837 DEBUG:boto:Data: 2023-10-12T01:25:05.837 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.838 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.839 DEBUG:boto:Params: {} 2023-10-12T01:25:05.839 DEBUG:boto:Token: None 2023-10-12T01:25:05.840 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11 2023-10-12T01:25:05.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q3HyTo7c8htP3brNYjCF2h1aec0= 2023-10-12T01:25:05.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q3HyTo7c8htP3brNYjCF2h1aec0=', 'Content-Length': '0'} 2023-10-12T01:25:05.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e3a6fa6643e1d8-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.843 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.844 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.844 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.845 DEBUG:boto:Path: /wafmoa-62/obj11 2023-10-12T01:25:05.845 DEBUG:boto:Data: 2023-10-12T01:25:05.846 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.847 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.847 DEBUG:boto:Params: {} 2023-10-12T01:25:05.847 DEBUG:boto:Token: None 2023-10-12T01:25:05.848 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11 2023-10-12T01:25:05.848 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:q3HyTo7c8htP3brNYjCF2h1aec0= 2023-10-12T01:25:05.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:q3HyTo7c8htP3brNYjCF2h1aec0=', 'Content-Length': '0'} 2023-10-12T01:25:05.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023098f98853f32d1-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:25:05.852 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.852 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.853 DEBUG:boto:Method: GET 2023-10-12T01:25:05.853 DEBUG:boto:Path: /wafmoa-62/obj11 2023-10-12T01:25:05.854 DEBUG:boto:Data: 2023-10-12T01:25:05.854 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.855 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.856 DEBUG:boto:Params: {} 2023-10-12T01:25:05.856 DEBUG:boto:Token: None 2023-10-12T01:25:05.857 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11 2023-10-12T01:25:05.857 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ko49cTuOY4DlJ2YANKI/ANCZDK0= 2023-10-12T01:25:05.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ko49cTuOY4DlJ2YANKI/ANCZDK0=', 'Content-Length': '0'} 2023-10-12T01:25:05.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9764a3a042fddc7-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.902 DEBUG:boto:path=/wafmoa-62/obj11 2023-10-12T01:25:05.903 DEBUG:boto:auth_path=/wafmoa-62/obj11 2023-10-12T01:25:05.903 DEBUG:boto:Method: GET 2023-10-12T01:25:05.904 DEBUG:boto:Path: /wafmoa-62/obj11 2023-10-12T01:25:05.904 DEBUG:boto:Data: 2023-10-12T01:25:05.905 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.906 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.906 DEBUG:boto:Params: {} 2023-10-12T01:25:05.907 DEBUG:boto:Token: None 2023-10-12T01:25:05.907 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj11 2023-10-12T01:25:05.908 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ko49cTuOY4DlJ2YANKI/ANCZDK0= 2023-10-12T01:25:05.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ko49cTuOY4DlJ2YANKI/ANCZDK0=', 'Content-Length': '0'} 2023-10-12T01:25:05.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb3150a238af346-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.911 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.913 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.914 DEBUG:boto:path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.914 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.915 DEBUG:boto:Method: GET 2023-10-12T01:25:05.915 DEBUG:boto:Path: /wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.916 DEBUG:boto:Data: 2023-10-12T01:25:05.916 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.917 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.917 DEBUG:boto:Params: {} 2023-10-12T01:25:05.918 DEBUG:boto:Token: None 2023-10-12T01:25:05.918 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.919 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de5d804e1fbb2d2-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.922 DEBUG:boto:b'wafmoa-62obj111000falseobj112023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.923 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.923 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.924 DEBUG:boto:path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.924 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.925 DEBUG:boto:Method: GET 2023-10-12T01:25:05.925 DEBUG:boto:Path: /wafmoa-62/?prefix=obj11 2023-10-12T01:25:05.926 DEBUG:boto:Data: 2023-10-12T01:25:05.926 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.927 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.927 DEBUG:boto:Params: {} 2023-10-12T01:25:05.928 DEBUG:boto:Token: None 2023-10-12T01:25:05.928 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.929 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006137672773ce723b-0065274af1-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.932 DEBUG:boto:b'wafmoa-62obj111000falseobj112023-10-12T01:23:13.747Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:05.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:25:05.934 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.934 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.935 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.935 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.935 DEBUG:boto:Method: GET 2023-10-12T01:25:05.936 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.936 DEBUG:boto:Data: 2023-10-12T01:25:05.937 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.938 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.938 DEBUG:boto:Params: {} 2023-10-12T01:25:05.939 DEBUG:boto:Token: None 2023-10-12T01:25:05.939 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.940 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc= 2023-10-12T01:25:05.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc=', 'Content-Length': '0'} 2023-10-12T01:25:05.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a582cebdc7b758b-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.943 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.944 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.944 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.944 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.945 DEBUG:boto:Method: GET 2023-10-12T01:25:05.945 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.946 DEBUG:boto:Data: 2023-10-12T01:25:05.946 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.947 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.947 DEBUG:boto:Params: {} 2023-10-12T01:25:05.948 DEBUG:boto:Token: None 2023-10-12T01:25:05.948 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc= 2023-10-12T01:25:05.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc=', 'Content-Length': '0'} 2023-10-12T01:25:05.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006deb6f343ff77a05-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.952 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.953 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.953 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.953 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.954 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.954 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.955 DEBUG:boto:Data: 2023-10-12T01:25:05.955 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.956 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.956 DEBUG:boto:Params: {} 2023-10-12T01:25:05.957 DEBUG:boto:Token: None 2023-10-12T01:25:05.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTaXLEo9aQuIJKNDG/7cJ9JY2J4= 2023-10-12T01:25:05.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTaXLEo9aQuIJKNDG/7cJ9JY2J4=', 'Content-Length': '0'} 2023-10-12T01:25:05.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e82b03b12199b0-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.961 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.961 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.962 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.962 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.963 DEBUG:boto:Method: HEAD 2023-10-12T01:25:05.963 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.963 DEBUG:boto:Data: 2023-10-12T01:25:05.964 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.965 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.965 DEBUG:boto:Params: {} 2023-10-12T01:25:05.965 DEBUG:boto:Token: None 2023-10-12T01:25:05.966 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.966 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FTaXLEo9aQuIJKNDG/7cJ9JY2J4= 2023-10-12T01:25:05.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FTaXLEo9aQuIJKNDG/7cJ9JY2J4=', 'Content-Length': '0'} 2023-10-12T01:25:05.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c5b9b919330b1c-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:25:05.970 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.970 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.971 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.971 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.972 DEBUG:boto:Method: GET 2023-10-12T01:25:05.972 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.973 DEBUG:boto:Data: 2023-10-12T01:25:05.973 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.974 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.974 DEBUG:boto:Params: {} 2023-10-12T01:25:05.975 DEBUG:boto:Token: None 2023-10-12T01:25:05.975 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.976 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc= 2023-10-12T01:25:05.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc=', 'Content-Length': '0'} 2023-10-12T01:25:05.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002652d52cf708509d-0065274af1-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.979 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:05.980 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:05.980 DEBUG:boto:path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.980 DEBUG:boto:auth_path=/wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.981 DEBUG:boto:Method: GET 2023-10-12T01:25:05.981 DEBUG:boto:Path: /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.982 DEBUG:boto:Data: 2023-10-12T01:25:05.982 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:05.983 DEBUG:boto:Port: 8000 2023-10-12T01:25:05.984 DEBUG:boto:Params: {} 2023-10-12T01:25:05.984 DEBUG:boto:Token: None 2023-10-12T01:25:05.985 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/obj22?versionId=null 2023-10-12T01:25:05.985 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc= 2023-10-12T01:25:05.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9GFpnRSv2YwtGmmg9XrusPuf9tc=', 'Content-Length': '0'} 2023-10-12T01:25:05.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9f238f353314fd-0065274af1-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:05.988 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:05.989 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:05.989 DEBUG:boto:path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:05.990 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:05.990 DEBUG:boto:Method: GET 2023-10-12T01:25:05.991 DEBUG:boto:Path: /wafmoa-62/?prefix=obj22 2023-10-12T01:25:05.991 DEBUG:boto:Data: 2023-10-12T01:25:05.991 DEBUG:boto:Headers: {} 2023-10-12T01:25:05.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:05.992 DEBUG:boto:Port: 8001 2023-10-12T01:25:05.993 DEBUG:boto:Params: {} 2023-10-12T01:25:05.993 DEBUG:boto:Token: None 2023-10-12T01:25:05.994 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:05 GMT /wafmoa-62/ 2023-10-12T01:25:05.994 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA= 2023-10-12T01:25:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:05 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5erPH9Oz2b4vAeScTgW0T1+lmdA=', 'Content-Length': '0'} 2023-10-12T01:25:05.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc7794cdabc37df-0065274af1-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.042 DEBUG:boto:b'wafmoa-62obj221000falseobj222023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.043 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.044 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.044 DEBUG:boto:path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.045 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.045 DEBUG:boto:Method: GET 2023-10-12T01:25:06.045 DEBUG:boto:Path: /wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.046 DEBUG:boto:Data: 2023-10-12T01:25:06.046 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.047 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.048 DEBUG:boto:Params: {} 2023-10-12T01:25:06.048 DEBUG:boto:Token: None 2023-10-12T01:25:06.049 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dc8dd58458fd1a-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.053 DEBUG:boto:b'wafmoa-62obj221000falseobj222023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.054 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:06.054 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:06.055 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.055 DEBUG:boto:Path: /wafmoa-62/obj22 2023-10-12T01:25:06.055 DEBUG:boto:Data: 2023-10-12T01:25:06.056 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.056 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.057 DEBUG:boto:Params: {} 2023-10-12T01:25:06.057 DEBUG:boto:Token: None 2023-10-12T01:25:06.058 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj22 2023-10-12T01:25:06.058 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NyRF3irWex3rH6wsGREHN4BPYso= 2023-10-12T01:25:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NyRF3irWex3rH6wsGREHN4BPYso=', 'Content-Length': '0'} 2023-10-12T01:25:06.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34ee07e91b054c4-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.061 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:06.062 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:06.062 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.063 DEBUG:boto:Path: /wafmoa-62/obj22 2023-10-12T01:25:06.063 DEBUG:boto:Data: 2023-10-12T01:25:06.063 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.064 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.065 DEBUG:boto:Params: {} 2023-10-12T01:25:06.065 DEBUG:boto:Token: None 2023-10-12T01:25:06.066 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj22 2023-10-12T01:25:06.066 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NyRF3irWex3rH6wsGREHN4BPYso= 2023-10-12T01:25:06.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NyRF3irWex3rH6wsGREHN4BPYso=', 'Content-Length': '0'} 2023-10-12T01:25:06.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007582250e1096a0d2-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:25:06.070 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:06.070 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:06.071 DEBUG:boto:Method: GET 2023-10-12T01:25:06.071 DEBUG:boto:Path: /wafmoa-62/obj22 2023-10-12T01:25:06.071 DEBUG:boto:Data: 2023-10-12T01:25:06.072 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.073 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.073 DEBUG:boto:Params: {} 2023-10-12T01:25:06.074 DEBUG:boto:Token: None 2023-10-12T01:25:06.074 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj22 2023-10-12T01:25:06.075 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yhprftz9h7hw8mo9PPU2N/xlGnQ= 2023-10-12T01:25:06.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yhprftz9h7hw8mo9PPU2N/xlGnQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed54a08fe8a3df33-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.078 DEBUG:boto:path=/wafmoa-62/obj22 2023-10-12T01:25:06.078 DEBUG:boto:auth_path=/wafmoa-62/obj22 2023-10-12T01:25:06.079 DEBUG:boto:Method: GET 2023-10-12T01:25:06.079 DEBUG:boto:Path: /wafmoa-62/obj22 2023-10-12T01:25:06.080 DEBUG:boto:Data: 2023-10-12T01:25:06.080 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.081 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.081 DEBUG:boto:Params: {} 2023-10-12T01:25:06.082 DEBUG:boto:Token: None 2023-10-12T01:25:06.082 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj22 2023-10-12T01:25:06.083 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yhprftz9h7hw8mo9PPU2N/xlGnQ= 2023-10-12T01:25:06.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yhprftz9h7hw8mo9PPU2N/xlGnQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaffc8b4d1a71e43-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.086 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.087 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.087 DEBUG:boto:path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.087 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.088 DEBUG:boto:Method: GET 2023-10-12T01:25:06.088 DEBUG:boto:Path: /wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.089 DEBUG:boto:Data: 2023-10-12T01:25:06.089 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.090 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.090 DEBUG:boto:Params: {} 2023-10-12T01:25:06.091 DEBUG:boto:Token: None 2023-10-12T01:25:06.091 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.092 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a578aaec945ae32-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.095 DEBUG:boto:b'wafmoa-62obj221000falseobj222023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.096 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.096 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.097 DEBUG:boto:path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.097 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.098 DEBUG:boto:Method: GET 2023-10-12T01:25:06.098 DEBUG:boto:Path: /wafmoa-62/?prefix=obj22 2023-10-12T01:25:06.099 DEBUG:boto:Data: 2023-10-12T01:25:06.099 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.100 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.100 DEBUG:boto:Params: {} 2023-10-12T01:25:06.101 DEBUG:boto:Token: None 2023-10-12T01:25:06.101 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.102 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002042294cae42b9ff-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.105 DEBUG:boto:b'wafmoa-62obj221000falseobj222023-10-12T01:23:13.768Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:25:06.106 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.107 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.107 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.107 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.108 DEBUG:boto:Method: GET 2023-10-12T01:25:06.108 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.109 DEBUG:boto:Data: 2023-10-12T01:25:06.109 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.110 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.111 DEBUG:boto:Params: {} 2023-10-12T01:25:06.111 DEBUG:boto:Token: None 2023-10-12T01:25:06.112 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ= 2023-10-12T01:25:06.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e0c99ffc2883db-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.118 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.118 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.119 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.119 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.120 DEBUG:boto:Method: GET 2023-10-12T01:25:06.120 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.121 DEBUG:boto:Data: 2023-10-12T01:25:06.121 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.122 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.122 DEBUG:boto:Params: {} 2023-10-12T01:25:06.123 DEBUG:boto:Token: None 2023-10-12T01:25:06.123 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.124 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ= 2023-10-12T01:25:06.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c020f136e4f7f2d0-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.128 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.128 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.129 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.129 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.129 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.130 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.130 DEBUG:boto:Data: 2023-10-12T01:25:06.131 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.132 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.132 DEBUG:boto:Params: {} 2023-10-12T01:25:06.133 DEBUG:boto:Token: None 2023-10-12T01:25:06.133 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.133 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LUSHFeqgVW0uHcIR5RHsBiEutK0= 2023-10-12T01:25:06.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LUSHFeqgVW0uHcIR5RHsBiEutK0=', 'Content-Length': '0'} 2023-10-12T01:25:06.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002447eaa449dc67ea-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.137 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.137 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.137 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.138 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.138 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.139 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.139 DEBUG:boto:Data: 2023-10-12T01:25:06.139 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.140 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.141 DEBUG:boto:Params: {} 2023-10-12T01:25:06.141 DEBUG:boto:Token: None 2023-10-12T01:25:06.142 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.142 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LUSHFeqgVW0uHcIR5RHsBiEutK0= 2023-10-12T01:25:06.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LUSHFeqgVW0uHcIR5RHsBiEutK0=', 'Content-Length': '0'} 2023-10-12T01:25:06.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c92cb02a051f94c-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:25:06.146 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.146 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.147 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.147 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.147 DEBUG:boto:Method: GET 2023-10-12T01:25:06.148 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.148 DEBUG:boto:Data: 2023-10-12T01:25:06.149 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.149 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.150 DEBUG:boto:Params: {} 2023-10-12T01:25:06.150 DEBUG:boto:Token: None 2023-10-12T01:25:06.151 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.151 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ= 2023-10-12T01:25:06.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026ea3abe1bc4e022-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.194 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.195 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.195 DEBUG:boto:path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.196 DEBUG:boto:auth_path=/wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.196 DEBUG:boto:Method: GET 2023-10-12T01:25:06.197 DEBUG:boto:Path: /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.197 DEBUG:boto:Data: 2023-10-12T01:25:06.197 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.198 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.199 DEBUG:boto:Params: {} 2023-10-12T01:25:06.199 DEBUG:boto:Token: None 2023-10-12T01:25:06.200 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33?versionId=null 2023-10-12T01:25:06.200 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ= 2023-10-12T01:25:06.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SnV2Rx6pSkeXjxEZ9rCULkiwbFQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000087fb5dbcf0ca5-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.204 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.204 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.205 DEBUG:boto:path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.205 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.205 DEBUG:boto:Method: GET 2023-10-12T01:25:06.206 DEBUG:boto:Path: /wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.206 DEBUG:boto:Data: 2023-10-12T01:25:06.207 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.208 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.208 DEBUG:boto:Params: {} 2023-10-12T01:25:06.208 DEBUG:boto:Token: None 2023-10-12T01:25:06.209 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.209 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc666d3f5fb072e-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.213 DEBUG:boto:b'wafmoa-62obj331000falseobj332023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.214 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.214 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.215 DEBUG:boto:path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.215 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.215 DEBUG:boto:Method: GET 2023-10-12T01:25:06.216 DEBUG:boto:Path: /wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.216 DEBUG:boto:Data: 2023-10-12T01:25:06.217 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.218 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.218 DEBUG:boto:Params: {} 2023-10-12T01:25:06.218 DEBUG:boto:Token: None 2023-10-12T01:25:06.219 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.219 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5623530efa5061-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.223 DEBUG:boto:b'wafmoa-62obj331000falseobj332023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.224 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.224 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.225 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.225 DEBUG:boto:Path: /wafmoa-62/obj33 2023-10-12T01:25:06.226 DEBUG:boto:Data: 2023-10-12T01:25:06.226 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.227 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.227 DEBUG:boto:Params: {} 2023-10-12T01:25:06.228 DEBUG:boto:Token: None 2023-10-12T01:25:06.228 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33 2023-10-12T01:25:06.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QxyLNdwwype2iXCAtou0vtch31o= 2023-10-12T01:25:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QxyLNdwwype2iXCAtou0vtch31o=', 'Content-Length': '0'} 2023-10-12T01:25:06.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c130103270319b19-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.231 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.232 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.232 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.233 DEBUG:boto:Path: /wafmoa-62/obj33 2023-10-12T01:25:06.233 DEBUG:boto:Data: 2023-10-12T01:25:06.233 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.234 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.235 DEBUG:boto:Params: {} 2023-10-12T01:25:06.235 DEBUG:boto:Token: None 2023-10-12T01:25:06.236 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33 2023-10-12T01:25:06.236 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QxyLNdwwype2iXCAtou0vtch31o= 2023-10-12T01:25:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QxyLNdwwype2iXCAtou0vtch31o=', 'Content-Length': '0'} 2023-10-12T01:25:06.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60fa58b023eb2b4-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:25:06.239 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.240 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.240 DEBUG:boto:Method: GET 2023-10-12T01:25:06.241 DEBUG:boto:Path: /wafmoa-62/obj33 2023-10-12T01:25:06.241 DEBUG:boto:Data: 2023-10-12T01:25:06.242 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.243 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.243 DEBUG:boto:Params: {} 2023-10-12T01:25:06.244 DEBUG:boto:Token: None 2023-10-12T01:25:06.244 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33 2023-10-12T01:25:06.244 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q5evM8K2Rb7jDIHLaHfp8mCyM6o= 2023-10-12T01:25:06.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q5evM8K2Rb7jDIHLaHfp8mCyM6o=', 'Content-Length': '0'} 2023-10-12T01:25:06.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e4c7bce85a8a62-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.248 DEBUG:boto:path=/wafmoa-62/obj33 2023-10-12T01:25:06.248 DEBUG:boto:auth_path=/wafmoa-62/obj33 2023-10-12T01:25:06.249 DEBUG:boto:Method: GET 2023-10-12T01:25:06.249 DEBUG:boto:Path: /wafmoa-62/obj33 2023-10-12T01:25:06.250 DEBUG:boto:Data: 2023-10-12T01:25:06.250 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.251 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.251 DEBUG:boto:Params: {} 2023-10-12T01:25:06.252 DEBUG:boto:Token: None 2023-10-12T01:25:06.252 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj33 2023-10-12T01:25:06.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Q5evM8K2Rb7jDIHLaHfp8mCyM6o= 2023-10-12T01:25:06.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Q5evM8K2Rb7jDIHLaHfp8mCyM6o=', 'Content-Length': '0'} 2023-10-12T01:25:06.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb5c98032a90daf-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.298 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.299 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.299 DEBUG:boto:path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.300 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.300 DEBUG:boto:Method: GET 2023-10-12T01:25:06.301 DEBUG:boto:Path: /wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.301 DEBUG:boto:Data: 2023-10-12T01:25:06.302 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.303 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.303 DEBUG:boto:Params: {} 2023-10-12T01:25:06.304 DEBUG:boto:Token: None 2023-10-12T01:25:06.304 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.305 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007238e9be24118853-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.309 DEBUG:boto:b'wafmoa-62obj331000falseobj332023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.309 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.310 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.310 DEBUG:boto:path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.311 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.311 DEBUG:boto:Method: GET 2023-10-12T01:25:06.312 DEBUG:boto:Path: /wafmoa-62/?prefix=obj33 2023-10-12T01:25:06.312 DEBUG:boto:Data: 2023-10-12T01:25:06.312 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.313 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.314 DEBUG:boto:Params: {} 2023-10-12T01:25:06.314 DEBUG:boto:Token: None 2023-10-12T01:25:06.315 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.315 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de5b3ff4f837f53-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.321 DEBUG:boto:b'wafmoa-62obj331000falseobj332023-10-12T01:23:13.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:25:06.323 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.323 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.324 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.324 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.324 DEBUG:boto:Method: GET 2023-10-12T01:25:06.325 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.325 DEBUG:boto:Data: 2023-10-12T01:25:06.326 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.327 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.327 DEBUG:boto:Params: {} 2023-10-12T01:25:06.328 DEBUG:boto:Token: None 2023-10-12T01:25:06.328 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.329 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY= 2023-10-12T01:25:06.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY=', 'Content-Length': '0'} 2023-10-12T01:25:06.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c86e5a9068e1c3-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.332 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.333 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.333 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.334 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.334 DEBUG:boto:Method: GET 2023-10-12T01:25:06.335 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.335 DEBUG:boto:Data: 2023-10-12T01:25:06.335 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.336 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.337 DEBUG:boto:Params: {} 2023-10-12T01:25:06.337 DEBUG:boto:Token: None 2023-10-12T01:25:06.338 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY= 2023-10-12T01:25:06.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY=', 'Content-Length': '0'} 2023-10-12T01:25:06.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51b1918f7b0774e-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.342 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.342 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.343 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.343 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.343 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.344 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.344 DEBUG:boto:Data: 2023-10-12T01:25:06.345 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.345 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.346 DEBUG:boto:Params: {} 2023-10-12T01:25:06.346 DEBUG:boto:Token: None 2023-10-12T01:25:06.347 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.347 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qvSY3ZTj7xqmfhqwtFRF48n+YRw= 2023-10-12T01:25:06.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qvSY3ZTj7xqmfhqwtFRF48n+YRw=', 'Content-Length': '0'} 2023-10-12T01:25:06.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006565a2db51b8d48d-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.350 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.351 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.351 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.352 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.352 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.353 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.353 DEBUG:boto:Data: 2023-10-12T01:25:06.353 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.354 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.355 DEBUG:boto:Params: {} 2023-10-12T01:25:06.355 DEBUG:boto:Token: None 2023-10-12T01:25:06.356 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.356 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qvSY3ZTj7xqmfhqwtFRF48n+YRw= 2023-10-12T01:25:06.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qvSY3ZTj7xqmfhqwtFRF48n+YRw=', 'Content-Length': '0'} 2023-10-12T01:25:06.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4db501a016642cf-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:25:06.359 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.360 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.360 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.361 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.361 DEBUG:boto:Method: GET 2023-10-12T01:25:06.362 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.362 DEBUG:boto:Data: 2023-10-12T01:25:06.363 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.364 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.364 DEBUG:boto:Params: {} 2023-10-12T01:25:06.364 DEBUG:boto:Token: None 2023-10-12T01:25:06.365 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.365 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY= 2023-10-12T01:25:06.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY=', 'Content-Length': '0'} 2023-10-12T01:25:06.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060731745462eb41e-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.410 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.411 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.412 DEBUG:boto:path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.412 DEBUG:boto:auth_path=/wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.412 DEBUG:boto:Method: GET 2023-10-12T01:25:06.413 DEBUG:boto:Path: /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.413 DEBUG:boto:Data: 2023-10-12T01:25:06.414 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.415 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.415 DEBUG:boto:Params: {} 2023-10-12T01:25:06.415 DEBUG:boto:Token: None 2023-10-12T01:25:06.416 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44?versionId=null 2023-10-12T01:25:06.417 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY= 2023-10-12T01:25:06.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DvX0H9qcuVXHoqaa1ieL5ty8lvY=', 'Content-Length': '0'} 2023-10-12T01:25:06.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b55ce5a2f7a438-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.420 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.420 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.421 DEBUG:boto:path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.421 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.422 DEBUG:boto:Method: GET 2023-10-12T01:25:06.422 DEBUG:boto:Path: /wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.423 DEBUG:boto:Data: 2023-10-12T01:25:06.423 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.424 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.424 DEBUG:boto:Params: {} 2023-10-12T01:25:06.425 DEBUG:boto:Token: None 2023-10-12T01:25:06.425 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.426 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a62e2e4b2b0c7a53-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.429 DEBUG:boto:b'wafmoa-62obj441000falseobj442023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.430 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.431 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.431 DEBUG:boto:path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.431 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.432 DEBUG:boto:Method: GET 2023-10-12T01:25:06.432 DEBUG:boto:Path: /wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.433 DEBUG:boto:Data: 2023-10-12T01:25:06.433 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.434 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.434 DEBUG:boto:Params: {} 2023-10-12T01:25:06.435 DEBUG:boto:Token: None 2023-10-12T01:25:06.435 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.436 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f78b6d27e87034-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.486 DEBUG:boto:b'wafmoa-62obj441000falseobj442023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.487 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.488 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.488 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.488 DEBUG:boto:Path: /wafmoa-62/obj44 2023-10-12T01:25:06.489 DEBUG:boto:Data: 2023-10-12T01:25:06.489 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.490 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.491 DEBUG:boto:Params: {} 2023-10-12T01:25:06.491 DEBUG:boto:Token: None 2023-10-12T01:25:06.492 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44 2023-10-12T01:25:06.492 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mgTdqNTDJ7PPxt0ueRaD95+HCno= 2023-10-12T01:25:06.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mgTdqNTDJ7PPxt0ueRaD95+HCno=', 'Content-Length': '0'} 2023-10-12T01:25:06.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdf1c2cd9eae2e6-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.496 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.496 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.496 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.497 DEBUG:boto:Path: /wafmoa-62/obj44 2023-10-12T01:25:06.497 DEBUG:boto:Data: 2023-10-12T01:25:06.498 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.499 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.499 DEBUG:boto:Params: {} 2023-10-12T01:25:06.499 DEBUG:boto:Token: None 2023-10-12T01:25:06.500 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44 2023-10-12T01:25:06.500 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mgTdqNTDJ7PPxt0ueRaD95+HCno= 2023-10-12T01:25:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mgTdqNTDJ7PPxt0ueRaD95+HCno=', 'Content-Length': '0'} 2023-10-12T01:25:06.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6642331570f11f-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:25:06.504 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.505 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.505 DEBUG:boto:Method: GET 2023-10-12T01:25:06.506 DEBUG:boto:Path: /wafmoa-62/obj44 2023-10-12T01:25:06.506 DEBUG:boto:Data: 2023-10-12T01:25:06.506 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.507 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.508 DEBUG:boto:Params: {} 2023-10-12T01:25:06.508 DEBUG:boto:Token: None 2023-10-12T01:25:06.509 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44 2023-10-12T01:25:06.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yX2+/mdHj2TiStg/LDGZKOgQ2Gk= 2023-10-12T01:25:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yX2+/mdHj2TiStg/LDGZKOgQ2Gk=', 'Content-Length': '0'} 2023-10-12T01:25:06.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0a520742f8accd-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.512 DEBUG:boto:path=/wafmoa-62/obj44 2023-10-12T01:25:06.513 DEBUG:boto:auth_path=/wafmoa-62/obj44 2023-10-12T01:25:06.513 DEBUG:boto:Method: GET 2023-10-12T01:25:06.514 DEBUG:boto:Path: /wafmoa-62/obj44 2023-10-12T01:25:06.514 DEBUG:boto:Data: 2023-10-12T01:25:06.515 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.516 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.516 DEBUG:boto:Params: {} 2023-10-12T01:25:06.517 DEBUG:boto:Token: None 2023-10-12T01:25:06.517 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj44 2023-10-12T01:25:06.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yX2+/mdHj2TiStg/LDGZKOgQ2Gk= 2023-10-12T01:25:06.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yX2+/mdHj2TiStg/LDGZKOgQ2Gk=', 'Content-Length': '0'} 2023-10-12T01:25:06.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007692746b37ea6b71-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.521 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.521 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.522 DEBUG:boto:path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.522 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.523 DEBUG:boto:Method: GET 2023-10-12T01:25:06.525 DEBUG:boto:Path: /wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.526 DEBUG:boto:Data: 2023-10-12T01:25:06.526 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.527 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.527 DEBUG:boto:Params: {} 2023-10-12T01:25:06.528 DEBUG:boto:Token: None 2023-10-12T01:25:06.528 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.529 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc1f4120ff76a97-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.533 DEBUG:boto:b'wafmoa-62obj441000falseobj442023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.533 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.534 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.534 DEBUG:boto:path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.535 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.535 DEBUG:boto:Method: GET 2023-10-12T01:25:06.536 DEBUG:boto:Path: /wafmoa-62/?prefix=obj44 2023-10-12T01:25:06.536 DEBUG:boto:Data: 2023-10-12T01:25:06.536 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.537 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.538 DEBUG:boto:Params: {} 2023-10-12T01:25:06.538 DEBUG:boto:Token: None 2023-10-12T01:25:06.539 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.539 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf03817b426fce5-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.543 DEBUG:boto:b'wafmoa-62obj441000falseobj442023-10-12T01:23:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:25:06.544 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.544 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.545 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.545 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.546 DEBUG:boto:Method: GET 2023-10-12T01:25:06.546 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.546 DEBUG:boto:Data: 2023-10-12T01:25:06.547 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.548 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.548 DEBUG:boto:Params: {} 2023-10-12T01:25:06.549 DEBUG:boto:Token: None 2023-10-12T01:25:06.549 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q= 2023-10-12T01:25:06.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q=', 'Content-Length': '0'} 2023-10-12T01:25:06.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a93af4709622f4cb-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.553 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.554 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.554 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.555 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.555 DEBUG:boto:Method: GET 2023-10-12T01:25:06.556 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.556 DEBUG:boto:Data: 2023-10-12T01:25:06.556 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.557 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.558 DEBUG:boto:Params: {} 2023-10-12T01:25:06.558 DEBUG:boto:Token: None 2023-10-12T01:25:06.559 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.559 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q= 2023-10-12T01:25:06.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q=', 'Content-Length': '0'} 2023-10-12T01:25:06.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d88fe6b03ca9b76-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.602 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.603 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.603 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.604 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.604 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.605 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.605 DEBUG:boto:Data: 2023-10-12T01:25:06.605 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.606 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.607 DEBUG:boto:Params: {} 2023-10-12T01:25:06.607 DEBUG:boto:Token: None 2023-10-12T01:25:06.608 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cqf1nLDH8B38TyvOweHgewjlZ4k= 2023-10-12T01:25:06.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cqf1nLDH8B38TyvOweHgewjlZ4k=', 'Content-Length': '0'} 2023-10-12T01:25:06.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030eb458c8899b596-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.611 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.612 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.612 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.612 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.613 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.613 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.614 DEBUG:boto:Data: 2023-10-12T01:25:06.614 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.615 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.615 DEBUG:boto:Params: {} 2023-10-12T01:25:06.616 DEBUG:boto:Token: None 2023-10-12T01:25:06.616 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.617 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cqf1nLDH8B38TyvOweHgewjlZ4k= 2023-10-12T01:25:06.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cqf1nLDH8B38TyvOweHgewjlZ4k=', 'Content-Length': '0'} 2023-10-12T01:25:06.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eec01e655333a26-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:25:06.620 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.621 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.621 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.622 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.622 DEBUG:boto:Method: GET 2023-10-12T01:25:06.622 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.623 DEBUG:boto:Data: 2023-10-12T01:25:06.623 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.624 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.625 DEBUG:boto:Params: {} 2023-10-12T01:25:06.625 DEBUG:boto:Token: None 2023-10-12T01:25:06.626 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.626 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q= 2023-10-12T01:25:06.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q=', 'Content-Length': '0'} 2023-10-12T01:25:06.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b97eb0df75f7b4-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.670 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.671 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.671 DEBUG:boto:path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.672 DEBUG:boto:auth_path=/wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.672 DEBUG:boto:Method: GET 2023-10-12T01:25:06.673 DEBUG:boto:Path: /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.673 DEBUG:boto:Data: 2023-10-12T01:25:06.674 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.675 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.675 DEBUG:boto:Params: {} 2023-10-12T01:25:06.676 DEBUG:boto:Token: None 2023-10-12T01:25:06.676 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55?versionId=null 2023-10-12T01:25:06.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q= 2023-10-12T01:25:06.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tooXVcwCjFqQu0rjwNVVm/Yjs7Q=', 'Content-Length': '0'} 2023-10-12T01:25:06.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0da8bc814b7161e-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.683 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.683 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.684 DEBUG:boto:path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.684 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.685 DEBUG:boto:Method: GET 2023-10-12T01:25:06.685 DEBUG:boto:Path: /wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.685 DEBUG:boto:Data: 2023-10-12T01:25:06.686 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.687 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.687 DEBUG:boto:Params: {} 2023-10-12T01:25:06.688 DEBUG:boto:Token: None 2023-10-12T01:25:06.688 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.689 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a27abee7c747392-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.692 DEBUG:boto:b'wafmoa-62obj551000falseobj552023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.693 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.694 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.694 DEBUG:boto:path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.695 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.695 DEBUG:boto:Method: GET 2023-10-12T01:25:06.695 DEBUG:boto:Path: /wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.696 DEBUG:boto:Data: 2023-10-12T01:25:06.696 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.697 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.698 DEBUG:boto:Params: {} 2023-10-12T01:25:06.698 DEBUG:boto:Token: None 2023-10-12T01:25:06.699 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.699 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665d50335704f8fb-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.703 DEBUG:boto:b'wafmoa-62obj551000falseobj552023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.704 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.704 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.705 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.705 DEBUG:boto:Path: /wafmoa-62/obj55 2023-10-12T01:25:06.706 DEBUG:boto:Data: 2023-10-12T01:25:06.706 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.707 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.707 DEBUG:boto:Params: {} 2023-10-12T01:25:06.708 DEBUG:boto:Token: None 2023-10-12T01:25:06.708 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55 2023-10-12T01:25:06.709 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6w1BuWn3igHY3xgSC/3hqGDxYQ= 2023-10-12T01:25:06.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6w1BuWn3igHY3xgSC/3hqGDxYQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850e36a1bc12c334-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.712 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.712 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.713 DEBUG:boto:Method: HEAD 2023-10-12T01:25:06.713 DEBUG:boto:Path: /wafmoa-62/obj55 2023-10-12T01:25:06.714 DEBUG:boto:Data: 2023-10-12T01:25:06.714 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.715 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.715 DEBUG:boto:Params: {} 2023-10-12T01:25:06.716 DEBUG:boto:Token: None 2023-10-12T01:25:06.716 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55 2023-10-12T01:25:06.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S6w1BuWn3igHY3xgSC/3hqGDxYQ= 2023-10-12T01:25:06.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S6w1BuWn3igHY3xgSC/3hqGDxYQ=', 'Content-Length': '0'} 2023-10-12T01:25:06.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006dbd9be7bf9b540-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:25:06.720 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.721 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.721 DEBUG:boto:Method: GET 2023-10-12T01:25:06.722 DEBUG:boto:Path: /wafmoa-62/obj55 2023-10-12T01:25:06.722 DEBUG:boto:Data: 2023-10-12T01:25:06.723 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.723 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.724 DEBUG:boto:Params: {} 2023-10-12T01:25:06.724 DEBUG:boto:Token: None 2023-10-12T01:25:06.725 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55 2023-10-12T01:25:06.725 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xjwqnvBlu49U4PDMkv1hlhiDVOs= 2023-10-12T01:25:06.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xjwqnvBlu49U4PDMkv1hlhiDVOs=', 'Content-Length': '0'} 2023-10-12T01:25:06.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-62:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3dae6a4417b996-0065274af2-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.731 DEBUG:boto:path=/wafmoa-62/obj55 2023-10-12T01:25:06.731 DEBUG:boto:auth_path=/wafmoa-62/obj55 2023-10-12T01:25:06.732 DEBUG:boto:Method: GET 2023-10-12T01:25:06.732 DEBUG:boto:Path: /wafmoa-62/obj55 2023-10-12T01:25:06.732 DEBUG:boto:Data: 2023-10-12T01:25:06.733 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.734 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.734 DEBUG:boto:Params: {} 2023-10-12T01:25:06.735 DEBUG:boto:Token: None 2023-10-12T01:25:06.735 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/obj55 2023-10-12T01:25:06.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xjwqnvBlu49U4PDMkv1hlhiDVOs= 2023-10-12T01:25:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xjwqnvBlu49U4PDMkv1hlhiDVOs=', 'Content-Length': '0'} 2023-10-12T01:25:06.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:23:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9819023ec38e1e0-0065274af2-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.782 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.783 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.784 DEBUG:boto:path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.784 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.784 DEBUG:boto:Method: GET 2023-10-12T01:25:06.785 DEBUG:boto:Path: /wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.785 DEBUG:boto:Data: 2023-10-12T01:25:06.786 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:25:06.787 DEBUG:boto:Port: 8001 2023-10-12T01:25:06.787 DEBUG:boto:Params: {} 2023-10-12T01:25:06.787 DEBUG:boto:Token: None 2023-10-12T01:25:06.788 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.788 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced80e4b59c828fb-0065274af2-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.792 DEBUG:boto:b'wafmoa-62obj551000falseobj552023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.793 DEBUG:boto:path=/wafmoa-62/ 2023-10-12T01:25:06.793 DEBUG:boto:auth_path=/wafmoa-62/ 2023-10-12T01:25:06.794 DEBUG:boto:path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.794 DEBUG:boto:auth_path=/wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.795 DEBUG:boto:Method: GET 2023-10-12T01:25:06.795 DEBUG:boto:Path: /wafmoa-62/?prefix=obj55 2023-10-12T01:25:06.796 DEBUG:boto:Data: 2023-10-12T01:25:06.796 DEBUG:boto:Headers: {} 2023-10-12T01:25:06.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:06.797 DEBUG:boto:Port: 8000 2023-10-12T01:25:06.797 DEBUG:boto:Params: {} 2023-10-12T01:25:06.798 DEBUG:boto:Token: None 2023-10-12T01:25:06.798 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:25:06 GMT /wafmoa-62/ 2023-10-12T01:25:06.799 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM= 2023-10-12T01:25:06.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:06 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KClGbPA9eIrGMAVA8a6cWhDJwuM=', 'Content-Length': '0'} 2023-10-12T01:25:06.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000edd146a6d5ab72e-0065274af2-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:06.802 DEBUG:boto:b'wafmoa-62obj551000falseobj552023-10-12T01:23:13.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:25:06.803 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-62 zones={a2, a1} 2023-10-12T01:25:06.804 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-12T01:25:06.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:06.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:06.805 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:06.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:06.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:06.914 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:25:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000023563", 2023-10-12T01:25:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:17:04.510533Z" 2023-10-12T01:25:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-12T01:25:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:23:13.851875Z" 2023-10-12T01:25:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033008", 2023-10-12T01:25:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:23:18.517434Z" 2023-10-12T01:25:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032773", 2023-10-12T01:25:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:23:18.519156Z" 2023-10-12T01:25:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:06.918 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:25:06.920 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037529', 2: '00000000000000000000:00000000000000033008', 3: '00000000000000000000:00000000000000032773'} 2023-10-12T01:25:06.920 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:25:06.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:06.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:06.922 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:06.973 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:06.974 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:07.020 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:25:07.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:25:07.021 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:25:07.021 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:25:07.021 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:25:07.022 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 15514057470930744787 2023-10-12T01:25:07.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.022 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:25:07.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:25:07.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.023 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000023563", 2023-10-12T01:25:07.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:25:07.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:17:04.510533Z" 2023-10-12T01:25:07.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.025 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:25:07.025 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-12T01:25:07.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 53, 2023-10-12T01:25:07.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:23:13.851875Z" 2023-10-12T01:25:07.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.028 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:25:07.028 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.028 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.028 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033008", 2023-10-12T01:25:07.028 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.029 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.029 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:23:18.517434Z" 2023-10-12T01:25:07.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.030 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:25:07.030 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.030 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.031 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032773", 2023-10-12T01:25:07.031 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.031 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.031 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.031 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:23:18.519156Z" 2023-10-12T01:25:07.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:25:07.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.032 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:25:07.033 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 62, 2023-10-12T01:25:07.033 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 62 2023-10-12T01:25:07.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.033 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:25:07Z" 2023-10-12T01:25:07.034 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:25:07.035 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15514057470930744787 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023563", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:17:04.510533Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-12T01:23:13.851875Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033008", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:23:18.517434Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032773", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:23:18.519156Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-12T01:25:07Z" } 2023-10-12T01:25:07.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023563', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:17:04.510533Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-12T01:23:13.851875Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033008', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:23:18.517434Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032773', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:23:18.519156Z'}}] 2023-10-12T01:25:07.036 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037529', 2: '00000000000000000000:00000000000000033008', 3: '00000000000000000000:00000000000000032773'} 2023-10-12T01:25:07.036 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000023563', 1: '00000000000000000000:00000000000000037529', 2: '00000000000000000000:00000000000000033008', 3: '00000000000000000000:00000000000000032773'} 2023-10-12T01:25:07.037 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:25:07.037 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-12T01:25:07.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.038 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:07.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:07.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:07.135 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:25:07.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-10-12T01:25:07.136 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:18:47.639595Z" 2023-10-12T01:25:07.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015306", 2023-10-12T01:25:07.137 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:24:48.413279Z" 2023-10-12T01:25:07.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014044", 2023-10-12T01:25:07.137 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:25:04.141656Z" 2023-10-12T01:25:07.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.138 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-10-12T01:25:07.138 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:25:04.143766Z" 2023-10-12T01:25:07.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.139 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:25:07.140 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015306', 2: '00000000000000000000:00000000000000014044', 3: '00000000000000000000:00000000000000011678'} 2023-10-12T01:25:07.140 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:25:07.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.142 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:07.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:07.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:07.236 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:25:07.237 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:25:07.237 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:25:07.237 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:25:07.238 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:25:07.238 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T01:25:07.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.238 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:25:07.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.239 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:25:07.239 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.239 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.239 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005723", 2023-10-12T01:25:07.240 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.240 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.240 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.240 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:18:47.639595Z" 2023-10-12T01:25:07.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.241 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:25:07.241 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.241 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.242 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015306", 2023-10-12T01:25:07.242 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.242 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.242 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.243 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:24:48.413279Z" 2023-10-12T01:25:07.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.243 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:25:07.244 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.244 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.244 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014044", 2023-10-12T01:25:07.244 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.244 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.245 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.245 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:25:04.141656Z" 2023-10-12T01:25:07.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.246 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:25:07.246 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.246 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:25:07.246 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-10-12T01:25:07.247 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.247 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.247 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.247 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:25:04.143766Z" 2023-10-12T01:25:07.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.248 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:25:07.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.248 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:25:07.249 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T01:25:07.249 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T01:25:07.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.249 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:25:07Z" 2023-10-12T01:25:07.249 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:25:07.250 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:18:47.639595Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:24:48.413279Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014044", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:25:04.141656Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011678", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:25:04.143766Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T01:25:07Z" } 2023-10-12T01:25:07.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:18:47.639595Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:24:48.413279Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014044', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:25:04.141656Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011678', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:25:04.143766Z'}}] 2023-10-12T01:25:07.251 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015306', 2: '00000000000000000000:00000000000000014044', 3: '00000000000000000000:00000000000000011678'} 2023-10-12T01:25:07.252 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005723', 1: '00000000000000000000:00000000000000015306', 2: '00000000000000000000:00000000000000014044', 3: '00000000000000000000:00000000000000011678'} 2023-10-12T01:25:07.252 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:25:07.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-10-12T01:25:07.254 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:07.254 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:07.255 DEBUG:boto:Method: PUT 2023-10-12T01:25:07.255 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:07.256 DEBUG:boto:Data: 2023-10-12T01:25:07.256 DEBUG:boto:Headers: {} 2023-10-12T01:25:07.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:07.257 DEBUG:boto:Port: 8000 2023-10-12T01:25:07.257 DEBUG:boto:Params: {} 2023-10-12T01:25:07.258 DEBUG:boto:Token: None 2023-10-12T01:25:07.258 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:25:07 GMT /wafmoa-63/ 2023-10-12T01:25:07.259 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4cTpPqbe4rTNgE6yo/fL3yMXG1Q= 2023-10-12T01:25:07.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4cTpPqbe4rTNgE6yo/fL3yMXG1Q=', 'Content-Length': '0'} 2023-10-12T01:25:07.274 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000764e5bb5cb966634-0065274af3-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:07.274 DEBUG:rgw_multi.tests:created bucket=wafmoa-63 2023-10-12T01:25:07.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.276 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:07.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:07.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:07.353 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:25:07.366 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:25:07.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:25:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:25:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:25:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:25:07.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.368 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073907.272991_694.1", 2023-10-12T01:25:07.368 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:25:07.272991Z" 2023-10-12T01:25:07.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:25:07.369 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:25:07.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.369 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:25:07.370 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:25:07.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:25:07.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:07.372 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:07.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:07.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:07.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:07.457+0000 7f4d81570780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:25:07.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:07.458+0000 7f4d81570780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:25:07.474 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:25:07.474 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:25:07.474 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:25:07.474 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:25:07.475 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:25:07.475 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:25:07.475 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:07.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.476 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:25:07.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.476 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:25:07.476 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.476 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:07.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:25:07.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.477 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:25:07.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.477 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:25:07.478 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:07.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:25:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:25:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073641.852711_690.1", 2023-10-12T01:25:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:20:41.852711Z", 2023-10-12T01:25:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:25:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:25:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:25:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:25:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:25:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:25:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:25:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:25:07Z" 2023-10-12T01:25:07.487 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:25:07.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073641.852711_690.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:20:41.852711Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:25:07Z" } 2023-10-12T01:25:07.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073641.852711_690.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:20:41.852711Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:25:07.490 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:25:07.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073641.852711_690.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:25:07.490 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697073907.272991_694.1 target=1_1697073641.852711_690.1 2023-10-12T01:25:12.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:12.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:25:12.493 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:25:12.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:12.549 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:12.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:12.579+0000 7f500174e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:25:12.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:12.580+0000 7f500174e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:25:12.594 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:25:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:25:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:25:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:25:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:25:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:25:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:25:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:25:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:25:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:25:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:25:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:12.600 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:25:12.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:12.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:12.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:25:12.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:12.601 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:12.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:12.602 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:25:12.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:12.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:12.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:12.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:25:12.604 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:12.605 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:12.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073907.272991_694.1", 2023-10-12T01:25:12.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:12.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:12.606 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:12.606 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:25:07.272991Z", 2023-10-12T01:25:12.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:12.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:12.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:25:12.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:25:12.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:25:12.609 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:25:12.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:25:12.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:25:12.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:25:12.610 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:25:12.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:25:12.611 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:25:12.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:12.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:25:12.612 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:25:12.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.612 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:25:12.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:25:12.613 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:25:12.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:25:12.613 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:25:12Z" 2023-10-12T01:25:12.613 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:25:12.615 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697073907.272991_694.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:25:07.272991Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:25:12Z" } 2023-10-12T01:25:12.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073907.272991_694.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:25:07.272991Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:25:12.616 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:25:12.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:25:12.616 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:25:12.617 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.617 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.618 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.618 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.619 DEBUG:boto:Data: 2023-10-12T01:25:12.619 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.620 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.621 DEBUG:boto:Params: {} 2023-10-12T01:25:12.622 DEBUG:boto:Token: None 2023-10-12T01:25:12.623 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000477a20c628141b25-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.628 DEBUG:boto:path=/wafmoa-63/obj11 2023-10-12T01:25:12.628 DEBUG:boto:auth_path=/wafmoa-63/obj11 2023-10-12T01:25:12.628 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.629 DEBUG:boto:Path: /wafmoa-63/obj11 2023-10-12T01:25:12.629 DEBUG:boto:Data: 2023-10-12T01:25:12.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.631 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.631 DEBUG:boto:Params: {} 2023-10-12T01:25:12.632 DEBUG:boto:Token: None 2023-10-12T01:25:12.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj11 2023-10-12T01:25:12.633 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C6wxCu0gqiry7gFYy+QKDhrBsPM= 2023-10-12T01:25:12.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C6wxCu0gqiry7gFYy+QKDhrBsPM='} 2023-10-12T01:25:12.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2e54c30ec478bcb-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.640 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.640 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.641 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.641 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.642 DEBUG:boto:Data: 2023-10-12T01:25:12.642 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.643 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.643 DEBUG:boto:Params: {} 2023-10-12T01:25:12.644 DEBUG:boto:Token: None 2023-10-12T01:25:12.644 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.645 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008141fe71a249f16f-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.648 DEBUG:boto:path=/wafmoa-63/obj22 2023-10-12T01:25:12.649 DEBUG:boto:auth_path=/wafmoa-63/obj22 2023-10-12T01:25:12.649 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.650 DEBUG:boto:Path: /wafmoa-63/obj22 2023-10-12T01:25:12.650 DEBUG:boto:Data: 2023-10-12T01:25:12.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.652 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.652 DEBUG:boto:Params: {} 2023-10-12T01:25:12.653 DEBUG:boto:Token: None 2023-10-12T01:25:12.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj22 2023-10-12T01:25:12.654 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8FULwpja/oK0HeQU9p8AsJjNtiI= 2023-10-12T01:25:12.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8FULwpja/oK0HeQU9p8AsJjNtiI='} 2023-10-12T01:25:12.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daf9ed7830c6bf6f-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.703 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.704 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.704 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.705 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.705 DEBUG:boto:Data: 2023-10-12T01:25:12.705 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.706 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.707 DEBUG:boto:Params: {} 2023-10-12T01:25:12.707 DEBUG:boto:Token: None 2023-10-12T01:25:12.708 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000330255d270983df7-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.711 DEBUG:boto:path=/wafmoa-63/obj33 2023-10-12T01:25:12.712 DEBUG:boto:auth_path=/wafmoa-63/obj33 2023-10-12T01:25:12.712 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.713 DEBUG:boto:Path: /wafmoa-63/obj33 2023-10-12T01:25:12.713 DEBUG:boto:Data: 2023-10-12T01:25:12.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.714 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.715 DEBUG:boto:Params: {} 2023-10-12T01:25:12.715 DEBUG:boto:Token: None 2023-10-12T01:25:12.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj33 2023-10-12T01:25:12.716 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GHYAsjxj1pnsiABrakIvCUDILnM= 2023-10-12T01:25:12.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GHYAsjxj1pnsiABrakIvCUDILnM='} 2023-10-12T01:25:12.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a2acdb477a53d5a-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.722 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.723 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.723 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.724 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.724 DEBUG:boto:Data: 2023-10-12T01:25:12.724 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.725 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.726 DEBUG:boto:Params: {} 2023-10-12T01:25:12.726 DEBUG:boto:Token: None 2023-10-12T01:25:12.727 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065373d4f0658d5e6-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.731 DEBUG:boto:path=/wafmoa-63/obj44 2023-10-12T01:25:12.731 DEBUG:boto:auth_path=/wafmoa-63/obj44 2023-10-12T01:25:12.732 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.732 DEBUG:boto:Path: /wafmoa-63/obj44 2023-10-12T01:25:12.733 DEBUG:boto:Data: 2023-10-12T01:25:12.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.734 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.735 DEBUG:boto:Params: {} 2023-10-12T01:25:12.735 DEBUG:boto:Token: None 2023-10-12T01:25:12.736 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj44 2023-10-12T01:25:12.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lvyzW1H7M6qVemD+wtU3RjqiEvg= 2023-10-12T01:25:12.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lvyzW1H7M6qVemD+wtU3RjqiEvg='} 2023-10-12T01:25:12.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd228e0b5ab0c79c-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.787 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.788 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.788 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.788 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.789 DEBUG:boto:Data: 2023-10-12T01:25:12.789 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.790 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.790 DEBUG:boto:Params: {} 2023-10-12T01:25:12.792 DEBUG:boto:Token: None 2023-10-12T01:25:12.792 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001944a7f56fcc8fa-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.797 DEBUG:boto:path=/wafmoa-63/obj55 2023-10-12T01:25:12.797 DEBUG:boto:auth_path=/wafmoa-63/obj55 2023-10-12T01:25:12.797 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.798 DEBUG:boto:Path: /wafmoa-63/obj55 2023-10-12T01:25:12.798 DEBUG:boto:Data: 2023-10-12T01:25:12.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.800 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.800 DEBUG:boto:Params: {} 2023-10-12T01:25:12.801 DEBUG:boto:Token: None 2023-10-12T01:25:12.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj55 2023-10-12T01:25:12.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xma0COUtyNWBbfy2d2pREhtegh8= 2023-10-12T01:25:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xma0COUtyNWBbfy2d2pREhtegh8='} 2023-10-12T01:25:12.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060276106f20a0a3f-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:12.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:12.808 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:12.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:12.863 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:12.904 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:12.904 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:12.908 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-10-12T01:25:12.944 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.944 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.945 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.945 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.946 DEBUG:boto:Data: 2023-10-12T01:25:12.946 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.947 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.948 DEBUG:boto:Params: {} 2023-10-12T01:25:12.948 DEBUG:boto:Token: None 2023-10-12T01:25:12.949 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.949 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c1791a02e592f7-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.954 DEBUG:boto:path=/wafmoa-63/obj17 2023-10-12T01:25:12.954 DEBUG:boto:auth_path=/wafmoa-63/obj17 2023-10-12T01:25:12.955 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.955 DEBUG:boto:Path: /wafmoa-63/obj17 2023-10-12T01:25:12.955 DEBUG:boto:Data: 2023-10-12T01:25:12.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.957 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.958 DEBUG:boto:Params: {} 2023-10-12T01:25:12.958 DEBUG:boto:Token: None 2023-10-12T01:25:12.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj17 2023-10-12T01:25:12.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xv1EDN5c70xQKxwH6kNmYjRD/WY= 2023-10-12T01:25:12.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xv1EDN5c70xQKxwH6kNmYjRD/WY='} 2023-10-12T01:25:12.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf6f47f7e8021af3-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.966 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:12.966 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:12.966 DEBUG:boto:Method: HEAD 2023-10-12T01:25:12.967 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:12.967 DEBUG:boto:Data: 2023-10-12T01:25:12.968 DEBUG:boto:Headers: {} 2023-10-12T01:25:12.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.969 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.970 DEBUG:boto:Params: {} 2023-10-12T01:25:12.970 DEBUG:boto:Token: None 2023-10-12T01:25:12.971 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/ 2023-10-12T01:25:12.971 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:25:12.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:25:12.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d18291a360482b27-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:12.975 DEBUG:boto:path=/wafmoa-63/obj34 2023-10-12T01:25:12.976 DEBUG:boto:auth_path=/wafmoa-63/obj34 2023-10-12T01:25:12.976 DEBUG:boto:Method: PUT 2023-10-12T01:25:12.976 DEBUG:boto:Path: /wafmoa-63/obj34 2023-10-12T01:25:12.977 DEBUG:boto:Data: 2023-10-12T01:25:12.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:12.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:12.978 DEBUG:boto:Port: 8000 2023-10-12T01:25:12.978 DEBUG:boto:Params: {} 2023-10-12T01:25:12.979 DEBUG:boto:Token: None 2023-10-12T01:25:12.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:12 GMT /wafmoa-63/obj34 2023-10-12T01:25:12.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wjnLKy8auPBzkwzXzEh/51RCmF0= 2023-10-12T01:25:12.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wjnLKy8auPBzkwzXzEh/51RCmF0='} 2023-10-12T01:25:13.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f04a63ad869a61d-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.023 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.024 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.024 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.024 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.025 DEBUG:boto:Data: 2023-10-12T01:25:13.025 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.026 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.027 DEBUG:boto:Params: {} 2023-10-12T01:25:13.027 DEBUG:boto:Token: None 2023-10-12T01:25:13.028 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.028 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfc87b1726f3bf44-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.032 DEBUG:boto:path=/wafmoa-63/obj51 2023-10-12T01:25:13.032 DEBUG:boto:auth_path=/wafmoa-63/obj51 2023-10-12T01:25:13.032 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.033 DEBUG:boto:Path: /wafmoa-63/obj51 2023-10-12T01:25:13.033 DEBUG:boto:Data: 2023-10-12T01:25:13.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.035 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.035 DEBUG:boto:Params: {} 2023-10-12T01:25:13.035 DEBUG:boto:Token: None 2023-10-12T01:25:13.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj51 2023-10-12T01:25:13.036 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0KD6wRT6tsOdt1Qe/Q8ecJLYLBw= 2023-10-12T01:25:13.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0KD6wRT6tsOdt1Qe/Q8ecJLYLBw='} 2023-10-12T01:25:13.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091be1681e679689d-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.042 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.043 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.043 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.044 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.044 DEBUG:boto:Data: 2023-10-12T01:25:13.044 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.045 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.046 DEBUG:boto:Params: {} 2023-10-12T01:25:13.046 DEBUG:boto:Token: None 2023-10-12T01:25:13.047 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.047 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052ed63fa72c97191-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.051 DEBUG:boto:path=/wafmoa-63/obj68 2023-10-12T01:25:13.051 DEBUG:boto:auth_path=/wafmoa-63/obj68 2023-10-12T01:25:13.052 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.052 DEBUG:boto:Path: /wafmoa-63/obj68 2023-10-12T01:25:13.052 DEBUG:boto:Data: 2023-10-12T01:25:13.053 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.054 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.054 DEBUG:boto:Params: {} 2023-10-12T01:25:13.055 DEBUG:boto:Token: None 2023-10-12T01:25:13.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj68 2023-10-12T01:25:13.056 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:21TxuEV9Ng1Q7srYtwpGkqrR7mM= 2023-10-12T01:25:13.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:21TxuEV9Ng1Q7srYtwpGkqrR7mM='} 2023-10-12T01:25:13.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5466cc15005f019-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.104 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.105 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.105 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.106 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.107 DEBUG:boto:Data: 2023-10-12T01:25:13.107 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.109 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.110 DEBUG:boto:Params: {} 2023-10-12T01:25:13.111 DEBUG:boto:Token: None 2023-10-12T01:25:13.111 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.112 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2697b764caebc12-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.117 DEBUG:boto:path=/wafmoa-63/obj85 2023-10-12T01:25:13.118 DEBUG:boto:auth_path=/wafmoa-63/obj85 2023-10-12T01:25:13.119 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.119 DEBUG:boto:Path: /wafmoa-63/obj85 2023-10-12T01:25:13.120 DEBUG:boto:Data: 2023-10-12T01:25:13.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.123 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.123 DEBUG:boto:Params: {} 2023-10-12T01:25:13.124 DEBUG:boto:Token: None 2023-10-12T01:25:13.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj85 2023-10-12T01:25:13.126 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SxwQRri9Jbc7v2042kDR3oULtTw= 2023-10-12T01:25:13.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SxwQRri9Jbc7v2042kDR3oULtTw='} 2023-10-12T01:25:13.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005effddd8e47d93c4-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.135 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:13.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:13.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:13.231 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:13.231 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:13.237 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-10-12T01:25:13.282 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.282 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.283 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.283 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.284 DEBUG:boto:Data: 2023-10-12T01:25:13.284 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.285 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.285 DEBUG:boto:Params: {} 2023-10-12T01:25:13.286 DEBUG:boto:Token: None 2023-10-12T01:25:13.286 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.287 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000712e4f22e9e38574-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.291 DEBUG:boto:path=/wafmoa-63/obj19 2023-10-12T01:25:13.291 DEBUG:boto:auth_path=/wafmoa-63/obj19 2023-10-12T01:25:13.292 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.292 DEBUG:boto:Path: /wafmoa-63/obj19 2023-10-12T01:25:13.293 DEBUG:boto:Data: 2023-10-12T01:25:13.293 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.295 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.295 DEBUG:boto:Params: {} 2023-10-12T01:25:13.296 DEBUG:boto:Token: None 2023-10-12T01:25:13.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj19 2023-10-12T01:25:13.297 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iXm827ttWxhZxBRmzYLMOudecsE= 2023-10-12T01:25:13.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iXm827ttWxhZxBRmzYLMOudecsE='} 2023-10-12T01:25:13.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e9fe7522d9a8399-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.343 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.344 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.344 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.344 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.345 DEBUG:boto:Data: 2023-10-12T01:25:13.345 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.346 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.346 DEBUG:boto:Params: {} 2023-10-12T01:25:13.347 DEBUG:boto:Token: None 2023-10-12T01:25:13.347 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.348 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4d4ee4c3b81d4e2-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.351 DEBUG:boto:path=/wafmoa-63/obj38 2023-10-12T01:25:13.352 DEBUG:boto:auth_path=/wafmoa-63/obj38 2023-10-12T01:25:13.352 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.353 DEBUG:boto:Path: /wafmoa-63/obj38 2023-10-12T01:25:13.353 DEBUG:boto:Data: 2023-10-12T01:25:13.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.355 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.355 DEBUG:boto:Params: {} 2023-10-12T01:25:13.356 DEBUG:boto:Token: None 2023-10-12T01:25:13.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj38 2023-10-12T01:25:13.358 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mSbSzOK2GYfFkQkz5hhYf6+Wpq4= 2023-10-12T01:25:13.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mSbSzOK2GYfFkQkz5hhYf6+Wpq4='} 2023-10-12T01:25:13.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a0e9e9943642a50-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.363 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.364 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.364 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.364 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.365 DEBUG:boto:Data: 2023-10-12T01:25:13.365 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.366 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.366 DEBUG:boto:Params: {} 2023-10-12T01:25:13.367 DEBUG:boto:Token: None 2023-10-12T01:25:13.367 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049060741b175eeee-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.371 DEBUG:boto:path=/wafmoa-63/obj57 2023-10-12T01:25:13.372 DEBUG:boto:auth_path=/wafmoa-63/obj57 2023-10-12T01:25:13.372 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.373 DEBUG:boto:Path: /wafmoa-63/obj57 2023-10-12T01:25:13.373 DEBUG:boto:Data: 2023-10-12T01:25:13.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.374 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.374 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.375 DEBUG:boto:Params: {} 2023-10-12T01:25:13.375 DEBUG:boto:Token: None 2023-10-12T01:25:13.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj57 2023-10-12T01:25:13.376 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7tuFEBepqrXQuE8ke5ranXqRR0A= 2023-10-12T01:25:13.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tuFEBepqrXQuE8ke5ranXqRR0A='} 2023-10-12T01:25:13.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa1b4265bedf362a-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.427 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.428 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.428 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.428 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.429 DEBUG:boto:Data: 2023-10-12T01:25:13.429 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.430 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.431 DEBUG:boto:Params: {} 2023-10-12T01:25:13.431 DEBUG:boto:Token: None 2023-10-12T01:25:13.431 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.432 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000607e0095deba56ee-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.436 DEBUG:boto:path=/wafmoa-63/obj76 2023-10-12T01:25:13.437 DEBUG:boto:auth_path=/wafmoa-63/obj76 2023-10-12T01:25:13.437 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.437 DEBUG:boto:Path: /wafmoa-63/obj76 2023-10-12T01:25:13.438 DEBUG:boto:Data: 2023-10-12T01:25:13.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.439 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.439 DEBUG:boto:Params: {} 2023-10-12T01:25:13.440 DEBUG:boto:Token: None 2023-10-12T01:25:13.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj76 2023-10-12T01:25:13.441 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2SEOW2CpapHhZrUNdy9oc8jf/D0= 2023-10-12T01:25:13.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2SEOW2CpapHhZrUNdy9oc8jf/D0='} 2023-10-12T01:25:13.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c270bdbfe99e1e-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.448 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.448 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.448 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.449 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.449 DEBUG:boto:Data: 2023-10-12T01:25:13.450 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.450 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.451 DEBUG:boto:Params: {} 2023-10-12T01:25:13.451 DEBUG:boto:Token: None 2023-10-12T01:25:13.452 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.452 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000069a06b439f45473-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.456 DEBUG:boto:path=/wafmoa-63/obj95 2023-10-12T01:25:13.456 DEBUG:boto:auth_path=/wafmoa-63/obj95 2023-10-12T01:25:13.457 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.457 DEBUG:boto:Path: /wafmoa-63/obj95 2023-10-12T01:25:13.457 DEBUG:boto:Data: 2023-10-12T01:25:13.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.459 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.459 DEBUG:boto:Params: {} 2023-10-12T01:25:13.460 DEBUG:boto:Token: None 2023-10-12T01:25:13.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj95 2023-10-12T01:25:13.461 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Hqf98kNnJG12G40H56glYwWGkfo= 2023-10-12T01:25:13.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hqf98kNnJG12G40H56glYwWGkfo='} 2023-10-12T01:25:13.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009667689a646eb89-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.512 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:13.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:13.568 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:13.614 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:13.615 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:13.621 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-10-12T01:25:13.668 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.668 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.669 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.669 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.670 DEBUG:boto:Data: 2023-10-12T01:25:13.670 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.671 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.671 DEBUG:boto:Params: {} 2023-10-12T01:25:13.672 DEBUG:boto:Token: None 2023-10-12T01:25:13.672 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.673 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004539ae95070403ac-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.677 DEBUG:boto:path=/wafmoa-63/obj23 2023-10-12T01:25:13.677 DEBUG:boto:auth_path=/wafmoa-63/obj23 2023-10-12T01:25:13.678 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.678 DEBUG:boto:Path: /wafmoa-63/obj23 2023-10-12T01:25:13.679 DEBUG:boto:Data: 2023-10-12T01:25:13.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.680 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.680 DEBUG:boto:Params: {} 2023-10-12T01:25:13.681 DEBUG:boto:Token: None 2023-10-12T01:25:13.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj23 2023-10-12T01:25:13.682 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:d9aTojXsW75gZhZ3b9ukzOpU64o= 2023-10-12T01:25:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d9aTojXsW75gZhZ3b9ukzOpU64o='} 2023-10-12T01:25:13.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000096167a90e33bd1f-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.688 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.688 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.689 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.689 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.690 DEBUG:boto:Data: 2023-10-12T01:25:13.690 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.691 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.691 DEBUG:boto:Params: {} 2023-10-12T01:25:13.692 DEBUG:boto:Token: None 2023-10-12T01:25:13.692 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.693 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031abb046b6d5be1f-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.697 DEBUG:boto:path=/wafmoa-63/obj46 2023-10-12T01:25:13.697 DEBUG:boto:auth_path=/wafmoa-63/obj46 2023-10-12T01:25:13.698 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.698 DEBUG:boto:Path: /wafmoa-63/obj46 2023-10-12T01:25:13.698 DEBUG:boto:Data: 2023-10-12T01:25:13.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.700 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.700 DEBUG:boto:Params: {} 2023-10-12T01:25:13.701 DEBUG:boto:Token: None 2023-10-12T01:25:13.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj46 2023-10-12T01:25:13.701 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yD83Nlgem3kO4gbCHmhWwbXfm6g= 2023-10-12T01:25:13.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yD83Nlgem3kO4gbCHmhWwbXfm6g='} 2023-10-12T01:25:13.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a027c83aa6f881e0-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.747 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.748 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.748 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.749 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.749 DEBUG:boto:Data: 2023-10-12T01:25:13.749 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.750 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.751 DEBUG:boto:Params: {} 2023-10-12T01:25:13.751 DEBUG:boto:Token: None 2023-10-12T01:25:13.752 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.752 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f4c51d61a0c7f4-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.756 DEBUG:boto:path=/wafmoa-63/obj69 2023-10-12T01:25:13.757 DEBUG:boto:auth_path=/wafmoa-63/obj69 2023-10-12T01:25:13.757 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.758 DEBUG:boto:Path: /wafmoa-63/obj69 2023-10-12T01:25:13.758 DEBUG:boto:Data: 2023-10-12T01:25:13.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.759 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.760 DEBUG:boto:Params: {} 2023-10-12T01:25:13.760 DEBUG:boto:Token: None 2023-10-12T01:25:13.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj69 2023-10-12T01:25:13.761 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sm4VpdGGPGQMd/Nc9ZTlu8xdRlk= 2023-10-12T01:25:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sm4VpdGGPGQMd/Nc9ZTlu8xdRlk='} 2023-10-12T01:25:13.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001eeb24404385dd82-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.767 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.768 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.768 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.769 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.769 DEBUG:boto:Data: 2023-10-12T01:25:13.770 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.770 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.771 DEBUG:boto:Params: {} 2023-10-12T01:25:13.771 DEBUG:boto:Token: None 2023-10-12T01:25:13.772 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd75d3658f3f1d03-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.776 DEBUG:boto:path=/wafmoa-63/obj92 2023-10-12T01:25:13.777 DEBUG:boto:auth_path=/wafmoa-63/obj92 2023-10-12T01:25:13.777 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.778 DEBUG:boto:Path: /wafmoa-63/obj92 2023-10-12T01:25:13.778 DEBUG:boto:Data: 2023-10-12T01:25:13.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.779 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.780 DEBUG:boto:Params: {} 2023-10-12T01:25:13.781 DEBUG:boto:Token: None 2023-10-12T01:25:13.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj92 2023-10-12T01:25:13.782 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZLm0vqJXX8gaieDI0s+KArCH92A= 2023-10-12T01:25:13.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZLm0vqJXX8gaieDI0s+KArCH92A='} 2023-10-12T01:25:13.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084fc3a3c82b81ef0-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.827 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:13.828 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:13.828 DEBUG:boto:Method: HEAD 2023-10-12T01:25:13.829 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:13.829 DEBUG:boto:Data: 2023-10-12T01:25:13.829 DEBUG:boto:Headers: {} 2023-10-12T01:25:13.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.830 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.831 DEBUG:boto:Params: {} 2023-10-12T01:25:13.831 DEBUG:boto:Token: None 2023-10-12T01:25:13.832 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/ 2023-10-12T01:25:13.832 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:25:13.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:25:13.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000409a3a499869be9b-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.836 DEBUG:boto:path=/wafmoa-63/obj115 2023-10-12T01:25:13.836 DEBUG:boto:auth_path=/wafmoa-63/obj115 2023-10-12T01:25:13.837 DEBUG:boto:Method: PUT 2023-10-12T01:25:13.837 DEBUG:boto:Path: /wafmoa-63/obj115 2023-10-12T01:25:13.837 DEBUG:boto:Data: 2023-10-12T01:25:13.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:13.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:13.839 DEBUG:boto:Port: 8000 2023-10-12T01:25:13.839 DEBUG:boto:Params: {} 2023-10-12T01:25:13.840 DEBUG:boto:Token: None 2023-10-12T01:25:13.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:13 GMT /wafmoa-63/obj115 2023-10-12T01:25:13.840 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xE5duTDERiOFpfg8UDtn2gT5dWA= 2023-10-12T01:25:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xE5duTDERiOFpfg8UDtn2gT5dWA='} 2023-10-12T01:25:13.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b8b72c8120d8e6d-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:13.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:13.847 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:13.900 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:13.900 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:13.947 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:13.948 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:13.955 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-10-12T01:25:14.006 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.007 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.007 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.008 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.008 DEBUG:boto:Data: 2023-10-12T01:25:14.008 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.009 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.010 DEBUG:boto:Params: {} 2023-10-12T01:25:14.010 DEBUG:boto:Token: None 2023-10-12T01:25:14.011 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.011 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee88bf8e6d911180-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.016 DEBUG:boto:path=/wafmoa-63/obj29 2023-10-12T01:25:14.016 DEBUG:boto:auth_path=/wafmoa-63/obj29 2023-10-12T01:25:14.016 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.017 DEBUG:boto:Path: /wafmoa-63/obj29 2023-10-12T01:25:14.017 DEBUG:boto:Data: 2023-10-12T01:25:14.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.019 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.019 DEBUG:boto:Params: {} 2023-10-12T01:25:14.019 DEBUG:boto:Token: None 2023-10-12T01:25:14.020 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj29 2023-10-12T01:25:14.020 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:IlgiGIujuh7KFp7hz23A83hVG6s= 2023-10-12T01:25:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IlgiGIujuh7KFp7hz23A83hVG6s='} 2023-10-12T01:25:14.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087806eed63e5efaf-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.067 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.068 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.068 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.068 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.069 DEBUG:boto:Data: 2023-10-12T01:25:14.069 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.070 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.071 DEBUG:boto:Params: {} 2023-10-12T01:25:14.071 DEBUG:boto:Token: None 2023-10-12T01:25:14.072 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.072 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd93f728c45f993-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.076 DEBUG:boto:path=/wafmoa-63/obj58 2023-10-12T01:25:14.077 DEBUG:boto:auth_path=/wafmoa-63/obj58 2023-10-12T01:25:14.077 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.078 DEBUG:boto:Path: /wafmoa-63/obj58 2023-10-12T01:25:14.078 DEBUG:boto:Data: 2023-10-12T01:25:14.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.080 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.080 DEBUG:boto:Params: {} 2023-10-12T01:25:14.081 DEBUG:boto:Token: None 2023-10-12T01:25:14.081 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj58 2023-10-12T01:25:14.081 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lzdBz5TpMXXW+PKvUp2DI0IXN0w= 2023-10-12T01:25:14.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lzdBz5TpMXXW+PKvUp2DI0IXN0w='} 2023-10-12T01:25:14.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0af9cfe501c0d6e-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.087 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.088 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.088 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.089 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.089 DEBUG:boto:Data: 2023-10-12T01:25:14.089 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.090 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.091 DEBUG:boto:Params: {} 2023-10-12T01:25:14.091 DEBUG:boto:Token: None 2023-10-12T01:25:14.094 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095ea9bcbe38cd249-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.099 DEBUG:boto:path=/wafmoa-63/obj87 2023-10-12T01:25:14.099 DEBUG:boto:auth_path=/wafmoa-63/obj87 2023-10-12T01:25:14.100 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.100 DEBUG:boto:Path: /wafmoa-63/obj87 2023-10-12T01:25:14.100 DEBUG:boto:Data: 2023-10-12T01:25:14.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.102 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.102 DEBUG:boto:Params: {} 2023-10-12T01:25:14.103 DEBUG:boto:Token: None 2023-10-12T01:25:14.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj87 2023-10-12T01:25:14.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:I4HMB/9WdA9GQp5RWMQv/TGy564= 2023-10-12T01:25:14.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I4HMB/9WdA9GQp5RWMQv/TGy564='} 2023-10-12T01:25:14.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a7c0a1981064446-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.147 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.148 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.148 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.148 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.149 DEBUG:boto:Data: 2023-10-12T01:25:14.149 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.150 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.150 DEBUG:boto:Params: {} 2023-10-12T01:25:14.151 DEBUG:boto:Token: None 2023-10-12T01:25:14.151 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.152 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023d891402eee6490-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.156 DEBUG:boto:path=/wafmoa-63/obj116 2023-10-12T01:25:14.157 DEBUG:boto:auth_path=/wafmoa-63/obj116 2023-10-12T01:25:14.157 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.158 DEBUG:boto:Path: /wafmoa-63/obj116 2023-10-12T01:25:14.158 DEBUG:boto:Data: 2023-10-12T01:25:14.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.159 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.160 DEBUG:boto:Params: {} 2023-10-12T01:25:14.160 DEBUG:boto:Token: None 2023-10-12T01:25:14.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj116 2023-10-12T01:25:14.161 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tG+EdQo21ImmTrgU4/PiV2XXQRs= 2023-10-12T01:25:14.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tG+EdQo21ImmTrgU4/PiV2XXQRs='} 2023-10-12T01:25:14.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000883b6129c3e4a9c6-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.167 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.168 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.168 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.169 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.169 DEBUG:boto:Data: 2023-10-12T01:25:14.169 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.170 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.171 DEBUG:boto:Params: {} 2023-10-12T01:25:14.171 DEBUG:boto:Token: None 2023-10-12T01:25:14.172 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.172 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038d7e57c68d7c917-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.176 DEBUG:boto:path=/wafmoa-63/obj145 2023-10-12T01:25:14.177 DEBUG:boto:auth_path=/wafmoa-63/obj145 2023-10-12T01:25:14.177 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.178 DEBUG:boto:Path: /wafmoa-63/obj145 2023-10-12T01:25:14.178 DEBUG:boto:Data: 2023-10-12T01:25:14.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.179 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.180 DEBUG:boto:Params: {} 2023-10-12T01:25:14.180 DEBUG:boto:Token: None 2023-10-12T01:25:14.180 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj145 2023-10-12T01:25:14.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EVhhuFslffKQwaWBZVt6LEW+heA= 2023-10-12T01:25:14.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVhhuFslffKQwaWBZVt6LEW+heA='} 2023-10-12T01:25:14.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed1c3d47fda2340-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:14.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:14.233 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:14.286 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:14.286 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:14.334 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:14.335 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:14.344 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-10-12T01:25:14.400 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.401 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.402 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.402 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.403 DEBUG:boto:Data: 2023-10-12T01:25:14.403 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.404 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.404 DEBUG:boto:Params: {} 2023-10-12T01:25:14.405 DEBUG:boto:Token: None 2023-10-12T01:25:14.405 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.406 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daf09a286872ba45-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.411 DEBUG:boto:path=/wafmoa-63/obj31 2023-10-12T01:25:14.412 DEBUG:boto:auth_path=/wafmoa-63/obj31 2023-10-12T01:25:14.412 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.413 DEBUG:boto:Path: /wafmoa-63/obj31 2023-10-12T01:25:14.413 DEBUG:boto:Data: 2023-10-12T01:25:14.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.414 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.415 DEBUG:boto:Params: {} 2023-10-12T01:25:14.415 DEBUG:boto:Token: None 2023-10-12T01:25:14.416 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj31 2023-10-12T01:25:14.416 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GFWhFD9cLoWZMh+XnFX+RSB7ruc= 2023-10-12T01:25:14.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFWhFD9cLoWZMh+XnFX+RSB7ruc='} 2023-10-12T01:25:14.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c0af0c1145ca8ca-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.423 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.424 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.424 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.425 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.425 DEBUG:boto:Data: 2023-10-12T01:25:14.425 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.426 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.427 DEBUG:boto:Params: {} 2023-10-12T01:25:14.428 DEBUG:boto:Token: None 2023-10-12T01:25:14.428 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3d8106cece000b4-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.433 DEBUG:boto:path=/wafmoa-63/obj62 2023-10-12T01:25:14.434 DEBUG:boto:auth_path=/wafmoa-63/obj62 2023-10-12T01:25:14.434 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.435 DEBUG:boto:Path: /wafmoa-63/obj62 2023-10-12T01:25:14.435 DEBUG:boto:Data: 2023-10-12T01:25:14.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.437 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.437 DEBUG:boto:Params: {} 2023-10-12T01:25:14.438 DEBUG:boto:Token: None 2023-10-12T01:25:14.438 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj62 2023-10-12T01:25:14.439 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:znP6iipYkJ4XwG4ojp4FjS8Xvv0= 2023-10-12T01:25:14.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:znP6iipYkJ4XwG4ojp4FjS8Xvv0='} 2023-10-12T01:25:14.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000addc2f286dca1abb-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.445 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.445 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.446 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.446 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.447 DEBUG:boto:Data: 2023-10-12T01:25:14.447 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.448 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.449 DEBUG:boto:Params: {} 2023-10-12T01:25:14.449 DEBUG:boto:Token: None 2023-10-12T01:25:14.449 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.450 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6c5de11d5d8d3fe-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.454 DEBUG:boto:path=/wafmoa-63/obj93 2023-10-12T01:25:14.455 DEBUG:boto:auth_path=/wafmoa-63/obj93 2023-10-12T01:25:14.455 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.456 DEBUG:boto:Path: /wafmoa-63/obj93 2023-10-12T01:25:14.456 DEBUG:boto:Data: 2023-10-12T01:25:14.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.458 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.458 DEBUG:boto:Params: {} 2023-10-12T01:25:14.459 DEBUG:boto:Token: None 2023-10-12T01:25:14.459 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj93 2023-10-12T01:25:14.460 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/XB5zulF3bTtLNAtqUq2qtLAFMA= 2023-10-12T01:25:14.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/XB5zulF3bTtLNAtqUq2qtLAFMA='} 2023-10-12T01:25:14.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0448fc2e964c74d-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.466 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.466 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.467 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.467 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.468 DEBUG:boto:Data: 2023-10-12T01:25:14.468 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.469 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.469 DEBUG:boto:Params: {} 2023-10-12T01:25:14.470 DEBUG:boto:Token: None 2023-10-12T01:25:14.470 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.471 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d24e2d1acd4b90-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.475 DEBUG:boto:path=/wafmoa-63/obj124 2023-10-12T01:25:14.475 DEBUG:boto:auth_path=/wafmoa-63/obj124 2023-10-12T01:25:14.476 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.476 DEBUG:boto:Path: /wafmoa-63/obj124 2023-10-12T01:25:14.477 DEBUG:boto:Data: 2023-10-12T01:25:14.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.478 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.479 DEBUG:boto:Params: {} 2023-10-12T01:25:14.479 DEBUG:boto:Token: None 2023-10-12T01:25:14.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj124 2023-10-12T01:25:14.480 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8cJU3DjEnWK58XslCHBbYVYyH20= 2023-10-12T01:25:14.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8cJU3DjEnWK58XslCHBbYVYyH20='} 2023-10-12T01:25:14.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e96b664b6e67f5e3-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.523 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.524 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.524 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.525 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.525 DEBUG:boto:Data: 2023-10-12T01:25:14.525 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.526 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.527 DEBUG:boto:Params: {} 2023-10-12T01:25:14.527 DEBUG:boto:Token: None 2023-10-12T01:25:14.528 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.528 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f23f3c046225583-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.533 DEBUG:boto:path=/wafmoa-63/obj155 2023-10-12T01:25:14.533 DEBUG:boto:auth_path=/wafmoa-63/obj155 2023-10-12T01:25:14.534 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.535 DEBUG:boto:Path: /wafmoa-63/obj155 2023-10-12T01:25:14.535 DEBUG:boto:Data: 2023-10-12T01:25:14.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.537 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.538 DEBUG:boto:Params: {} 2023-10-12T01:25:14.539 DEBUG:boto:Token: None 2023-10-12T01:25:14.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj155 2023-10-12T01:25:14.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7fYxoerX0Ps6YRdc3J/DyWsIAMw= 2023-10-12T01:25:14.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fYxoerX0Ps6YRdc3J/DyWsIAMw='} 2023-10-12T01:25:14.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b7e5cd91c0a8139-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:14.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:25:14.550 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:25:14.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:14.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:14.655 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:25:14.656 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-63 2023-10-12T01:25:14.665 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-10-12T01:25:14.726 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.727 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.727 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.727 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.728 DEBUG:boto:Data: 2023-10-12T01:25:14.728 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.729 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.730 DEBUG:boto:Params: {} 2023-10-12T01:25:14.730 DEBUG:boto:Token: None 2023-10-12T01:25:14.731 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.731 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d1dc8c5485fb11-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.736 DEBUG:boto:path=/wafmoa-63/obj37 2023-10-12T01:25:14.736 DEBUG:boto:auth_path=/wafmoa-63/obj37 2023-10-12T01:25:14.737 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.737 DEBUG:boto:Path: /wafmoa-63/obj37 2023-10-12T01:25:14.738 DEBUG:boto:Data: 2023-10-12T01:25:14.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.739 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.740 DEBUG:boto:Params: {} 2023-10-12T01:25:14.740 DEBUG:boto:Token: None 2023-10-12T01:25:14.741 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj37 2023-10-12T01:25:14.741 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fshpsWlmaC/Fei8FkluWyGV22e4= 2023-10-12T01:25:14.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fshpsWlmaC/Fei8FkluWyGV22e4='} 2023-10-12T01:25:14.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb0311eaa1113bf2-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.787 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.788 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.788 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.788 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.789 DEBUG:boto:Data: 2023-10-12T01:25:14.789 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.790 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.791 DEBUG:boto:Params: {} 2023-10-12T01:25:14.792 DEBUG:boto:Token: None 2023-10-12T01:25:14.793 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.793 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3ce7f8724f5b7a5-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.799 DEBUG:boto:path=/wafmoa-63/obj74 2023-10-12T01:25:14.799 DEBUG:boto:auth_path=/wafmoa-63/obj74 2023-10-12T01:25:14.800 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.800 DEBUG:boto:Path: /wafmoa-63/obj74 2023-10-12T01:25:14.801 DEBUG:boto:Data: 2023-10-12T01:25:14.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.802 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.802 DEBUG:boto:Params: {} 2023-10-12T01:25:14.803 DEBUG:boto:Token: None 2023-10-12T01:25:14.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj74 2023-10-12T01:25:14.804 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+SQYFAZOsYgf7aXyF7Fc4jZxh2s= 2023-10-12T01:25:14.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+SQYFAZOsYgf7aXyF7Fc4jZxh2s='} 2023-10-12T01:25:14.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb688eb02d416489-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.810 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.811 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.811 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.812 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.812 DEBUG:boto:Data: 2023-10-12T01:25:14.813 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.813 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.814 DEBUG:boto:Params: {} 2023-10-12T01:25:14.814 DEBUG:boto:Token: None 2023-10-12T01:25:14.815 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.815 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026a90689ab34eb32-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.820 DEBUG:boto:path=/wafmoa-63/obj111 2023-10-12T01:25:14.820 DEBUG:boto:auth_path=/wafmoa-63/obj111 2023-10-12T01:25:14.820 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.821 DEBUG:boto:Path: /wafmoa-63/obj111 2023-10-12T01:25:14.821 DEBUG:boto:Data: 2023-10-12T01:25:14.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.823 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.823 DEBUG:boto:Params: {} 2023-10-12T01:25:14.823 DEBUG:boto:Token: None 2023-10-12T01:25:14.824 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj111 2023-10-12T01:25:14.824 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wsKDnnf/2F+WewbxqOSGUbsH5UA= 2023-10-12T01:25:14.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wsKDnnf/2F+WewbxqOSGUbsH5UA='} 2023-10-12T01:25:14.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023786c046015ecd5-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.867 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.867 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.868 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.868 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.869 DEBUG:boto:Data: 2023-10-12T01:25:14.869 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.870 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.870 DEBUG:boto:Params: {} 2023-10-12T01:25:14.871 DEBUG:boto:Token: None 2023-10-12T01:25:14.871 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.872 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd450ef88ca394e7-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.878 DEBUG:boto:path=/wafmoa-63/obj148 2023-10-12T01:25:14.878 DEBUG:boto:auth_path=/wafmoa-63/obj148 2023-10-12T01:25:14.879 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.880 DEBUG:boto:Path: /wafmoa-63/obj148 2023-10-12T01:25:14.881 DEBUG:boto:Data: 2023-10-12T01:25:14.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.883 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.883 DEBUG:boto:Params: {} 2023-10-12T01:25:14.884 DEBUG:boto:Token: None 2023-10-12T01:25:14.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj148 2023-10-12T01:25:14.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AGghmVGrUASOKD9z+8p4UeHKNTY= 2023-10-12T01:25:14.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AGghmVGrUASOKD9z+8p4UeHKNTY='} 2023-10-12T01:25:14.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074b7a90db89ac803-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.891 DEBUG:boto:path=/wafmoa-63/ 2023-10-12T01:25:14.892 DEBUG:boto:auth_path=/wafmoa-63/ 2023-10-12T01:25:14.892 DEBUG:boto:Method: HEAD 2023-10-12T01:25:14.893 DEBUG:boto:Path: /wafmoa-63/ 2023-10-12T01:25:14.894 DEBUG:boto:Data: 2023-10-12T01:25:14.894 DEBUG:boto:Headers: {} 2023-10-12T01:25:14.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.896 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.896 DEBUG:boto:Params: {} 2023-10-12T01:25:14.897 DEBUG:boto:Token: None 2023-10-12T01:25:14.898 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/ 2023-10-12T01:25:14.899 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:25:14.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:25:14.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c64fab5317b65ae5-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.904 DEBUG:boto:path=/wafmoa-63/obj185 2023-10-12T01:25:14.904 DEBUG:boto:auth_path=/wafmoa-63/obj185 2023-10-12T01:25:14.905 DEBUG:boto:Method: PUT 2023-10-12T01:25:14.906 DEBUG:boto:Path: /wafmoa-63/obj185 2023-10-12T01:25:14.906 DEBUG:boto:Data: 2023-10-12T01:25:14.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:25:14.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:25:14.910 DEBUG:boto:Port: 8000 2023-10-12T01:25:14.913 DEBUG:boto:Params: {} 2023-10-12T01:25:14.915 DEBUG:boto:Token: None 2023-10-12T01:25:14.915 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:25:14 GMT /wafmoa-63/obj185 2023-10-12T01:25:14.916 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GMb8M6GM3ps8MeCNn7vdEdKwj7k= 2023-10-12T01:25:14.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GMb8M6GM3ps8MeCNn7vdEdKwj7k='} 2023-10-12T01:25:14.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7ac61e2cdee3e35-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:25:14.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-63 2023-10-12T01:25:14.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:25:14.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:25:14.961 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:25:15.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:15.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:15.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:15.052+0000 7f9f7b8ff780 1 waiting to reach incremental sync.. 2023-10-12T01:25:20.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:20.053+0000 7f9f7b8ff780 1 waiting to reach incremental sync.. 2023-10-12T01:25:25.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:25.055+0000 7f9f7b8ff780 1 bucket sync caught up with source: 2023-10-12T01:25:25.057 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.88.6, , , , , , 00000000002.120.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.90.6, , , , 00000000001.128.6] 2023-10-12T01:25:25.057 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.88.6, , , , , , 00000000002.120.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.90.6, , , , 00000000001.128.6] 2023-10-12T01:25:25.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:25.055+0000 7f9f7b8ff780 0 bucket checkpoint complete 2023-10-12T01:25:25.070 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-63 2023-10-12T01:25:25.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:25:25.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:25:25.072 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:25:25.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:25:25.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:25:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:25.166+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:30.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:30.168+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:35.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:35.169+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:40.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:40.168+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:45.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:45.169+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:50.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:50.170+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:25:55.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:25:55.171+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:00.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:00.171+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:05.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:05.172+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:10.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:10.173+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:15.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:15.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:20.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:25.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:25.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:30.175+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:35.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:35.176+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:40.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:40.176+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:45.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:45.178+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:50.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:50.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:26:55.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:26:55.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:00.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:00.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:05.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:05.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:10.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:10.180+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:15.181+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:20.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:20.183+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:25.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:25.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:30.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:35.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:35.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:40.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:40.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:45.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:45.185+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:50.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:50.185+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:27:55.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:27:55.187+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:00.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:00.188+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:05.188+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:05.272 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:28:05.272+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:28:10.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:10.189+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:15.189+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:20.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:20.190+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:25.190+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:30.191+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:35.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:35.192+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:40.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:40.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:45.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:45.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:50.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:50.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:28:55.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:28:55.194+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:00.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:00.194+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:05.195+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:10.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:10.196+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:15.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:15.197+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:20.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:20.198+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:25.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:30.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:35.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:35.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:40.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:40.200+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:45.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:45.200+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:50.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:50.201+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:29:55.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:29:55.202+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:30:00.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:00.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:30:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:05.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:30:10.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:10.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:30:15.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:15.204+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:30:20.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.205+0000 7f48f099e780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-12T01:30:20.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.205+0000 7f48f099e780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:30:20.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.205+0000 7f48f099e780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:30:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-12T01:30:20.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-10-12T01:30:20.227 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.228 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.228 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.229 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.229 DEBUG:boto:Data: 2023-10-12T01:30:20.229 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.230 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.231 DEBUG:boto:Params: {} 2023-10-12T01:30:20.231 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:30:20.232 DEBUG:boto:Token: None 2023-10-12T01:30:20.232 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.233 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EprK931kEfLwdn4fhLNmw0HHAEQ= 2023-10-12T01:30:20.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EprK931kEfLwdn4fhLNmw0HHAEQ=', 'Content-Length': '0'} 2023-10-12T01:30:20.257 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c5c17e232d7dba7-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.258 DEBUG:rgw_multi.tests:created bucket=wafmoa-64 2023-10-12T01:30:20.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:30:20.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:30:20.260 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:30:20.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:20.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:20.346 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:30:20.361 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:30:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:30:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:30:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:30:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:30:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074220.255567_728.1", 2023-10-12T01:30:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:30:20.255567Z" 2023-10-12T01:30:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:30:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:30:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.365 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:30:20.366 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:30:20.366 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:30:20.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:30:20.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:30:20.368 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:30:20.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:20.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:20.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.448+0000 7f0bf8eeb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:30:20.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.448+0000 7f0bf8eeb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:30:20.462 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:30:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:30:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:30:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:30:20.463 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:30:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:30:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:30:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.464 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:30:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:30:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:30:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:30:20.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:30:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:30:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:30:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:30:20.466 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:30:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:30:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.467 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:30:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:30:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:30:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:30:20.468 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:30:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:30:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:30:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:30:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:30:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:30:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:30:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:30:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074220.255567_728.1", 2023-10-12T01:30:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:30:20.471 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:30:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:30:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:20.255567Z", 2023-10-12T01:30:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:30:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:30:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:30:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:30:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:30:20.473 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:30:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:30:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:30:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:30:20.474 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:30:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:30:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:30:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:30:20.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:30:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:30:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:30:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:30:20.476 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:30:20Z" 2023-10-12T01:30:20.477 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:30:20.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074220.255567_728.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:20.255567Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:30:20Z" } 2023-10-12T01:30:20.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074220.255567_728.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:20.255567Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:30:20.479 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:30:20.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:30:20.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:30:20.480 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.481 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.481 DEBUG:boto:Method: HEAD 2023-10-12T01:30:20.481 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.482 DEBUG:boto:Data: 2023-10-12T01:30:20.482 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.483 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.483 DEBUG:boto:Params: {} 2023-10-12T01:30:20.484 DEBUG:boto:Token: None 2023-10-12T01:30:20.484 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.485 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:30:20.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:30:20.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ef27b0ee737079-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.489 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:20.489 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:20.490 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.490 DEBUG:boto:Path: /wafmoa-64/obj11 2023-10-12T01:30:20.490 DEBUG:boto:Data: 2023-10-12T01:30:20.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:20.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.492 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.492 DEBUG:boto:Params: {} 2023-10-12T01:30:20.492 DEBUG:boto:Token: None 2023-10-12T01:30:20.493 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/obj11 2023-10-12T01:30:20.493 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NCUlPQFyDU1TE5Fe5zKe+ld6SzA= 2023-10-12T01:30:20.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NCUlPQFyDU1TE5Fe5zKe+ld6SzA='} 2023-10-12T01:30:20.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087de8b05277a155a-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.501 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.501 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.502 DEBUG:boto:Method: HEAD 2023-10-12T01:30:20.502 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.503 DEBUG:boto:Data: 2023-10-12T01:30:20.503 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.504 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.504 DEBUG:boto:Params: {} 2023-10-12T01:30:20.505 DEBUG:boto:Token: None 2023-10-12T01:30:20.505 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.506 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:30:20.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:30:20.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b0fedd5f950a5dc-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.509 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:20.509 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:20.510 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.510 DEBUG:boto:Path: /wafmoa-64/obj22 2023-10-12T01:30:20.511 DEBUG:boto:Data: 2023-10-12T01:30:20.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:20.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.512 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.512 DEBUG:boto:Params: {} 2023-10-12T01:30:20.513 DEBUG:boto:Token: None 2023-10-12T01:30:20.513 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/obj22 2023-10-12T01:30:20.514 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:87oPWsLqTMEej8BVCDz3tuVA/8M= 2023-10-12T01:30:20.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:87oPWsLqTMEej8BVCDz3tuVA/8M='} 2023-10-12T01:30:20.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d91d09c89e6a1e4b-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.519 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.519 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.519 DEBUG:boto:Method: HEAD 2023-10-12T01:30:20.520 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.520 DEBUG:boto:Data: 2023-10-12T01:30:20.521 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.522 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.522 DEBUG:boto:Params: {} 2023-10-12T01:30:20.522 DEBUG:boto:Token: None 2023-10-12T01:30:20.523 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.523 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:30:20.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:30:20.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d9be4e2b531672c-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.527 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:20.527 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:20.528 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.528 DEBUG:boto:Path: /wafmoa-64/obj33 2023-10-12T01:30:20.529 DEBUG:boto:Data: 2023-10-12T01:30:20.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:20.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.530 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.530 DEBUG:boto:Params: {} 2023-10-12T01:30:20.531 DEBUG:boto:Token: None 2023-10-12T01:30:20.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/obj33 2023-10-12T01:30:20.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:wLmEm+f7riW5E1a2foZucljf6wU= 2023-10-12T01:30:20.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLmEm+f7riW5E1a2foZucljf6wU='} 2023-10-12T01:30:20.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006323aef16ad79298-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.538 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.539 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.539 DEBUG:boto:Method: HEAD 2023-10-12T01:30:20.539 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.540 DEBUG:boto:Data: 2023-10-12T01:30:20.540 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.541 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.541 DEBUG:boto:Params: {} 2023-10-12T01:30:20.542 DEBUG:boto:Token: None 2023-10-12T01:30:20.542 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.543 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:30:20.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:30:20.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a94e30c473916127-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.546 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:20.547 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:20.547 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.548 DEBUG:boto:Path: /wafmoa-64/obj44 2023-10-12T01:30:20.548 DEBUG:boto:Data: 2023-10-12T01:30:20.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:20.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.549 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.550 DEBUG:boto:Params: {} 2023-10-12T01:30:20.550 DEBUG:boto:Token: None 2023-10-12T01:30:20.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/obj44 2023-10-12T01:30:20.551 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kBbN+pbJI86/ff9n5J0wop+6iDM= 2023-10-12T01:30:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kBbN+pbJI86/ff9n5J0wop+6iDM='} 2023-10-12T01:30:20.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000748e219bec4a4ae4-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.595 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:20.596 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:20.596 DEBUG:boto:Method: HEAD 2023-10-12T01:30:20.596 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:20.597 DEBUG:boto:Data: 2023-10-12T01:30:20.597 DEBUG:boto:Headers: {} 2023-10-12T01:30:20.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.598 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.599 DEBUG:boto:Params: {} 2023-10-12T01:30:20.599 DEBUG:boto:Token: None 2023-10-12T01:30:20.600 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/ 2023-10-12T01:30:20.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:30:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:30:20.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca3d6cb9e49edc7-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.604 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:20.604 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:20.604 DEBUG:boto:Method: PUT 2023-10-12T01:30:20.605 DEBUG:boto:Path: /wafmoa-64/obj55 2023-10-12T01:30:20.605 DEBUG:boto:Data: 2023-10-12T01:30:20.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:20.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:20.607 DEBUG:boto:Port: 8000 2023-10-12T01:30:20.607 DEBUG:boto:Params: {} 2023-10-12T01:30:20.607 DEBUG:boto:Token: None 2023-10-12T01:30:20.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:20 GMT /wafmoa-64/obj55 2023-10-12T01:30:20.608 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jayQ9u/JPcxUVz2loRgGb4ycmj8= 2023-10-12T01:30:20.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jayQ9u/JPcxUVz2loRgGb4ycmj8='} 2023-10-12T01:30:20.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8c0da041751dbda-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:20.613 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-64 2023-10-12T01:30:20.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:20.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:20.615 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:30:20.671 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:20.671 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:20.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:20.705+0000 7f2ea8280780 1 waiting to reach incremental sync.. 2023-10-12T01:30:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:25.706+0000 7f2ea8280780 1 waiting to reach incremental sync.. 2023-10-12T01:30:30.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:30.709+0000 7f2ea8280780 1 bucket sync caught up with source: 2023-10-12T01:30:30.709 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.107.6, 00000000002.126.6, 00000000002.109.6] 2023-10-12T01:30:30.709 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.126.6, 00000000002.109.6] 2023-10-12T01:30:30.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:30.709+0000 7f2ea8280780 0 bucket checkpoint complete 2023-10-12T01:30:30.726 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=wafmoa-64 2023-10-12T01:30:30.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:30.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:30.728 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:30:30.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:30.792 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:30.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:30.829+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:30:35.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:35.830+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:30:40.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:40.830+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:30:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:45.830+0000 7f5040497780 1 bucket sync caught up with source: 2023-10-12T01:30:45.832 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.362.6, 00000000002.147.6, 00000000002.378.6] 2023-10-12T01:30:45.832 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.362.6, 00000000002.147.6, 00000000002.378.6] 2023-10-12T01:30:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:45.830+0000 7f5040497780 0 bucket checkpoint complete 2023-10-12T01:30:45.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wafmoa-64 zones={a2, a1} 2023-10-12T01:30:45.846 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.847 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.847 DEBUG:boto:Method: HEAD 2023-10-12T01:30:45.847 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:45.848 DEBUG:boto:Data: 2023-10-12T01:30:45.848 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.849 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.849 DEBUG:boto:Params: {} 2023-10-12T01:30:45.850 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:30:45.851 DEBUG:boto:Token: None 2023-10-12T01:30:45.851 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/ 2023-10-12T01:30:45.852 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY= 2023-10-12T01:30:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY=', 'Content-Length': '0'} 2023-10-12T01:30:45.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008252213f32c32a6d-0065274c45-1fd8-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.856 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.857 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.857 DEBUG:boto:Method: HEAD 2023-10-12T01:30:45.858 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:45.858 DEBUG:boto:Data: 2023-10-12T01:30:45.858 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.859 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.860 DEBUG:boto:Params: {} 2023-10-12T01:30:45.860 DEBUG:boto:Token: None 2023-10-12T01:30:45.861 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/ 2023-10-12T01:30:45.861 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY= 2023-10-12T01:30:45.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY=', 'Content-Length': '0'} 2023-10-12T01:30:45.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc5155fd8e8768b9-0065274c45-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.865 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-12T01:30:45.865 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.865 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.866 DEBUG:boto:path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.866 DEBUG:boto:auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.867 DEBUG:boto:Method: GET 2023-10-12T01:30:45.867 DEBUG:boto:Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.868 DEBUG:boto:Data: 2023-10-12T01:30:45.868 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.869 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.869 DEBUG:boto:Params: {} 2023-10-12T01:30:45.870 DEBUG:boto:Token: None 2023-10-12T01:30:45.870 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/?versions 2023-10-12T01:30:45.871 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:30:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:30:45.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c96d658d38488c-0065274c45-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.874 DEBUG:boto:b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:45.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:30:45.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:30:45.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:30:45.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:30:45.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:30:45.877 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-12T01:30:45.877 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.878 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.878 DEBUG:boto:path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.878 DEBUG:boto:auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.879 DEBUG:boto:Method: GET 2023-10-12T01:30:45.879 DEBUG:boto:Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.880 DEBUG:boto:Data: 2023-10-12T01:30:45.880 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.881 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.881 DEBUG:boto:Params: {} 2023-10-12T01:30:45.882 DEBUG:boto:Token: None 2023-10-12T01:30:45.882 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/?versions 2023-10-12T01:30:45.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:30:45.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:30:45.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783cd5e8e30dd3ed-0065274c45-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.886 DEBUG:boto:b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:45.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-12T01:30:45.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-12T01:30:45.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-12T01:30:45.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-12T01:30:45.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-12T01:30:45.889 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.889 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.890 DEBUG:boto:path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.890 DEBUG:boto:auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.891 DEBUG:boto:Method: GET 2023-10-12T01:30:45.891 DEBUG:boto:Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.892 DEBUG:boto:Data: 2023-10-12T01:30:45.892 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.893 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.893 DEBUG:boto:Params: {} 2023-10-12T01:30:45.894 DEBUG:boto:Token: None 2023-10-12T01:30:45.894 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/?versions 2023-10-12T01:30:45.895 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:30:45.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:30:45.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5c9bdb5488be8b-0065274c45-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.898 DEBUG:boto:b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:45.899 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:45.899 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:45.900 DEBUG:boto:path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.900 DEBUG:boto:auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.901 DEBUG:boto:Method: GET 2023-10-12T01:30:45.901 DEBUG:boto:Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:30:45.901 DEBUG:boto:Data: 2023-10-12T01:30:45.902 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.903 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.903 DEBUG:boto:Params: {} 2023-10-12T01:30:45.904 DEBUG:boto:Token: None 2023-10-12T01:30:45.904 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/?versions 2023-10-12T01:30:45.904 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:30:45.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:30:45.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c24dc747f77b5b1-0065274c45-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.908 DEBUG:boto:b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:45.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:30:45.909 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.909 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.910 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.910 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.911 DEBUG:boto:Method: GET 2023-10-12T01:30:45.911 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.912 DEBUG:boto:Data: 2023-10-12T01:30:45.912 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.913 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.913 DEBUG:boto:Params: {} 2023-10-12T01:30:45.914 DEBUG:boto:Token: None 2023-10-12T01:30:45.914 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.915 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:30:45.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:30:45.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f9202f6b7dae85-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.918 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.918 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.919 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.919 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.919 DEBUG:boto:Method: GET 2023-10-12T01:30:45.920 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.920 DEBUG:boto:Data: 2023-10-12T01:30:45.921 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.922 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.922 DEBUG:boto:Params: {} 2023-10-12T01:30:45.923 DEBUG:boto:Token: None 2023-10-12T01:30:45.923 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.924 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:30:45.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:30:45.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa8e0f03fed8b05-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.966 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.967 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.967 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.967 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.968 DEBUG:boto:Method: HEAD 2023-10-12T01:30:45.968 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.969 DEBUG:boto:Data: 2023-10-12T01:30:45.969 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.970 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.970 DEBUG:boto:Params: {} 2023-10-12T01:30:45.971 DEBUG:boto:Token: None 2023-10-12T01:30:45.971 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.972 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY= 2023-10-12T01:30:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY=', 'Content-Length': '0'} 2023-10-12T01:30:45.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c350ce304ee182b4-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.975 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.975 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.975 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.976 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.976 DEBUG:boto:Method: HEAD 2023-10-12T01:30:45.977 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.977 DEBUG:boto:Data: 2023-10-12T01:30:45.977 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.978 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.979 DEBUG:boto:Params: {} 2023-10-12T01:30:45.979 DEBUG:boto:Token: None 2023-10-12T01:30:45.980 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.980 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY= 2023-10-12T01:30:45.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY=', 'Content-Length': '0'} 2023-10-12T01:30:45.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1428ed02c557151-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:30:45.983 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.984 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.984 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.985 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.985 DEBUG:boto:Method: GET 2023-10-12T01:30:45.985 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.986 DEBUG:boto:Data: 2023-10-12T01:30:45.986 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:45.987 DEBUG:boto:Port: 8001 2023-10-12T01:30:45.988 DEBUG:boto:Params: {} 2023-10-12T01:30:45.988 DEBUG:boto:Token: None 2023-10-12T01:30:45.989 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.989 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:30:45.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:30:45.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac620c7a80bd08eb-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:45.992 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:45.992 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:45.993 DEBUG:boto:path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.993 DEBUG:boto:auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.994 DEBUG:boto:Method: GET 2023-10-12T01:30:45.994 DEBUG:boto:Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.995 DEBUG:boto:Data: 2023-10-12T01:30:45.995 DEBUG:boto:Headers: {} 2023-10-12T01:30:45.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:45.996 DEBUG:boto:Port: 8000 2023-10-12T01:30:45.996 DEBUG:boto:Params: {} 2023-10-12T01:30:45.997 DEBUG:boto:Token: None 2023-10-12T01:30:45.997 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:45 GMT /wafmoa-64/obj11?versionId=null 2023-10-12T01:30:45.998 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:30:45.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:30:46.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc79641a99699d1-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.001 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.001 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.002 DEBUG:boto:path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.002 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.002 DEBUG:boto:Method: GET 2023-10-12T01:30:46.003 DEBUG:boto:Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.003 DEBUG:boto:Data: 2023-10-12T01:30:46.004 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.005 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.005 DEBUG:boto:Params: {} 2023-10-12T01:30:46.005 DEBUG:boto:Token: None 2023-10-12T01:30:46.006 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.006 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f40d50a3601ef0-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.010 DEBUG:boto:b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.010 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.011 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.011 DEBUG:boto:path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.012 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.012 DEBUG:boto:Method: GET 2023-10-12T01:30:46.013 DEBUG:boto:Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.013 DEBUG:boto:Data: 2023-10-12T01:30:46.013 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.014 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.015 DEBUG:boto:Params: {} 2023-10-12T01:30:46.015 DEBUG:boto:Token: None 2023-10-12T01:30:46.016 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6795281344c943c-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.020 DEBUG:boto:b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.020 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:46.021 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:46.021 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.022 DEBUG:boto:Path: /wafmoa-64/obj11 2023-10-12T01:30:46.022 DEBUG:boto:Data: 2023-10-12T01:30:46.022 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.023 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.023 DEBUG:boto:Params: {} 2023-10-12T01:30:46.024 DEBUG:boto:Token: None 2023-10-12T01:30:46.024 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj11 2023-10-12T01:30:46.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU= 2023-10-12T01:30:46.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU=', 'Content-Length': '0'} 2023-10-12T01:30:46.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85933b2af40f926-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.027 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:46.028 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:46.028 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.029 DEBUG:boto:Path: /wafmoa-64/obj11 2023-10-12T01:30:46.029 DEBUG:boto:Data: 2023-10-12T01:30:46.030 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.030 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.031 DEBUG:boto:Params: {} 2023-10-12T01:30:46.031 DEBUG:boto:Token: None 2023-10-12T01:30:46.032 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj11 2023-10-12T01:30:46.032 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU= 2023-10-12T01:30:46.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU=', 'Content-Length': '0'} 2023-10-12T01:30:46.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009605c67a523e7456-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-12T01:30:46.036 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:46.036 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:46.036 DEBUG:boto:Method: GET 2023-10-12T01:30:46.037 DEBUG:boto:Path: /wafmoa-64/obj11 2023-10-12T01:30:46.037 DEBUG:boto:Data: 2023-10-12T01:30:46.038 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.039 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.039 DEBUG:boto:Params: {} 2023-10-12T01:30:46.040 DEBUG:boto:Token: None 2023-10-12T01:30:46.040 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj11 2023-10-12T01:30:46.041 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8= 2023-10-12T01:30:46.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8=', 'Content-Length': '0'} 2023-10-12T01:30:46.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb6d39d5126985b-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.044 DEBUG:boto:path=/wafmoa-64/obj11 2023-10-12T01:30:46.044 DEBUG:boto:auth_path=/wafmoa-64/obj11 2023-10-12T01:30:46.045 DEBUG:boto:Method: GET 2023-10-12T01:30:46.045 DEBUG:boto:Path: /wafmoa-64/obj11 2023-10-12T01:30:46.045 DEBUG:boto:Data: 2023-10-12T01:30:46.046 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.047 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.047 DEBUG:boto:Params: {} 2023-10-12T01:30:46.048 DEBUG:boto:Token: None 2023-10-12T01:30:46.048 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj11 2023-10-12T01:30:46.049 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8= 2023-10-12T01:30:46.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8=', 'Content-Length': '0'} 2023-10-12T01:30:46.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346fd2af61236e50-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.094 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.095 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.095 DEBUG:boto:path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.096 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.096 DEBUG:boto:Method: GET 2023-10-12T01:30:46.097 DEBUG:boto:Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.097 DEBUG:boto:Data: 2023-10-12T01:30:46.097 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.098 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.099 DEBUG:boto:Params: {} 2023-10-12T01:30:46.099 DEBUG:boto:Token: None 2023-10-12T01:30:46.100 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.100 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141941e23c04493d-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.104 DEBUG:boto:b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.104 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.105 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.105 DEBUG:boto:path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.106 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.106 DEBUG:boto:Method: GET 2023-10-12T01:30:46.107 DEBUG:boto:Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:30:46.107 DEBUG:boto:Data: 2023-10-12T01:30:46.107 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.108 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.109 DEBUG:boto:Params: {} 2023-10-12T01:30:46.109 DEBUG:boto:Token: None 2023-10-12T01:30:46.110 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.110 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007adb96c1e1a26be-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.114 DEBUG:boto:b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:30:46.115 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.115 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.116 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.116 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.117 DEBUG:boto:Method: GET 2023-10-12T01:30:46.117 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.117 DEBUG:boto:Data: 2023-10-12T01:30:46.118 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.119 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.119 DEBUG:boto:Params: {} 2023-10-12T01:30:46.120 DEBUG:boto:Token: None 2023-10-12T01:30:46.120 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:30:46.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:30:46.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf0ee7f5db19442-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.123 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.124 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.124 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.125 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.125 DEBUG:boto:Method: GET 2023-10-12T01:30:46.126 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.126 DEBUG:boto:Data: 2023-10-12T01:30:46.127 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.128 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.128 DEBUG:boto:Params: {} 2023-10-12T01:30:46.128 DEBUG:boto:Token: None 2023-10-12T01:30:46.129 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:30:46.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:30:46.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2d53b20ceea4fa-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.133 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.133 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.133 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.134 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.134 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.135 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.135 DEBUG:boto:Data: 2023-10-12T01:30:46.135 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.136 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.137 DEBUG:boto:Params: {} 2023-10-12T01:30:46.137 DEBUG:boto:Token: None 2023-10-12T01:30:46.138 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.138 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY= 2023-10-12T01:30:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY=', 'Content-Length': '0'} 2023-10-12T01:30:46.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096a228288570c87-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.141 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.141 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.142 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.142 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.143 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.143 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.143 DEBUG:boto:Data: 2023-10-12T01:30:46.144 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.145 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.145 DEBUG:boto:Params: {} 2023-10-12T01:30:46.146 DEBUG:boto:Token: None 2023-10-12T01:30:46.146 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.146 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY= 2023-10-12T01:30:46.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY=', 'Content-Length': '0'} 2023-10-12T01:30:46.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81ba5929dbb4201-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:30:46.150 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.150 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.151 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.151 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.152 DEBUG:boto:Method: GET 2023-10-12T01:30:46.152 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.152 DEBUG:boto:Data: 2023-10-12T01:30:46.153 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.154 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.154 DEBUG:boto:Params: {} 2023-10-12T01:30:46.155 DEBUG:boto:Token: None 2023-10-12T01:30:46.155 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:30:46.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:30:46.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e9d517f5a2e2e68-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.198 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.199 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.199 DEBUG:boto:path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.199 DEBUG:boto:auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.200 DEBUG:boto:Method: GET 2023-10-12T01:30:46.200 DEBUG:boto:Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.201 DEBUG:boto:Data: 2023-10-12T01:30:46.201 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.202 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.203 DEBUG:boto:Params: {} 2023-10-12T01:30:46.203 DEBUG:boto:Token: None 2023-10-12T01:30:46.204 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22?versionId=null 2023-10-12T01:30:46.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:30:46.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:30:46.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b4d07e0cf785fe-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.207 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.208 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.208 DEBUG:boto:path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.209 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.209 DEBUG:boto:Method: GET 2023-10-12T01:30:46.209 DEBUG:boto:Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.210 DEBUG:boto:Data: 2023-10-12T01:30:46.210 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.211 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.212 DEBUG:boto:Params: {} 2023-10-12T01:30:46.212 DEBUG:boto:Token: None 2023-10-12T01:30:46.213 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be81250c274cebb6-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.216 DEBUG:boto:b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.217 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.218 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.218 DEBUG:boto:path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.219 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.219 DEBUG:boto:Method: GET 2023-10-12T01:30:46.219 DEBUG:boto:Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.220 DEBUG:boto:Data: 2023-10-12T01:30:46.220 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.221 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.222 DEBUG:boto:Params: {} 2023-10-12T01:30:46.222 DEBUG:boto:Token: None 2023-10-12T01:30:46.223 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.223 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b73a68c5da8ca64-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.226 DEBUG:boto:b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.227 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.227 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.228 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.228 DEBUG:boto:Path: /wafmoa-64/obj22 2023-10-12T01:30:46.229 DEBUG:boto:Data: 2023-10-12T01:30:46.229 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.230 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.230 DEBUG:boto:Params: {} 2023-10-12T01:30:46.231 DEBUG:boto:Token: None 2023-10-12T01:30:46.231 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22 2023-10-12T01:30:46.232 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo= 2023-10-12T01:30:46.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo=', 'Content-Length': '0'} 2023-10-12T01:30:46.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2327a02570a56ca-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.234 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.235 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.235 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.236 DEBUG:boto:Path: /wafmoa-64/obj22 2023-10-12T01:30:46.236 DEBUG:boto:Data: 2023-10-12T01:30:46.237 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.237 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.238 DEBUG:boto:Params: {} 2023-10-12T01:30:46.238 DEBUG:boto:Token: None 2023-10-12T01:30:46.239 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22 2023-10-12T01:30:46.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo= 2023-10-12T01:30:46.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo=', 'Content-Length': '0'} 2023-10-12T01:30:46.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9ce54aa2b31d83-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-12T01:30:46.243 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.243 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.243 DEBUG:boto:Method: GET 2023-10-12T01:30:46.244 DEBUG:boto:Path: /wafmoa-64/obj22 2023-10-12T01:30:46.244 DEBUG:boto:Data: 2023-10-12T01:30:46.245 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.246 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.246 DEBUG:boto:Params: {} 2023-10-12T01:30:46.247 DEBUG:boto:Token: None 2023-10-12T01:30:46.247 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22 2023-10-12T01:30:46.248 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM= 2023-10-12T01:30:46.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM=', 'Content-Length': '0'} 2023-10-12T01:30:46.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045b45c3f715bcc6-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.251 DEBUG:boto:path=/wafmoa-64/obj22 2023-10-12T01:30:46.251 DEBUG:boto:auth_path=/wafmoa-64/obj22 2023-10-12T01:30:46.251 DEBUG:boto:Method: GET 2023-10-12T01:30:46.252 DEBUG:boto:Path: /wafmoa-64/obj22 2023-10-12T01:30:46.252 DEBUG:boto:Data: 2023-10-12T01:30:46.253 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.254 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.254 DEBUG:boto:Params: {} 2023-10-12T01:30:46.255 DEBUG:boto:Token: None 2023-10-12T01:30:46.255 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj22 2023-10-12T01:30:46.256 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM= 2023-10-12T01:30:46.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM=', 'Content-Length': '0'} 2023-10-12T01:30:46.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6d5789956a6feb-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.259 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.259 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.260 DEBUG:boto:path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.260 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.260 DEBUG:boto:Method: GET 2023-10-12T01:30:46.261 DEBUG:boto:Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.261 DEBUG:boto:Data: 2023-10-12T01:30:46.262 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.263 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.263 DEBUG:boto:Params: {} 2023-10-12T01:30:46.264 DEBUG:boto:Token: None 2023-10-12T01:30:46.264 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cf78ffcee407245-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.268 DEBUG:boto:b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.269 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.269 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.270 DEBUG:boto:path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.270 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.271 DEBUG:boto:Method: GET 2023-10-12T01:30:46.271 DEBUG:boto:Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:30:46.271 DEBUG:boto:Data: 2023-10-12T01:30:46.272 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.273 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.273 DEBUG:boto:Params: {} 2023-10-12T01:30:46.274 DEBUG:boto:Token: None 2023-10-12T01:30:46.274 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.275 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003058291796839c2d-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.318 DEBUG:boto:b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:30:46.319 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.320 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.320 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.321 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.321 DEBUG:boto:Method: GET 2023-10-12T01:30:46.321 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.322 DEBUG:boto:Data: 2023-10-12T01:30:46.322 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.323 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.324 DEBUG:boto:Params: {} 2023-10-12T01:30:46.324 DEBUG:boto:Token: None 2023-10-12T01:30:46.324 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.325 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:30:46.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:30:46.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055d22dbb5c2ad19-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.328 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.328 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.329 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.329 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.330 DEBUG:boto:Method: GET 2023-10-12T01:30:46.330 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.330 DEBUG:boto:Data: 2023-10-12T01:30:46.331 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.332 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.332 DEBUG:boto:Params: {} 2023-10-12T01:30:46.333 DEBUG:boto:Token: None 2023-10-12T01:30:46.333 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.334 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:30:46.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:30:46.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005657c0fd5d20e0f0-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.337 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.337 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.338 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.338 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.338 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.339 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.339 DEBUG:boto:Data: 2023-10-12T01:30:46.340 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.340 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.341 DEBUG:boto:Params: {} 2023-10-12T01:30:46.341 DEBUG:boto:Token: None 2023-10-12T01:30:46.342 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.342 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs= 2023-10-12T01:30:46.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs=', 'Content-Length': '0'} 2023-10-12T01:30:46.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038e7a5b6fcc3499-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.345 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.345 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.346 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.346 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.347 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.347 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.347 DEBUG:boto:Data: 2023-10-12T01:30:46.348 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.349 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.349 DEBUG:boto:Params: {} 2023-10-12T01:30:46.349 DEBUG:boto:Token: None 2023-10-12T01:30:46.350 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.350 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs= 2023-10-12T01:30:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs=', 'Content-Length': '0'} 2023-10-12T01:30:46.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098edff317e648137-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:30:46.354 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.354 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.355 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.355 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.355 DEBUG:boto:Method: GET 2023-10-12T01:30:46.356 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.356 DEBUG:boto:Data: 2023-10-12T01:30:46.357 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.357 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.358 DEBUG:boto:Params: {} 2023-10-12T01:30:46.358 DEBUG:boto:Token: None 2023-10-12T01:30:46.359 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:30:46.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:30:46.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d6b1e70f195937-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.362 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.363 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.363 DEBUG:boto:path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.364 DEBUG:boto:auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.364 DEBUG:boto:Method: GET 2023-10-12T01:30:46.365 DEBUG:boto:Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.365 DEBUG:boto:Data: 2023-10-12T01:30:46.365 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.366 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.367 DEBUG:boto:Params: {} 2023-10-12T01:30:46.367 DEBUG:boto:Token: None 2023-10-12T01:30:46.368 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33?versionId=null 2023-10-12T01:30:46.368 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:30:46.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:30:46.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2cdbaec3652ee2-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.371 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.372 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.372 DEBUG:boto:path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.373 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.373 DEBUG:boto:Method: GET 2023-10-12T01:30:46.374 DEBUG:boto:Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.374 DEBUG:boto:Data: 2023-10-12T01:30:46.374 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.375 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.376 DEBUG:boto:Params: {} 2023-10-12T01:30:46.376 DEBUG:boto:Token: None 2023-10-12T01:30:46.377 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.377 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ce9167101e0fa9-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.381 DEBUG:boto:b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.381 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.382 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.382 DEBUG:boto:path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.383 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.383 DEBUG:boto:Method: GET 2023-10-12T01:30:46.384 DEBUG:boto:Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.384 DEBUG:boto:Data: 2023-10-12T01:30:46.384 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.385 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.386 DEBUG:boto:Params: {} 2023-10-12T01:30:46.386 DEBUG:boto:Token: None 2023-10-12T01:30:46.387 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.387 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed52c923f6df55f9-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.391 DEBUG:boto:b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.391 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.392 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.392 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.393 DEBUG:boto:Path: /wafmoa-64/obj33 2023-10-12T01:30:46.393 DEBUG:boto:Data: 2023-10-12T01:30:46.393 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.394 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.395 DEBUG:boto:Params: {} 2023-10-12T01:30:46.395 DEBUG:boto:Token: None 2023-10-12T01:30:46.396 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33 2023-10-12T01:30:46.396 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM= 2023-10-12T01:30:46.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM=', 'Content-Length': '0'} 2023-10-12T01:30:46.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ab83ffb423f414-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.399 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.399 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.399 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.400 DEBUG:boto:Path: /wafmoa-64/obj33 2023-10-12T01:30:46.400 DEBUG:boto:Data: 2023-10-12T01:30:46.401 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.402 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.402 DEBUG:boto:Params: {} 2023-10-12T01:30:46.402 DEBUG:boto:Token: None 2023-10-12T01:30:46.403 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33 2023-10-12T01:30:46.403 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM= 2023-10-12T01:30:46.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM=', 'Content-Length': '0'} 2023-10-12T01:30:46.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfce61710d88113e-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-12T01:30:46.406 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.407 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.407 DEBUG:boto:Method: GET 2023-10-12T01:30:46.408 DEBUG:boto:Path: /wafmoa-64/obj33 2023-10-12T01:30:46.408 DEBUG:boto:Data: 2023-10-12T01:30:46.409 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.409 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.410 DEBUG:boto:Params: {} 2023-10-12T01:30:46.410 DEBUG:boto:Token: None 2023-10-12T01:30:46.411 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33 2023-10-12T01:30:46.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0= 2023-10-12T01:30:46.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0=', 'Content-Length': '0'} 2023-10-12T01:30:46.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c975552d468d407-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.414 DEBUG:boto:path=/wafmoa-64/obj33 2023-10-12T01:30:46.415 DEBUG:boto:auth_path=/wafmoa-64/obj33 2023-10-12T01:30:46.415 DEBUG:boto:Method: GET 2023-10-12T01:30:46.415 DEBUG:boto:Path: /wafmoa-64/obj33 2023-10-12T01:30:46.416 DEBUG:boto:Data: 2023-10-12T01:30:46.416 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.417 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.418 DEBUG:boto:Params: {} 2023-10-12T01:30:46.418 DEBUG:boto:Token: None 2023-10-12T01:30:46.419 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj33 2023-10-12T01:30:46.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0= 2023-10-12T01:30:46.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0=', 'Content-Length': '0'} 2023-10-12T01:30:46.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf56e3556589180-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.462 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.463 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.463 DEBUG:boto:path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.464 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.464 DEBUG:boto:Method: GET 2023-10-12T01:30:46.464 DEBUG:boto:Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.465 DEBUG:boto:Data: 2023-10-12T01:30:46.465 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.466 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.467 DEBUG:boto:Params: {} 2023-10-12T01:30:46.467 DEBUG:boto:Token: None 2023-10-12T01:30:46.468 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.468 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3003ac0dc2c350-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.472 DEBUG:boto:b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.472 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.473 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.473 DEBUG:boto:path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.474 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.474 DEBUG:boto:Method: GET 2023-10-12T01:30:46.474 DEBUG:boto:Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:30:46.475 DEBUG:boto:Data: 2023-10-12T01:30:46.475 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.476 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.477 DEBUG:boto:Params: {} 2023-10-12T01:30:46.477 DEBUG:boto:Token: None 2023-10-12T01:30:46.478 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.478 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2bc581debb6dd5-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.481 DEBUG:boto:b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:30:46.483 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.483 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.483 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.484 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.484 DEBUG:boto:Method: GET 2023-10-12T01:30:46.485 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.485 DEBUG:boto:Data: 2023-10-12T01:30:46.486 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.486 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.487 DEBUG:boto:Params: {} 2023-10-12T01:30:46.487 DEBUG:boto:Token: None 2023-10-12T01:30:46.488 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:30:46.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:30:46.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db77de1f4753c536-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.491 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.492 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.492 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.493 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.493 DEBUG:boto:Method: GET 2023-10-12T01:30:46.493 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.494 DEBUG:boto:Data: 2023-10-12T01:30:46.494 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.495 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.496 DEBUG:boto:Params: {} 2023-10-12T01:30:46.496 DEBUG:boto:Token: None 2023-10-12T01:30:46.497 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:30:46.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:30:46.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfaab50f0215f68-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.500 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.500 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.500 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.501 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.501 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.502 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.502 DEBUG:boto:Data: 2023-10-12T01:30:46.502 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.503 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.504 DEBUG:boto:Params: {} 2023-10-12T01:30:46.504 DEBUG:boto:Token: None 2023-10-12T01:30:46.505 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.505 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8= 2023-10-12T01:30:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8=', 'Content-Length': '0'} 2023-10-12T01:30:46.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af25cf56fa23568-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.508 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.508 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.509 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.509 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.509 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.510 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.510 DEBUG:boto:Data: 2023-10-12T01:30:46.511 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.511 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.512 DEBUG:boto:Params: {} 2023-10-12T01:30:46.512 DEBUG:boto:Token: None 2023-10-12T01:30:46.513 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.513 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8= 2023-10-12T01:30:46.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8=', 'Content-Length': '0'} 2023-10-12T01:30:46.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bd68a7dd51a865-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:30:46.516 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.517 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.517 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.518 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.518 DEBUG:boto:Method: GET 2023-10-12T01:30:46.518 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.519 DEBUG:boto:Data: 2023-10-12T01:30:46.519 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.520 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.521 DEBUG:boto:Params: {} 2023-10-12T01:30:46.521 DEBUG:boto:Token: None 2023-10-12T01:30:46.522 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.522 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:30:46.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:30:46.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7b8232c55429fe-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.566 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.567 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.567 DEBUG:boto:path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.568 DEBUG:boto:auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.568 DEBUG:boto:Method: GET 2023-10-12T01:30:46.568 DEBUG:boto:Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.569 DEBUG:boto:Data: 2023-10-12T01:30:46.569 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.570 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.571 DEBUG:boto:Params: {} 2023-10-12T01:30:46.571 DEBUG:boto:Token: None 2023-10-12T01:30:46.571 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44?versionId=null 2023-10-12T01:30:46.572 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:30:46.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:30:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d12c1b5808d4507-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.575 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.575 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.576 DEBUG:boto:path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.576 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.577 DEBUG:boto:Method: GET 2023-10-12T01:30:46.577 DEBUG:boto:Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.578 DEBUG:boto:Data: 2023-10-12T01:30:46.578 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.579 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.579 DEBUG:boto:Params: {} 2023-10-12T01:30:46.580 DEBUG:boto:Token: None 2023-10-12T01:30:46.580 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.581 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc36c2a06d72044-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.584 DEBUG:boto:b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.585 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.585 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.586 DEBUG:boto:path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.586 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.587 DEBUG:boto:Method: GET 2023-10-12T01:30:46.587 DEBUG:boto:Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.588 DEBUG:boto:Data: 2023-10-12T01:30:46.588 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.589 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.589 DEBUG:boto:Params: {} 2023-10-12T01:30:46.590 DEBUG:boto:Token: None 2023-10-12T01:30:46.590 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.591 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b2581414a266e1-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.594 DEBUG:boto:b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.595 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.596 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.596 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.596 DEBUG:boto:Path: /wafmoa-64/obj44 2023-10-12T01:30:46.597 DEBUG:boto:Data: 2023-10-12T01:30:46.597 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.598 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.598 DEBUG:boto:Params: {} 2023-10-12T01:30:46.599 DEBUG:boto:Token: None 2023-10-12T01:30:46.599 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44 2023-10-12T01:30:46.600 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA= 2023-10-12T01:30:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA=', 'Content-Length': '0'} 2023-10-12T01:30:46.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b482ee95adf23e0f-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.603 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.603 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.603 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.604 DEBUG:boto:Path: /wafmoa-64/obj44 2023-10-12T01:30:46.604 DEBUG:boto:Data: 2023-10-12T01:30:46.605 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.605 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.606 DEBUG:boto:Params: {} 2023-10-12T01:30:46.606 DEBUG:boto:Token: None 2023-10-12T01:30:46.607 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44 2023-10-12T01:30:46.607 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA= 2023-10-12T01:30:46.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA=', 'Content-Length': '0'} 2023-10-12T01:30:46.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9b22d467aaf8941-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-12T01:30:46.610 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.611 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.611 DEBUG:boto:Method: GET 2023-10-12T01:30:46.612 DEBUG:boto:Path: /wafmoa-64/obj44 2023-10-12T01:30:46.612 DEBUG:boto:Data: 2023-10-12T01:30:46.612 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.613 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.614 DEBUG:boto:Params: {} 2023-10-12T01:30:46.614 DEBUG:boto:Token: None 2023-10-12T01:30:46.615 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44 2023-10-12T01:30:46.615 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os= 2023-10-12T01:30:46.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os=', 'Content-Length': '0'} 2023-10-12T01:30:46.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c47c7ee4825327-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.618 DEBUG:boto:path=/wafmoa-64/obj44 2023-10-12T01:30:46.619 DEBUG:boto:auth_path=/wafmoa-64/obj44 2023-10-12T01:30:46.619 DEBUG:boto:Method: GET 2023-10-12T01:30:46.620 DEBUG:boto:Path: /wafmoa-64/obj44 2023-10-12T01:30:46.620 DEBUG:boto:Data: 2023-10-12T01:30:46.620 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.621 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.622 DEBUG:boto:Params: {} 2023-10-12T01:30:46.622 DEBUG:boto:Token: None 2023-10-12T01:30:46.623 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj44 2023-10-12T01:30:46.623 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os= 2023-10-12T01:30:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os=', 'Content-Length': '0'} 2023-10-12T01:30:46.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9db33dfe4eb6a94-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.626 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.626 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.627 DEBUG:boto:path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.627 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.628 DEBUG:boto:Method: GET 2023-10-12T01:30:46.628 DEBUG:boto:Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.629 DEBUG:boto:Data: 2023-10-12T01:30:46.629 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.630 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.630 DEBUG:boto:Params: {} 2023-10-12T01:30:46.631 DEBUG:boto:Token: None 2023-10-12T01:30:46.631 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.632 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c687a102cd4ec5-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.635 DEBUG:boto:b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.635 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.636 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.636 DEBUG:boto:path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.637 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.637 DEBUG:boto:Method: GET 2023-10-12T01:30:46.637 DEBUG:boto:Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:30:46.638 DEBUG:boto:Data: 2023-10-12T01:30:46.638 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.639 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.640 DEBUG:boto:Params: {} 2023-10-12T01:30:46.640 DEBUG:boto:Token: None 2023-10-12T01:30:46.640 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.641 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9ce3c0ade60833-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.644 DEBUG:boto:b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:30:46.645 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.646 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.646 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.647 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.647 DEBUG:boto:Method: GET 2023-10-12T01:30:46.648 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.648 DEBUG:boto:Data: 2023-10-12T01:30:46.648 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.649 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.650 DEBUG:boto:Params: {} 2023-10-12T01:30:46.650 DEBUG:boto:Token: None 2023-10-12T01:30:46.651 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.651 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:30:46.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:30:46.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009255e70e42139756-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.654 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.655 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.655 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.656 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.656 DEBUG:boto:Method: GET 2023-10-12T01:30:46.657 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.657 DEBUG:boto:Data: 2023-10-12T01:30:46.657 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.658 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.659 DEBUG:boto:Params: {} 2023-10-12T01:30:46.659 DEBUG:boto:Token: None 2023-10-12T01:30:46.660 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.660 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:30:46.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:30:46.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80334c494f13c5c-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.663 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.663 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.664 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.664 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.665 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.665 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.666 DEBUG:boto:Data: 2023-10-12T01:30:46.666 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.667 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.667 DEBUG:boto:Params: {} 2023-10-12T01:30:46.668 DEBUG:boto:Token: None 2023-10-12T01:30:46.668 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.668 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no= 2023-10-12T01:30:46.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no=', 'Content-Length': '0'} 2023-10-12T01:30:46.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e64fbf0f6211e9-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.671 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.672 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.672 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.673 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.673 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.673 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.674 DEBUG:boto:Data: 2023-10-12T01:30:46.674 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.675 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.675 DEBUG:boto:Params: {} 2023-10-12T01:30:46.676 DEBUG:boto:Token: None 2023-10-12T01:30:46.676 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no= 2023-10-12T01:30:46.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no=', 'Content-Length': '0'} 2023-10-12T01:30:46.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9ab6f75c5cfac3-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:30:46.680 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.680 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.681 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.681 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.682 DEBUG:boto:Method: GET 2023-10-12T01:30:46.682 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.683 DEBUG:boto:Data: 2023-10-12T01:30:46.683 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.684 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.684 DEBUG:boto:Params: {} 2023-10-12T01:30:46.685 DEBUG:boto:Token: None 2023-10-12T01:30:46.685 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.686 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:30:46.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:30:46.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a184c47a20edec21-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.730 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.731 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.731 DEBUG:boto:path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.732 DEBUG:boto:auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.732 DEBUG:boto:Method: GET 2023-10-12T01:30:46.733 DEBUG:boto:Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.733 DEBUG:boto:Data: 2023-10-12T01:30:46.733 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.734 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.735 DEBUG:boto:Params: {} 2023-10-12T01:30:46.735 DEBUG:boto:Token: None 2023-10-12T01:30:46.736 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55?versionId=null 2023-10-12T01:30:46.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:30:46.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:30:46.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818ecb80056ba36f-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.739 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.740 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.740 DEBUG:boto:path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.740 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.741 DEBUG:boto:Method: GET 2023-10-12T01:30:46.741 DEBUG:boto:Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.742 DEBUG:boto:Data: 2023-10-12T01:30:46.742 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.743 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.744 DEBUG:boto:Params: {} 2023-10-12T01:30:46.744 DEBUG:boto:Token: None 2023-10-12T01:30:46.744 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.745 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb5d1a5e0d95bc4-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.748 DEBUG:boto:b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.749 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.749 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.750 DEBUG:boto:path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.750 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.750 DEBUG:boto:Method: GET 2023-10-12T01:30:46.751 DEBUG:boto:Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.751 DEBUG:boto:Data: 2023-10-12T01:30:46.752 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.753 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.753 DEBUG:boto:Params: {} 2023-10-12T01:30:46.754 DEBUG:boto:Token: None 2023-10-12T01:30:46.754 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.755 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8abe559d8168dd-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.758 DEBUG:boto:b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.759 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.759 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.759 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.760 DEBUG:boto:Path: /wafmoa-64/obj55 2023-10-12T01:30:46.760 DEBUG:boto:Data: 2023-10-12T01:30:46.760 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.761 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.762 DEBUG:boto:Params: {} 2023-10-12T01:30:46.762 DEBUG:boto:Token: None 2023-10-12T01:30:46.763 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55 2023-10-12T01:30:46.763 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw= 2023-10-12T01:30:46.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw=', 'Content-Length': '0'} 2023-10-12T01:30:46.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619807bc81e95e56-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.766 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.766 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.767 DEBUG:boto:Method: HEAD 2023-10-12T01:30:46.767 DEBUG:boto:Path: /wafmoa-64/obj55 2023-10-12T01:30:46.767 DEBUG:boto:Data: 2023-10-12T01:30:46.768 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.769 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.769 DEBUG:boto:Params: {} 2023-10-12T01:30:46.769 DEBUG:boto:Token: None 2023-10-12T01:30:46.770 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55 2023-10-12T01:30:46.770 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw= 2023-10-12T01:30:46.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw=', 'Content-Length': '0'} 2023-10-12T01:30:46.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3d9f913e780189-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-12T01:30:46.774 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.774 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.774 DEBUG:boto:Method: GET 2023-10-12T01:30:46.775 DEBUG:boto:Path: /wafmoa-64/obj55 2023-10-12T01:30:46.775 DEBUG:boto:Data: 2023-10-12T01:30:46.776 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.777 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.777 DEBUG:boto:Params: {} 2023-10-12T01:30:46.777 DEBUG:boto:Token: None 2023-10-12T01:30:46.778 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55 2023-10-12T01:30:46.778 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8= 2023-10-12T01:30:46.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8=', 'Content-Length': '0'} 2023-10-12T01:30:46.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb97b347ba9b2cf5-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.781 DEBUG:boto:path=/wafmoa-64/obj55 2023-10-12T01:30:46.782 DEBUG:boto:auth_path=/wafmoa-64/obj55 2023-10-12T01:30:46.782 DEBUG:boto:Method: GET 2023-10-12T01:30:46.783 DEBUG:boto:Path: /wafmoa-64/obj55 2023-10-12T01:30:46.783 DEBUG:boto:Data: 2023-10-12T01:30:46.784 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.784 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.785 DEBUG:boto:Params: {} 2023-10-12T01:30:46.785 DEBUG:boto:Token: None 2023-10-12T01:30:46.786 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/obj55 2023-10-12T01:30:46.786 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8= 2023-10-12T01:30:46.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8=', 'Content-Length': '0'} 2023-10-12T01:30:46.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b790f1a83b99d0c-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.789 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.790 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.790 DEBUG:boto:path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.791 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.791 DEBUG:boto:Method: GET 2023-10-12T01:30:46.792 DEBUG:boto:Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.792 DEBUG:boto:Data: 2023-10-12T01:30:46.792 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:46.793 DEBUG:boto:Port: 8001 2023-10-12T01:30:46.794 DEBUG:boto:Params: {} 2023-10-12T01:30:46.794 DEBUG:boto:Token: None 2023-10-12T01:30:46.795 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097005e3ed657e1a4-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.798 DEBUG:boto:b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.799 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:46.799 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:46.800 DEBUG:boto:path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.800 DEBUG:boto:auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.801 DEBUG:boto:Method: GET 2023-10-12T01:30:46.801 DEBUG:boto:Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:30:46.802 DEBUG:boto:Data: 2023-10-12T01:30:46.802 DEBUG:boto:Headers: {} 2023-10-12T01:30:46.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:46.803 DEBUG:boto:Port: 8000 2023-10-12T01:30:46.803 DEBUG:boto:Params: {} 2023-10-12T01:30:46.804 DEBUG:boto:Token: None 2023-10-12T01:30:46.804 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:30:46 GMT /wafmoa-64/ 2023-10-12T01:30:46.805 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:30:46.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:30:46.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bda87f5e0c09a84-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:46.808 DEBUG:boto:b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:30:46.808 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wafmoa-64 zones={a2, a1} 2023-10-12T01:30:46.809 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:30:46.809 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:30:46.810 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:30:46.809+0000 7f2e18867700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 178629) UID: 0 2023-10-12T01:30:46.811 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:30:46.809+0000 7f2e1cad07c0 -1 shutting down 2023-10-12T01:30:48.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:30:52.908 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:30:52.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:52.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:52.910 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:52.970 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:52.970 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:53.016 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:53.016 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:53.020 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-10-12T01:30:53.060 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.061 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.061 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.062 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.062 DEBUG:boto:Data: 2023-10-12T01:30:53.063 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.064 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.064 DEBUG:boto:Params: {} 2023-10-12T01:30:53.065 DEBUG:boto:Token: None 2023-10-12T01:30:53.065 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.065 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd33e2c1be0ba157-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.069 DEBUG:boto:path=/wafmoa-64/obj17 2023-10-12T01:30:53.070 DEBUG:boto:auth_path=/wafmoa-64/obj17 2023-10-12T01:30:53.070 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.071 DEBUG:boto:Path: /wafmoa-64/obj17 2023-10-12T01:30:53.071 DEBUG:boto:Data: 2023-10-12T01:30:53.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.072 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.073 DEBUG:boto:Params: {} 2023-10-12T01:30:53.073 DEBUG:boto:Token: None 2023-10-12T01:30:53.074 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj17 2023-10-12T01:30:53.074 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y+9gxoADktU1EOFmsLgoplKXBDo= 2023-10-12T01:30:53.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+9gxoADktU1EOFmsLgoplKXBDo='} 2023-10-12T01:30:53.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f8b8bb5e6d4d634-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.081 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.081 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.081 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.082 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.082 DEBUG:boto:Data: 2023-10-12T01:30:53.083 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.083 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.084 DEBUG:boto:Params: {} 2023-10-12T01:30:53.084 DEBUG:boto:Token: None 2023-10-12T01:30:53.085 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000984494561bbd4a96-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.089 DEBUG:boto:path=/wafmoa-64/obj34 2023-10-12T01:30:53.089 DEBUG:boto:auth_path=/wafmoa-64/obj34 2023-10-12T01:30:53.090 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.090 DEBUG:boto:Path: /wafmoa-64/obj34 2023-10-12T01:30:53.091 DEBUG:boto:Data: 2023-10-12T01:30:53.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.092 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.092 DEBUG:boto:Params: {} 2023-10-12T01:30:53.093 DEBUG:boto:Token: None 2023-10-12T01:30:53.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj34 2023-10-12T01:30:53.094 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8VivdHw/npnYSuaxPHpWxA/PmRk= 2023-10-12T01:30:53.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8VivdHw/npnYSuaxPHpWxA/PmRk='} 2023-10-12T01:30:53.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c5cb222a1f5d30c-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.100 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.100 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.101 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.101 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.102 DEBUG:boto:Data: 2023-10-12T01:30:53.102 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.103 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.103 DEBUG:boto:Params: {} 2023-10-12T01:30:53.104 DEBUG:boto:Token: None 2023-10-12T01:30:53.104 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.105 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093eadfde7275f515-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.108 DEBUG:boto:path=/wafmoa-64/obj51 2023-10-12T01:30:53.108 DEBUG:boto:auth_path=/wafmoa-64/obj51 2023-10-12T01:30:53.109 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.109 DEBUG:boto:Path: /wafmoa-64/obj51 2023-10-12T01:30:53.110 DEBUG:boto:Data: 2023-10-12T01:30:53.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.111 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.111 DEBUG:boto:Params: {} 2023-10-12T01:30:53.112 DEBUG:boto:Token: None 2023-10-12T01:30:53.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj51 2023-10-12T01:30:53.113 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RiTEWW98Durl/yBJfVDlRtEIH7k= 2023-10-12T01:30:53.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RiTEWW98Durl/yBJfVDlRtEIH7k='} 2023-10-12T01:30:53.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085356dc858a167fd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.163 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.164 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.164 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.165 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.165 DEBUG:boto:Data: 2023-10-12T01:30:53.166 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.166 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.167 DEBUG:boto:Params: {} 2023-10-12T01:30:53.167 DEBUG:boto:Token: None 2023-10-12T01:30:53.168 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.168 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1c1a27b78578cd-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.172 DEBUG:boto:path=/wafmoa-64/obj68 2023-10-12T01:30:53.172 DEBUG:boto:auth_path=/wafmoa-64/obj68 2023-10-12T01:30:53.173 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.173 DEBUG:boto:Path: /wafmoa-64/obj68 2023-10-12T01:30:53.173 DEBUG:boto:Data: 2023-10-12T01:30:53.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.175 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.175 DEBUG:boto:Params: {} 2023-10-12T01:30:53.176 DEBUG:boto:Token: None 2023-10-12T01:30:53.176 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj68 2023-10-12T01:30:53.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OXjWTh+n1faeeNGyQnzjOLde/Ws= 2023-10-12T01:30:53.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXjWTh+n1faeeNGyQnzjOLde/Ws='} 2023-10-12T01:30:53.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e272704c848f5606-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.182 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.183 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.183 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.184 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.184 DEBUG:boto:Data: 2023-10-12T01:30:53.184 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.185 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.186 DEBUG:boto:Params: {} 2023-10-12T01:30:53.186 DEBUG:boto:Token: None 2023-10-12T01:30:53.187 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.187 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055653d3dfd9bcc1c-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.190 DEBUG:boto:path=/wafmoa-64/obj85 2023-10-12T01:30:53.191 DEBUG:boto:auth_path=/wafmoa-64/obj85 2023-10-12T01:30:53.191 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.192 DEBUG:boto:Path: /wafmoa-64/obj85 2023-10-12T01:30:53.192 DEBUG:boto:Data: 2023-10-12T01:30:53.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.193 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.194 DEBUG:boto:Params: {} 2023-10-12T01:30:53.194 DEBUG:boto:Token: None 2023-10-12T01:30:53.195 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj85 2023-10-12T01:30:53.195 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:55vB6yN5bPn6CYwf+GOhInx5GZI= 2023-10-12T01:30:53.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:55vB6yN5bPn6CYwf+GOhInx5GZI='} 2023-10-12T01:30:53.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f81f7333d4bb99-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.240 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:53.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:53.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:53.336 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:53.337 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:53.342 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-10-12T01:30:53.386 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.386 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.387 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.387 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.387 DEBUG:boto:Data: 2023-10-12T01:30:53.388 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.389 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.389 DEBUG:boto:Params: {} 2023-10-12T01:30:53.390 DEBUG:boto:Token: None 2023-10-12T01:30:53.390 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.391 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000006f891ab3667341-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.396 DEBUG:boto:path=/wafmoa-64/obj19 2023-10-12T01:30:53.397 DEBUG:boto:auth_path=/wafmoa-64/obj19 2023-10-12T01:30:53.397 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.397 DEBUG:boto:Path: /wafmoa-64/obj19 2023-10-12T01:30:53.398 DEBUG:boto:Data: 2023-10-12T01:30:53.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.399 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.399 DEBUG:boto:Params: {} 2023-10-12T01:30:53.400 DEBUG:boto:Token: None 2023-10-12T01:30:53.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj19 2023-10-12T01:30:53.401 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j7KKiwxZwWs7guuzoePfECfbF1s= 2023-10-12T01:30:53.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7KKiwxZwWs7guuzoePfECfbF1s='} 2023-10-12T01:30:53.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000658133ee74edbecd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.407 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.407 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.407 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.408 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.408 DEBUG:boto:Data: 2023-10-12T01:30:53.409 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.409 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.410 DEBUG:boto:Params: {} 2023-10-12T01:30:53.410 DEBUG:boto:Token: None 2023-10-12T01:30:53.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009964b636605cf823-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.415 DEBUG:boto:path=/wafmoa-64/obj38 2023-10-12T01:30:53.415 DEBUG:boto:auth_path=/wafmoa-64/obj38 2023-10-12T01:30:53.416 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.416 DEBUG:boto:Path: /wafmoa-64/obj38 2023-10-12T01:30:53.416 DEBUG:boto:Data: 2023-10-12T01:30:53.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.418 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.418 DEBUG:boto:Params: {} 2023-10-12T01:30:53.418 DEBUG:boto:Token: None 2023-10-12T01:30:53.419 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj38 2023-10-12T01:30:53.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:y0fNnsQhdaN/56pBTFkreeQigy0= 2023-10-12T01:30:53.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y0fNnsQhdaN/56pBTFkreeQigy0='} 2023-10-12T01:30:53.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6249e4d63c482e-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.424 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.425 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.425 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.426 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.426 DEBUG:boto:Data: 2023-10-12T01:30:53.426 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.427 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.428 DEBUG:boto:Params: {} 2023-10-12T01:30:53.428 DEBUG:boto:Token: None 2023-10-12T01:30:53.429 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.429 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d8b1b3e59278d94-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.432 DEBUG:boto:path=/wafmoa-64/obj57 2023-10-12T01:30:53.433 DEBUG:boto:auth_path=/wafmoa-64/obj57 2023-10-12T01:30:53.433 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.434 DEBUG:boto:Path: /wafmoa-64/obj57 2023-10-12T01:30:53.434 DEBUG:boto:Data: 2023-10-12T01:30:53.434 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.435 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.436 DEBUG:boto:Params: {} 2023-10-12T01:30:53.436 DEBUG:boto:Token: None 2023-10-12T01:30:53.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj57 2023-10-12T01:30:53.437 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ZeL5KijJWbL9eX7mROeZGT6bmRI= 2023-10-12T01:30:53.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZeL5KijJWbL9eX7mROeZGT6bmRI='} 2023-10-12T01:30:53.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006540aa415bd229e3-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.483 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.483 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.484 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.484 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.485 DEBUG:boto:Data: 2023-10-12T01:30:53.485 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.486 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.487 DEBUG:boto:Params: {} 2023-10-12T01:30:53.487 DEBUG:boto:Token: None 2023-10-12T01:30:53.487 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c445f9b58b0568c7-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.492 DEBUG:boto:path=/wafmoa-64/obj76 2023-10-12T01:30:53.492 DEBUG:boto:auth_path=/wafmoa-64/obj76 2023-10-12T01:30:53.493 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.493 DEBUG:boto:Path: /wafmoa-64/obj76 2023-10-12T01:30:53.493 DEBUG:boto:Data: 2023-10-12T01:30:53.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.495 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.495 DEBUG:boto:Params: {} 2023-10-12T01:30:53.495 DEBUG:boto:Token: None 2023-10-12T01:30:53.496 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj76 2023-10-12T01:30:53.496 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3znAqQfSX9UmV9vdQwaRpWtF6QQ= 2023-10-12T01:30:53.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3znAqQfSX9UmV9vdQwaRpWtF6QQ='} 2023-10-12T01:30:53.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f4f36e375fea6cd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.502 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.503 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.503 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.504 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.504 DEBUG:boto:Data: 2023-10-12T01:30:53.504 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.505 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.506 DEBUG:boto:Params: {} 2023-10-12T01:30:53.506 DEBUG:boto:Token: None 2023-10-12T01:30:53.506 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.507 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001ef78a619f6159e-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.510 DEBUG:boto:path=/wafmoa-64/obj95 2023-10-12T01:30:53.511 DEBUG:boto:auth_path=/wafmoa-64/obj95 2023-10-12T01:30:53.511 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.512 DEBUG:boto:Path: /wafmoa-64/obj95 2023-10-12T01:30:53.512 DEBUG:boto:Data: 2023-10-12T01:30:53.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.513 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.514 DEBUG:boto:Params: {} 2023-10-12T01:30:53.514 DEBUG:boto:Token: None 2023-10-12T01:30:53.515 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj95 2023-10-12T01:30:53.515 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yf12Uz5NugW4dr984cIm0Ro3njk= 2023-10-12T01:30:53.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yf12Uz5NugW4dr984cIm0Ro3njk='} 2023-10-12T01:30:53.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f106f042b8f74416-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.560 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:53.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:53.613 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:53.659 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:53.659 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:53.665 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-10-12T01:30:53.713 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.714 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.714 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.715 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.715 DEBUG:boto:Data: 2023-10-12T01:30:53.716 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.716 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.717 DEBUG:boto:Params: {} 2023-10-12T01:30:53.717 DEBUG:boto:Token: None 2023-10-12T01:30:53.718 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.718 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002284e3f17b0f1f03-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.723 DEBUG:boto:path=/wafmoa-64/obj23 2023-10-12T01:30:53.723 DEBUG:boto:auth_path=/wafmoa-64/obj23 2023-10-12T01:30:53.723 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.724 DEBUG:boto:Path: /wafmoa-64/obj23 2023-10-12T01:30:53.724 DEBUG:boto:Data: 2023-10-12T01:30:53.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.726 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.726 DEBUG:boto:Params: {} 2023-10-12T01:30:53.726 DEBUG:boto:Token: None 2023-10-12T01:30:53.727 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj23 2023-10-12T01:30:53.727 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7PC279C39He0WlgtSmXb/8M2FgM= 2023-10-12T01:30:53.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7PC279C39He0WlgtSmXb/8M2FgM='} 2023-10-12T01:30:53.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a571d47392a13ac7-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.734 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.734 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.735 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.735 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.735 DEBUG:boto:Data: 2023-10-12T01:30:53.736 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.737 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.737 DEBUG:boto:Params: {} 2023-10-12T01:30:53.737 DEBUG:boto:Token: None 2023-10-12T01:30:53.738 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.738 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078662b189597c252-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.742 DEBUG:boto:path=/wafmoa-64/obj46 2023-10-12T01:30:53.743 DEBUG:boto:auth_path=/wafmoa-64/obj46 2023-10-12T01:30:53.743 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.743 DEBUG:boto:Path: /wafmoa-64/obj46 2023-10-12T01:30:53.744 DEBUG:boto:Data: 2023-10-12T01:30:53.744 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.745 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.745 DEBUG:boto:Params: {} 2023-10-12T01:30:53.746 DEBUG:boto:Token: None 2023-10-12T01:30:53.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj46 2023-10-12T01:30:53.747 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:itUlQ96VUE/eezMgE72iGKlAm2M= 2023-10-12T01:30:53.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:itUlQ96VUE/eezMgE72iGKlAm2M='} 2023-10-12T01:30:53.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:30:53.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bd44ade2dafeb33-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.753 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.753 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.754 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.754 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.754 DEBUG:boto:Data: 2023-10-12T01:30:53.755 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.756 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.756 DEBUG:boto:Params: {} 2023-10-12T01:30:53.756 DEBUG:boto:Token: None 2023-10-12T01:30:53.757 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.757 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e2d77ecbd83b495-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.761 DEBUG:boto:path=/wafmoa-64/obj69 2023-10-12T01:30:53.761 DEBUG:boto:auth_path=/wafmoa-64/obj69 2023-10-12T01:30:53.762 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.762 DEBUG:boto:Path: /wafmoa-64/obj69 2023-10-12T01:30:53.763 DEBUG:boto:Data: 2023-10-12T01:30:53.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.764 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.764 DEBUG:boto:Params: {} 2023-10-12T01:30:53.765 DEBUG:boto:Token: None 2023-10-12T01:30:53.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj69 2023-10-12T01:30:53.766 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6zwjVPJstomIW1B1nWI8vMtDAew= 2023-10-12T01:30:53.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6zwjVPJstomIW1B1nWI8vMtDAew='} 2023-10-12T01:30:53.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002497179e64c3567f-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.772 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.772 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.773 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.773 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.773 DEBUG:boto:Data: 2023-10-12T01:30:53.774 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.775 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.775 DEBUG:boto:Params: {} 2023-10-12T01:30:53.778 DEBUG:boto:Token: None 2023-10-12T01:30:53.778 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.779 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b61745888b51baec-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.783 DEBUG:boto:path=/wafmoa-64/obj92 2023-10-12T01:30:53.783 DEBUG:boto:auth_path=/wafmoa-64/obj92 2023-10-12T01:30:53.784 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.784 DEBUG:boto:Path: /wafmoa-64/obj92 2023-10-12T01:30:53.784 DEBUG:boto:Data: 2023-10-12T01:30:53.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.786 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.786 DEBUG:boto:Params: {} 2023-10-12T01:30:53.787 DEBUG:boto:Token: None 2023-10-12T01:30:53.787 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj92 2023-10-12T01:30:53.787 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KLwTrV0Ek4W29RfzsW8NTHf2/Vw= 2023-10-12T01:30:53.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KLwTrV0Ek4W29RfzsW8NTHf2/Vw='} 2023-10-12T01:30:53.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001304fe6e0da42af4-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.831 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:53.832 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:53.832 DEBUG:boto:Method: HEAD 2023-10-12T01:30:53.832 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:53.833 DEBUG:boto:Data: 2023-10-12T01:30:53.833 DEBUG:boto:Headers: {} 2023-10-12T01:30:53.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.834 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.834 DEBUG:boto:Params: {} 2023-10-12T01:30:53.835 DEBUG:boto:Token: None 2023-10-12T01:30:53.835 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/ 2023-10-12T01:30:53.836 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:30:53.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:30:53.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c16a49ed4c90fc63-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.840 DEBUG:boto:path=/wafmoa-64/obj115 2023-10-12T01:30:53.840 DEBUG:boto:auth_path=/wafmoa-64/obj115 2023-10-12T01:30:53.841 DEBUG:boto:Method: PUT 2023-10-12T01:30:53.841 DEBUG:boto:Path: /wafmoa-64/obj115 2023-10-12T01:30:53.841 DEBUG:boto:Data: 2023-10-12T01:30:53.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:53.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:53.843 DEBUG:boto:Port: 8000 2023-10-12T01:30:53.843 DEBUG:boto:Params: {} 2023-10-12T01:30:53.844 DEBUG:boto:Token: None 2023-10-12T01:30:53.844 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:53 GMT /wafmoa-64/obj115 2023-10-12T01:30:53.844 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QI7aq3+GfGKJjx7XV97dTGh5UAE= 2023-10-12T01:30:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QI7aq3+GfGKJjx7XV97dTGh5UAE='} 2023-10-12T01:30:53.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63c6f779a862dc0-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:53.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:53.852 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:53.951 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:53.951 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:53.959 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-10-12T01:30:54.011 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.012 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.012 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.012 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.013 DEBUG:boto:Data: 2023-10-12T01:30:54.013 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.014 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.015 DEBUG:boto:Params: {} 2023-10-12T01:30:54.015 DEBUG:boto:Token: None 2023-10-12T01:30:54.016 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.016 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d252b3b11f6ffe-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.020 DEBUG:boto:path=/wafmoa-64/obj29 2023-10-12T01:30:54.021 DEBUG:boto:auth_path=/wafmoa-64/obj29 2023-10-12T01:30:54.021 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.022 DEBUG:boto:Path: /wafmoa-64/obj29 2023-10-12T01:30:54.022 DEBUG:boto:Data: 2023-10-12T01:30:54.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.023 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.024 DEBUG:boto:Params: {} 2023-10-12T01:30:54.024 DEBUG:boto:Token: None 2023-10-12T01:30:54.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj29 2023-10-12T01:30:54.025 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zK373+42CpGPQr/eytzAls5ELM4= 2023-10-12T01:30:54.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zK373+42CpGPQr/eytzAls5ELM4='} 2023-10-12T01:30:54.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfe84279932b1b6d-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.071 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.071 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.072 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.072 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.073 DEBUG:boto:Data: 2023-10-12T01:30:54.073 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.074 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.075 DEBUG:boto:Params: {} 2023-10-12T01:30:54.075 DEBUG:boto:Token: None 2023-10-12T01:30:54.075 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.076 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900b9ebc159875a3-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.080 DEBUG:boto:path=/wafmoa-64/obj58 2023-10-12T01:30:54.080 DEBUG:boto:auth_path=/wafmoa-64/obj58 2023-10-12T01:30:54.081 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.081 DEBUG:boto:Path: /wafmoa-64/obj58 2023-10-12T01:30:54.081 DEBUG:boto:Data: 2023-10-12T01:30:54.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.083 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.083 DEBUG:boto:Params: {} 2023-10-12T01:30:54.084 DEBUG:boto:Token: None 2023-10-12T01:30:54.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj58 2023-10-12T01:30:54.085 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L4b5/NvjQJqIAvd6w7DywHXC7uQ= 2023-10-12T01:30:54.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L4b5/NvjQJqIAvd6w7DywHXC7uQ='} 2023-10-12T01:30:54.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c14eced8b4a3eada-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.091 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.091 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.092 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.092 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.092 DEBUG:boto:Data: 2023-10-12T01:30:54.093 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.094 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.094 DEBUG:boto:Params: {} 2023-10-12T01:30:54.095 DEBUG:boto:Token: None 2023-10-12T01:30:54.095 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.095 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021cbac25fdf7daf2-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.100 DEBUG:boto:path=/wafmoa-64/obj87 2023-10-12T01:30:54.100 DEBUG:boto:auth_path=/wafmoa-64/obj87 2023-10-12T01:30:54.100 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.101 DEBUG:boto:Path: /wafmoa-64/obj87 2023-10-12T01:30:54.101 DEBUG:boto:Data: 2023-10-12T01:30:54.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.103 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.103 DEBUG:boto:Params: {} 2023-10-12T01:30:54.103 DEBUG:boto:Token: None 2023-10-12T01:30:54.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj87 2023-10-12T01:30:54.104 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WAh6CmTZsUqo4I2QWx8Xu92XZgY= 2023-10-12T01:30:54.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WAh6CmTZsUqo4I2QWx8Xu92XZgY='} 2023-10-12T01:30:54.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a01ce57d91cc4033-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.151 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.152 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.152 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.152 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.153 DEBUG:boto:Data: 2023-10-12T01:30:54.153 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.154 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.154 DEBUG:boto:Params: {} 2023-10-12T01:30:54.155 DEBUG:boto:Token: None 2023-10-12T01:30:54.155 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.156 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc10d140381c332-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.160 DEBUG:boto:path=/wafmoa-64/obj116 2023-10-12T01:30:54.160 DEBUG:boto:auth_path=/wafmoa-64/obj116 2023-10-12T01:30:54.161 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.161 DEBUG:boto:Path: /wafmoa-64/obj116 2023-10-12T01:30:54.162 DEBUG:boto:Data: 2023-10-12T01:30:54.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.163 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.163 DEBUG:boto:Params: {} 2023-10-12T01:30:54.164 DEBUG:boto:Token: None 2023-10-12T01:30:54.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj116 2023-10-12T01:30:54.165 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:paK3Nb4KIax5TKlGfOg7lY5z7co= 2023-10-12T01:30:54.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:paK3Nb4KIax5TKlGfOg7lY5z7co='} 2023-10-12T01:30:54.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e58d52e3ac4cbb8-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.171 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.171 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.172 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.172 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.173 DEBUG:boto:Data: 2023-10-12T01:30:54.173 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.174 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.174 DEBUG:boto:Params: {} 2023-10-12T01:30:54.175 DEBUG:boto:Token: None 2023-10-12T01:30:54.175 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.176 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000796e5bbb12d8f9da-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.180 DEBUG:boto:path=/wafmoa-64/obj145 2023-10-12T01:30:54.180 DEBUG:boto:auth_path=/wafmoa-64/obj145 2023-10-12T01:30:54.180 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.181 DEBUG:boto:Path: /wafmoa-64/obj145 2023-10-12T01:30:54.181 DEBUG:boto:Data: 2023-10-12T01:30:54.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.182 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.183 DEBUG:boto:Params: {} 2023-10-12T01:30:54.183 DEBUG:boto:Token: None 2023-10-12T01:30:54.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj145 2023-10-12T01:30:54.184 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FAjCk59qfuEi15RKsHE8BIUw1/M= 2023-10-12T01:30:54.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FAjCk59qfuEi15RKsHE8BIUw1/M='} 2023-10-12T01:30:54.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d28a495c9f86f2af-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.228 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:54.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:54.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:54.335 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:54.335 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:54.345 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-10-12T01:30:54.406 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.407 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.407 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.408 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.408 DEBUG:boto:Data: 2023-10-12T01:30:54.409 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.409 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.410 DEBUG:boto:Params: {} 2023-10-12T01:30:54.410 DEBUG:boto:Token: None 2023-10-12T01:30:54.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ef247736c992347-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.415 DEBUG:boto:path=/wafmoa-64/obj31 2023-10-12T01:30:54.416 DEBUG:boto:auth_path=/wafmoa-64/obj31 2023-10-12T01:30:54.416 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.417 DEBUG:boto:Path: /wafmoa-64/obj31 2023-10-12T01:30:54.417 DEBUG:boto:Data: 2023-10-12T01:30:54.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.418 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.419 DEBUG:boto:Params: {} 2023-10-12T01:30:54.419 DEBUG:boto:Token: None 2023-10-12T01:30:54.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj31 2023-10-12T01:30:54.420 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3EcHhEgPjPz7HNAEk934d/M8ADU= 2023-10-12T01:30:54.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3EcHhEgPjPz7HNAEk934d/M8ADU='} 2023-10-12T01:30:54.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000674024992b469160-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.426 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.427 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.427 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.427 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.428 DEBUG:boto:Data: 2023-10-12T01:30:54.428 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.429 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.430 DEBUG:boto:Params: {} 2023-10-12T01:30:54.430 DEBUG:boto:Token: None 2023-10-12T01:30:54.431 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.431 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058025e962717bbe0-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.435 DEBUG:boto:path=/wafmoa-64/obj62 2023-10-12T01:30:54.436 DEBUG:boto:auth_path=/wafmoa-64/obj62 2023-10-12T01:30:54.436 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.436 DEBUG:boto:Path: /wafmoa-64/obj62 2023-10-12T01:30:54.437 DEBUG:boto:Data: 2023-10-12T01:30:54.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.438 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.439 DEBUG:boto:Params: {} 2023-10-12T01:30:54.439 DEBUG:boto:Token: None 2023-10-12T01:30:54.439 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj62 2023-10-12T01:30:54.440 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3iWbGe82NWz8QfIR/4lEgENFD8E= 2023-10-12T01:30:54.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3iWbGe82NWz8QfIR/4lEgENFD8E='} 2023-10-12T01:30:54.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d084f0a64d962a01-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.483 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.484 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.484 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.484 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.485 DEBUG:boto:Data: 2023-10-12T01:30:54.485 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.486 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.486 DEBUG:boto:Params: {} 2023-10-12T01:30:54.487 DEBUG:boto:Token: None 2023-10-12T01:30:54.487 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.488 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b5d1053e8440ec6-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.492 DEBUG:boto:path=/wafmoa-64/obj93 2023-10-12T01:30:54.493 DEBUG:boto:auth_path=/wafmoa-64/obj93 2023-10-12T01:30:54.493 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.494 DEBUG:boto:Path: /wafmoa-64/obj93 2023-10-12T01:30:54.494 DEBUG:boto:Data: 2023-10-12T01:30:54.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.495 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.496 DEBUG:boto:Params: {} 2023-10-12T01:30:54.496 DEBUG:boto:Token: None 2023-10-12T01:30:54.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj93 2023-10-12T01:30:54.497 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VH4dBKt95Vhzs0qhHC1A/DHbBqY= 2023-10-12T01:30:54.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VH4dBKt95Vhzs0qhHC1A/DHbBqY='} 2023-10-12T01:30:54.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091de2933594fa48d-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.504 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.505 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.505 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.506 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.506 DEBUG:boto:Data: 2023-10-12T01:30:54.507 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.507 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.508 DEBUG:boto:Params: {} 2023-10-12T01:30:54.508 DEBUG:boto:Token: None 2023-10-12T01:30:54.509 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.509 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c897b46f1d325354-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.513 DEBUG:boto:path=/wafmoa-64/obj124 2023-10-12T01:30:54.513 DEBUG:boto:auth_path=/wafmoa-64/obj124 2023-10-12T01:30:54.514 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.514 DEBUG:boto:Path: /wafmoa-64/obj124 2023-10-12T01:30:54.515 DEBUG:boto:Data: 2023-10-12T01:30:54.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.516 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.516 DEBUG:boto:Params: {} 2023-10-12T01:30:54.517 DEBUG:boto:Token: None 2023-10-12T01:30:54.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj124 2023-10-12T01:30:54.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Fx1Iy6tadhlgC+d0wKv56s9G0ac= 2023-10-12T01:30:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fx1Iy6tadhlgC+d0wKv56s9G0ac='} 2023-10-12T01:30:54.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095a8fe5f3d06cb29-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.563 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.564 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.564 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.564 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.565 DEBUG:boto:Data: 2023-10-12T01:30:54.565 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.566 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.566 DEBUG:boto:Params: {} 2023-10-12T01:30:54.567 DEBUG:boto:Token: None 2023-10-12T01:30:54.567 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.568 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e681b5bd019c67f-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.572 DEBUG:boto:path=/wafmoa-64/obj155 2023-10-12T01:30:54.572 DEBUG:boto:auth_path=/wafmoa-64/obj155 2023-10-12T01:30:54.573 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.573 DEBUG:boto:Path: /wafmoa-64/obj155 2023-10-12T01:30:54.574 DEBUG:boto:Data: 2023-10-12T01:30:54.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.575 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.575 DEBUG:boto:Params: {} 2023-10-12T01:30:54.576 DEBUG:boto:Token: None 2023-10-12T01:30:54.576 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj155 2023-10-12T01:30:54.577 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tg5d0RJ5b4xF63VTQZgzCnybL6g= 2023-10-12T01:30:54.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tg5d0RJ5b4xF63VTQZgzCnybL6g='} 2023-10-12T01:30:54.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046b960fd4e38cf45-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.584 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:54.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:54.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:54.687 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-10-12T01:30:54.688 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: wafmoa-64 2023-10-12T01:30:54.698 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-10-12T01:30:54.761 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.761 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.762 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.762 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.762 DEBUG:boto:Data: 2023-10-12T01:30:54.763 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.764 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.764 DEBUG:boto:Params: {} 2023-10-12T01:30:54.765 DEBUG:boto:Token: None 2023-10-12T01:30:54.765 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.765 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a39f25b0bc4ceb8a-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.770 DEBUG:boto:path=/wafmoa-64/obj37 2023-10-12T01:30:54.771 DEBUG:boto:auth_path=/wafmoa-64/obj37 2023-10-12T01:30:54.771 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.772 DEBUG:boto:Path: /wafmoa-64/obj37 2023-10-12T01:30:54.772 DEBUG:boto:Data: 2023-10-12T01:30:54.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.773 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.774 DEBUG:boto:Params: {} 2023-10-12T01:30:54.774 DEBUG:boto:Token: None 2023-10-12T01:30:54.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj37 2023-10-12T01:30:54.775 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rx217jRTLSvA+gn6REt7Xveno88= 2023-10-12T01:30:54.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rx217jRTLSvA+gn6REt7Xveno88='} 2023-10-12T01:30:54.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e63993f1035cd4f3-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.781 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.781 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.782 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.782 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.782 DEBUG:boto:Data: 2023-10-12T01:30:54.783 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.784 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.784 DEBUG:boto:Params: {} 2023-10-12T01:30:54.784 DEBUG:boto:Token: None 2023-10-12T01:30:54.785 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.785 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7e214a4f7edc436-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.790 DEBUG:boto:path=/wafmoa-64/obj74 2023-10-12T01:30:54.790 DEBUG:boto:auth_path=/wafmoa-64/obj74 2023-10-12T01:30:54.791 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.791 DEBUG:boto:Path: /wafmoa-64/obj74 2023-10-12T01:30:54.792 DEBUG:boto:Data: 2023-10-12T01:30:54.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.793 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.793 DEBUG:boto:Params: {} 2023-10-12T01:30:54.794 DEBUG:boto:Token: None 2023-10-12T01:30:54.794 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj74 2023-10-12T01:30:54.795 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ymrqEYbc012h1z/+Yre9jMQNJWk= 2023-10-12T01:30:54.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ymrqEYbc012h1z/+Yre9jMQNJWk='} 2023-10-12T01:30:54.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f8f02964dab2b9e-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.839 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.840 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.840 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.841 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.841 DEBUG:boto:Data: 2023-10-12T01:30:54.842 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.843 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.843 DEBUG:boto:Params: {} 2023-10-12T01:30:54.844 DEBUG:boto:Token: None 2023-10-12T01:30:54.844 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.845 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a8427edcab942b-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.849 DEBUG:boto:path=/wafmoa-64/obj111 2023-10-12T01:30:54.850 DEBUG:boto:auth_path=/wafmoa-64/obj111 2023-10-12T01:30:54.850 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.851 DEBUG:boto:Path: /wafmoa-64/obj111 2023-10-12T01:30:54.851 DEBUG:boto:Data: 2023-10-12T01:30:54.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.853 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.853 DEBUG:boto:Params: {} 2023-10-12T01:30:54.854 DEBUG:boto:Token: None 2023-10-12T01:30:54.854 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj111 2023-10-12T01:30:54.855 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S4y335bPAcSVe+wWrIQsyqbOsG0= 2023-10-12T01:30:54.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S4y335bPAcSVe+wWrIQsyqbOsG0='} 2023-10-12T01:30:54.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a962c2e5804a81c6-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.860 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.861 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.861 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.862 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.862 DEBUG:boto:Data: 2023-10-12T01:30:54.863 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.863 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.864 DEBUG:boto:Params: {} 2023-10-12T01:30:54.864 DEBUG:boto:Token: None 2023-10-12T01:30:54.865 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.865 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1c0245834264cf3-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.870 DEBUG:boto:path=/wafmoa-64/obj148 2023-10-12T01:30:54.870 DEBUG:boto:auth_path=/wafmoa-64/obj148 2023-10-12T01:30:54.871 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.871 DEBUG:boto:Path: /wafmoa-64/obj148 2023-10-12T01:30:54.871 DEBUG:boto:Data: 2023-10-12T01:30:54.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.873 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.873 DEBUG:boto:Params: {} 2023-10-12T01:30:54.874 DEBUG:boto:Token: None 2023-10-12T01:30:54.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj148 2023-10-12T01:30:54.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qkt9NkTiJb/0TqOYbR1EQ+nOdDc= 2023-10-12T01:30:54.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qkt9NkTiJb/0TqOYbR1EQ+nOdDc='} 2023-10-12T01:30:54.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049cd0b4ec20fc27c-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.881 DEBUG:boto:path=/wafmoa-64/ 2023-10-12T01:30:54.881 DEBUG:boto:auth_path=/wafmoa-64/ 2023-10-12T01:30:54.882 DEBUG:boto:Method: HEAD 2023-10-12T01:30:54.882 DEBUG:boto:Path: /wafmoa-64/ 2023-10-12T01:30:54.882 DEBUG:boto:Data: 2023-10-12T01:30:54.883 DEBUG:boto:Headers: {} 2023-10-12T01:30:54.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.884 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.884 DEBUG:boto:Params: {} 2023-10-12T01:30:54.884 DEBUG:boto:Token: None 2023-10-12T01:30:54.885 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/ 2023-10-12T01:30:54.885 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:30:54.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:30:54.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000830c18989217778b-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.890 DEBUG:boto:path=/wafmoa-64/obj185 2023-10-12T01:30:54.890 DEBUG:boto:auth_path=/wafmoa-64/obj185 2023-10-12T01:30:54.891 DEBUG:boto:Method: PUT 2023-10-12T01:30:54.891 DEBUG:boto:Path: /wafmoa-64/obj185 2023-10-12T01:30:54.891 DEBUG:boto:Data: 2023-10-12T01:30:54.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:30:54.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:30:54.893 DEBUG:boto:Port: 8000 2023-10-12T01:30:54.893 DEBUG:boto:Params: {} 2023-10-12T01:30:54.893 DEBUG:boto:Token: None 2023-10-12T01:30:54.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Thu, 12 Oct 2023 01:30:54 GMT /wafmoa-64/obj185 2023-10-12T01:30:54.894 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:SGacSw9bzgkhlRb5uMkDvXhe6p4= 2023-10-12T01:30:54.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SGacSw9bzgkhlRb5uMkDvXhe6p4='} 2023-10-12T01:30:54.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc4060a51c1412c0-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:30:54.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:30:54.940 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:30:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:55.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:55.020+0000 7fc4490d2780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-12T01:30:55.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:55.021+0000 7fc4490d2780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-12T01:30:55.043 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-12T01:30:55.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:30:55.045 INFO:tasks.rgw.c1.client.1:Started 2023-10-12T01:30:55.046 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:30:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-12T01:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:30:56.070 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:30:56.096 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:30:56.097 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-10-12T01:30:56.097 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:30:56.098 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-10-12T01:30:56.098 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=wafmoa-64 2023-10-12T01:30:56.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:56.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:30:56.099 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:30:56.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:30:56.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:30:56.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:30:56.235+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:01.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:01.235+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:06.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:06.236+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:11.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:11.238+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:16.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:16.238+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:21.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:21.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:26.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:26.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:31.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:36.240+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:41.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:41.240+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:46.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:46.242+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:51.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:51.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:31:56.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:31:56.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:32:01.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:01.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:06.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:06.244+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:11.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:11.245+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:16.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:16.245+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:21.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:21.246+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:26.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:26.248+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:31.248+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:36.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:36.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:41.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:41.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:46.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:46.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:51.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:51.250+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:32:56.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:32:56.250+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:01.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:01.252+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:05.282 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:33:05.282+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:33:06.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:06.253+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:11.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:11.253+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:16.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:16.254+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:21.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:21.254+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:26.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:26.255+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:31.256+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:36.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:36.256+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:41.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:41.258+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:46.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:46.258+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:51.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:51.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:33:56.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:33:56.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:01.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:01.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:06.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:06.260+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:11.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:11.261+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:16.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:16.261+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:21.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:21.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:26.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:26.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:31.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:31.264+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:36.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:36.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:41.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:41.264+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:46.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:46.265+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:51.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:51.265+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:34:56.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:34:56.267+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:01.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:01.267+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:06.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:06.268+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:11.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:11.268+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:16.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:16.269+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:21.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:21.269+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:26.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:26.270+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:31.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:31.271+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:36.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:36.272+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:41.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:41.273+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:46.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:46.273+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:35:51.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:51.273+0000 7efd02122780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-12T01:35:51.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:51.273+0000 7efd02122780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:35:51.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:51.273+0000 7efd02122780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:35:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-12T01:35:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-10-12T01:35:51.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:51.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:51.297 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:35:51.349 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.350 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.382 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:35:51.395 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:35:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:35:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:35:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:35:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:30:54.707300Z" 2023-10-12T01:35:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:35:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:35:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.399 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:35:51.400 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:51.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:35:51.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:51.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:51.401 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:35:51.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:51.481+0000 7fbe5b2ea780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:51.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:51.483+0000 7fbe5b2ea780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:51.497 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:35:51.498 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:35:51.498 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:35:51.498 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:35:51.499 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:35:51.499 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:35:51.499 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:51.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.499 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:35:51.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.500 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:35:51.500 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:51.500 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:51.500 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:35:51.501 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:51.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:35:51.501 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:51.501 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:35:51.501 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:51.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.502 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:51.503 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:51.504 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:35:51.504 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:51.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.505 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:35:51.505 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:51.505 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:51.505 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:35:51.505 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:51.506 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:51.506 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:51.506 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:35:51.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:51.507 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:51.508 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:35:51.508 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:51.508 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:51.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:51.508 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:35:51.509 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:51.509 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.509 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:51.509 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:51.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.510 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:35:51.510 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:35:51.510 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:35:51.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:51.510 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:35:51Z" 2023-10-12T01:35:51.511 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:35:51.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:35:51Z" } 2023-10-12T01:35:51.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:35:51.512 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:51.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:51.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:35:51.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.514 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:51.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.606 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:35:51.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.607 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:51.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.702 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:35:51.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.703 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.797 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:51.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:51.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.891 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-12T01:35:51.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.892 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:51.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:51.938 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:51.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:51.983 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:52.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:52.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:52.085 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:52.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.087 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:52.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:52.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:52.186 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:52.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.187 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:52.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:52.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:52.415 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:35:52.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.416 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:52.468 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:52.469 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:52.511 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-10-12T01:35:52.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.513 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:52.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:52.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:52.612 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:35:52.612 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:35:52.612 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13, 2023-10-12T01:35:52.612 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:35:52.613 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:35:52.613 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:35:52.613 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:35:52.613 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:35:52.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.614 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:35:52.614 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:35:52.614 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:35:52.615 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:35:52.615 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.615 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:35:52.615 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.615 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:35:52.616 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:35:52.616 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.616 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:35:52.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.616 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.617 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:35:52.617 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.617 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:35:52.617 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.618 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:35:52.618 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:35:52.618 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:35:52.618 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:35:52.619 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:35:52.619 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:35:52.619 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:35:52.619 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:35:52.619 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:35:52.620 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:35:52.620 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.620 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.621 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:35:52.621 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:35:52.621 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.621 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:35:52.621 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.622 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:35:52.622 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:35:52.622 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:35:52.622 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:35:52.622 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:35:52.623 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:35:52.623 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:35:52.623 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:35:52.623 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:35:52.623 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:35:52.624 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.624 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.624 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.624 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:35:52.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.625 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:35:52.625 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:35:52.625 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:35:52.625 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:35:52.626 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.626 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.626 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:35:52.627 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:35:52.627 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:35:52.627 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:35:52.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.627 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:35:52.628 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:35:52.628 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:35:52.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.628 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:35:52.628 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:35:52.629 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.629 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:35:52.629 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.629 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.630 INFO:teuthology.orchestra.run.smithi112.stdout: "directional": [ 2023-10-12T01:35:52.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.630 INFO:teuthology.orchestra.run.smithi112.stdout: "source_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.630 INFO:teuthology.orchestra.run.smithi112.stdout: "dest_zone": "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.631 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:35:52.632 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:35:52.632 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:35:52.632 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:35:52.632 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.632 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:35:52.633 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.634 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:35:52.634 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.634 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:35:52.635 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:35:52.635 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:35:52.635 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:35:52.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.636 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:35:52.636 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:35:52.636 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:35:52.636 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:35:52.636 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.638 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:35:52.638 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.638 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.639 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:35:52.639 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:35:52.639 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:35:52.639 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:35:52.639 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.640 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:35:52.640 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.640 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:35:52.640 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:35:52.640 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:35:52.641 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:35:52.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.641 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:35:52.641 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:35:52.641 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.642 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:35:52.642 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.642 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:35:52.642 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:35:52.643 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:35:52.643 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:35:52.643 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:35:52.643 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:35:52.643 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:35:52.644 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:35:52.644 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:35:52.644 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:35:52.644 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.644 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.645 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:35:52.645 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:35:52.646 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:35:52.646 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:35:52.646 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.646 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:35:52.647 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:35:52.647 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:35:52.647 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:35:52.647 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:35:52.647 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:35:52.648 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:35:52.648 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:35:52.648 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:35:52.648 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:35:52.648 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.649 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.649 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.649 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:35:52.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.650 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:35:52.650 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:35:52.650 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:35:52.650 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:35:52.651 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.651 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.651 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:35:52.651 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:35:52.652 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:35:52.652 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:35:52.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.652 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:35:52.653 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:35:52.653 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:35:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:35:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:35:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:35:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:35:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:35:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:35:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:35:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:35:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:35:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:35:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:35:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:35:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:35:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:35:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:35:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:35:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:35:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:35:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:35:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:35:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:35:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:35:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:35:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:35:52.663 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:35:52.663 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:35:52.663 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:35:52.663 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:35:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:35:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:35:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:35:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:35:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:35:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:35:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:35:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:35:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:35:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:35:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:35:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:35:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:35:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:35:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:35:52.668 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:52.668 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:35:52.668 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-65 2023-10-12T01:35:52.669 DEBUG:boto:path=/wafmoa-65/ 2023-10-12T01:35:52.669 DEBUG:boto:auth_path=/wafmoa-65/ 2023-10-12T01:35:52.670 DEBUG:boto:Method: PUT 2023-10-12T01:35:52.670 DEBUG:boto:Path: /wafmoa-65/ 2023-10-12T01:35:52.670 DEBUG:boto:Data: 2023-10-12T01:35:52.671 DEBUG:boto:Headers: {} 2023-10-12T01:35:52.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:35:52.672 DEBUG:boto:Port: 8000 2023-10-12T01:35:52.672 DEBUG:boto:Params: {} 2023-10-12T01:35:52.673 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:35:52.673 DEBUG:boto:Token: None 2023-10-12T01:35:52.674 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:35:52 GMT /wafmoa-65/ 2023-10-12T01:35:52.675 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TtVdaKfGKnAMKEaPGt015aTHsdU= 2023-10-12T01:35:52.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:35:52 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TtVdaKfGKnAMKEaPGt015aTHsdU=', 'Content-Length': '0'} 2023-10-12T01:35:52.951 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7e720c14e95facb-0065274d78-2130-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:35:52.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:52.953 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.229 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:35:53.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.231 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.309 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.309 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.387 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:53.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.389 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.524 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-12T01:35:53.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.525 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.574 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.619 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:53.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.620 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.676 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.725 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:53.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.726 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.782 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.782 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.830 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:35:53.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.832 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.884 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:53.938 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:35:53.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:35:53.940 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:35:53.989 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:53.990 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:54.036 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-10-12T01:35:54.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:54.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:54.037 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:35:54.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:54.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:35:54.129 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:35:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:35:54.129 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:35:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:35:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:35:54.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:35:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:35:53.924941Z" 2023-10-12T01:35:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:35:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:35:54.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.132 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:35:54.133 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:54.134 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:35:54.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:54.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:54.135 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:35:54.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:54.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:54.213+0000 7f2ff3cbc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:54.214+0000 7f2ff3cbc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:54.227 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:35:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:35:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:35:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:35:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:35:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:35:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:35:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:35:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:35:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:35:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:35:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:35:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:35:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:35:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.237 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:35:54.238 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:54.238 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:54.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:35:54.238 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:54.239 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:54.239 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:54.239 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:35:54.239 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:54.239 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:54.240 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:35:54.240 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:54.241 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:54.241 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:35:54.241 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:54.241 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:54.241 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:54.242 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:35:54.242 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:54.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:54.242 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:54.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.243 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:35:54.243 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:35:54.243 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:35:54.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:54.244 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:35:54Z" 2023-10-12T01:35:54.244 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:35:54.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:35:54Z" } 2023-10-12T01:35:54.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:35:54.246 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:54.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:54.247 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:35:59.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:59.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:35:59.249 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:35:59.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:35:59.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:35:59.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:59.339+0000 7fa570b30780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:59.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:35:59.340+0000 7fa570b30780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:35:59.354 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:35:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:35:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:35:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:35:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:35:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:35:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:35:59.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:35:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:35:59.357 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:35:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:35:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:59.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:35:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:35:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:59.360 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:35:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:35:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:35:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:35:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:35:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:35:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:35:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:35:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:35:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:35:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:35:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:35:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:35:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:35:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:35:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:35:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:35:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:35:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:35:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:35:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:35:59Z" 2023-10-12T01:35:59.368 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:35:59.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:35:59Z" } 2023-10-12T01:35:59.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:35:59.370 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:59.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:35:59.371 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:04.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:04.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:04.373 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:04.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:04.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:04.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:04.464+0000 7f4af6bc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:04.465+0000 7f4af6bc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:04.479 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:04.480 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.481 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:04.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:04.485 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:04.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:04.487 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:04.488 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:04.489 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:04.490 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:04.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:04.492 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:04.493 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:04Z" 2023-10-12T01:36:04.493 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:04.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:04Z" } 2023-10-12T01:36:04.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:04.495 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:04.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:04.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:09.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:09.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:09.499 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:09.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:09.554 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:09.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:09.583+0000 7f8de3bc2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:09.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:09.584+0000 7f8de3bc2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:09.598 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:09.606 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:09.606 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:09.606 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:09.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:09.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:09.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:09.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:09.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:09.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:09.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:09.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:09.609 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:09.609 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:09.609 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:09.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:09.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:09.610 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:09.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.610 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:09.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:09.611 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:09.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:09.611 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:09Z" 2023-10-12T01:36:09.611 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:09.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:09Z" } 2023-10-12T01:36:09.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:09.613 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:09.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:09.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:14.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:14.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:14.616 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:14.673 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:14.673 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:14.705+0000 7f75eb458780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:14.707+0000 7f75eb458780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:14.720 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:14.721 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:14.721 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:14.721 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:14.721 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:14.733 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:14.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.733 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:14.733 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:14.733 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:14.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:14.734 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:14Z" 2023-10-12T01:36:14.734 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:14.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:14Z" } 2023-10-12T01:36:14.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:14.736 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:14.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:14.737 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:19.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:19.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:19.739 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:19.795 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:19.796 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:19.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:19.828+0000 7f3b20ee6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:19.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:19.829+0000 7f3b20ee6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:19.843 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:19.843 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:19.843 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:19.843 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:19.844 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:19.844 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:19.844 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:19.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.844 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:19.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:19.845 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:19.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:19.845 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:19.845 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:19.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:19.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:19.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:19.846 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:19.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:19.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:19.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:19.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:19.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:19.848 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:19.848 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:19.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:19.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:19.849 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:19.849 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:19.849 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:19.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:19.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:19.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:19.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:19.851 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:19.851 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:19.851 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:19.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:19.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:19.852 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:19.852 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:19.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:19.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:19.853 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:19.853 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:19.853 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:19.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:19.854 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:19.854 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:19.854 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:19.855 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:19.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:19.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:19.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:19.855 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:19.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.856 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:19.856 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:19.856 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:19.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:19.857 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:19Z" 2023-10-12T01:36:19.857 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:19.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:19Z" } 2023-10-12T01:36:19.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:19.859 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:19.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:19.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:24.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:24.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:24.863 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:24.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:24.953+0000 7fd925162780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:24.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:24.954+0000 7fd925162780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:24.967 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:24.968 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:24.969 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:24.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.969 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:24.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:24.969 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:24.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:24.970 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:24.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:24.970 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:24.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:24.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:24.971 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:24.971 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:24.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:24.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:24.972 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:24.972 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:24.973 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:24.974 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:24.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:24.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:24.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:24.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:24.975 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:24.975 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:24.976 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:24.976 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:24.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:24.976 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:24.976 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:24.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:24.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:24.977 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:24.977 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:24.978 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:24.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:24.978 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:24.978 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:24.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:24.979 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:24.979 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:24.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:24.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:24.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:24.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.980 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:24.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:24.981 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:24.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:24.981 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:24Z" 2023-10-12T01:36:24.981 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:24.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:24Z" } 2023-10-12T01:36:24.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:24.983 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:24.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:24.984 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:29.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:29.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:29.987 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:30.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:30.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:30.077+0000 7f9503df0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:30.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:30.078+0000 7f9503df0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:30.092 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:30.103 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:30.103 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:30.103 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:30.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:30.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:30.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:30.104 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:30.104 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:30.104 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:30.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:30.105 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:30.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.105 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:30.106 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:30.106 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:30.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:30.106 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:30Z" 2023-10-12T01:36:30.107 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:30.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:30Z" } 2023-10-12T01:36:30.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:30.109 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:30.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:30.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:35.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:35.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:35.113 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:35.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:35.170 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:35.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:35.201+0000 7f0cf33c0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:35.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:35.202+0000 7f0cf33c0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:35.216 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:35.217 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:35.217 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:35.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:35.217 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:35.217 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:35.218 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:35.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.218 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:35.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:35.219 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:35.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:35.219 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:35.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:35.220 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:35.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:35.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:35.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:35.220 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:35.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:35.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:35.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:35.222 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:35.222 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:35.222 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:35.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:35.223 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:35.223 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:35.223 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:35.224 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:35.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:35.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:35.224 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:35.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:35.225 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:35.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:35.225 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:35.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:35.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:35.226 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:35.226 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:35.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:35.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:35.227 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:35.227 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:35.227 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:35.227 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:35.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:35.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:35.228 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:35.228 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:35.229 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:35.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:35.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:35.229 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:35.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.230 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:35.230 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:35.230 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:35.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:35.230 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:35Z" 2023-10-12T01:36:35.231 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:35.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:35Z" } 2023-10-12T01:36:35.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:35.233 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:35.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:35.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:40.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:40.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:40.236 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:40.326+0000 7f639a2c5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:40.327+0000 7f639a2c5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:40.348 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:40.349 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:40.349 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:40.349 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:40.349 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:40.350 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:40.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:40.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.351 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:40.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:40.351 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:40.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:40.352 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:40.352 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:40.352 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:40.352 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:40.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:40.353 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:40.353 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:40.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:40.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:40.354 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:40.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:40.354 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:40.355 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:40.355 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:40.355 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:40.355 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:40.355 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:40.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:40.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:40.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:40.356 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:40.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:40.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:40.357 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:40.357 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:40.357 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:40.358 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:40.358 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:40.358 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:40.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:40.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:40.359 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:40.359 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:40.359 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:40.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:40.360 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:40.360 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:40.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:40.360 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:40.361 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:40.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:40.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:40.361 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:40.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.362 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:40.362 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:40.362 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:40.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:40.363 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:40Z" 2023-10-12T01:36:40.363 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:40.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:40Z" } 2023-10-12T01:36:40.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:40.365 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:40.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:40.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:45.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:45.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:45.368 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:45.427 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:45.428 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:45.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:45.460+0000 7fd149ab4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:45.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:45.461+0000 7fd149ab4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:45.476 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:45.491 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:45.492 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:45.493 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:45.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:45.495 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:45.496 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:45.497 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:45.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:45.499 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:45.500 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:45Z" 2023-10-12T01:36:45.501 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:45.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:45Z" } 2023-10-12T01:36:45.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:45.503 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:45.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:45.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:50.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:50.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:50.507 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:50.563 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:50.563 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:50.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:50.595+0000 7fc533d2d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:50.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:50.596+0000 7fc533d2d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:50.610 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:50.610 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:50.610 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:50.611 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:50.611 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:50.611 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:50.611 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:50.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.612 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:50.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:50.612 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:50.612 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:50.612 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:50.613 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:50.613 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:50.613 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:50.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:50.613 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:50.614 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:50.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:50.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:50.614 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:50.615 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:50.615 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:50.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:50.615 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:50.615 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:50.616 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:50.616 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:50.616 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:50.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:50.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:50.617 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:50.617 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:50.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:50.617 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:50.618 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:50.620 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:50.620 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:50.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:50.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:50.620 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:50.621 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:50.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:50.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:50.621 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:50.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:50Z" 2023-10-12T01:36:50.623 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:50.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:50Z" } 2023-10-12T01:36:50.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:50.624 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:50.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:50.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:36:55.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:55.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:36:55.627 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:36:55.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:36:55.684 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:36:55.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:55.716+0000 7f7120be9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:55.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:36:55.717+0000 7f7120be9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:36:55.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:36:55.731 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:36:55.731 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:36:55.732 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:36:55.732 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:36:55.732 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:36:55.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:55.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.733 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:36:55.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:55.733 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:36:55.733 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:55.734 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:55.734 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:36:55.734 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:55.734 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:36:55.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:55.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:36:55.735 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:55.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:55.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:55.736 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:36:55.736 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:55.736 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:55.736 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:36:55.736 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:36:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:36:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:55.738 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:36:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:55.739 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:36:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:36:55.740 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:36:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:36:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:36:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:36:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:36:55.741 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:36:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:36:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:36:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:36:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:55.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:36:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:36:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:36:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:36:55.743 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:36:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:36:55.744 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:36:55Z" 2023-10-12T01:36:55.744 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:36:55.745 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:36:55Z" } 2023-10-12T01:36:55.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:36:55.746 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:55.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:36:55.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:37:00.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:00.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:00.749 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:00.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:00.807 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:00.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:00.839+0000 7f24664fd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:00.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:00.840+0000 7f24664fd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:00.854 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:00.855 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:00.855 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:00.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:00.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:00.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:00.857 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:00.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:00.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:00.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:00.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:00.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:00.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:00.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:00.859 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:00.859 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:00.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:00.860 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:00.860 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:00.860 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:00.860 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:00.860 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:00.861 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074254.707300_752.1", 2023-10-12T01:37:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:00.862 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:00.863 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:00.863 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.707300Z", 2023-10-12T01:37:00.863 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:00.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:00.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:00.864 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:00.864 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:00.864 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:00.864 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:00.865 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:00.865 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:00.865 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:00.865 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:00.866 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:00.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:00.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:00.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:00.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.867 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:00.867 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:00.867 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:00.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:00.867 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:00Z" 2023-10-12T01:37:00.868 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:00.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074254.707300_752.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.707300Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:00Z" } 2023-10-12T01:37:00.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074254.707300_752.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.707300Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:00.869 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:00.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074254.707300_752.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:00.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074254.707300_752.1 2023-10-12T01:37:05.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:05.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:05.873 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:05.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:05.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:05.961+0000 7ff0e02d0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:05.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:05.962+0000 7ff0e02d0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:05.976 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:05.983 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:05.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.708070_782.1", 2023-10-12T01:37:05.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:05.984 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:05.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:05.984 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.708070Z", 2023-10-12T01:37:05.985 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:05.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:05.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:05.985 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:05Z" 2023-10-12T01:37:05.989 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:05.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.708070_782.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.708070Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:05Z" } 2023-10-12T01:37:05.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.708070_782.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.708070Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:05.991 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:05.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.708070_782.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:05.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074553.708070_782.1 2023-10-12T01:37:10.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:10.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:10.994 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:11.052 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:11.052 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:11.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:11.084+0000 7f4f23afb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:11.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:11.085+0000 7f4f23afb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:11.100 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:11.100 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:11.101 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:11.101 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:11.101 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:11.101 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:11.101 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:11.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.102 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:11.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:11.102 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:11.103 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:11.103 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:11.103 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:11.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:11.104 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:11.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:11.104 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:11.104 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:11.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:11.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:11.105 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:11.106 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:11.106 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:11.106 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:11.106 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:11.107 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:11.107 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:11.107 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:11.107 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:11.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:11.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:11.108 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:11.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:11.109 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:11.109 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.182833_775.1", 2023-10-12T01:37:11.109 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:11.109 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:11.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:11.110 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.182833Z", 2023-10-12T01:37:11.110 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:11.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:11.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:11.111 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:11.111 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:11Z" 2023-10-12T01:37:11.115 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:11.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.182833_775.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.182833Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:11Z" } 2023-10-12T01:37:11.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.182833_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.182833Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:11.117 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:11.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.182833_775.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:11.118 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074553.924941_787.1 target=1_1697074553.182833_775.1 2023-10-12T01:37:16.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:16.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:16.210+0000 7ff0ad244780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:16.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:16.211+0000 7ff0ad244780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:16.225 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:16.227 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.227 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:16.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.229 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:16.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.229 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:16.229 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.230 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:16.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.230 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.231 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:16.231 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.231 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.231 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:16.232 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.232 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:16.233 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.233 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:37:16.233 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.234 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.234 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.234 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.924941Z", 2023-10-12T01:37:16.234 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.235 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:16.235 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.235 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.236 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:16.236 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.236 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.236 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.236 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:16.237 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:16.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:16Z" 2023-10-12T01:37:16.239 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:16.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.924941_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.924941Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:16Z" } 2023-10-12T01:37:16.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.924941_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.924941Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:16.241 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:16.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:16.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:37:16.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.243 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:16.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.346 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:16.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:16.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.404 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.456 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:16.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.457 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:16.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.562 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:16.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.564 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:16.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.661 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:37:16.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:16.662 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:16.710 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.756 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:37:16.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.758 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:16.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:37:16.849 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:37:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:37:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:37:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:16.647858Z" 2023-10-12T01:37:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:37:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.853 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:37:16.854 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:16.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:37:16.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:16.856 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:16.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:16.932+0000 7f671b4e9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:16.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:16.933+0000 7f671b4e9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:16.946 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:37:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.924941Z", 2023-10-12T01:37:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:16Z" 2023-10-12T01:37:16.961 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:16.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.924941_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.924941Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:16Z" } 2023-10-12T01:37:16.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.924941_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.924941Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:16.963 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:16.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:16.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074636.647858_795.1 target=1_1697074553.924941_787.1 2023-10-12T01:37:21.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:21.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:21.967 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:22.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:22.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:22.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:22.056+0000 7f1304010780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:22.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:22.057+0000 7f1304010780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:22.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:22.072 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:22.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:22.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:22.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:22.076 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:22.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:22.079 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.924941Z", 2023-10-12T01:37:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:22.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:22.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:22.082 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:22.083 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:22.084 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:22Z" 2023-10-12T01:37:22.084 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:22.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.924941_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.924941Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:22Z" } 2023-10-12T01:37:22.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.924941_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.924941Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:22.086 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:22.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:22.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074636.647858_795.1 target=1_1697074553.924941_787.1 2023-10-12T01:37:27.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:27.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:27.089 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:27.179+0000 7f6cebfd2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:27.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:27.180+0000 7f6cebfd2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:27.193 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:27.193 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:27.201 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:27.201 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:37:27.201 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:27.201 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:27.201 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:27.202 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.924941Z", 2023-10-12T01:37:27.202 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:27.202 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:27.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:27.203 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:27.203 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:27.203 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:27.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:27.204 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:27.204 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:27.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:27.204 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:27.204 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:27.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:27.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:27.205 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:27.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.205 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:27.206 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:27.206 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:27.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:27.206 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:27Z" 2023-10-12T01:37:27.206 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:27.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.924941_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.924941Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:27Z" } 2023-10-12T01:37:27.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.924941_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.924941Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:27.208 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:27.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:27.209 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074636.647858_795.1 target=1_1697074553.924941_787.1 2023-10-12T01:37:32.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:32.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:32.211 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:32.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:32.302+0000 7f4187283780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:32.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:32.303+0000 7f4187283780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:32.317 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074553.924941_787.1", 2023-10-12T01:37:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:35:53.924941Z", 2023-10-12T01:37:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:32Z" 2023-10-12T01:37:32.331 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:32.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074553.924941_787.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:35:53.924941Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:32Z" } 2023-10-12T01:37:32.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074553.924941_787.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:35:53.924941Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:32.333 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:32.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074553.924941_787.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:32.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074636.647858_795.1 target=1_1697074553.924941_787.1 2023-10-12T01:37:37.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:37.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:37.336 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:37.395 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.395 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:37.428+0000 7f8a4bb8b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:37.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:37.429+0000 7f8a4bb8b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:37.446 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:37Z" 2023-10-12T01:37:37.461 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:37.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:37Z" } 2023-10-12T01:37:37.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:37.463 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:37.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:37.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:37:37.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.465 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.563 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:37.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.565 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.619 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.619 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.664 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:37.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.666 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.763 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:37:37.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.764 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.812 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.812 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.858 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:37:37.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.860 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:37.949 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:37:37.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:37.951 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:37.999 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.050 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:38.050 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:38.050 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14, 2023-10-12T01:37:38.050 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:37:38.051 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:37:38.051 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:37:38.051 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:38.051 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:37:38.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.052 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:37:38.052 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:37:38.052 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:37:38.052 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:37:38.053 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.053 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:37:38.053 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.053 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:37:38.053 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:37:38.054 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:38.054 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:38.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.054 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:38.055 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:37:38.055 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.055 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:37:38.055 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.055 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:38.056 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:38.056 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:38.056 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:38.056 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:38.057 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:38.057 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:38.057 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:38.057 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:38.057 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:38.058 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.058 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.058 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:37:38.059 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:37:38.059 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.059 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:37:38.059 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.060 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:38.060 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:38.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:38.060 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:38.060 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:38.062 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.062 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.062 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.063 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:37:38.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.063 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:37:38.063 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:37:38.063 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:37:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:37:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.064 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:37:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:37:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:37:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.065 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:37:38.066 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.067 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:37:38.067 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:37:38.067 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:37:38.067 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:37:38.067 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.068 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:37:38.068 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.068 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:37:38.068 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:37:38.068 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:38.069 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:38.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.069 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:37:38.069 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:37:38.069 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.070 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:37:38.070 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.070 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:38.070 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:38.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:38.072 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:37:38.074 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:38.075 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:38.076 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.078 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:37:38.079 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:38.080 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:38.081 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.082 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.083 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:38.084 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:37:38.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.086 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.086 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:37:38.086 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:38.086 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:37:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:37:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:37:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:37:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:37:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:37:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:37:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:37:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:37:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:37:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:37:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:37:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:37:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:37:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:37:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:37:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.095 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:38.096 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-10-12T01:37:38.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:38.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:38.098 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:37:38.268 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:37:38.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:38.269 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:37:38.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:38.270 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:37:38.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:38.271 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:16.647858Z" 2023-10-12T01:37:38.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:38.271 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:37:38.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.272 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:37:38.273 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:38.273 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:37:38.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:38.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:38.275 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:38.357 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:38.470+0000 7f2e9ea0b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:38.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:38.473+0000 7f2e9ea0b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:38.526 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:38.527 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:38.527 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:38.527 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:38.527 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:38.528 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:38.528 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.528 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:38.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:38.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:38.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:38.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:38.535 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:38.535 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:38.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:38.536 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:38.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:38.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:38.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:38Z" 2023-10-12T01:37:38.540 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:38.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:38Z" } 2023-10-12T01:37:38.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:38.542 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:38.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:38.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:37:38.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.544 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.712 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:37:38.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.714 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:38.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.847 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:37:38.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.849 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:38.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:38.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:38.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:37:38.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:38.961 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:39.013 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.014 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.059 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:37:39.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.060 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:39.111 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.111 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.156 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:37:39.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.157 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:39.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.258 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 15, 2023-10-12T01:37:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:37:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:37:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:37:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:37:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:37:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:37:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:37:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:37:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:37:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:37:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:37:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:39.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:37:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.263 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:37:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:39.264 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:39.265 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:37:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:37:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.267 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:37:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:39.268 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:39.269 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:37:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:37:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:37:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:37:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:37:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:37:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:37:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:37:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:37:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:37:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:37:39.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:37:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:37:39.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:37:39.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:37:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:37:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:37:39.277 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:37:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.278 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:37:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:37:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:37:39.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:37:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:37:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:37:39.280 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:37:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:37:39.281 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:37:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:37:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:37:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.282 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:37:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.283 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:37:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:37:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:37:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:37:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:37:39.285 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:37:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:37:39.286 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:37:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:37:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:37:39.287 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:37:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:37:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:39.288 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:39.289 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.290 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:39.291 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:37:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:37:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:37:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:37:39.292 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:37:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:37:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:37:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:37:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:37:39.293 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:37:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:37:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:37:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:37:39.294 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:37:39.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:37:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:37:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:37:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:37:39.296 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:37:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:39.297 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:37:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:37:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:37:39.298 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.299 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:37:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:37:39.302 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:37:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:37:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:37:39.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:37:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:37:39.304 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:37:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:37:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:37:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:37:39.305 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:37:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:37:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:37:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:37:39.306 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:37:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:37:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:37:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:37:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:37:39.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:37:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:39.308 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:37:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:39.309 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:39.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:37:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:37:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:37:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:37:39.311 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:37:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:37:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:37:39.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:37:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:39.313 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:39.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:37:39.314 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:37:39.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.491 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:37:39.492 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-66 2023-10-12T01:37:39.492 DEBUG:boto:path=/wafmoa-66/ 2023-10-12T01:37:39.493 DEBUG:boto:auth_path=/wafmoa-66/ 2023-10-12T01:37:39.493 DEBUG:boto:Method: PUT 2023-10-12T01:37:39.494 DEBUG:boto:Path: /wafmoa-66/ 2023-10-12T01:37:39.494 DEBUG:boto:Data: 2023-10-12T01:37:39.495 DEBUG:boto:Headers: {} 2023-10-12T01:37:39.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:37:39.496 DEBUG:boto:Port: 8000 2023-10-12T01:37:39.496 DEBUG:boto:Params: {} 2023-10-12T01:37:39.497 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:37:39.497 DEBUG:boto:Token: None 2023-10-12T01:37:39.498 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-66/ 2023-10-12T01:37:39.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oBkvySX8vY0qq02wTM/RtuAFJuk= 2023-10-12T01:37:39.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oBkvySX8vY0qq02wTM/RtuAFJuk=', 'Content-Length': '0'} 2023-10-12T01:37:39.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ace6dbe41d43fa7-0065274de3-22d1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.536 DEBUG:boto:path=/wafmoa-66/ 2023-10-12T01:37:39.536 DEBUG:boto:auth_path=/wafmoa-66/ 2023-10-12T01:37:39.537 DEBUG:boto:Method: HEAD 2023-10-12T01:37:39.537 DEBUG:boto:Path: /wafmoa-66/ 2023-10-12T01:37:39.538 DEBUG:boto:Data: 2023-10-12T01:37:39.538 DEBUG:boto:Headers: {} 2023-10-12T01:37:39.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:37:39.539 DEBUG:boto:Port: 8000 2023-10-12T01:37:39.539 DEBUG:boto:Params: {} 2023-10-12T01:37:39.540 DEBUG:boto:Token: None 2023-10-12T01:37:39.540 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-66/ 2023-10-12T01:37:39.541 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vA/qsYoVuE1iI4UKhCnqw/VQyfA= 2023-10-12T01:37:39.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vA/qsYoVuE1iI4UKhCnqw/VQyfA=', 'Content-Length': '0'} 2023-10-12T01:37:39.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003173662458a0ea12-0065274de3-22d1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.545 DEBUG:boto:path=/wafmoa-66/obj1 2023-10-12T01:37:39.545 DEBUG:boto:auth_path=/wafmoa-66/obj1 2023-10-12T01:37:39.546 DEBUG:boto:Method: PUT 2023-10-12T01:37:39.546 DEBUG:boto:Path: /wafmoa-66/obj1 2023-10-12T01:37:39.547 DEBUG:boto:Data: 2023-10-12T01:37:39.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:37:39.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:37:39.548 DEBUG:boto:Port: 8000 2023-10-12T01:37:39.548 DEBUG:boto:Params: {} 2023-10-12T01:37:39.549 DEBUG:boto:Token: None 2023-10-12T01:37:39.549 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-66/obj1 2023-10-12T01:37:39.550 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B3T4ab1vO7hhInVOUNiJhKS9RrI= 2023-10-12T01:37:39.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B3T4ab1vO7hhInVOUNiJhKS9RrI='} 2023-10-12T01:37:39.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dda0006d707b9bf8-0065274de3-22d1-a1'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.565 INFO:rgw_multi.tests:create bucket zone=a2 name=wafmoa-67 2023-10-12T01:37:39.565 DEBUG:boto:path=/wafmoa-67/ 2023-10-12T01:37:39.565 DEBUG:boto:auth_path=/wafmoa-67/ 2023-10-12T01:37:39.566 DEBUG:boto:Method: PUT 2023-10-12T01:37:39.566 DEBUG:boto:Path: /wafmoa-67/ 2023-10-12T01:37:39.567 DEBUG:boto:Data: 2023-10-12T01:37:39.567 DEBUG:boto:Headers: {} 2023-10-12T01:37:39.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:37:39.568 DEBUG:boto:Port: 8001 2023-10-12T01:37:39.568 DEBUG:boto:Params: {} 2023-10-12T01:37:39.569 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:37:39.569 DEBUG:boto:Token: None 2023-10-12T01:37:39.570 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-67/ 2023-10-12T01:37:39.570 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:kIA27UUySqeYaTZc8SS0wbqP3DM= 2023-10-12T01:37:39.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kIA27UUySqeYaTZc8SS0wbqP3DM=', 'Content-Length': '0'} 2023-10-12T01:37:39.662 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7847b9d9790c194-0065274de3-22dd-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.662 DEBUG:boto:path=/wafmoa-67/ 2023-10-12T01:37:39.663 DEBUG:boto:auth_path=/wafmoa-67/ 2023-10-12T01:37:39.663 DEBUG:boto:Method: HEAD 2023-10-12T01:37:39.664 DEBUG:boto:Path: /wafmoa-67/ 2023-10-12T01:37:39.664 DEBUG:boto:Data: 2023-10-12T01:37:39.664 DEBUG:boto:Headers: {} 2023-10-12T01:37:39.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:37:39.665 DEBUG:boto:Port: 8001 2023-10-12T01:37:39.666 DEBUG:boto:Params: {} 2023-10-12T01:37:39.666 DEBUG:boto:Token: None 2023-10-12T01:37:39.667 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-67/ 2023-10-12T01:37:39.667 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XCKQfdgInRHaVGLHT4I3ESADsCk= 2023-10-12T01:37:39.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XCKQfdgInRHaVGLHT4I3ESADsCk=', 'Content-Length': '0'} 2023-10-12T01:37:39.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a70371b30a302f3e-0065274de3-22dd-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.672 DEBUG:boto:path=/wafmoa-67/obj2 2023-10-12T01:37:39.673 DEBUG:boto:auth_path=/wafmoa-67/obj2 2023-10-12T01:37:39.673 DEBUG:boto:Method: PUT 2023-10-12T01:37:39.673 DEBUG:boto:Path: /wafmoa-67/obj2 2023-10-12T01:37:39.674 DEBUG:boto:Data: 2023-10-12T01:37:39.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:37:39.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:37:39.675 DEBUG:boto:Port: 8001 2023-10-12T01:37:39.675 DEBUG:boto:Params: {} 2023-10-12T01:37:39.676 DEBUG:boto:Token: None 2023-10-12T01:37:39.676 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:37:39 GMT /wafmoa-67/obj2 2023-10-12T01:37:39.677 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2zAgsalRq5Cq+WBWnI/8xplp1hE= 2023-10-12T01:37:39.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:37:39 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2zAgsalRq5Cq+WBWnI/8xplp1hE='} 2023-10-12T01:37:39.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f3f0fa7ed3f38ea-0065274de3-22dd-a2'), ('Date', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:37:39.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:39.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:39.699 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:39.777 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.777 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.844 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:37:39.865 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:37:39.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.866 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:39.866 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:37:39.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.866 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:39.867 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:37:39.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.867 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074659.637690_839.1", 2023-10-12T01:37:39.868 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:39.637690Z" 2023-10-12T01:37:39.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:39.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:39.868 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:39.868 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:37:39.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:39.869 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:37:39.870 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:39.870 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:37:39.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:39.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:39.872 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:39.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:39.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:39.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:39.986+0000 7f8d632df780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:39.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:39.988+0000 7f8d632df780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:40.006 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:40.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:40.014 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:40.015 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:40.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:40.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.019 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:40.020 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:39Z" 2023-10-12T01:37:40.020 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:40.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:39Z" } 2023-10-12T01:37:40.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:40.023 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:40.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:40.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:37:45.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:45.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:45.026 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:45.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:45.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:45.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:45.114+0000 7fcf5c8d4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:45.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:45.115+0000 7fcf5c8d4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:45.130 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:45.130 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:45.131 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:45.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.132 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:45.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:45.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:45.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:45.133 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:45.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:45.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:45.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:45.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:45.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:45.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:45.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:45.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:45.135 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:45.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:45.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:45.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:45.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:45.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:45.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:45.136 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:45.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:45.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:45.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:45.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:45.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:45.138 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:45.138 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:45.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:45.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:45.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:45.141 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:45.141 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:45.141 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:45.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:45.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:45.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:45.142 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:45.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.142 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:45.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:45.143 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:45.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:45.143 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:45Z" 2023-10-12T01:37:45.144 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:45.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:45Z" } 2023-10-12T01:37:45.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:45.146 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:45.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:45.146 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:37:50.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:50.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:50.149 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:50.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:50.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:50.237+0000 7f4105e3c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:50.238+0000 7f4105e3c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:50.253 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:50.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:50.255 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:50.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:50.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:50.258 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:50.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:50.260 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:50.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:50.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:50.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:50.266 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:50Z" 2023-10-12T01:37:50.267 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:50.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:50Z" } 2023-10-12T01:37:50.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:50.269 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:50.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:50.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:37:55.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:55.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:37:55.272 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:37:55.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:37:55.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:37:55.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:55.361+0000 7f163fe40780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:55.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:37:55.362+0000 7f163fe40780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:37:55.376 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:37:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:37:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:37:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:37:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:37:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:37:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:37:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:37:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:37:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:37:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:37:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:37:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:37:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:37:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:37:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:37:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:37:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:37:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:37:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:37:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:37:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:37:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:37:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:37:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:37:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:37:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:37:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:37:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:37:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:37:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:37:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:37:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:37:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:37:55Z" 2023-10-12T01:37:55.390 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:37:55.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:37:55Z" } 2023-10-12T01:37:55.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:37:55.392 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:55.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:37:55.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:00.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:00.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:00.395 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:00.452 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:00.452 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:00.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:00.483+0000 7f0003049780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:00.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:00.484+0000 7f0003049780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:00.498 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:00.498 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:00.499 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:00.499 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:00.499 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:00.499 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:00.500 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:00.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.500 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:00.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:00.500 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:00.501 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:00.501 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:00.501 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:00.501 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:00.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:00.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:00.502 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:00.502 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:00.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:00.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:00.503 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:00.503 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:00.503 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:00.504 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:00.504 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:00.504 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:00.504 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:00.504 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:00.505 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:00.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:00.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:00.506 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:00.507 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:00.507 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:00.507 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:00.507 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:00.507 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:00.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:00.508 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:00.508 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:00.508 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:00.509 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:00.509 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:00.509 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:00.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:00.510 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:00.510 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:00.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:00.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:00.510 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:00.512 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:00Z" 2023-10-12T01:38:00.512 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:00.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:00Z" } 2023-10-12T01:38:00.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:00.514 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:00.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:00.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:05.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:05.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:05.517 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:05.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:05.606+0000 7f827b8ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:05.607+0000 7f827b8ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:05.620 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:05.620 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:05.621 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:05.621 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:05.621 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:05.621 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:05Z" 2023-10-12T01:38:05.634 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:05.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:05Z" } 2023-10-12T01:38:05.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:05.636 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:05.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:05.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:10.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:10.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:10.639 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:10.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:10.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:10.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:10.725+0000 7f9ed5a52780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:10.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:10.726+0000 7f9ed5a52780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:10.739 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:10.739 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:10.739 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:10.740 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:10.740 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:10.740 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:10.740 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:10.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.741 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:10.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:10.741 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:10.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:10.742 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:10.742 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:10.742 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:10.742 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:10.743 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:10.743 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:10.743 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:10.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:10.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:10.744 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:10.744 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:10.744 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:10.745 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:10.745 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:10.745 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:10.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:10.745 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:10.746 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:10.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:10.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:10.747 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:10.747 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:10.747 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:10.747 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:10.747 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:10.748 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:10.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:10.748 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:10.748 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:10.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:10.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:10.749 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:10.749 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:10.749 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:10.750 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:10.750 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:10.750 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:10.750 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:10.751 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:10.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:10.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:10.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:10.751 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:10.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.752 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:10.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:10.752 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:10.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:10.753 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:10Z" 2023-10-12T01:38:10.753 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:10.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:10Z" } 2023-10-12T01:38:10.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:10.757 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:10.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:10.758 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:15.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:15.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:15.761 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:15.818 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:15.819 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:15.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:15.850+0000 7fb7d1dd9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:15.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:15.851+0000 7fb7d1dd9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:15.865 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:15.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:15.865 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:15.866 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:15.866 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:15.866 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:15.866 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:15.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.867 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:15.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:15.867 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:15.867 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:15.868 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:15.868 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:15.868 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:15.868 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:15.868 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:15.869 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:15.869 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:15.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:15.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:15.870 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:15.870 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:15.870 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:15.870 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:15.871 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:15.871 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:15.871 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:15.871 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:15.871 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:15.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:15.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:15.872 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:15.872 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:15.873 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:15.873 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:15.873 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:15.873 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:15.873 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:15.874 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:15.874 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:15.874 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:15.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:15.875 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:15.875 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:15.875 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:15.875 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:15.875 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:15.876 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:15.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:15.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:15.876 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:15.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:15.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:15.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:15.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.877 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:15.878 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:15.878 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:15.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:15.878 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:15Z" 2023-10-12T01:38:15.878 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:15.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:15Z" } 2023-10-12T01:38:15.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:15.880 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:15.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:15.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:20.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:20.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:20.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:20.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:20.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:20.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:20.975+0000 7f89b4b7f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:20.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:20.976+0000 7f89b4b7f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:20.990 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:20.990 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:20.990 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:20.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:20.991 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:20.991 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:20.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:20.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:20.992 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:20.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:20.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:20.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:20.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:20.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:20.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:20.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:20.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:20.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:20.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:20.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:20.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:20.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:20.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:20.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:20.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:20.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:20.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:20.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:20.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:20.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:20.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:20.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:20.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:20.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:20.997 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:20.997 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:20.998 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:20.998 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:20.998 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:20.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:20.999 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:20.999 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:20.999 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:20.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:20.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:21.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:21.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:21.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:21.000 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:21.000 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:21.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:21.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:21.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:21.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:21.003 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:21.003 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:21.003 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:21.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:21.003 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:20Z" 2023-10-12T01:38:21.004 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:21.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:20Z" } 2023-10-12T01:38:21.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:21.006 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:21.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:21.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:26.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:26.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:26.009 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:26.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:26.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:26.097+0000 7fcba73c5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:26.098+0000 7fcba73c5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:26.113 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:26.113 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:26.114 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:26.114 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:26.114 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:26.114 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:26.115 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:26.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.115 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:26.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:26.115 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:26.116 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:26.116 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:26.116 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:26.116 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:26.116 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:26.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:26.117 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:26.117 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:26.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:26.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:26.118 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:26.118 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:26.118 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:26.119 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:26.119 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:26.119 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:26.119 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:26.119 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:26.120 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:26.120 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:26.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:26.120 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:26.121 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:26.121 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:26.121 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:26.121 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:26.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:26.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:26.122 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:26.122 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:26.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:26.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:26.123 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:26.123 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:26.123 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:26.124 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:26.124 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:26.124 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:26.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:26.124 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:26.125 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:26.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:26.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:26.125 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:26.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.126 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:26.126 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:26.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:26.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:26.127 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:26Z" 2023-10-12T01:38:26.127 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:26.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:26Z" } 2023-10-12T01:38:26.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:26.129 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:26.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:26.130 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:31.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:31.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:31.132 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:31.189 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:31.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:31.220+0000 7f9416686780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:31.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:31.221+0000 7f9416686780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:31.236 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:31Z" 2023-10-12T01:38:31.249 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:31.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:31Z" } 2023-10-12T01:38:31.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:31.251 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:31.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:31.252 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:36.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:36.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:36.255 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:36.311 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:36.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:36.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:36.342+0000 7f6757aa8780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:36.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:36.343+0000 7f6757aa8780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:36.357 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:36.357 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:36.357 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:36.358 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:36.358 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:36.358 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:36.358 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:36.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.359 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:36.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:36.359 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:36.359 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:36.360 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:36.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:36.360 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:36.360 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:36.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:36.362 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:36.362 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:36.362 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:36.362 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:36.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:36.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:36.364 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:36.364 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:36.365 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:36.365 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:36.365 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:36.365 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:36.366 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:36.366 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:36.366 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:36.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:36.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:36.367 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:36.367 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:36.367 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:36.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:36.368 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:36.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:36.368 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:36.368 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:36.369 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:36.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:36.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:36.369 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:36.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:36.370 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:36Z" 2023-10-12T01:38:36.371 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:36.372 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:36Z" } 2023-10-12T01:38:36.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:36.373 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:36.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:36.374 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:41.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:41.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:41.376 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:41.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:41.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:41.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:41.466+0000 7f4445d59780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:41.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:41.467+0000 7f4445d59780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:41.481 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:41.481 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:41.481 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:41.482 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:41.482 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:41.482 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:41.482 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:41.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.483 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:41.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:41.483 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:41.483 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:41.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:41.485 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:41.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:41.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:41.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:41.486 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:41.486 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:41.486 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:41.486 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:41.487 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:41.487 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:41.487 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:41.487 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:41.489 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:41.489 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:41.489 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:41.489 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:41.489 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:41.490 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:41.490 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:41.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:41.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:41Z" 2023-10-12T01:38:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:41.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:41Z" } 2023-10-12T01:38:41.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:41.496 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:41.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:41.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:46.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:46.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:46.499 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:46.558 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:46.558 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:46.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:46.592+0000 7f6dc3c62780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:46.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:46.593+0000 7f6dc3c62780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:46.606 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:46.607 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:46.607 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:46.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:46.608 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:46.608 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:46.608 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:46.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.608 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:46.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:46.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:46.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:46.609 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:46.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:46.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:46.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:46.610 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:46.610 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:46.611 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:46.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:46.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:46.612 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:46.613 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:46.614 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:46.615 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:46.616 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:46.616 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:46.616 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:46.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:46.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:46.617 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:46.617 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:46.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:46.618 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:46.618 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:46.618 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:46.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:46.618 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:46.619 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:46.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:46.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:46.619 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:46.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.620 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:46.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:46.620 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:46.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:46.621 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:46Z" 2023-10-12T01:38:46.621 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:46.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:46Z" } 2023-10-12T01:38:46.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:46.623 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:46.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:46.624 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:51.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:51.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:51.626 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:51.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:51.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:51.714+0000 7f811ad49780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:51.715+0000 7f811ad49780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:51.729 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:51.729 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:51.730 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:51.730 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:51.730 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:51.730 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:51.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:51.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.731 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:51.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:51.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:51.732 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:51.732 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:51.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:51.732 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:51.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:51.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:51.733 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:51.733 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:51.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:51.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:51.734 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:51.734 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:51.734 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:51.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:51.736 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:51.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:51.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:51.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:51.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:51.737 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:51.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:51.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:51.738 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:51.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:51.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:51.738 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:51.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:51.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:51.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:51.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:51.739 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:51.740 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:51.740 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:51.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:51.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:51.740 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:51.741 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:51.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:51.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:51.741 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:51.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.742 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:51.742 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:51.742 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:51.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:51.743 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:51Z" 2023-10-12T01:38:51.743 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:51.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:51Z" } 2023-10-12T01:38:51.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:51.745 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:51.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:51.746 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:38:56.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:56.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:38:56.748 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:38:56.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:38:56.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:38:56.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:56.838+0000 7f58b7375780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:56.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:38:56.839+0000 7f58b7375780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:38:56.852 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:38:56.853 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:38:56.853 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:38:56.853 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:38:56.853 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:38:56.853 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:38:56.854 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:56.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.854 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:38:56.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:56.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:38:56.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:56.855 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:56.855 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:38:56.856 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:56.856 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:38:56.856 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:56.856 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:38:56.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:56.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:56.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:56.857 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:38:56.858 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:56.858 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:56.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:38:56.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:56.859 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:56.859 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:56.859 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:38:56.859 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:56.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:56.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:56.860 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:38:56.860 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:56.860 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:56.861 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:38:56.861 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:56.861 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:56.861 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:56.862 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:38:56.862 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:56.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:56.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:38:56.863 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:38:56.863 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:38:56.863 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:38:56.863 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:38:56.863 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:38:56.864 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:38:56.864 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:38:56.864 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:38:56.864 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:38:56.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:56.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:38:56.865 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:38:56.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.866 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:38:56.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:38:56.866 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:38:56.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:38:56.866 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:38:56Z" 2023-10-12T01:38:56.867 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:38:56.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:38:56Z" } 2023-10-12T01:38:56.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:38:56.869 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:56.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:38:56.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:39:01.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:01.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:01.873 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:01.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:01.962+0000 7f4c944f4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:01.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:01.963+0000 7f4c944f4780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:01.977 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:01.977 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:01.980 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:01.980 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:01.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:01.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:01.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:01.981 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:01.981 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:01.981 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:01.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:01.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:01.982 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:01.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:01.982 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:01.983 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:01.983 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:01.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:01.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:01.985 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:01.985 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:01.985 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:01.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:39:01.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:01.986 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:01.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:01.986 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:39:01.986 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:01.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:01.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:01.987 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:01.987 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:01.988 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:01.988 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:01.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:01.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:01.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:01Z" 2023-10-12T01:39:01.991 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:01.992 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:01Z" } 2023-10-12T01:39:01.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:01.993 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:01.993 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:01.994 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:39:06.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:06.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:06.996 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:07.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:07.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:07.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:07.086+0000 7f7b344d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:07.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:07.087+0000 7f7b344d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:07.102 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:07.102 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:07.102 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:07.103 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:07.103 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:07.103 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:07.103 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:07.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.104 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:07.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:07.104 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:07.104 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:07.105 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:07.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:07.105 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:07.105 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:07.105 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:07.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:07.106 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:07.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:07.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:07.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:07.107 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:07.107 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:07.107 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:07.108 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:07.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:07.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:07.108 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:07.108 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:07.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:07.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:07.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:07.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:07.110 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:07.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:39:07.110 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:07.110 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:07.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:07.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:39:07.111 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:07.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:07.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:07.112 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:07.112 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:07.112 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:07.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:07.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:07.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:07.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:07.113 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:07.113 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:07.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:07.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:07.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:07.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.115 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:07.115 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:07.115 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:07.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:07.115 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:07Z" 2023-10-12T01:39:07.116 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:07.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:07Z" } 2023-10-12T01:39:07.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:07.118 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:07.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:07.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:39:12.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:12.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:12.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:12.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:12.178 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:12.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:12.209+0000 7fcb66f0f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:12.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:12.210+0000 7fcb66f0f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:12.225 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:12.225 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:12.225 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:12.226 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:12.226 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:12.226 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:12.227 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:12.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.227 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:12.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:12.227 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:12.228 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:12.228 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:12.228 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:12.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:12.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:12.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:12.229 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:12.229 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:12.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:12.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:12.230 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:12.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:12.230 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:39:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:39:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:12.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:12.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.238 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:12.238 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:12.239 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:12.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:12.239 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:12Z" 2023-10-12T01:39:12.239 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:12.242 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:12Z" } 2023-10-12T01:39:12.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:12.243 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:12.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:12.244 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:39:17.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:17.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:17.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:17.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:17.338+0000 7fac8d8a7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:17.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:17.339+0000 7fac8d8a7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:17.355 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:17.358 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:17.358 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:17.358 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:17.358 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:17.359 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:17.359 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:17.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:17.359 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:17.359 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:17.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:17.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:17.360 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:17.360 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:17.361 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:17.361 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:17.361 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:17.361 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:17.362 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:17.362 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:17.362 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:17.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:17.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:17.363 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:17.363 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:17.363 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:17.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074636.647858_795.1", 2023-10-12T01:39:17.364 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:17.364 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:17.364 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:17.364 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:16.647858Z", 2023-10-12T01:39:17.365 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:17.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:17.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:17.365 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:17.366 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:17.366 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:17.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:17.366 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:17.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:17.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:17.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:17.367 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:17.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:17.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:17.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:17.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.368 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:17.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:17.369 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:17.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:17.369 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:17Z" 2023-10-12T01:39:17.369 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:17.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074636.647858_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:16.647858Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:17Z" } 2023-10-12T01:39:17.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074636.647858_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:16.647858Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:17.371 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:17.372 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074636.647858_795.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:17.372 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074659.637690_839.1 target=1_1697074636.647858_795.1 2023-10-12T01:39:22.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.375 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:22.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:22.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:22.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:22.466+0000 7f68b29cf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:22.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:22.467+0000 7f68b29cf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:22.481 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:22.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.485 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:22.485 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:22.486 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.486 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:22.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.486 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:22.487 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:22.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.487 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:22.488 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.488 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:22.488 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:22.488 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.488 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:22.489 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.489 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:22.489 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:22.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.490 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:22.490 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.490 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:22.491 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074659.637690_839.1", 2023-10-12T01:39:22.491 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.491 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:22.491 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.491 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.637690Z", 2023-10-12T01:39:22.492 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:22.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.492 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:22.493 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.493 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:22.493 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:22.493 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:22.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.494 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:22.494 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:22.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.495 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:22.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.495 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:22.495 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:22.495 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:22.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.496 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:22Z" 2023-10-12T01:39:22.496 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:22.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074659.637690_839.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:39.637690Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:22Z" } 2023-10-12T01:39:22.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074659.637690_839.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.637690Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:22.498 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:22.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:22.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:39:22.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.500 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:22.599 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:39:22.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:22.599 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:22.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:39:22.600 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:39:22.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:22.601 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:22.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:39:22.602 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:39:22.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.603 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:39:22.604 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:22.604 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:39:22.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:22.605 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:22.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:22.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:22.700 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:22.700 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:22.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:22.703 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:22.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.705 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050828", 2023-10-12T01:39:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:22.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.900060Z" 2023-10-12T01:39:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:22.708 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:22.710 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045518", 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.860208Z" 2023-10-12T01:39:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:22.712 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:22.713 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:22Z" 2023-10-12T01:39:22.713 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:22.715 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050828", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:30:54.900060Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045518", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.860208Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:22Z" } 2023-10-12T01:39:22.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050828', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.900060Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045518', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.860208Z'}}] 2023-10-12T01:39:22.715 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:22.716 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050828', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045518'} 2023-10-12T01:39:22.716 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050970 target=00000000000000000000:00000000000000050828, shard=3 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045518 2023-10-12T01:39:27.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:27.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:27.719 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:27.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:27.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:27.829 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:27.830 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:27.830 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:27.830 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:27.831 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:27.831 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:27.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.832 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:27.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:27.833 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:27.833 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:27.834 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:27.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:27.835 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:27.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:27.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:27.836 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:27.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:27.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:27.837 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:27.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:27.838 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:27.839 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050828", 2023-10-12T01:39:27.839 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:27.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:27.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:27.841 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.900060Z" 2023-10-12T01:39:27.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:27.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:27.842 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:27.842 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:27.842 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:27.843 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:27.843 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:27.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:27.843 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:27.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:27.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:27.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:27.844 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:27.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:27.845 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:27.845 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045518", 2023-10-12T01:39:27.845 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:27.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:27.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:27.846 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.860208Z" 2023-10-12T01:39:27.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:27.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:27.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:27.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.847 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:27.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:27.847 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:27.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:27.848 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:27Z" 2023-10-12T01:39:27.848 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:27.849 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050828", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:30:54.900060Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045518", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.860208Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:27Z" } 2023-10-12T01:39:27.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050828', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.900060Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045518', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.860208Z'}}] 2023-10-12T01:39:27.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:27.851 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050828', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045518'} 2023-10-12T01:39:27.851 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050970 target=00000000000000000000:00000000000000050828, shard=3 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045518 2023-10-12T01:39:32.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:32.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:32.854 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:32.913 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:32.913 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:32.961 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:32.961 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:32.961 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:32.962 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:32.962 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:32.962 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:32.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.963 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:32.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:32.963 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:32.963 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:32.963 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:32.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:32.964 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:32.964 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:32.964 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:32.964 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:32.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:32.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:32.965 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:32.966 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:32.966 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:32.966 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050828", 2023-10-12T01:39:32.966 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:32.966 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:32.967 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:32.967 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.900060Z" 2023-10-12T01:39:32.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:32.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:32.968 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:32.968 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:32.968 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:32.968 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:32.969 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:32.969 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:32.969 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:32.969 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:32.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:32.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:32.970 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:32.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:32.971 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:32.971 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045518", 2023-10-12T01:39:32.971 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:32.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:32.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:32.972 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.860208Z" 2023-10-12T01:39:32.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:32.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:32.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:32.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.973 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:32.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:32.973 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:32.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:32.974 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:32Z" 2023-10-12T01:39:32.974 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:32.975 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050828", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:30:54.900060Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045518", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.860208Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:32Z" } 2023-10-12T01:39:32.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050828', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.900060Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045518', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.860208Z'}}] 2023-10-12T01:39:32.977 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:32.977 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050828', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045518'} 2023-10-12T01:39:32.977 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050970 target=00000000000000000000:00000000000000050828, shard=3 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045518 2023-10-12T01:39:37.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:37.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:37.980 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:38.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:38.040 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:38.088 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:38.088 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:38.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:38.090 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:38.091 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050828", 2023-10-12T01:39:38.093 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.900060Z" 2023-10-12T01:39:38.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:38.095 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:38.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:38.096 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:38.096 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:38.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:38.096 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:38.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:38.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:38.097 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:38.098 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:38.098 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:38.098 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045518", 2023-10-12T01:39:38.098 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:38.098 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:38.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:38.099 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.860208Z" 2023-10-12T01:39:38.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:38.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:38.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:38.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.100 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:38.100 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:38.100 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:38.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:38.101 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:38Z" 2023-10-12T01:39:38.101 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:38.102 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050828", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:30:54.900060Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045518", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.860208Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:38Z" } 2023-10-12T01:39:38.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050828', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.900060Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045518', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.860208Z'}}] 2023-10-12T01:39:38.103 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:38.104 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050828', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045518'} 2023-10-12T01:39:38.104 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000050970 target=00000000000000000000:00000000000000050828, shard=3 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045518 2023-10-12T01:39:43.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.106 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:43.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.214 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:43.215 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:43.215 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:43.215 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:43.215 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:43.216 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:43.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.216 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:43.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.216 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:43.217 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:43.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:43.218 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.218 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:43.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.218 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:43.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.219 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:43.220 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.220 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:43.220 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:39:43.220 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:43.221 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.221 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:39:43.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.222 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:43.222 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.223 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:43.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:43.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.223 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:43.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:43.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:43.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.225 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:43.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:39:43.226 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.226 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:43.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.226 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:39:43.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.227 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:43.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.228 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:43.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:43.229 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:43.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.229 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:43Z" 2023-10-12T01:39:43.229 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:43.230 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:43Z" } 2023-10-12T01:39:43.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:39:43.232 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:43.232 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:43.232 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:39:43.233 DEBUG:boto:path=/wafmoa-66/ 2023-10-12T01:39:43.233 DEBUG:boto:auth_path=/wafmoa-66/ 2023-10-12T01:39:43.234 DEBUG:boto:Method: HEAD 2023-10-12T01:39:43.234 DEBUG:boto:Path: /wafmoa-66/ 2023-10-12T01:39:43.235 DEBUG:boto:Data: 2023-10-12T01:39:43.235 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:39:43.236 DEBUG:boto:Port: 8001 2023-10-12T01:39:43.236 DEBUG:boto:Params: {} 2023-10-12T01:39:43.237 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:39:43.237 DEBUG:boto:Token: None 2023-10-12T01:39:43.238 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-66/ 2023-10-12T01:39:43.239 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mujv+/raxY6lYJi7vFBbz38DdqA= 2023-10-12T01:39:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mujv+/raxY6lYJi7vFBbz38DdqA=', 'Content-Length': '0'} 2023-10-12T01:39:43.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6805a834f6dead7-0065274e5f-22dd-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.245 DEBUG:boto:path=/wafmoa-66/obj1 2023-10-12T01:39:43.245 DEBUG:boto:auth_path=/wafmoa-66/obj1 2023-10-12T01:39:43.245 DEBUG:boto:Method: HEAD 2023-10-12T01:39:43.246 DEBUG:boto:Path: /wafmoa-66/obj1 2023-10-12T01:39:43.246 DEBUG:boto:Data: 2023-10-12T01:39:43.247 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:39:43.248 DEBUG:boto:Port: 8001 2023-10-12T01:39:43.248 DEBUG:boto:Params: {} 2023-10-12T01:39:43.248 DEBUG:boto:Token: None 2023-10-12T01:39:43.249 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-66/obj1 2023-10-12T01:39:43.249 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:+DFYFnOCKlXF+7rdSH7b6x+cDNQ= 2023-10-12T01:39:43.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+DFYFnOCKlXF+7rdSH7b6x+cDNQ=', 'Content-Length': '0'} 2023-10-12T01:39:43.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-66:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000036d2e0edc2100da-0065274e5f-22dd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.253 DEBUG:boto:path=/wafmoa-66/obj1 2023-10-12T01:39:43.253 DEBUG:boto:auth_path=/wafmoa-66/obj1 2023-10-12T01:39:43.253 DEBUG:boto:Method: GET 2023-10-12T01:39:43.254 DEBUG:boto:Path: /wafmoa-66/obj1 2023-10-12T01:39:43.254 DEBUG:boto:Data: 2023-10-12T01:39:43.255 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:39:43.256 DEBUG:boto:Port: 8001 2023-10-12T01:39:43.256 DEBUG:boto:Params: {} 2023-10-12T01:39:43.257 DEBUG:boto:Token: None 2023-10-12T01:39:43.257 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-66/obj1 2023-10-12T01:39:43.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:XMLLXvbjpH27OqiCuk2besktf3Q= 2023-10-12T01:39:43.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:XMLLXvbjpH27OqiCuk2besktf3Q=', 'Content-Length': '0'} 2023-10-12T01:39:43.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-66:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f88c16b98849c67-0065274e5f-22dd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.261 DEBUG:boto:path=/wafmoa-67/ 2023-10-12T01:39:43.261 DEBUG:boto:auth_path=/wafmoa-67/ 2023-10-12T01:39:43.262 DEBUG:boto:Method: HEAD 2023-10-12T01:39:43.262 DEBUG:boto:Path: /wafmoa-67/ 2023-10-12T01:39:43.263 DEBUG:boto:Data: 2023-10-12T01:39:43.263 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:43.264 DEBUG:boto:Port: 8000 2023-10-12T01:39:43.264 DEBUG:boto:Params: {} 2023-10-12T01:39:43.265 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:39:43.265 DEBUG:boto:Token: None 2023-10-12T01:39:43.266 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-67/ 2023-10-12T01:39:43.266 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:TZQEd0H317QJkiYGUIO8wNXJyYE= 2023-10-12T01:39:43.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:TZQEd0H317QJkiYGUIO8wNXJyYE=', 'Content-Length': '0'} 2023-10-12T01:39:43.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af9e84dde1d2c380-0065274e5f-22d1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.271 DEBUG:boto:path=/wafmoa-67/obj2 2023-10-12T01:39:43.272 DEBUG:boto:auth_path=/wafmoa-67/obj2 2023-10-12T01:39:43.272 DEBUG:boto:Method: HEAD 2023-10-12T01:39:43.273 DEBUG:boto:Path: /wafmoa-67/obj2 2023-10-12T01:39:43.273 DEBUG:boto:Data: 2023-10-12T01:39:43.273 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:43.274 DEBUG:boto:Port: 8000 2023-10-12T01:39:43.275 DEBUG:boto:Params: {} 2023-10-12T01:39:43.275 DEBUG:boto:Token: None 2023-10-12T01:39:43.276 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-67/obj2 2023-10-12T01:39:43.276 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LrnV/BbOaow5Ye4IAvG6tvDrcSY= 2023-10-12T01:39:43.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LrnV/BbOaow5Ye4IAvG6tvDrcSY=', 'Content-Length': '0'} 2023-10-12T01:39:43.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-67:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7edeaca09ee99f4-0065274e5f-22d1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.279 DEBUG:boto:path=/wafmoa-67/obj2 2023-10-12T01:39:43.280 DEBUG:boto:auth_path=/wafmoa-67/obj2 2023-10-12T01:39:43.280 DEBUG:boto:Method: GET 2023-10-12T01:39:43.281 DEBUG:boto:Path: /wafmoa-67/obj2 2023-10-12T01:39:43.281 DEBUG:boto:Data: 2023-10-12T01:39:43.282 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:43.282 DEBUG:boto:Port: 8000 2023-10-12T01:39:43.283 DEBUG:boto:Params: {} 2023-10-12T01:39:43.283 DEBUG:boto:Token: None 2023-10-12T01:39:43.284 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-67/obj2 2023-10-12T01:39:43.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:s4ByDkywv7P9CUyBK6yTxUvBiIw= 2023-10-12T01:39:43.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:s4ByDkywv7P9CUyBK6yTxUvBiIw=', 'Content-Length': '0'} 2023-10-12T01:39:43.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:37:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-67:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006805ff9ea38bb242-0065274e5f-22d1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.289 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.385 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:39:43.386 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-10-12T01:39:43.387 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-12T01:39:43.388 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-12T01:39:43.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.389 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:43.439 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.439 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.470 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:39:43.484 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:39:43.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.485 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:43.485 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:39:43.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.486 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:43.486 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:39:43.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.486 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074659.637690_839.1", 2023-10-12T01:39:43.487 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:37:39.637690Z" 2023-10-12T01:39:43.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.487 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:43.488 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:39:43.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.488 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:39:43.489 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:43.490 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:39:43.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.491 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:43.540 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:43.569+0000 7fbc049a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:43.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:43.569+0000 7fbc049a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:43.583 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:43.587 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.587 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:43.587 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:43.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.588 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:43.588 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.588 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:43.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:43.589 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:43.589 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.589 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:43.590 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:43.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.590 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:43.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.591 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:43.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074659.637690_839.1", 2023-10-12T01:39:43.591 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:43.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.637690Z", 2023-10-12T01:39:43.592 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:43.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:43.593 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:43.593 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:43.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:43.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:43.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:43.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:43.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:43.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:43.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.595 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.597 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:43Z" 2023-10-12T01:39:43.597 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:43.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074659.637690_839.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:39.637690Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:43Z" } 2023-10-12T01:39:43.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074659.637690_839.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.637690Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:43.599 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:43.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074659.637690_839.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:43.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:39:43.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:43.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.691 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:39:43.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:43.693 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:43.738 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.738 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.782 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:39:43.783 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-68 2023-10-12T01:39:43.783 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:39:43.784 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:39:43.784 DEBUG:boto:Method: PUT 2023-10-12T01:39:43.785 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:39:43.785 DEBUG:boto:Data: 2023-10-12T01:39:43.786 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:43.787 DEBUG:boto:Port: 8000 2023-10-12T01:39:43.787 DEBUG:boto:Params: {} 2023-10-12T01:39:43.788 DEBUG:boto:Token: None 2023-10-12T01:39:43.788 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-68/ 2023-10-12T01:39:43.789 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LLwVAPX4VskL5S1dhekI7juZoyk= 2023-10-12T01:39:43.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LLwVAPX4VskL5S1dhekI7juZoyk=', 'Content-Length': '0'} 2023-10-12T01:39:43.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e568b336e3fc6eb3-0065274e5f-22d1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.810 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-69 2023-10-12T01:39:43.811 DEBUG:boto:path=/wafmoa-69/ 2023-10-12T01:39:43.812 DEBUG:boto:auth_path=/wafmoa-69/ 2023-10-12T01:39:43.812 DEBUG:boto:Method: PUT 2023-10-12T01:39:43.813 DEBUG:boto:Path: /wafmoa-69/ 2023-10-12T01:39:43.813 DEBUG:boto:Data: 2023-10-12T01:39:43.814 DEBUG:boto:Headers: {} 2023-10-12T01:39:43.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:43.815 DEBUG:boto:Port: 8000 2023-10-12T01:39:43.815 DEBUG:boto:Params: {} 2023-10-12T01:39:43.816 DEBUG:boto:Token: None 2023-10-12T01:39:43.816 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:39:43 GMT /wafmoa-69/ 2023-10-12T01:39:43.817 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eJxMsJyppE8m0lth/hgSX3O4wlQ= 2023-10-12T01:39:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:43 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eJxMsJyppE8m0lth/hgSX3O4wlQ=', 'Content-Length': '0'} 2023-10-12T01:39:43.832 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f49b9b16d55b66b0-0065274e5f-22d1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:43.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.834 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:43.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:43.917 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:39:43.930 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:39:43.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.931 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:43.931 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:39:43.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.932 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:43.932 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:39:43.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:39:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:43.831059Z" 2023-10-12T01:39:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:39:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:43.934 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:39:43.935 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:43.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:39:43.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:43.937 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:43.986 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:43.987 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:44.014+0000 7f67261b9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:44.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:44.015+0000 7f67261b9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:44.029 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:44.029 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:44.030 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:44.030 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:44.030 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:44.030 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:44.031 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.031 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:44.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.032 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:44.032 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:44.032 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:44.034 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.035 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:44.035 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:44.035 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:44.035 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:44.036 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:44.036 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:44.036 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:44.036 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:44.036 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.037 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:44.038 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:44.038 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:44.038 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:39:44.038 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:44.038 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:44.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:44.039 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:39:44.039 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.040 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:44.040 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:44.040 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:44.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:44.041 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:44.041 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:44.041 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:44.042 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:44.042 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.043 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:44.043 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:44.044 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:44.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.044 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:44Z" 2023-10-12T01:39:44.044 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:44.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:44Z" } 2023-10-12T01:39:44.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:44.048 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:44.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:44.049 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:39:44.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.145 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:39:44.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.146 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.244 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:39:44.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=wafmoa-68', '--dest-bucket=wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=wafmoa-68', '--dest-bucket=wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=wafmoa-68 --dest-bucket=wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.301 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.301 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-68","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:39:44.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.397 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.398 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.444 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-68","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:39:44.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.445 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.491 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.491 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.535 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-68","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:39:44.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.537 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.638 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:44.638 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:44.638 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 16, 2023-10-12T01:39:44.639 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:39:44.639 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:39:44.639 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:39:44.639 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:44.639 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:39:44.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.640 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:39:44.640 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:39:44.640 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:39:44.640 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:39:44.641 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.641 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:39:44.641 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.641 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:39:44.641 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:39:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:39:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:39:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:39:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.644 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:39:44.644 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:39:44.644 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:39:44.644 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:39:44.645 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:39:44.645 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:39:44.645 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:39:44.645 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:39:44.645 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:39:44.646 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:39:44.646 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.647 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:39:44.647 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:39:44.647 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.647 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:39:44.647 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.648 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:39:44.648 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:39:44.648 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:39:44.648 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:39:44.649 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:39:44.649 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:39:44.649 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:39:44.649 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:39:44.649 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:39:44.650 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:39:44.650 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.650 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.650 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.650 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.651 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:39:44.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.651 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:39:44.651 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:39:44.651 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:39:44.652 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:39:44.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.652 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.653 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:39:44.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:39:44.653 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:39:44.653 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:39:44.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.654 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:39:44.654 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:39:44.654 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:39:44.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.655 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:39:44.655 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:39:44.655 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.655 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:39:44.656 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.656 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.656 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.656 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:39:44.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.657 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:39:44.657 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:39:44.657 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-68", 2023-10-12T01:39:44.657 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:39:44.658 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.658 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:39:44.658 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.658 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:39:44.659 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-68", 2023-10-12T01:39:44.659 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:39:44.659 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.659 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:39:44.659 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.660 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:39:44.660 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:39:44.660 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:39:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:39:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:39:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:39:44.662 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:39:44.662 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:39:44.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.662 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.663 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:39:44.663 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.663 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.664 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:39:44.664 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.664 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.665 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:39:44.665 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:39:44.665 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:39:44.665 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:39:44.665 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.666 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:39:44.666 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.666 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:39:44.666 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:39:44.666 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:39:44.667 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:39:44.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.667 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:39:44.667 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:39:44.668 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.668 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:39:44.668 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.668 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:39:44.668 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:39:44.669 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:39:44.669 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:39:44.669 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:39:44.669 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:39:44.669 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:39:44.670 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:39:44.670 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:39:44.670 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:39:44.670 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.670 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.671 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:39:44.671 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:39:44.671 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:39:44.672 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:39:44.672 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.672 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:39:44.672 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:39:44.673 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:39:44.673 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:39:44.673 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:39:44.673 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:39:44.673 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:39:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:39:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:39:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:39:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:39:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:39:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:39:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:39:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:39:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:39:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:39:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:39:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:39:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:39:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:39:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:39:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:39:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:39:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:39:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:39:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:39:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:39:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:39:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:39:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.684 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:44.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.684 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:39:44.684 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:39:44.684 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:39:44.685 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:39:44.685 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:39:44.685 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:39:44.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:39:44.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:39:44.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:39:44.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.686 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:39:44.686 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:39:44.687 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:39:44.687 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:39:44.687 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:39:44.687 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:39:44.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:39:44.689 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:39:44.689 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:39:44.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.689 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:39:44.689 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:39:44.690 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:39:44.690 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:39:44.690 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:39:44.690 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:39:44.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.691 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:39:44.691 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:39:44.691 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:39:44.691 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:39:44.691 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:39:44.692 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:39:44.692 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:44.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:44.692 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:39:44.692 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:39:44.693 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:44.693 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:44.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:39:44.694 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:39:44.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:44.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"wafmoa-68"},"dest":{"zone":"a1","bucket":"wafmoa-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"wafmoa-68"},"dest":{"zone":"a2","bucket":"wafmoa-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-10-12T01:39:44.862 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:39:44.863 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:39:44.863 DEBUG:boto:Method: HEAD 2023-10-12T01:39:44.863 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:39:44.864 DEBUG:boto:Data: 2023-10-12T01:39:44.864 DEBUG:boto:Headers: {} 2023-10-12T01:39:44.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:44.865 DEBUG:boto:Port: 8000 2023-10-12T01:39:44.866 DEBUG:boto:Params: {} 2023-10-12T01:39:44.866 DEBUG:boto:Token: None 2023-10-12T01:39:44.867 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:44 GMT /wafmoa-68/ 2023-10-12T01:39:44.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5AoH78XIx33O4tmfhIsXm57RzQs= 2023-10-12T01:39:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5AoH78XIx33O4tmfhIsXm57RzQs=', 'Content-Length': '0'} 2023-10-12T01:39:45.045 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000052259798fb75089-0065274e61-2435-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.046 DEBUG:boto:path=/wafmoa-68/obj1 2023-10-12T01:39:45.047 DEBUG:boto:auth_path=/wafmoa-68/obj1 2023-10-12T01:39:45.047 DEBUG:boto:Method: PUT 2023-10-12T01:39:45.047 DEBUG:boto:Path: /wafmoa-68/obj1 2023-10-12T01:39:45.048 DEBUG:boto:Data: 2023-10-12T01:39:45.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:39:45.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.049 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.050 DEBUG:boto:Params: {} 2023-10-12T01:39:45.050 DEBUG:boto:Token: None 2023-10-12T01:39:45.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-68/obj1 2023-10-12T01:39:45.051 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oOllkx3S14/iwCaO+ELpHqarPMY= 2023-10-12T01:39:45.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oOllkx3S14/iwCaO+ELpHqarPMY='} 2023-10-12T01:39:45.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbbf59ff30609020-0065274e61-2435-a1'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.073 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:39:45.074 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:39:45.074 DEBUG:boto:Method: HEAD 2023-10-12T01:39:45.075 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:39:45.075 DEBUG:boto:Data: 2023-10-12T01:39:45.075 DEBUG:boto:Headers: {} 2023-10-12T01:39:45.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.076 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.077 DEBUG:boto:Params: {} 2023-10-12T01:39:45.077 DEBUG:boto:Token: None 2023-10-12T01:39:45.077 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-68/ 2023-10-12T01:39:45.078 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YUdKYjcMXJ1ZAcgSjM/Ux3D0guo= 2023-10-12T01:39:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YUdKYjcMXJ1ZAcgSjM/Ux3D0guo=', 'Content-Length': '0'} 2023-10-12T01:39:45.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af7bdca51bb8f7e2-0065274e61-2435-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.085 DEBUG:boto:path=/wafmoa-68/obj2 2023-10-12T01:39:45.085 DEBUG:boto:auth_path=/wafmoa-68/obj2 2023-10-12T01:39:45.086 DEBUG:boto:Method: PUT 2023-10-12T01:39:45.086 DEBUG:boto:Path: /wafmoa-68/obj2 2023-10-12T01:39:45.087 DEBUG:boto:Data: 2023-10-12T01:39:45.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:39:45.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.088 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.088 DEBUG:boto:Params: {} 2023-10-12T01:39:45.089 DEBUG:boto:Token: None 2023-10-12T01:39:45.089 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-68/obj2 2023-10-12T01:39:45.090 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ze9Qu4OOsEB/wjf87DL75jW52DQ= 2023-10-12T01:39:45.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ze9Qu4OOsEB/wjf87DL75jW52DQ='} 2023-10-12T01:39:45.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000492727de2eb971cd-0065274e61-2435-a1'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.135 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:39:45.136 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:39:45.136 DEBUG:boto:Method: HEAD 2023-10-12T01:39:45.136 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:39:45.137 DEBUG:boto:Data: 2023-10-12T01:39:45.137 DEBUG:boto:Headers: {} 2023-10-12T01:39:45.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.138 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.138 DEBUG:boto:Params: {} 2023-10-12T01:39:45.139 DEBUG:boto:Token: None 2023-10-12T01:39:45.139 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-68/ 2023-10-12T01:39:45.140 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YUdKYjcMXJ1ZAcgSjM/Ux3D0guo= 2023-10-12T01:39:45.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YUdKYjcMXJ1ZAcgSjM/Ux3D0guo=', 'Content-Length': '0'} 2023-10-12T01:39:45.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006138ac8d1aafb3b9-0065274e61-2435-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.145 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:39:45.145 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:39:45.146 DEBUG:boto:Method: PUT 2023-10-12T01:39:45.146 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:39:45.147 DEBUG:boto:Data: 2023-10-12T01:39:45.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:39:45.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.148 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.148 DEBUG:boto:Params: {} 2023-10-12T01:39:45.148 DEBUG:boto:Token: None 2023-10-12T01:39:45.149 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-68/obj3 2023-10-12T01:39:45.149 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:PAtjypn7ZfjECz6Q7EELzU9qrYY= 2023-10-12T01:39:45.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:PAtjypn7ZfjECz6Q7EELzU9qrYY='} 2023-10-12T01:39:45.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047e439461148b092-0065274e61-2435-a1'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.165 DEBUG:boto:path=/wafmoa-69/ 2023-10-12T01:39:45.166 DEBUG:boto:auth_path=/wafmoa-69/ 2023-10-12T01:39:45.166 DEBUG:boto:Method: HEAD 2023-10-12T01:39:45.166 DEBUG:boto:Path: /wafmoa-69/ 2023-10-12T01:39:45.167 DEBUG:boto:Data: 2023-10-12T01:39:45.167 DEBUG:boto:Headers: {} 2023-10-12T01:39:45.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.168 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.168 DEBUG:boto:Params: {} 2023-10-12T01:39:45.169 DEBUG:boto:Token: None 2023-10-12T01:39:45.169 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-69/ 2023-10-12T01:39:45.170 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UwlTlSXDdEToOre/GXHq2l5jCYo= 2023-10-12T01:39:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UwlTlSXDdEToOre/GXHq2l5jCYo=', 'Content-Length': '0'} 2023-10-12T01:39:45.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c28ce12453f1d873-0065274e61-2435-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.177 DEBUG:boto:path=/wafmoa-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-12T01:39:45.178 DEBUG:boto:auth_path=/wafmoa-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-12T01:39:45.178 DEBUG:boto:Method: PUT 2023-10-12T01:39:45.179 DEBUG:boto:Path: /wafmoa-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-12T01:39:45.179 DEBUG:boto:Data: 2023-10-12T01:39:45.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:39:45.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:39:45.180 DEBUG:boto:Port: 8000 2023-10-12T01:39:45.181 DEBUG:boto:Params: {} 2023-10-12T01:39:45.181 DEBUG:boto:Token: None 2023-10-12T01:39:45.182 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:39:45 GMT /wafmoa-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-12T01:39:45.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:sIrJbgG+97Ny6W5G7JnaAXW4s1s= 2023-10-12T01:39:45.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:39:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:sIrJbgG+97Ny6W5G7JnaAXW4s1s='} 2023-10-12T01:39:45.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4e2470c73c7ac06-0065274e61-2435-a1'), ('Date', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:39:45.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.228 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:45.300 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:45.301 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:43.831059Z" 2023-10-12T01:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:39:45.405 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:45.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:39:45.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.407 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:45.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:45.468 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:45.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:45.505+0000 7fd022c3f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:39:45.507+0000 7fd022c3f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:39:45.523 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:45.523 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:39:45.524 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:39:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:39:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:39:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:45.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:39:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.529 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:39:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:39:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:39:45.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:45.533 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:39:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:39:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:39:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:39:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:39:45.536 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:39:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.537 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:45Z" 2023-10-12T01:39:45.537 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:45.538 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:39:45Z" } 2023-10-12T01:39:45.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:39:45.539 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:45.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:39:45.540 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:39:45.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.541 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:45.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:45.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:45.641 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:39:45.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.642 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:45.642 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:45.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.643 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-12T01:39:45.643 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.161892Z" 2023-10-12T01:39:45.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:39:45.644 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:39:45.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:39:45.645 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:39:45.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.645 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:39:45.646 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:39:45.647 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:39:45.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:45.648 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:45.699 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:45.699 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:45.743 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:45.744 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:45.744 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:45.744 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:45.744 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:45.745 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:45.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.745 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:45.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.745 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:45.746 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.746 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:45.746 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:45.746 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.747 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:45.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.747 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:45.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.748 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:45.748 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.748 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:45.749 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:39:45.749 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.749 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:45.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.749 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:39:45.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.750 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:45.750 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.751 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:45.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.752 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:45.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:45.753 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:45.753 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:45.753 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:45.753 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:39:45.753 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:45.754 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:45.754 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:45.754 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:39:45.754 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:45.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:45.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.755 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:45.755 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:45.756 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:45.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:45.756 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:45Z" 2023-10-12T01:39:45.756 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:45.757 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:45Z" } 2023-10-12T01:39:45.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:39:45.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:39:45.759 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:45.759 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:39:50.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:50.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:50.762 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:50.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:50.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:50.869 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:50.870 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:50.870 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:50.870 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:50.870 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:50.871 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:50.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.871 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:50.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:50.871 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:50.872 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:50.872 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:50.872 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:50.872 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:50.872 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:50.873 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:50.873 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:50.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:50.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:50.874 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:50.874 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:50.874 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:50.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:39:50.875 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:50.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:50.875 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:50.875 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:39:50.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:50.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:50.876 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:50.876 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:50.876 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:50.877 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:50.877 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:50.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:50.877 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:50.877 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:50.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:50.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:50.878 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:50.878 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:50.879 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:50.879 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:39:50.879 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:50.879 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:50.880 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:50.880 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:39:50.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:50.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:50.880 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:50.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.881 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:50.881 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:50.881 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:50.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:50.882 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:50Z" 2023-10-12T01:39:50.882 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:50.883 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:50Z" } 2023-10-12T01:39:50.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:39:50.884 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:39:50.884 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:50.885 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:39:55.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:55.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:39:55.887 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:39:55.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:39:55.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:39:55.999 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:39:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:39:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:39:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:39:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:39:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:39:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:39:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:39:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:39:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:39:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:39:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:39:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:56.008 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:56.008 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:39:56.008 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:39:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:39:56.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:56.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:56.012 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:39:56.012 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:56.012 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:56.012 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:39:56.013 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:56.013 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:56.013 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:56.013 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:39:56.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:56.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:39:56.014 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:39:56.014 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:39:56.015 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:39:56.015 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:39:56.015 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:39:56.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:39:56.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:39:56.016 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:39:56.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:56.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:39:56.017 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:39:56.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.017 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:39:56.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:39:56.017 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:39:56.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:39:56.018 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:39:55Z" 2023-10-12T01:39:56.018 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:39:56.019 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:39:55Z" } 2023-10-12T01:39:56.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:39:56.020 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:39:56.021 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:39:56.021 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:01.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:01.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:01.023 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:01.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:01.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:01.127 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:01.127 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.128 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:01.129 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:01.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:01.131 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:01.132 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:01.134 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:01.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:01.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:01.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:01.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:01.139 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:01Z" 2023-10-12T01:40:01.139 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:01.140 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:01Z" } 2023-10-12T01:40:01.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:01.141 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:01.142 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:01.142 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:06.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:06.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:06.144 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:06.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:06.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:06.252 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:06.253 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:06.253 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:06.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:06.253 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:06.254 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:06.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.254 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:06.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:06.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:06.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:06.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:06.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:06.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:06.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:06.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:06.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:06.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:06.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:06.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:06.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:06.257 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:06.258 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:06.258 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:06.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:06.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:06.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:06.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:06.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:06.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:06.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:06.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:06.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:06.260 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:06.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:06.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:06.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:06.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:06.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:06.262 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:06.262 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:06.262 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:06.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:06.263 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:06.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:06.263 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:06.263 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:06Z" 2023-10-12T01:40:06.265 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:06.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:06Z" } 2023-10-12T01:40:06.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:06.268 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:06.268 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:06.269 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:11.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:11.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:11.271 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:11.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:11.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:11.381 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:11Z" 2023-10-12T01:40:11.394 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:11.395 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:11Z" } 2023-10-12T01:40:11.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:11.396 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:11.397 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:11.400 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:16.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:16.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:16.403 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:16.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:16.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:16.510 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:16.516 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:16.516 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:16.516 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:16.516 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:16.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:16.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:16.517 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:16.517 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:16.518 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:16.518 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:16.518 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:16.518 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:16.518 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:16.519 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:16.519 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:16.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:16Z" 2023-10-12T01:40:16.523 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:16.524 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:16Z" } 2023-10-12T01:40:16.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:16.525 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:16.526 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:16.526 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:21.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:21.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:21.528 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:21.588 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:21.588 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:21.637 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:21.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:21.644 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:21.647 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:21.651 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:21Z" 2023-10-12T01:40:21.651 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:21.652 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:21Z" } 2023-10-12T01:40:21.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:21.653 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:21.653 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:21.654 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:26.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:26.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:26.657 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:26.716 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:26.716 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:26.764 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:26.764 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:26.764 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:26.765 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:26.765 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:26.765 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:26.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.766 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:26.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:26.766 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:26.766 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:26.767 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:26.767 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:26.767 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:26.767 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:26.768 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:26.768 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:26.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:26.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:26Z" 2023-10-12T01:40:26.778 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:26.780 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:26Z" } 2023-10-12T01:40:26.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:26.781 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:26.781 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:26.781 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:31.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:31.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:31.783 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:31.843 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:31.890 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:31.902 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:31.902 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:31.902 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:31.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:31.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:31.903 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:31.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.903 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:31.903 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:31.903 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:31.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:31.904 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:31Z" 2023-10-12T01:40:31.904 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:31.906 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:31Z" } 2023-10-12T01:40:31.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:31.907 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:31.907 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:31.908 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:36.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:36.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:36.910 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:36.969 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:36.970 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:37.018 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:37Z" 2023-10-12T01:40:37.032 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:37.033 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:37Z" } 2023-10-12T01:40:37.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:37.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:37.034 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:37.035 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:42.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:42.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:42.038 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:42.096 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:42.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:42.144 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:42.145 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:42.145 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:42.145 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:42.145 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:42.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:42.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:42.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:42.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:42.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:42.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:42.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:42.149 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:42.150 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:42.150 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:42.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:42.151 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:42.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:42.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:42.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:42.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:42.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:42.152 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:42.153 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:42.153 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:42.154 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:42.156 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:42.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:42.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:42.156 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:42.157 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:42.157 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:42.157 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:42.158 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:42.158 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:42.158 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:42.158 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:42.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:42.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:42.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:42.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.160 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:42.160 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:42.160 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:42.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:42.161 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:42Z" 2023-10-12T01:40:42.161 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:42.163 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:42Z" } 2023-10-12T01:40:42.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:42.165 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:42.166 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:42.166 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:47.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:47.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:47.169 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:47.228 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:47.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:47.278 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:47.279 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:47.279 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:47.279 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:47.280 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:47.280 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:47.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.280 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:47.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:47.281 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:47.281 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:47.282 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:47.282 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:47.282 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:47.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:47.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:47.283 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:47.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:47.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:47.284 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:47.284 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:47.285 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:47.285 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:47.285 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:47.285 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:47.285 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:47.286 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:47.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:47.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:47.287 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:47.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:47.287 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:47.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:47.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:47.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:47.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:47.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:47.289 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:47.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:47.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:47.290 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:47.290 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:47.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:47.290 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:47.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:47.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:47.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:47.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:47.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:47.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:47.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.292 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:47.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:47.294 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:47.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:47.294 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:47Z" 2023-10-12T01:40:47.294 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:47.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:47Z" } 2023-10-12T01:40:47.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:47.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:47.297 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:47.297 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:52.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:52.299 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:52.359 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:52.359 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:52.407 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:52.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:52.408 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:52.408 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:52.409 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:52.409 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:52.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.409 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:52.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:52.410 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:52.410 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:52.410 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:52.410 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:52.411 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:52.411 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:52.411 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:52.411 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:52.414 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:52.414 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:52.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:52.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:52.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:52.415 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:52.415 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:52.415 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:52.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:52.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:52.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:52.416 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:52.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:52.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:52.417 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:52.417 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:52.417 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:52.418 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:52.418 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:52.418 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:52.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:52.418 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:52.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:52.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:52.419 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:52.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.419 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:52.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:52.420 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:52.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:52.420 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:52Z" 2023-10-12T01:40:52.421 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:52.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:52Z" } 2023-10-12T01:40:52.422 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:52.423 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:52.423 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:52.423 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:40:57.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:57.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:40:57.426 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:40:57.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:40:57.487 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:40:57.537 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:40:57.537 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:40:57.538 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:40:57.538 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:40:57.538 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:40:57.538 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:40:57.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.539 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:40:57.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:57.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:40:57.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:57.540 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:57.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:40:57.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:57.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:40:57.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:57.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:40:57.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:57.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:57.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:40:57.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:57.542 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:57.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:40:57.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:57.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:40:57.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:57.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:40:57.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:57.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:57.544 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:40:57.544 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:57.544 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:57.544 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:40:57.545 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:57.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:57.545 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:57.545 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:40:57.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:57.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:40:57.546 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:40:57.546 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:40:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:40:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:40:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:40:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:40:57.547 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:40:57.548 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:40:57.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:57.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:40:57.548 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:40:57.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.549 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:40:57.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:40:57.549 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:40:57.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:40:57.549 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:40:57Z" 2023-10-12T01:40:57.550 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:40:57.551 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:40:57Z" } 2023-10-12T01:40:57.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:40:57.552 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:40:57.552 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:40:57.553 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:02.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:02.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:02.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:02.617 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:02.664 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:02.665 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:02.665 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:02.665 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:02.665 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:02.666 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:02.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.666 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:02.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:02.666 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:02.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:02.667 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:02.667 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:02.667 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:02.668 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:02.668 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:02.668 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:02.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:02.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:02.669 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:02.669 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:02.669 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:02.669 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:02.670 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:02.670 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:02.670 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:02.670 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:02.670 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:02.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:02.671 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:02.671 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:02.671 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:02.672 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:02.672 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:02.672 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:02.672 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:02.673 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:02.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:02.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:02.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:02.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:02.674 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:02.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:02.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:02.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:02.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:02.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:02.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:02.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:02.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:02.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.676 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:02.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:02.676 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:02.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:02.677 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:02Z" 2023-10-12T01:41:02.677 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:02.678 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:02Z" } 2023-10-12T01:41:02.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:02.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:02.680 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:02.680 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:07.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:07.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:07.682 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:07.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:07.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:07.789 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:07.790 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:07.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:07.792 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:07.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:07.794 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:07.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:07.796 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:07.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:07.799 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:07.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:07.801 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:07.802 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:07Z" 2023-10-12T01:41:07.802 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:07.803 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:07Z" } 2023-10-12T01:41:07.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:07.804 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:07.805 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:07.805 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:12.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:12.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:12.808 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:12.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:12.867 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:12.915 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:12.915 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.916 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:12.917 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:12.918 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:12.919 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:12.920 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:12.922 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:12.923 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:12.924 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:12.925 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:12.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:12.927 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:12.928 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:12Z" 2023-10-12T01:41:12.928 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:12.929 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:12Z" } 2023-10-12T01:41:12.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:12.930 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:12.931 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:12.931 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:17.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:17.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:17.934 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:17.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:17.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:18.042 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:18Z" 2023-10-12T01:41:18.055 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:18.056 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:18Z" } 2023-10-12T01:41:18.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:18.057 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:18.057 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:18.058 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:23.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:23.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:23.060 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:23.167 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:23.167 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:23.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:23.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:23.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:23.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:23.172 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:23.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:23.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:23.175 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:23.176 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:23.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:23.178 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:23.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:23.180 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:23Z" 2023-10-12T01:41:23.180 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:23.181 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:23Z" } 2023-10-12T01:41:23.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:23.185 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:23.186 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:23.186 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:28.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:28.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:28.189 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:28.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:28.248 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:28.295 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:28.296 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:28.296 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:28.296 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:28.297 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:28.297 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:28.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.297 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:28.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:28.298 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:28.298 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:28.298 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:28.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:28.298 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:28.299 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:28.299 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:28.299 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:28.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:28.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:28.300 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:28.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:28.300 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:28.301 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:28.301 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:28.301 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:28.301 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:28.302 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:28.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:28.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:28.302 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:28.303 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:28.303 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:28.303 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:28.303 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:28.303 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:28.304 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:28.304 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:28.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:28.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:28.305 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:28.305 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:28.305 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:28.305 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:28.306 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:28.306 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:28.306 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:28.306 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:28.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:28.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:28.307 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:28.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.307 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:28.307 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:28.308 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:28.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:28.308 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:28Z" 2023-10-12T01:41:28.308 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:28.309 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:28Z" } 2023-10-12T01:41:28.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:28.310 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:28.311 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:28.311 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:33.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:33.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:33.314 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:33.373 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:33.373 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:33.426 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:33.427 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:33.427 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:33.427 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:33.428 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:33.428 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:33.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.428 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:33.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:33.429 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:33.429 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:33.429 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:33.429 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:33.430 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:33.430 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:33.430 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:33.430 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:33.431 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:33.432 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:33.432 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:33.432 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:33.433 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:33.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:33.433 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:33.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:33.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:33.434 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:33.434 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:33.436 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:33.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:33.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:33.437 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:33.437 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:33.437 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:33.437 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:33.438 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:33.438 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:33.438 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:33.438 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:33.438 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:33.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:33.439 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:33.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.439 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:33.440 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:33.440 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:33.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:33.440 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:33Z" 2023-10-12T01:41:33.440 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:33.441 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:33Z" } 2023-10-12T01:41:33.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:33.442 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:33.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:33.443 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:38.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:38.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:38.447 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:38.506 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:38.506 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:38.554 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:38.554 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:38.555 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:38.555 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:38.555 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:38.555 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:38.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.556 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:38.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:38.556 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:38.556 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:38.557 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:38.557 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:38.557 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:38.557 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:38.557 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:38.558 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:38.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:38.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:38.559 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:38.559 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:38.559 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:38.559 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:38.559 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:38.560 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:38.560 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:38.560 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:38.560 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:38.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:38.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:38.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:38.561 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:38.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:38.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:38.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:38.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:38.562 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:38.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:38.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:38.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:38.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:38.564 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:38.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:38.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:38.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:38.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:38.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:38.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:38.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:38.567 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:38.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.567 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:38.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:38.568 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:38.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:38.568 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:38Z" 2023-10-12T01:41:38.569 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:38.570 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:38Z" } 2023-10-12T01:41:38.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:38.571 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:38.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:38.572 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:43.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:43.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:43.574 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:43.681 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:43.681 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-12T01:41:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:39.563613Z" 2023-10-12T01:41:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:43.689 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:43.689 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:43.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044323", 2023-10-12T01:41:43.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:43.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:43.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:43.690 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:30:54.780559Z" 2023-10-12T01:41:43.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:43.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:43.691 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:43.691 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:43.691 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:43.692 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-12T01:41:43.692 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:43.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:43.692 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:43.693 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:37:46.722600Z" 2023-10-12T01:41:43.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:43.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:43.693 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:43.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.694 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:43.694 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:43.694 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:43.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:43.695 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:43Z" 2023-10-12T01:41:43.695 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:43.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:37:39.563613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044323", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:30:54.780559Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:37:46.722600Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:43Z" } 2023-10-12T01:41:43.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:37:39.563613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044323', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:54.780559Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:37:46.722600Z'}}] 2023-10-12T01:41:43.697 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:43.698 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000050970', 2: '00000000000000000000:00000000000000044323', 3: '00000000000000000000:00000000000000045659'} 2023-10-12T01:41:43.698 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=2 master=00000000000000000000:00000000000000044465 target=00000000000000000000:00000000000000044323, shard=3 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659 2023-10-12T01:41:46.071 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:41:46.070+0000 7fa14d828700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:41:48.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:48.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:48.701 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:48.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:48.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:48.809 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:48.809 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:48.810 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:48.810 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:48.810 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:48.810 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:41:48.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.811 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:48.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:48.811 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:48.811 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:48.812 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:48.812 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:41:48.812 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:48.812 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:41:48.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:48.813 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:41:48.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:48.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:48.814 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:48.814 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:48.814 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:48.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-12T01:41:48.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:48.815 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:41:48.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:48.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.161892Z" 2023-10-12T01:41:48.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:48.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:48.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:48.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:48.816 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:48.817 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:41:48.817 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:48.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:48.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:48.817 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:41:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:41:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:41:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:41:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:41:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:41:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:48Z" 2023-10-12T01:41:48.822 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:48.823 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:39:45.161892Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:41:48Z" } 2023-10-12T01:41:48.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.161892Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:41:48.824 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:48.824 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051250', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:41:48.825 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:41:48.825 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:41:48.825 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:41:48.826 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.826 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:41:48.827 DEBUG:boto:Data: 2023-10-12T01:41:48.827 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.828 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.828 DEBUG:boto:Params: {} 2023-10-12T01:41:48.829 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:41:48.829 DEBUG:boto:Token: None 2023-10-12T01:41:48.830 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/ 2023-10-12T01:41:48.831 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VmyEswSGOEDqTe4AcNea147N1R8= 2023-10-12T01:41:48.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VmyEswSGOEDqTe4AcNea147N1R8=', 'Content-Length': '0'} 2023-10-12T01:41:48.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f28c90420a82ce2-0065274edc-2429-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.837 DEBUG:boto:path=/wafmoa-68/obj1 2023-10-12T01:41:48.837 DEBUG:boto:auth_path=/wafmoa-68/obj1 2023-10-12T01:41:48.838 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.838 DEBUG:boto:Path: /wafmoa-68/obj1 2023-10-12T01:41:48.839 DEBUG:boto:Data: 2023-10-12T01:41:48.839 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.840 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.840 DEBUG:boto:Params: {} 2023-10-12T01:41:48.841 DEBUG:boto:Token: None 2023-10-12T01:41:48.841 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj1 2023-10-12T01:41:48.842 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:KwWG9QW6FlgC35ci/L/6NexwpZc= 2023-10-12T01:41:48.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KwWG9QW6FlgC35ci/L/6NexwpZc=', 'Content-Length': '0'} 2023-10-12T01:41:48.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2d2c7f6ad444914-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.845 DEBUG:boto:path=/wafmoa-68/obj1 2023-10-12T01:41:48.846 DEBUG:boto:auth_path=/wafmoa-68/obj1 2023-10-12T01:41:48.846 DEBUG:boto:Method: GET 2023-10-12T01:41:48.847 DEBUG:boto:Path: /wafmoa-68/obj1 2023-10-12T01:41:48.847 DEBUG:boto:Data: 2023-10-12T01:41:48.848 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.849 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.849 DEBUG:boto:Params: {} 2023-10-12T01:41:48.849 DEBUG:boto:Token: None 2023-10-12T01:41:48.850 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj1 2023-10-12T01:41:48.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r5LvaFd8/RS61BO5rNbZ5JWGIXE= 2023-10-12T01:41:48.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r5LvaFd8/RS61BO5rNbZ5JWGIXE=', 'Content-Length': '0'} 2023-10-12T01:41:48.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022853e352959c106-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.854 DEBUG:boto:path=/wafmoa-68/obj2 2023-10-12T01:41:48.854 DEBUG:boto:auth_path=/wafmoa-68/obj2 2023-10-12T01:41:48.855 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.855 DEBUG:boto:Path: /wafmoa-68/obj2 2023-10-12T01:41:48.856 DEBUG:boto:Data: 2023-10-12T01:41:48.856 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.857 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.857 DEBUG:boto:Params: {} 2023-10-12T01:41:48.858 DEBUG:boto:Token: None 2023-10-12T01:41:48.858 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj2 2023-10-12T01:41:48.859 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:V49GGDyBZVrU5GpnkhamaDqUhpE= 2023-10-12T01:41:48.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:V49GGDyBZVrU5GpnkhamaDqUhpE=', 'Content-Length': '0'} 2023-10-12T01:41:48.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000929c41b756ce4778-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.861 DEBUG:boto:path=/wafmoa-68/obj2 2023-10-12T01:41:48.862 DEBUG:boto:auth_path=/wafmoa-68/obj2 2023-10-12T01:41:48.862 DEBUG:boto:Method: GET 2023-10-12T01:41:48.863 DEBUG:boto:Path: /wafmoa-68/obj2 2023-10-12T01:41:48.863 DEBUG:boto:Data: 2023-10-12T01:41:48.864 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.864 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.865 DEBUG:boto:Params: {} 2023-10-12T01:41:48.866 DEBUG:boto:Token: None 2023-10-12T01:41:48.866 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj2 2023-10-12T01:41:48.867 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bfrAnT9te9YAyEtlCqAUdDUtkVw= 2023-10-12T01:41:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bfrAnT9te9YAyEtlCqAUdDUtkVw=', 'Content-Length': '0'} 2023-10-12T01:41:48.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e94df7da57cd54-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.870 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:41:48.870 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:41:48.871 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.871 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:41:48.872 DEBUG:boto:Data: 2023-10-12T01:41:48.872 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.873 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.873 DEBUG:boto:Params: {} 2023-10-12T01:41:48.874 DEBUG:boto:Token: None 2023-10-12T01:41:48.874 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj3 2023-10-12T01:41:48.875 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8LRyLbyaWwPOIEUWLYuDDZn9mpU= 2023-10-12T01:41:48.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8LRyLbyaWwPOIEUWLYuDDZn9mpU=', 'Content-Length': '0'} 2023-10-12T01:41:48.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e92ae15857b8ecbd-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.878 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:41:48.879 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:41:48.879 DEBUG:boto:Method: GET 2023-10-12T01:41:48.880 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:41:48.880 DEBUG:boto:Data: 2023-10-12T01:41:48.881 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.881 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.882 DEBUG:boto:Params: {} 2023-10-12T01:41:48.882 DEBUG:boto:Token: None 2023-10-12T01:41:48.883 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-68/obj3 2023-10-12T01:41:48.883 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QwH3D7xHIedaE+cjj0OWohNXrmM= 2023-10-12T01:41:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QwH3D7xHIedaE+cjj0OWohNXrmM=', 'Content-Length': '0'} 2023-10-12T01:41:48.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083c0971ca301966c-0065274edc-2429-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.890 DEBUG:boto:path=/wafmoa-69/ 2023-10-12T01:41:48.891 DEBUG:boto:auth_path=/wafmoa-69/ 2023-10-12T01:41:48.891 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.892 DEBUG:boto:Path: /wafmoa-69/ 2023-10-12T01:41:48.893 DEBUG:boto:Data: 2023-10-12T01:41:48.893 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.894 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.894 DEBUG:boto:Params: {} 2023-10-12T01:41:48.895 DEBUG:boto:Token: None 2023-10-12T01:41:48.895 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-69/ 2023-10-12T01:41:48.896 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lio9a+QT5XgbZAZ2AIk7bbaCxNs= 2023-10-12T01:41:48.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lio9a+QT5XgbZAZ2AIk7bbaCxNs=', 'Content-Length': '0'} 2023-10-12T01:41:48.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c4b8a7835b2c38a-0065274edc-2429-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.900 DEBUG:boto:path=/wafmoa-69/obj1 2023-10-12T01:41:48.901 DEBUG:boto:auth_path=/wafmoa-69/obj1 2023-10-12T01:41:48.901 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.902 DEBUG:boto:Path: /wafmoa-69/obj1 2023-10-12T01:41:48.902 DEBUG:boto:Data: 2023-10-12T01:41:48.903 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.904 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.904 DEBUG:boto:Params: {} 2023-10-12T01:41:48.905 DEBUG:boto:Token: None 2023-10-12T01:41:48.905 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-69/obj1 2023-10-12T01:41:48.906 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ohHvERy871Nx+YsIehsDD6f9chk= 2023-10-12T01:41:48.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ohHvERy871Nx+YsIehsDD6f9chk=', 'Content-Length': '0'} 2023-10-12T01:41:48.908 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000020affe0981def012-0065274edc-2429-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.908 DEBUG:boto:path=/wafmoa-69/obj2 2023-10-12T01:41:48.909 DEBUG:boto:auth_path=/wafmoa-69/obj2 2023-10-12T01:41:48.909 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.910 DEBUG:boto:Path: /wafmoa-69/obj2 2023-10-12T01:41:48.910 DEBUG:boto:Data: 2023-10-12T01:41:48.910 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.911 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.912 DEBUG:boto:Params: {} 2023-10-12T01:41:48.912 DEBUG:boto:Token: None 2023-10-12T01:41:48.913 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-69/obj2 2023-10-12T01:41:48.913 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lfTUJ2TH11kWaOS9GIPYkkaZgq8= 2023-10-12T01:41:48.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lfTUJ2TH11kWaOS9GIPYkkaZgq8=', 'Content-Length': '0'} 2023-10-12T01:41:48.915 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000834cf43755390f1e-0065274edc-2429-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.916 DEBUG:boto:path=/wafmoa-69/obj3 2023-10-12T01:41:48.916 DEBUG:boto:auth_path=/wafmoa-69/obj3 2023-10-12T01:41:48.917 DEBUG:boto:Method: HEAD 2023-10-12T01:41:48.917 DEBUG:boto:Path: /wafmoa-69/obj3 2023-10-12T01:41:48.918 DEBUG:boto:Data: 2023-10-12T01:41:48.918 DEBUG:boto:Headers: {} 2023-10-12T01:41:48.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:41:48.919 DEBUG:boto:Port: 8001 2023-10-12T01:41:48.919 DEBUG:boto:Params: {} 2023-10-12T01:41:48.920 DEBUG:boto:Token: None 2023-10-12T01:41:48.920 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:48 GMT /wafmoa-69/obj3 2023-10-12T01:41:48.921 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hBXvuDH2uvCF6MYcvMbAy+yQatU= 2023-10-12T01:41:48.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:48 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hBXvuDH2uvCF6MYcvMbAy+yQatU=', 'Content-Length': '0'} 2023-10-12T01:41:48.923 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002fecdab60e630f10-0065274edc-2429-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:41:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:48.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:48.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:48.924 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:48.977 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:48.977 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.022 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:41:49.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.024 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.077 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.121 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:41:49.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.123 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.171 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.216 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:41:49.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.218 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.319 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:49.320 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:41:49.320 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17, 2023-10-12T01:41:49.320 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:41:49.320 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:41:49.321 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:41:49.321 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:41:49.321 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:41:49.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.322 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:41:49.322 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:41:49.322 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:41:49.322 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:41:49.323 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.323 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:41:49.323 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.323 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:41:49.323 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:41:49.324 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.324 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:41:49.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.324 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.324 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:41:49.325 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.325 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:41:49.325 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.325 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:41:49.325 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:41:49.326 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:41:49.326 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:41:49.326 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:41:49.326 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:41:49.327 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:41:49.327 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:41:49.327 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:41:49.328 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:41:49.328 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.328 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.329 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:41:49.329 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:41:49.330 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.330 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:41:49.330 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.330 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:41:49.330 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:41:49.331 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:41:49.331 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:41:49.331 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:41:49.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:41:49.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:41:49.332 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:41:49.332 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:41:49.332 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:41:49.332 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.333 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.333 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.333 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:41:49.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.334 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:41:49.334 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:41:49.334 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:41:49.334 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:41:49.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.335 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.335 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:41:49.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:41:49.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:41:49.336 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:41:49.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.337 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:41:49.337 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:41:49.337 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:41:49.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.339 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:41:49.339 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:41:49.339 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.340 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:41:49.340 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.340 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:41:49.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.341 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:41:49.341 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:41:49.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:41:49.342 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:41:49.342 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.342 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:41:49.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.343 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:41:49.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:41:49.344 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:41:49.344 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.344 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:41:49.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.345 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:41:49.345 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:41:49.345 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:41:49.346 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:41:49.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.346 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:41:49.347 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:41:49.347 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:41:49.347 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:41:49.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.348 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.348 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-10-12T01:41:49.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.349 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.349 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:41:49.349 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.350 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.350 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:41:49.351 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:41:49.351 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:41:49.351 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:41:49.351 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.352 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:41:49.352 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.352 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:41:49.352 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:41:49.352 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:41:49.353 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:41:49.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.353 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:41:49.353 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:41:49.354 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.354 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:41:49.354 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.354 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:41:49.355 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:41:49.355 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:41:49.355 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:41:49.356 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:41:49.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:41:49.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:41:49.356 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:41:49.357 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:41:49.357 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:41:49.357 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.357 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.358 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:41:49.358 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:41:49.358 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:41:49.359 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:41:49.359 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.359 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:41:49.359 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:41:49.359 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:41:49.360 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:41:49.360 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:41:49.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:41:49.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:41:49.361 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:41:49.361 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:41:49.361 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:41:49.361 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.362 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.362 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.362 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:41:49.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.363 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:41:49.363 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:41:49.363 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:41:49.364 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:41:49.364 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.364 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.365 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:41:49.365 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:41:49.365 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:41:49.365 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:41:49.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.366 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:41:49.366 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:41:49.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.367 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:41:49.367 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:41:49.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.367 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:41:49.367 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:41:49.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:41:49.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.370 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:41:49.370 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:41:49.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:49.371 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:41:49.371 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:41:49.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:49.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.372 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:41:49.373 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:41:49.373 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:41:49.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:41:49.373 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:41:49.374 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:41:49.374 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:41:49.374 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:41:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:41:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:41:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:41:49.375 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:41:49.376 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:41:49.376 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:41:49.376 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:41:49.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.377 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:41:49.377 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:41:49.377 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:41:49.377 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:41:49.377 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:41:49.378 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:41:49.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.378 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:41:49.378 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:41:49.379 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:41:49.379 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:41:49.379 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:41:49.379 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:41:49.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.380 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:41:49.380 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:41:49.380 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:41:49.380 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:41:49.381 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:41:49.381 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:41:49.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:49.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:49.381 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:41:49.382 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:41:49.382 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:49.382 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:49.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.384 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.437 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.498 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:41:49.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.503 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.711 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:41:49.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.712 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:49.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:49.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:49.928 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:41:49.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:49.930 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:50.006 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:50.007 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:50.077 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:41:50.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:50.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:50.079 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:50.204 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:41:50.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:50.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:41:50.206 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:41:50.253 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:50.254 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:50.302 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:41:50.302 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:41:50.303 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:41:50.303 DEBUG:boto:Method: HEAD 2023-10-12T01:41:50.304 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:41:50.304 DEBUG:boto:Data: 2023-10-12T01:41:50.304 DEBUG:boto:Headers: {} 2023-10-12T01:41:50.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:41:50.305 DEBUG:boto:Port: 8000 2023-10-12T01:41:50.306 DEBUG:boto:Params: {} 2023-10-12T01:41:50.306 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:41:50.307 DEBUG:boto:Token: None 2023-10-12T01:41:50.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:50 GMT /wafmoa-68/ 2023-10-12T01:41:50.308 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7w6fWOdknUqiya2caiejcGkiSqs= 2023-10-12T01:41:50.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7w6fWOdknUqiya2caiejcGkiSqs=', 'Content-Length': '0'} 2023-10-12T01:41:50.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5ee01a961a8ff4c-0065274ede-2544-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:41:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:50.317 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:41:50.318 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:41:50.318 DEBUG:boto:Method: PUT 2023-10-12T01:41:50.319 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:41:50.319 DEBUG:boto:Data: 2023-10-12T01:41:50.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:41:50.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:41:50.320 DEBUG:boto:Port: 8000 2023-10-12T01:41:50.321 DEBUG:boto:Params: {} 2023-10-12T01:41:50.321 DEBUG:boto:Token: None 2023-10-12T01:41:50.322 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:41:50 GMT /wafmoa-68/obj3 2023-10-12T01:41:50.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zCBxihN8IQfcpo3NkjIyZimIF9s= 2023-10-12T01:41:50.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:41:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zCBxihN8IQfcpo3NkjIyZimIF9s='} 2023-10-12T01:41:50.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009acb4205c18c920e-0065274ede-2544-a1'), ('Date', 'Thu, 12 Oct 2023 01:41:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:50.333 DEBUG:boto:path=/wafmoa-69/ 2023-10-12T01:41:50.333 DEBUG:boto:auth_path=/wafmoa-69/ 2023-10-12T01:41:50.334 DEBUG:boto:Method: HEAD 2023-10-12T01:41:50.334 DEBUG:boto:Path: /wafmoa-69/ 2023-10-12T01:41:50.335 DEBUG:boto:Data: 2023-10-12T01:41:50.335 DEBUG:boto:Headers: {} 2023-10-12T01:41:50.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:41:50.336 DEBUG:boto:Port: 8000 2023-10-12T01:41:50.336 DEBUG:boto:Params: {} 2023-10-12T01:41:50.337 DEBUG:boto:Token: None 2023-10-12T01:41:50.337 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:41:50 GMT /wafmoa-69/ 2023-10-12T01:41:50.338 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:38qpT2yjhzIQbZU6ngqDWxvEkEY= 2023-10-12T01:41:50.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:41:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:38qpT2yjhzIQbZU6ngqDWxvEkEY=', 'Content-Length': '0'} 2023-10-12T01:41:50.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2fa65f23e9008e8-0065274ede-2544-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:41:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:50.342 DEBUG:boto:path=/wafmoa-69/obj3 2023-10-12T01:41:50.342 DEBUG:boto:auth_path=/wafmoa-69/obj3 2023-10-12T01:41:50.343 DEBUG:boto:Method: PUT 2023-10-12T01:41:50.343 DEBUG:boto:Path: /wafmoa-69/obj3 2023-10-12T01:41:50.343 DEBUG:boto:Data: 2023-10-12T01:41:50.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:41:50.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:41:50.345 DEBUG:boto:Port: 8000 2023-10-12T01:41:50.345 DEBUG:boto:Params: {} 2023-10-12T01:41:50.345 DEBUG:boto:Token: None 2023-10-12T01:41:50.346 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:41:50 GMT /wafmoa-69/obj3 2023-10-12T01:41:50.346 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:7EAS4zy0NCLapEVvU9Nm/t5AqjU= 2023-10-12T01:41:50.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:41:50 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7EAS4zy0NCLapEVvU9Nm/t5AqjU='} 2023-10-12T01:41:50.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b210556dd47bfaf-0065274ede-2544-a1'), ('Date', 'Thu, 12 Oct 2023 01:41:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:41:50.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:50.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:50.354 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:50.432 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:41:50.445 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:41:50.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:41:50.446 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:41:50.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:41:50.447 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:41:50.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.448 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:41:50.448 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:41:50.188707Z" 2023-10-12T01:41:50.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:41:50.449 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:41:50.449 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.449 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:41:50.450 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:41:50.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:41:50.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:50.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:50.452 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:50.501 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:50.502 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:50.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:41:50.530+0000 7ffb2ecdb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:41:50.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:41:50.531+0000 7ffb2ecdb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:41:50.544 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:50.545 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:50.545 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:50.545 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:50.545 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:50.546 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:41:50.546 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:50.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.546 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:50.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.547 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:50.547 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:50.547 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:50.547 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:41:50.548 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:50.548 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:41:50.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:50.548 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:41:50.548 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:41:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:41:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:41:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:41:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:41:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:41:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:41:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:41:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:50Z" 2023-10-12T01:41:50.558 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:50.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:41:50Z" } 2023-10-12T01:41:50.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:41:50.560 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:41:50.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:41:50.561 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:41:55.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:55.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:41:55.563 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:41:55.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:41:55.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:41:55.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:41:55.652+0000 7f1f1e15a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:41:55.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:41:55.654+0000 7f1f1e15a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:41:55.667 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:41:55.668 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:41:55.668 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:41:55.668 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:41:55.669 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:41:55.669 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:41:55.669 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:55.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.670 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:41:55.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:55.670 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:41:55.670 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:55.671 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:55.671 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:41:55.671 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:55.671 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:41:55.672 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:55.672 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:41:55.672 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:55.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:55.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:55.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:41:55.673 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:55.674 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:55.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:41:55.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:55.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:55.674 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:55.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:41:55.675 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:55.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:55.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:55.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:41:55.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:55.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:55.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:41:55.677 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:55.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:55.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:55.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:41:55.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:55.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:55.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:41:55.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:41:55.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:41:55.679 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:41:55.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:41:55.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:41:55.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:41:55.680 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:41:55.680 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:41:55.681 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:41:55.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:55.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:41:55.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:41:55.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.682 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:41:55.682 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:41:55.682 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:41:55.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:41:55.683 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:41:55Z" 2023-10-12T01:41:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:41:55.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:41:55Z" } 2023-10-12T01:41:55.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:41:55.685 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:41:55.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:41:55.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:00.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:00.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:00.688 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:00.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:00.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:00.779+0000 7f6079440780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:00.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:00.780+0000 7f6079440780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:00.794 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:00.795 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:00Z" 2023-10-12T01:42:00.808 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:00.809 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:00Z" } 2023-10-12T01:42:00.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:00.810 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:00.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:00.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:05.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:05.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:05.813 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:05.872 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:05.872 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:05.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:05.904+0000 7f02876d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:05.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:05.905+0000 7f02876d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:05.919 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:05Z" 2023-10-12T01:42:05.933 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:05.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:05Z" } 2023-10-12T01:42:05.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:05.935 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:05.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:05.936 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:10.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:10.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:10.938 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:10.992 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:10.992 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:11.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:11.021+0000 7f9eed2cc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:11.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:11.022+0000 7f9eed2cc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:11.036 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:11.036 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:11.036 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:11.037 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:11.038 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:11.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:11.041 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:11.042 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:11.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:11.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:11.046 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:11.047 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:11Z" 2023-10-12T01:42:11.049 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:11.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:11Z" } 2023-10-12T01:42:11.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:11.052 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:11.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:11.052 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:16.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:16.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:16.054 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:16.112 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:16.112 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:16.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:16.144+0000 7fcd102fb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:16.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:16.145+0000 7fcd102fb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:16.160 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:16.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:16.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:16.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:16.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.172 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:16.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:16.173 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:16.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:16.173 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:16Z" 2023-10-12T01:42:16.174 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:16.175 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:16Z" } 2023-10-12T01:42:16.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:16.176 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:16.176 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:16.176 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:21.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:21.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:21.179 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:21.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:21.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:21.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:21.268+0000 7fe1b1853780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:21.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:21.269+0000 7fe1b1853780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:21.284 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:21.284 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:21.284 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:21.284 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:21.285 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:21.285 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:21.285 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:21.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.285 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:21.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:21.286 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:21.286 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:21.286 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:21.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:21.287 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:21.287 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:21.287 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:21.287 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:21.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:21.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:21.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:21.288 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:21.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:21.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:21.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:21.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:21.290 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:21.290 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:21.290 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:21.290 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:21.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:21.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:21.291 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:21.291 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:21.291 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:21.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:21.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:21.292 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:21.292 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:21.292 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:21.293 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:21.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:21.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:21.293 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:21.294 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:21.294 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:21.294 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:21.294 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:21.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:21.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:21.295 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:21.295 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:21.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:21.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:21.296 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:21.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.296 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:21.296 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:21.297 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:21.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:21.297 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:21Z" 2023-10-12T01:42:21.297 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:21.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:21Z" } 2023-10-12T01:42:21.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:21.299 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:21.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:21.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:26.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:26.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:26.302 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:26.359 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:26.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:26.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:26.392+0000 7f04dcbcd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:26.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:26.393+0000 7f04dcbcd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:26.407 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:26.409 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:26.409 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:26.409 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:26.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.409 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:26.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:26.410 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:26.410 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:26.410 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:26.411 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:26.411 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:26.411 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:26.411 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:26.412 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:26.412 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:26.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:26.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:26.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:26.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:26.413 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:26.413 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:26.414 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:26.414 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:26.414 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:26.414 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:26.414 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:26.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:26.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:26.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:26.415 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:26.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:26.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:26.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:26.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:26.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:26.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:26.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:26.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:26.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:26.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:26.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:26.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:26.418 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:26.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:26.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:26.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:26.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:26.419 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:26.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:26.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:26.420 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:26.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.420 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:26.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:26.421 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:26.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:26.421 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:26Z" 2023-10-12T01:42:26.421 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:26.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:26Z" } 2023-10-12T01:42:26.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:26.424 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:26.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:26.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:31.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:31.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:31.427 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:31.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:31.520+0000 7f3bf1000780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:31.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:31.521+0000 7f3bf1000780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:31.534 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:31.535 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:31.535 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:31.535 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:31.536 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:31.536 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:31.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:31.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.537 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:31.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:31.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:31.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:31.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:31.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:31.538 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:31.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:31.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:31.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:31.539 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:31.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:31.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:31.540 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:31.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:31.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:31.541 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:31.541 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:31.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:31.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:31.542 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:31.542 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:31.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:31.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:31.543 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:31.543 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:31.544 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:31.544 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:31.544 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:31.544 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:31.545 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:31.545 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:31.545 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:31.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:31.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:31.547 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:31.547 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:31.548 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:31.548 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:31.549 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:31.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:31.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:31.550 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:31.550 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:31.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:31.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:31.551 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:31.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.552 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:31.552 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:31.553 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:31.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:31.553 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:31Z" 2023-10-12T01:42:31.554 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:31.556 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:31Z" } 2023-10-12T01:42:31.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:31.557 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:31.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:31.559 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:36.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:36.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:36.562 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:36.619 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:36.619 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:36.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:36.653+0000 7f10832e0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:36.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:36.654+0000 7f10832e0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:36.667 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:36.668 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:36.668 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:36.668 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:36.669 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:36.669 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:36.669 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:36.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.669 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:36.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:36.670 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:36.670 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:36.670 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:36.671 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:36.671 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:36.671 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:36.671 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:36.671 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:36.672 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:36.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:36.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:36.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:36.673 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:36.673 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:36.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:36.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:36.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:36.674 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:36.674 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:36.675 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:36.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:36.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:36.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:36.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:36.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:36.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:36.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:36.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:36.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:36.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:36.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:36.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:36.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:36.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:36.678 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:36.678 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:36.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:36.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:36.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:36.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:36.680 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:36.680 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:36.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:36.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:36.680 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:36.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.681 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:36.681 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:36.681 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:36.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:36.682 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:36Z" 2023-10-12T01:42:36.682 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:36.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:36Z" } 2023-10-12T01:42:36.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:36.685 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:36.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:36.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:41.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:41.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:41.688 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:41.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:41.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:41.781+0000 7f8fb5c5d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:41.782+0000 7f8fb5c5d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:41.795 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:41Z" 2023-10-12T01:42:41.809 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:41.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:41Z" } 2023-10-12T01:42:41.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:41.812 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:41.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:41.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:46.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:46.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:46.815 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:46.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:46.874 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:46.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:46.906+0000 7f5f51006780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:46.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:46.907+0000 7f5f51006780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:46.921 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:46.921 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:46.921 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:46.922 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:46.922 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:46.922 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:46.922 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:46.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.923 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:46.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:46.923 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:46.923 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:46.924 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:46.924 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:46.924 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:46.924 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:46.924 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:46.925 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:46.925 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:46.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:46.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:46.926 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:46.926 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:46.926 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:46.926 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:46.927 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:46.927 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:46.927 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:46.927 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:46.927 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:46.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:46.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:46.928 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:46.928 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:46.929 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:46.929 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:46.929 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:46.929 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:46.929 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:46.930 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:46.930 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:46.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:46.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:46.931 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:46.931 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:46.931 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:46.931 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:46.932 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:46.933 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:46.934 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:46.934 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:46.934 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:46.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:46.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:46.935 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:46.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.936 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:46.936 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:46.936 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:46.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:46.936 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:46Z" 2023-10-12T01:42:46.937 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:46.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:46Z" } 2023-10-12T01:42:46.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:46.939 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:46.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:46.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:51.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:51.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:51.942 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:52.000 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:52.000 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:52.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:52.032+0000 7f896ab61780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:52.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:52.033+0000 7f896ab61780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:52.048 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:52.048 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:52.049 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:52.049 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:52.049 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:52.050 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:52.050 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:52.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.051 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:52.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:52.052 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:52.053 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:52.053 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:52.053 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:52.061 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:52.061 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:52.062 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:52.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:52.062 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:52.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:52.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:52.064 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:52.064 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:52.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:52.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:52.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:52.066 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:52.066 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:52.066 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:52.067 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:52.067 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:52.068 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:52.068 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:52.068 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:52.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:52.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:52.072 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:52Z" 2023-10-12T01:42:52.072 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:52.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:52Z" } 2023-10-12T01:42:52.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:52.076 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:52.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:52.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:42:57.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:57.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:42:57.079 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:42:57.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:42:57.138 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:42:57.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:57.172+0000 7f694c122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:57.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:42:57.173+0000 7f694c122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:42:57.198 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:42:57.198 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:42:57.199 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:42:57.199 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:42:57.199 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:42:57.199 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:42:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:42:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:42:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:42:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:42:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:42:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:42:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:42:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:42:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:42:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:42:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:42:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:42:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:42:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:42:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:42:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:42:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:42:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:42:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:42:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:42:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:42:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:42:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:42:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:42:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:42:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:42:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:42:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:42:57Z" 2023-10-12T01:42:57.214 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:42:57.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:42:57Z" } 2023-10-12T01:42:57.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:42:57.216 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:57.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:42:57.216 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:02.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:02.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:02.219 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:02.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:02.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:02.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:02.313+0000 7fbe932ad780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:02.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:02.314+0000 7fbe932ad780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:02.328 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:02.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:02.340 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:02.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:02.340 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:02.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:02.341 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:02.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:02.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:02.341 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:02.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:02.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:02.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:02.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:02.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.343 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:02.343 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:02.344 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:02.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:02.345 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:02Z" 2023-10-12T01:43:02.345 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:02.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:02Z" } 2023-10-12T01:43:02.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:02.347 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:02.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:02.349 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:07.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:07.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:07.352 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:07.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:07.409 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:07.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:07.440+0000 7f19333bc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:07.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:07.441+0000 7f19333bc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:07.459 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:07.470 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:07.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:07.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:07.470 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:07.471 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:07.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:07.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:07.471 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:07.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.472 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:07.472 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:07.472 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:07.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:07.473 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:07Z" 2023-10-12T01:43:07.473 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:07.474 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:07Z" } 2023-10-12T01:43:07.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:07.475 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:07.475 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:07.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:12.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:12.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:12.478 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:12.538 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:12.538 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:12.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:12.567+0000 7ff9e4ecc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:12.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:12.568+0000 7ff9e4ecc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:12.582 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:12.582 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:12Z" 2023-10-12T01:43:12.596 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:12.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:12Z" } 2023-10-12T01:43:12.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:12.598 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:12.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:12.598 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:17.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:17.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:17.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:17.692+0000 7f0719b8c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:17.693+0000 7f0719b8c780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:17.708 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:17.708 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:17.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:17.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:17.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:17.713 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:17Z" 2023-10-12T01:43:17.722 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:17.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:17Z" } 2023-10-12T01:43:17.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:17.724 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:17.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:17.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:22.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:22.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:22.727 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:22.784 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:22.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:22.817+0000 7fcbd9a53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:22.818+0000 7fcbd9a53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:22.831 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:22.832 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:22.832 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:22.832 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:22.833 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:22.833 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:22.833 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:22.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.833 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:22.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:22.834 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:22.834 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:22.834 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:22.835 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:22.835 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:22.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:22.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:22.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:22Z" 2023-10-12T01:43:22.845 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:22.846 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:22Z" } 2023-10-12T01:43:22.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:22.847 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:22.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:22.848 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:27.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:27.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:27.851 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:27.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:27.940+0000 7fa93fa4e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:27.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:27.941+0000 7fa93fa4e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:27.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:27.956 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:27.956 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:27.956 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:27.957 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:27.957 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:27.957 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:27.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.958 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:27.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:27.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:27.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:27.958 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:27.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:27.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:27.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:27.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:27.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:27.960 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:27.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:27.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:27.960 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:27.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:27.961 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:27.961 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:27.961 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:27.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:27.962 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:27.962 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:27.962 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:27.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:27Z" 2023-10-12T01:43:27.969 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:27.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:27Z" } 2023-10-12T01:43:27.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:27.971 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:27.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:27.972 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:32.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:32.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:32.975 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:33.031 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:33.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:33.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:33.061+0000 7f8dd7827780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:33.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:33.062+0000 7f8dd7827780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:33.077 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:33.077 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:33.078 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:33.078 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:33.078 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:33.078 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:33.079 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:33.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.079 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:33.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:33.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:33.080 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:33.081 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:33.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:33.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:33.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:33.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:33.082 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:33.082 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:33.084 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:33.084 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:33.084 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:33.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:33.085 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:33.085 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:33.086 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:33.086 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:33.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:33.087 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:33.087 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:33.087 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:33.088 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:33.088 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:33.088 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:33.088 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:33.088 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:33Z" 2023-10-12T01:43:33.093 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:33.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:33Z" } 2023-10-12T01:43:33.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:33.096 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:33.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:33.097 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:38.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:38.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:38.099 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:38.187+0000 7f1823292780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:38.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:38.188+0000 7f1823292780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:38.204 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:38.204 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:38.204 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:38.204 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:38.205 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:38.205 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:38.205 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:38.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.206 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:38.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:38.206 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:38.206 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:38.206 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:38.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:38.207 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:38.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:38.207 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:38.208 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:38.208 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:38.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:38.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:38.209 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:38.209 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:38.209 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:38.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:38.210 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:38.210 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:38.210 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:38.210 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:38.210 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:38.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:38.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:38.211 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:38.211 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:38.212 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:38.212 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:38.212 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:38.212 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:38.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:38.213 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:38.213 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:38.213 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:38.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:38.214 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:38.214 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:38.214 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:38.214 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:38.215 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:38.215 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:38.215 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:38.215 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:38.215 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:38.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:38.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:38.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:38.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.217 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:38.217 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:38.217 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:38.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:38.217 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:38Z" 2023-10-12T01:43:38.218 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:38.219 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:38Z" } 2023-10-12T01:43:38.220 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:38.220 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:38.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:38.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:43.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:43.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:43.223 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:43.281 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:43.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:43.313+0000 7fa7c3cc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:43.314+0000 7fa7c3cc6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:43.329 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:43.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:43.329 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:43.329 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:43.330 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:43.330 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:43.330 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:43.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.331 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:43.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:43.331 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:43.331 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:43.331 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:43.332 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:43.332 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:43.332 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:43.332 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:43.333 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:43.333 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:43.333 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:43.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:43.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:43.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:43.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:43.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:43.334 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:43.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:43.335 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:43.335 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:43.335 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:43.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:43.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:43.336 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:43.336 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:43.337 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:43.337 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:43.337 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:43.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:43.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:43.338 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:43.338 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:43.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:43.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:43.339 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:43.339 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:43.339 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:43.339 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:43.339 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:43.340 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:43.340 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:43.340 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:43.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:43.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:43.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:43.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:43.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.341 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:43.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:43.342 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:43.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:43.342 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:43Z" 2023-10-12T01:43:43.342 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:43.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:43Z" } 2023-10-12T01:43:43.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:43.344 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:43.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:43.345 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:48.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:48.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:48.348 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:48.404 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:48.404 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:48.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:48.441+0000 7f64293e0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:48.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:48.442+0000 7f64293e0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:48.456 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:48.457 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:48.457 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:48.458 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:48.458 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:48.458 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:48.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:48.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.459 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:48.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:48.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:48.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:48.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:48.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:48.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:48.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:48.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:48.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:48.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:48.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:48.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:48.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:48.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:48.462 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:48.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:48.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:48.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:48.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:48.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:48.464 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:48.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:48.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:48.465 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:48.465 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:48.465 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:48.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074783.831059_854.1", 2023-10-12T01:43:48.465 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:48.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:48.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:48.466 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:43.831059Z", 2023-10-12T01:43:48.466 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:48.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:48.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:48.467 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:48.467 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:48.468 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:48.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:48.468 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:48.468 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:48.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:48.469 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:48.469 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:48.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:48.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:48.469 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:48.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.470 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:48.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:48.470 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:48.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:48.471 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:48Z" 2023-10-12T01:43:48.471 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:48.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074783.831059_854.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:43.831059Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:48Z" } 2023-10-12T01:43:48.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074783.831059_854.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:43.831059Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:48.473 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:48.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074783.831059_854.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:48.474 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697074910.188707_904.1 target=1_1697074783.831059_854.1 2023-10-12T01:43:53.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.502 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:53.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:53.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:53.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:53.593+0000 7febd7f81780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:53.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:53.594+0000 7febd7f81780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:53.608 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:53.608 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:53.608 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:53.608 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:53.608 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:53.609 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:53.609 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:53.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.609 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:53.611 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:53.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:53.611 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:53.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.612 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:53.612 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.613 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:53.613 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:53.613 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.613 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:53.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.614 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:53.615 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:53.615 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.616 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:53.616 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:53.617 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:43:53.617 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.617 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:53.617 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.618 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.188707Z", 2023-10-12T01:43:53.618 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:53.618 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.619 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:53.619 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.620 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:53.620 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:53.620 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:53.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.621 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:53.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.622 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:53.622 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:53.622 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:53.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.623 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:53Z" 2023-10-12T01:43:53.623 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:53.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074910.188707_904.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:41:50.188707Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:53Z" } 2023-10-12T01:43:53.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074910.188707_904.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.188707Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:53.627 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:53.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:53.628 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:43:53.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.629 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:53.729 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:43:53.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:43:53.730 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:43:53.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:43:53.731 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:43:53.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:43:53.732 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:43:53.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:43:53.733 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:43:53.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.733 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:43:53.734 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:43:53.735 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:43:53.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:53.737 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:53.787 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:53.788 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:53.831 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:43:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:43:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:43:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:43:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:43:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:43:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:43:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:43:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:43:53.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:43:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:43:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:43:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:43:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:43:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:43:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:43:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:43:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:53Z" 2023-10-12T01:43:53.845 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:53.846 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:43:53Z" } 2023-10-12T01:43:53.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:43:53.847 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:43:53.848 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:43:53.848 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:43:53.848 DEBUG:boto:path=/wafmoa-68/ 2023-10-12T01:43:53.849 DEBUG:boto:auth_path=/wafmoa-68/ 2023-10-12T01:43:53.849 DEBUG:boto:Method: HEAD 2023-10-12T01:43:53.850 DEBUG:boto:Path: /wafmoa-68/ 2023-10-12T01:43:53.850 DEBUG:boto:Data: 2023-10-12T01:43:53.851 DEBUG:boto:Headers: {} 2023-10-12T01:43:53.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:43:53.851 DEBUG:boto:Port: 8001 2023-10-12T01:43:53.852 DEBUG:boto:Params: {} 2023-10-12T01:43:53.852 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:43:53.853 DEBUG:boto:Token: None 2023-10-12T01:43:53.853 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:43:53 GMT /wafmoa-68/ 2023-10-12T01:43:53.854 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:cFfkor9rls6/SCe1LCNqM9zK6JA= 2023-10-12T01:43:53.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:43:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:cFfkor9rls6/SCe1LCNqM9zK6JA=', 'Content-Length': '0'} 2023-10-12T01:43:53.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086a6e11dd17180af-0065274f59-2539-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:43:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:43:53.861 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:43:53.861 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:43:53.861 DEBUG:boto:Method: HEAD 2023-10-12T01:43:53.862 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:43:53.862 DEBUG:boto:Data: 2023-10-12T01:43:53.863 DEBUG:boto:Headers: {} 2023-10-12T01:43:53.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:43:53.863 DEBUG:boto:Port: 8001 2023-10-12T01:43:53.864 DEBUG:boto:Params: {} 2023-10-12T01:43:53.864 DEBUG:boto:Token: None 2023-10-12T01:43:53.865 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:43:53 GMT /wafmoa-68/obj3 2023-10-12T01:43:53.866 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:FU2fH44oEkHpQqA4v5+2NikmXFM= 2023-10-12T01:43:53.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:43:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FU2fH44oEkHpQqA4v5+2NikmXFM=', 'Content-Length': '0'} 2023-10-12T01:43:53.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ab3a70aa6694738-0065274f59-2539-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:43:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:43:53.869 DEBUG:boto:path=/wafmoa-68/obj3 2023-10-12T01:43:53.869 DEBUG:boto:auth_path=/wafmoa-68/obj3 2023-10-12T01:43:53.870 DEBUG:boto:Method: GET 2023-10-12T01:43:53.870 DEBUG:boto:Path: /wafmoa-68/obj3 2023-10-12T01:43:53.871 DEBUG:boto:Data: 2023-10-12T01:43:53.871 DEBUG:boto:Headers: {} 2023-10-12T01:43:53.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:43:53.872 DEBUG:boto:Port: 8001 2023-10-12T01:43:53.872 DEBUG:boto:Params: {} 2023-10-12T01:43:53.873 DEBUG:boto:Token: None 2023-10-12T01:43:53.873 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:43:53 GMT /wafmoa-68/obj3 2023-10-12T01:43:53.874 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bYC7hPaRPvxjDm7oqaIzgzJcN2M= 2023-10-12T01:43:53.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:43:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bYC7hPaRPvxjDm7oqaIzgzJcN2M=', 'Content-Length': '0'} 2023-10-12T01:43:53.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:39:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-68:677f5a1d-3ab4-4b5f-af15-9890c2c79648.8915.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025de5fbbbd4548d6-0065274f59-2539-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:43:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:43:53.877 DEBUG:boto:path=/wafmoa-69/ 2023-10-12T01:43:53.877 DEBUG:boto:auth_path=/wafmoa-69/ 2023-10-12T01:43:53.877 DEBUG:boto:Method: HEAD 2023-10-12T01:43:53.878 DEBUG:boto:Path: /wafmoa-69/ 2023-10-12T01:43:53.878 DEBUG:boto:Data: 2023-10-12T01:43:53.879 DEBUG:boto:Headers: {} 2023-10-12T01:43:53.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:43:53.879 DEBUG:boto:Port: 8001 2023-10-12T01:43:53.880 DEBUG:boto:Params: {} 2023-10-12T01:43:53.880 DEBUG:boto:Token: None 2023-10-12T01:43:53.881 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:43:53 GMT /wafmoa-69/ 2023-10-12T01:43:53.881 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:foX7S2KISEgTTwyzjgUT7u7u/D4= 2023-10-12T01:43:53.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:43:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:foX7S2KISEgTTwyzjgUT7u7u/D4=', 'Content-Length': '0'} 2023-10-12T01:43:53.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c2fd2743e85e8be-0065274f59-2539-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:43:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:43:53.884 DEBUG:boto:path=/wafmoa-69/obj3 2023-10-12T01:43:53.885 DEBUG:boto:auth_path=/wafmoa-69/obj3 2023-10-12T01:43:53.885 DEBUG:boto:Method: HEAD 2023-10-12T01:43:53.886 DEBUG:boto:Path: /wafmoa-69/obj3 2023-10-12T01:43:53.887 DEBUG:boto:Data: 2023-10-12T01:43:53.887 DEBUG:boto:Headers: {} 2023-10-12T01:43:53.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:43:53.888 DEBUG:boto:Port: 8001 2023-10-12T01:43:53.888 DEBUG:boto:Params: {} 2023-10-12T01:43:53.889 DEBUG:boto:Token: None 2023-10-12T01:43:53.889 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:43:53 GMT /wafmoa-69/obj3 2023-10-12T01:43:53.890 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:oYLCHocT4obVV85QAYOlMInFd6c= 2023-10-12T01:43:53.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:43:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:oYLCHocT4obVV85QAYOlMInFd6c=', 'Content-Length': '0'} 2023-10-12T01:43:53.892 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ff2309b6ae1d4bbd-0065274f59-2539-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:43:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:43:53.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:43:53.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:43:53.893 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket wafmoa-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:43:53.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:53.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:53.993 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:43:53.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:43:53.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:43:53.995 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:43:54.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:54.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:54.090 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:43:54.091 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-10-12T01:43:54.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:54.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:54.093 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:54.141 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:54.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:54.172 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:43:54.185 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:43:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:43:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:43:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.187 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075033.979983_906.1", 2023-10-12T01:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:43:53.979983Z" 2023-10-12T01:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:43:54.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.189 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:43:54.190 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:54.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:43:54.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:54.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:54.192 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:54.241 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:54.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:54.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:54.271+0000 7f27e1b61780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:54.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:54.272+0000 7f27e1b61780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:54.284 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:54.285 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:54.285 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:54.285 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:54.285 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:54.286 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:54.286 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:54.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.286 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:54.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:54.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:54.290 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:43:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.188707Z", 2023-10-12T01:43:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:54.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:54.294 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:54.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:54.296 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:54.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:54.298 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:54Z" 2023-10-12T01:43:54.298 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:54.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074910.188707_904.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:41:50.188707Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:54Z" } 2023-10-12T01:43:54.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074910.188707_904.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.188707Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:54.300 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:54.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:54.301 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075033.979983_906.1 target=1_1697074910.188707_904.1 2023-10-12T01:43:59.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:59.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:43:59.303 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:43:59.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:43:59.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:43:59.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:59.398+0000 7f39ab76a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:59.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:43:59.400+0000 7f39ab76a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:43:59.413 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:43:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:43:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:43:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:43:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:43:59.414 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:43:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:43:59.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:43:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:43:59.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:43:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:43:59.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:59.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:43:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:43:59.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:43:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:59.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:43:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:59.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:59.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:43:59.422 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:59.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:59.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:59.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.188707Z", 2023-10-12T01:43:59.423 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:59.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:59.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:43:59.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:43:59.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:43:59.424 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:43:59.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:43:59.424 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:43:59.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:43:59.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:43:59.425 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:43:59.425 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:43:59.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:59.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:43:59.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:43:59.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:43:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:43:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:43:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:43:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:43:59Z" 2023-10-12T01:43:59.428 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:43:59.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074910.188707_904.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:41:50.188707Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:43:59Z" } 2023-10-12T01:43:59.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074910.188707_904.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.188707Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:43:59.430 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:59.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:43:59.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075033.979983_906.1 target=1_1697074910.188707_904.1 2023-10-12T01:44:04.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:04.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:04.433 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:04.490 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:04.490 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:04.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:04.521+0000 7fd4a8d3b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:04.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:04.522+0000 7fd4a8d3b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:04.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:04.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:44:04.544 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.188707Z", 2023-10-12T01:44:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:04.546 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:04.547 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:04.548 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:04.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:04.550 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:04Z" 2023-10-12T01:44:04.550 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:04.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074910.188707_904.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:41:50.188707Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:04Z" } 2023-10-12T01:44:04.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074910.188707_904.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.188707Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:04.552 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:04.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:04.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075033.979983_906.1 target=1_1697074910.188707_904.1 2023-10-12T01:44:09.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:09.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:09.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:09.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:09.613 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:09.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:09.644+0000 7f7b005ec780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:09.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:09.645+0000 7f7b005ec780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:09.659 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:09.660 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.661 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:09.662 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:09.663 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:09.664 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:09.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:09.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:09.667 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697074910.188707_904.1", 2023-10-12T01:44:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:09.668 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.188707Z", 2023-10-12T01:44:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:09.669 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:09.670 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:09.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:09.672 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:09.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:09.673 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:09Z" 2023-10-12T01:44:09.673 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:09.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697074910.188707_904.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:41:50.188707Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:09Z" } 2023-10-12T01:44:09.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074910.188707_904.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.188707Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:09.675 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:09.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074910.188707_904.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:09.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075033.979983_906.1 target=1_1697074910.188707_904.1 2023-10-12T01:44:14.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:14.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:14.678 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:14.736 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:14.736 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:14.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:14.767+0000 7f4d3134a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:14.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:14.769+0000 7f4d3134a780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:14.783 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:14.784 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:14.784 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:14.784 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:14.784 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:14.784 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:14.785 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:14.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.785 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:14.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:14.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:14.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:14.786 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:14.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:14.786 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:14.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:14.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:14.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:14.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:14.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:14.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:14.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:14.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075033.979983_906.1", 2023-10-12T01:44:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:43:53.979983Z", 2023-10-12T01:44:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:14.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:14.798 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:14Z" 2023-10-12T01:44:14.798 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:14.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075033.979983_906.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:43:53.979983Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:14Z" } 2023-10-12T01:44:14.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075033.979983_906.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:43:53.979983Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:14.800 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:14.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:14.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:44:14.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:14.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:14.803 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:14.849 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:14.849 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:14.894 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:44:14.894 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-70 2023-10-12T01:44:14.895 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:44:14.895 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:44:14.896 DEBUG:boto:Method: PUT 2023-10-12T01:44:14.897 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:44:14.897 DEBUG:boto:Data: 2023-10-12T01:44:14.898 DEBUG:boto:Headers: {} 2023-10-12T01:44:14.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:14.899 DEBUG:boto:Port: 8000 2023-10-12T01:44:14.899 DEBUG:boto:Params: {} 2023-10-12T01:44:14.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:44:14.900 DEBUG:boto:Token: None 2023-10-12T01:44:14.901 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:44:14 GMT /wafmoa-70/ 2023-10-12T01:44:14.901 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Vg3w/p3UKisUgs40ZqepSuG1GIM= 2023-10-12T01:44:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:44:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Vg3w/p3UKisUgs40ZqepSuG1GIM=', 'Content-Length': '0'} 2023-10-12T01:44:14.919 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1dd35527c3d5491-0065274f6e-2544-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:44:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:14.920 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-71 2023-10-12T01:44:14.920 DEBUG:boto:path=/wafmoa-71/ 2023-10-12T01:44:14.921 DEBUG:boto:auth_path=/wafmoa-71/ 2023-10-12T01:44:14.921 DEBUG:boto:Method: PUT 2023-10-12T01:44:14.922 DEBUG:boto:Path: /wafmoa-71/ 2023-10-12T01:44:14.922 DEBUG:boto:Data: 2023-10-12T01:44:14.922 DEBUG:boto:Headers: {} 2023-10-12T01:44:14.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:14.923 DEBUG:boto:Port: 8000 2023-10-12T01:44:14.924 DEBUG:boto:Params: {} 2023-10-12T01:44:14.924 DEBUG:boto:Token: None 2023-10-12T01:44:14.924 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:44:14 GMT /wafmoa-71/ 2023-10-12T01:44:14.925 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:osGx6MruYKVLtUe50UNAzPWgVH0= 2023-10-12T01:44:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:44:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:osGx6MruYKVLtUe50UNAzPWgVH0=', 'Content-Length': '0'} 2023-10-12T01:44:14.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077eaf93ee7790c75-0065274f6e-2544-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:44:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:14.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:14.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:14.942 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:14.997 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:14.997 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:15.028 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:44:15.041 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:44:15.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:15.042 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:44:15.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.042 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:15.043 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:44:15.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.043 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:44:15.043 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:44:14.939913Z" 2023-10-12T01:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.045 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:44:15.051 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:15.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:44:15.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:15.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:15.053 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:15.102 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:15.103 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:15.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:15.130+0000 7fb51ffcd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:15.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:15.130+0000 7fb51ffcd780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:15.144 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:15.144 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:15.145 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:15.145 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:15.145 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:15.145 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:15.146 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:15.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.146 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:15.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:15.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:15.147 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:15.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:15.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:15.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:15.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:15.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:15.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:15.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.149 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:15.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:15.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:15.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:15.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:15.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:15.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:15.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:15.151 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:15.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.152 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:15.152 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:15.152 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:15.152 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075033.979983_906.1", 2023-10-12T01:44:15.152 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:43:53.979983Z", 2023-10-12T01:44:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:15.154 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:15.154 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:15.154 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:15.155 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:15.155 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:15.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:15.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:15.155 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:15.156 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:15.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:15.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:15.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.157 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:15.157 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:15.157 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:15.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:15.158 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:15Z" 2023-10-12T01:44:15.158 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:15.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075033.979983_906.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:43:53.979983Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:15Z" } 2023-10-12T01:44:15.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075033.979983_906.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:43:53.979983Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:15.160 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:15.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075033.979983_906.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:15.161 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075054.939913_914.1 target=1_1697075033.979983_906.1 2023-10-12T01:44:20.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:20.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:20.163 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:20.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.223 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:20.254+0000 7fbeafd22780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:20.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:20.255+0000 7fbeafd22780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:20.270 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:20.270 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:20.271 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:20.271 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:20.271 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:20.271 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:20.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.272 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:20.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:20.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:20.273 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:20.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:20.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:20.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:44:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:44:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:20Z" 2023-10-12T01:44:20.289 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:20.290 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:20Z" } 2023-10-12T01:44:20.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:20.291 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:20.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:20.292 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:44:20.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.293 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.347 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.391 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:44:20.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.393 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.446 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.447 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.492 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:44:20.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-70', '--dest-bucket=wafmoa-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-70', '--dest-bucket=wafmoa-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.494 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-70 --dest-bucket=wafmoa-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.593 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-70","zones":["a1"]},"dest":{"bucket":"wafmoa-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:44:20.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.688 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-70","zones":["a1"]},"dest":{"bucket":"wafmoa-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:44:20.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.690 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.739 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.791 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18, 2023-10-12T01:44:20.792 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:44:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:44:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:44:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:20.793 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:44:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:44:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:44:20.794 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:44:20.795 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:44:20.795 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.795 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:44:20.795 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:44:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:44:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:44:20.796 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:44:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:44:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:44:20.797 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:44:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.798 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:44:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:44:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:44:20.799 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:44:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:44:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:44:20.800 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:44:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:44:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:44:20.801 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:44:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:44:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:44:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.803 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:44:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:44:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:44:20.804 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:44:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:44:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:44:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:44:20.805 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:44:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:44:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:44:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:44:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.806 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:44:20.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:44:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:44:20.808 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:44:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:44:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.810 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:44:20.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:44:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:44:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:44:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:44:20.812 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:44:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:44:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:44:20.813 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:44:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:44:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:44:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:44:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:44:20.815 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:44:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-70", 2023-10-12T01:44:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:44:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:44:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:44:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-71", 2023-10-12T01:44:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:44:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:44:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:44:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:44:20.818 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:44:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:44:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:44:20.819 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:44:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:44:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:44:20.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.822 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:44:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.824 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:44:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.825 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:44:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:44:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:44:20.826 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:44:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:44:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.827 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:44:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:44:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:44:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:44:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.828 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:44:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:44:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:44:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.829 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:44:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:44:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:44:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:44:20.830 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:44:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:44:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:44:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:44:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:44:20.831 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:44:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:44:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:44:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:44:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:44:20.833 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:44:20.834 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:44:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:44:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:44:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:44:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:44:20.835 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:44:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:44:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:44:20.836 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:44:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.837 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:44:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:44:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:44:20.838 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:44:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:44:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.839 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:44:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:44:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:44:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:44:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.840 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:44:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:44:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:44:20.841 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:44:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:44:20.842 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:44:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:44:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:44:20.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:44:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:44:20.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:44:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:44:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:44:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:44:20.848 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:44:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:44:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:44:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:44:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:44:20.849 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:44:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:44:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.850 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:44:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:44:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:44:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:44:20.851 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:44:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:44:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:44:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:44:20.852 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:44:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:44:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:44:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:44:20.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:44:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:44:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:44:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:44:20.854 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:44:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:44:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.855 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:44:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:44:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:44:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:44:20.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:44:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:44:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:20.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:44:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:44:20.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:20.858 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:20.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:44:20.860 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:44:20.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:20.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:20.952 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-70","zones":["a1"]},"dest":{"bucket":"wafmoa-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:44:20.953 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:44:20.953 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:44:20.954 DEBUG:boto:Method: HEAD 2023-10-12T01:44:20.954 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:44:20.955 DEBUG:boto:Data: 2023-10-12T01:44:20.955 DEBUG:boto:Headers: {} 2023-10-12T01:44:20.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:20.956 DEBUG:boto:Port: 8000 2023-10-12T01:44:20.956 DEBUG:boto:Params: {} 2023-10-12T01:44:20.957 DEBUG:boto:Token: None 2023-10-12T01:44:20.957 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:44:20 GMT /wafmoa-70/ 2023-10-12T01:44:20.958 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:mGGPMY0EFfLXpncoXckPfImJWOw= 2023-10-12T01:44:20.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:44:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mGGPMY0EFfLXpncoXckPfImJWOw=', 'Content-Length': '0'} 2023-10-12T01:44:21.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eef82f1693d9e742-0065274f75-26df-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:21.057 DEBUG:boto:path=/wafmoa-70/obj1 2023-10-12T01:44:21.058 DEBUG:boto:auth_path=/wafmoa-70/obj1 2023-10-12T01:44:21.058 DEBUG:boto:Method: PUT 2023-10-12T01:44:21.059 DEBUG:boto:Path: /wafmoa-70/obj1 2023-10-12T01:44:21.059 DEBUG:boto:Data: 2023-10-12T01:44:21.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:44:21.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:21.060 DEBUG:boto:Port: 8000 2023-10-12T01:44:21.061 DEBUG:boto:Params: {} 2023-10-12T01:44:21.061 DEBUG:boto:Token: None 2023-10-12T01:44:21.062 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:44:21 GMT /wafmoa-70/obj1 2023-10-12T01:44:21.062 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8rArS25ojcB0pW8rzCGXQ65OZ9s= 2023-10-12T01:44:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:44:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8rArS25ojcB0pW8rzCGXQ65OZ9s='} 2023-10-12T01:44:21.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018b638e9e3cc8ca3-0065274f75-26df-a1'), ('Date', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:21.083 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:44:21.084 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:44:21.084 DEBUG:boto:Method: HEAD 2023-10-12T01:44:21.084 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:44:21.085 DEBUG:boto:Data: 2023-10-12T01:44:21.085 DEBUG:boto:Headers: {} 2023-10-12T01:44:21.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:21.086 DEBUG:boto:Port: 8000 2023-10-12T01:44:21.086 DEBUG:boto:Params: {} 2023-10-12T01:44:21.087 DEBUG:boto:Token: None 2023-10-12T01:44:21.087 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:44:21 GMT /wafmoa-70/ 2023-10-12T01:44:21.088 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:eSvnFWybRD0zaJLvbNlfxU4gSkY= 2023-10-12T01:44:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:44:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eSvnFWybRD0zaJLvbNlfxU4gSkY=', 'Content-Length': '0'} 2023-10-12T01:44:21.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afecb0e2f8f20c42-0065274f75-26df-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:21.092 DEBUG:boto:path=/wafmoa-70/obj2 2023-10-12T01:44:21.092 DEBUG:boto:auth_path=/wafmoa-70/obj2 2023-10-12T01:44:21.093 DEBUG:boto:Method: PUT 2023-10-12T01:44:21.093 DEBUG:boto:Path: /wafmoa-70/obj2 2023-10-12T01:44:21.093 DEBUG:boto:Data: 2023-10-12T01:44:21.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:44:21.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:44:21.095 DEBUG:boto:Port: 8000 2023-10-12T01:44:21.095 DEBUG:boto:Params: {} 2023-10-12T01:44:21.096 DEBUG:boto:Token: None 2023-10-12T01:44:21.096 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:44:21 GMT /wafmoa-70/obj2 2023-10-12T01:44:21.096 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:pjj7PfrelZ35FCOIqqRi/fdnm6Y= 2023-10-12T01:44:21.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:44:21 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pjj7PfrelZ35FCOIqqRi/fdnm6Y='} 2023-10-12T01:44:21.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029ca2eac8a547186-0065274f75-26df-a1'), ('Date', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:44:21.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.144 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:21.216 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:21.217 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:44:21.370 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:44:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:44:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:44:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:44:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:44:14.939913Z" 2023-10-12T01:44:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:44:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.374 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:44:21.375 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:21.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:44:21.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.376 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:21.458 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:21.459 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:21.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:21.557+0000 7f73b30cf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:21.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:44:21.566+0000 7f73b30cf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:44:21.607 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:44:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:44:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:44:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:44:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:44:21.613 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:44:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:21.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.615 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:44:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.616 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:44:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:44:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:44:21.618 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:44:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:44:21.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:21.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:44:21.621 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:44:21.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.621 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:21Z" 2023-10-12T01:44:21.621 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:21.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:44:21Z" } 2023-10-12T01:44:21.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:44:21.625 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:21.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:44:21.626 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:44:21.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.627 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:21.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:21.696 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:21.771 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:44:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:44:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:44:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:44:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:44:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.775 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:44:21.776 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:21.776 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:44:21.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:21.778 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:21.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:21.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:21.888 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:21.889 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:21.889 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:21.889 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:21.890 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:21.890 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:21.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.890 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:21.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:21.891 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.891 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:21.891 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:21.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:21.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.892 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:21.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.893 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:21.893 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.894 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:21.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:21.894 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.894 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:21.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:21.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:21.896 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.896 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:21.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:21.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:21.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:21.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:21.898 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:21.899 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:21.899 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:21.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:21.899 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:21.900 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:21.900 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:21.900 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:21.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:21.901 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:21.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.901 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:21.902 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:21.902 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:21.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:21.902 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:21Z" 2023-10-12T01:44:21.902 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:21.904 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:21Z" } 2023-10-12T01:44:21.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:21.905 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:21.905 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:21.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:26.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:26.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:26.908 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:26.967 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:26.967 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:27.016 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:27.017 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:27.017 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:27.017 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:27.017 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:27.018 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:27.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.018 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:27.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:27.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:27.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:27.019 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:27.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:27.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:27.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:27.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:27.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:27.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:27.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:27.021 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:27.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:27.021 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:27.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:27.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:27.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:27.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:27.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:27.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:27.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:27.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:27.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:27.024 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:27.024 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:27.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:27.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:27.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:27.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:27.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:27.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:27.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:27.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:27.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:27.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:27.027 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:27.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:27.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:27.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:27.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:27.029 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:27.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:27.029 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:27Z" 2023-10-12T01:44:27.029 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:27.030 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:27Z" } 2023-10-12T01:44:27.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:27.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:27.032 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:27.033 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:32.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:32.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:32.035 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:32.093 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:32.094 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:32.143 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:32.144 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:32.144 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:32.144 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:32Z" 2023-10-12T01:44:32.158 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:32.160 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:32Z" } 2023-10-12T01:44:32.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:32.161 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:32.161 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:32.162 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:37.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:37.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:37.164 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:37.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:37.223 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:37.270 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:37.271 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:37.271 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:37.271 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:37.271 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:37.272 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:37.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.272 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:37.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:37.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:37.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:37.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:37.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:37.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:37.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:37.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:37.275 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:37.275 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:37.275 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:37.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:37.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:37.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:37.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:37.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:37.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:37.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:37.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:37.277 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:37.278 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:37.278 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:37.278 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:37.278 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:37.278 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:37.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:37.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:37.279 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:37.280 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:37.280 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:37.280 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:37.280 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:37.280 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:37.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.282 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:37.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:37.282 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:37.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:37.283 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:37Z" 2023-10-12T01:44:37.283 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:37.284 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:37Z" } 2023-10-12T01:44:37.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:37.285 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:37.285 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:37.286 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:42.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:42.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:42.288 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:42.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:42.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:42.396 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:42.396 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:42.396 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:42.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:42.397 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:42.397 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:42.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.398 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:42.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:42.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:42.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:42.398 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:42.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:42.399 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:42.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:42.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:42.400 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:42.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:42.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:42.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:42.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:42.401 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:42.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:42.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:42.401 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:42.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:42.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:42.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:42.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:42.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:42.403 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:42.403 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:42.403 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:42.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:42.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:42.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:42.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:42.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:42.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:42.406 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:42.406 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:42.406 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:42.406 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:42.407 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:42.407 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:42.407 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:42.407 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:42.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:42.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:42.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:42.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.408 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:42.409 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:42.409 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:42.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:42.409 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:42Z" 2023-10-12T01:44:42.409 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:42.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:42Z" } 2023-10-12T01:44:42.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:42.411 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:42.412 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:42.412 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:47.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:47.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:47.415 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:47.475 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:47.475 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:47.525 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:47.526 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:47.526 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:47.526 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:47.527 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:47.527 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:47.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.528 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:47.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:47.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:47.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:47.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:47.529 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:47.529 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:47.529 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:47.530 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:47.530 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:47.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:47.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:47.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:47.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:47.532 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:47.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:47.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:47.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:47.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:47.533 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:47.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:47.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:47.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:47.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:47.535 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:47.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:47.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:47.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:47.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:47.536 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:47.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:47.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:47.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:47.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:47.537 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:47.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:47.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:47.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:47.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:47.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:47.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:47.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:47.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:47.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.540 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:47.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:47.541 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:47.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:47.541 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:47Z" 2023-10-12T01:44:47.542 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:47.543 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:47Z" } 2023-10-12T01:44:47.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:47.544 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:47.544 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:47.545 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:52.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:52.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:52.547 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:52.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:52.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:52.654 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:52.654 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:52.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:52.656 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:52.657 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:52.659 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:52.660 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:52.661 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:52.662 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:52.663 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:52.664 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:52.665 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:52.666 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:52.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:52.668 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:52Z" 2023-10-12T01:44:52.668 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:52.669 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:52Z" } 2023-10-12T01:44:52.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:52.670 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:52.670 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:52.671 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:44:57.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:57.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:44:57.673 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:44:57.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:44:57.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:44:57.784 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:44:57.784 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:44:57.784 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:44:57.785 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:44:57.785 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:44:57.785 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:44:57.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.785 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:44:57.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:57.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:44:57.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:57.786 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:57.787 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:44:57.787 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:57.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:44:57.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:57.788 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:44:57.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:57.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:57.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:44:57.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:57.789 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:57.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:44:57.789 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:57.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:44:57.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:57.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:44:57.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:57.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:57.791 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:44:57.791 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:57.792 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:57.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:44:57.792 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:57.792 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:57.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:57.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:44:57.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:57.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:44:57.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:44:57.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:44:57.794 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:44:57.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:44:57.795 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:44:57.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:44:57.795 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:44:57.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:44:57.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:57.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:44:57.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:44:57.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.796 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:44:57.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:44:57.797 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:44:57.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:44:57.797 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:44:57Z" 2023-10-12T01:44:57.798 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:44:57.799 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:44:57Z" } 2023-10-12T01:44:57.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:44:57.800 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:44:57.800 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:44:57.801 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:02.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:02.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:02.803 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:02.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:02.863 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:02.911 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:02.912 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:02.912 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:02.912 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:02.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:02.914 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:02.915 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:02.916 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:02.917 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:02.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:02.919 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:02.920 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:02.922 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:02.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:02.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:02.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:02.926 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:02Z" 2023-10-12T01:45:02.926 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:02.927 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:02Z" } 2023-10-12T01:45:02.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:02.928 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:02.928 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:02.929 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:07.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:07.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:07.931 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:07.990 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:07.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:08.039 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:08.039 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:08.039 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:08.040 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:08.040 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:08.040 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:08.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.040 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:08.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:08.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:08.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:08.041 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:08.042 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:08.042 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:08.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:08.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:08.042 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:08.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:08.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:08.043 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:08.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:08.044 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:08.044 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:08.044 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:08.044 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:08.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:08.045 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:08.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:08.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:08.046 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:08.046 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:08.046 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:08.047 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:08.047 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:08.047 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:08.047 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:08.047 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:08.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:08.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:08.049 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:08.049 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:08.049 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:08.050 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:08.050 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:08.050 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:08.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:08.051 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:08.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:08.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:08.051 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:08.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.052 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:08.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:08.052 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:08.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:08.052 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:08Z" 2023-10-12T01:45:08.053 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:08.054 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:08Z" } 2023-10-12T01:45:08.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:08.055 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:08.055 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:08.056 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:13.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:13.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:13.058 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:13.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:13.118 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:13.166 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:13.166 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:13.167 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:13.168 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:13.169 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:13.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:13.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:13.173 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:13.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:13.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:13Z" 2023-10-12T01:45:13.179 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:13.180 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:13Z" } 2023-10-12T01:45:13.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:13.181 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:13.182 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:13.182 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:18.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:18.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:18.184 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:18.243 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:18.243 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:18.291 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:18.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:18.296 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:18.297 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:18.298 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:18.300 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:18.301 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:18.302 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:18.302 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:18.302 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:18.302 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:18.303 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:18.303 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:18.303 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:18.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:18.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:18.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:18.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.304 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:18.304 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:18.305 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:18.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:18.305 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:18Z" 2023-10-12T01:45:18.305 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:18.306 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:18Z" } 2023-10-12T01:45:18.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:18.307 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:18.308 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:18.308 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:23.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:23.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:23.310 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:23.417 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:23.417 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:23.418 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:23.418 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:23.418 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:23.418 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:23.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.419 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:23.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:23.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:23.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:23.420 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:23.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:23.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:23.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:23.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:23.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:23.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:23.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:23.422 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:23.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:23.422 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:23.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:23.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:23.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:23.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:23.423 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:23.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:23.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:23.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:23.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:23.424 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:23.425 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:23.425 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:23.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:23.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:23.425 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:23.427 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:23Z" 2023-10-12T01:45:23.433 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:23.435 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:23Z" } 2023-10-12T01:45:23.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:23.436 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:23.436 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:23.436 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:28.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:28.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:28.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:28.499 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:28.548 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:28.548 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:28.548 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:28.548 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:28.549 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:28.549 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:28.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.549 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:28.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:28.550 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:28.550 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:28.550 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:28.551 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:28.551 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:28.551 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:28.552 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:28.552 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:28.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:28.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:28.553 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:28.553 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:28.553 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:28.554 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:28.554 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:28.555 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:28.556 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:28.556 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:28.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:28.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:28.557 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:28.557 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:28.557 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:28.558 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:28.558 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:28.558 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:28.558 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:28.561 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:28.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:28.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:28.562 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:28.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:28.563 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:28.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:28.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:28.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:28.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:28.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:28.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:28.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:28.564 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:28.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.565 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:28.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:28.565 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:28.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:28.566 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:28Z" 2023-10-12T01:45:28.566 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:28.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:28Z" } 2023-10-12T01:45:28.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:28.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:28.569 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:28.569 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:33.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:33.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:33.571 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:33.679 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:33.679 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:33.680 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:33.680 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:33.680 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:33.680 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:33.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.681 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:33.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:33.681 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:33.681 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:33.682 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:33.682 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:33.682 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:33.682 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:33.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:33.683 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:33.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:33.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:33.686 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:33.687 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:33.687 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:33.687 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:33.687 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:33.688 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:33.688 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:33.688 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:33.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:33.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:33.689 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:33.689 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:33.689 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:33.690 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:33.690 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:33.690 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:33.691 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:33.691 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:33.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:33.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:33.692 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:33.692 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:33.693 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:33.693 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:33.693 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:33.693 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:33.693 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:33.694 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:33.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:33.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:33.694 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:33.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.695 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:33.695 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:33.695 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:33.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:33.696 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:33Z" 2023-10-12T01:45:33.696 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:33.697 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:33Z" } 2023-10-12T01:45:33.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:33.698 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:33.698 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:33.699 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:38.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:38.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:38.701 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:38.808 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:38.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:38.811 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:38.811 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:38.811 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:38.811 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:38.812 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:38.812 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:38.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:38.812 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:38.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:38.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:38.813 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:38.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:38.814 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:38.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:38.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:38.815 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:38.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:38.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:38.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:38.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:38.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:38.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:38.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:38.817 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:38.817 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:38.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:38.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:38.818 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:38.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:38.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:38.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:38.819 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:38.819 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:38.819 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:38.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:38.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:38.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:38.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:38.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:38.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:38.821 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:38.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.822 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:38.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:38.822 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:38.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:38.823 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:38Z" 2023-10-12T01:45:38.823 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:38.824 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:38Z" } 2023-10-12T01:45:38.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:38.825 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:38.826 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:38.826 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:43.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:43.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:43.828 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:43.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:43.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:43.936 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:43.946 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:43.946 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:43.946 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:43.946 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:43.947 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:43.947 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:43.947 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:43.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:43.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:43.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:43.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.948 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:43.948 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:43.948 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:43.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:43.949 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:43Z" 2023-10-12T01:45:43.949 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:43.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:43Z" } 2023-10-12T01:45:43.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:43.951 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:43.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:43.952 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:48.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:48.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:48.958 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:49.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:49.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:49.065 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:49.066 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:49.067 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:49.067 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:49.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.067 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:49.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:49.068 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:49.068 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:49.068 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:49.068 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:49.068 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:49.069 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:49.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:49.069 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:49.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:49.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:49.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:49.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-12T01:45:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:41:50.352717Z" 2023-10-12T01:45:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-12T01:45:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.110273Z" 2023-10-12T01:45:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:49Z" 2023-10-12T01:45:49.079 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:49.080 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:41:50.352717Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.110273Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:49Z" } 2023-10-12T01:45:49.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:41:50.352717Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.110273Z'}}] 2023-10-12T01:45:49.081 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:49.081 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051530', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045799'} 2023-10-12T01:45:49.082 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051670 target=00000000000000000000:00000000000000051530, shard=3 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-12T01:45:54.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:54.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:54.084 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:54.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.143 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.191 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:54.193 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:45:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.194 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:54.195 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:45:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:54.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:45:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:45:54.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:54.199 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:45:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:45:54.200 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:45:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:45:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:45:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:45:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:45:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:54.208 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:45:54.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.210 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:45:54.211 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:45:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.212 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:54Z" 2023-10-12T01:45:54.212 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:54.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:45:54Z" } 2023-10-12T01:45:54.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:45:54.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:54.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:45:54.215 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:45:54.216 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:45:54.216 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:45:54.217 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.217 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:45:54.218 DEBUG:boto:Data: 2023-10-12T01:45:54.218 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.219 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.220 DEBUG:boto:Params: {} 2023-10-12T01:45:54.220 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:45:54.221 DEBUG:boto:Token: None 2023-10-12T01:45:54.222 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-70/ 2023-10-12T01:45:54.222 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:bOyxwe1BAXA9JO18i03681jRexk= 2023-10-12T01:45:54.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:bOyxwe1BAXA9JO18i03681jRexk=', 'Content-Length': '0'} 2023-10-12T01:45:54.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000549d5074f5a7018-0065274fd2-26e3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.229 DEBUG:boto:path=/wafmoa-70/obj1 2023-10-12T01:45:54.229 DEBUG:boto:auth_path=/wafmoa-70/obj1 2023-10-12T01:45:54.230 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.230 DEBUG:boto:Path: /wafmoa-70/obj1 2023-10-12T01:45:54.231 DEBUG:boto:Data: 2023-10-12T01:45:54.231 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.232 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.233 DEBUG:boto:Params: {} 2023-10-12T01:45:54.234 DEBUG:boto:Token: None 2023-10-12T01:45:54.234 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-70/obj1 2023-10-12T01:45:54.235 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:gI8jyvGS43jA5k6Oo5K6CJ5X0hU= 2023-10-12T01:45:54.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gI8jyvGS43jA5k6Oo5K6CJ5X0hU=', 'Content-Length': '0'} 2023-10-12T01:45:54.238 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006cef8f281382918b-0065274fd2-26e3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.239 DEBUG:boto:path=/wafmoa-70/obj2 2023-10-12T01:45:54.240 DEBUG:boto:auth_path=/wafmoa-70/obj2 2023-10-12T01:45:54.240 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.241 DEBUG:boto:Path: /wafmoa-70/obj2 2023-10-12T01:45:54.242 DEBUG:boto:Data: 2023-10-12T01:45:54.243 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.244 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.244 DEBUG:boto:Params: {} 2023-10-12T01:45:54.245 DEBUG:boto:Token: None 2023-10-12T01:45:54.245 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-70/obj2 2023-10-12T01:45:54.246 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:OIClS1xNw1TYZg/rYBgLYVpkiAE= 2023-10-12T01:45:54.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OIClS1xNw1TYZg/rYBgLYVpkiAE=', 'Content-Length': '0'} 2023-10-12T01:45:54.248 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000356829ecbd911f96-0065274fd2-26e3-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.248 DEBUG:boto:path=/wafmoa-71/ 2023-10-12T01:45:54.249 DEBUG:boto:auth_path=/wafmoa-71/ 2023-10-12T01:45:54.249 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.250 DEBUG:boto:Path: /wafmoa-71/ 2023-10-12T01:45:54.250 DEBUG:boto:Data: 2023-10-12T01:45:54.250 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.251 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.252 DEBUG:boto:Params: {} 2023-10-12T01:45:54.252 DEBUG:boto:Token: None 2023-10-12T01:45:54.253 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-71/ 2023-10-12T01:45:54.253 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9ll/OEViO1V8YU2RAwhzCVDJ9gc= 2023-10-12T01:45:54.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9ll/OEViO1V8YU2RAwhzCVDJ9gc=', 'Content-Length': '0'} 2023-10-12T01:45:54.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c5b6c5d7ae18a43-0065274fd2-26e3-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.257 DEBUG:boto:path=/wafmoa-71/obj1 2023-10-12T01:45:54.258 DEBUG:boto:auth_path=/wafmoa-71/obj1 2023-10-12T01:45:54.258 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.259 DEBUG:boto:Path: /wafmoa-71/obj1 2023-10-12T01:45:54.260 DEBUG:boto:Data: 2023-10-12T01:45:54.260 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.262 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.262 DEBUG:boto:Params: {} 2023-10-12T01:45:54.263 DEBUG:boto:Token: None 2023-10-12T01:45:54.264 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-71/obj1 2023-10-12T01:45:54.264 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Laso9N9vW85WKXSfxIbeY+CN+X0= 2023-10-12T01:45:54.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Laso9N9vW85WKXSfxIbeY+CN+X0=', 'Content-Length': '0'} 2023-10-12T01:45:54.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000820f309b5e4673b-0065274fd2-26e3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.268 DEBUG:boto:path=/wafmoa-71/obj1 2023-10-12T01:45:54.268 DEBUG:boto:auth_path=/wafmoa-71/obj1 2023-10-12T01:45:54.269 DEBUG:boto:Method: GET 2023-10-12T01:45:54.269 DEBUG:boto:Path: /wafmoa-71/obj1 2023-10-12T01:45:54.270 DEBUG:boto:Data: 2023-10-12T01:45:54.270 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.271 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.272 DEBUG:boto:Params: {} 2023-10-12T01:45:54.272 DEBUG:boto:Token: None 2023-10-12T01:45:54.273 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-71/obj1 2023-10-12T01:45:54.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Qd4TkvooOuKaSFW//ZIoKTyPyIc= 2023-10-12T01:45:54.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Qd4TkvooOuKaSFW//ZIoKTyPyIc=', 'Content-Length': '0'} 2023-10-12T01:45:54.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e11ee2c3e11b9fc-0065274fd2-26e3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.277 DEBUG:boto:path=/wafmoa-71/obj2 2023-10-12T01:45:54.277 DEBUG:boto:auth_path=/wafmoa-71/obj2 2023-10-12T01:45:54.277 DEBUG:boto:Method: HEAD 2023-10-12T01:45:54.278 DEBUG:boto:Path: /wafmoa-71/obj2 2023-10-12T01:45:54.278 DEBUG:boto:Data: 2023-10-12T01:45:54.279 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.280 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.280 DEBUG:boto:Params: {} 2023-10-12T01:45:54.280 DEBUG:boto:Token: None 2023-10-12T01:45:54.281 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-71/obj2 2023-10-12T01:45:54.281 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:aHVQLj+5XHSgNEGiKY4rEoAd56E= 2023-10-12T01:45:54.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:aHVQLj+5XHSgNEGiKY4rEoAd56E=', 'Content-Length': '0'} 2023-10-12T01:45:54.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000217b94a3eacb12f-0065274fd2-26e3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.285 DEBUG:boto:path=/wafmoa-71/obj2 2023-10-12T01:45:54.285 DEBUG:boto:auth_path=/wafmoa-71/obj2 2023-10-12T01:45:54.286 DEBUG:boto:Method: GET 2023-10-12T01:45:54.286 DEBUG:boto:Path: /wafmoa-71/obj2 2023-10-12T01:45:54.287 DEBUG:boto:Data: 2023-10-12T01:45:54.287 DEBUG:boto:Headers: {} 2023-10-12T01:45:54.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:45:54.288 DEBUG:boto:Port: 8001 2023-10-12T01:45:54.288 DEBUG:boto:Params: {} 2023-10-12T01:45:54.289 DEBUG:boto:Token: None 2023-10-12T01:45:54.289 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:45:54 GMT /wafmoa-71/obj2 2023-10-12T01:45:54.290 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:L3kHyUaMc4Q9RHIaZgy6zl9QVEo= 2023-10-12T01:45:54.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:45:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L3kHyUaMc4Q9RHIaZgy6zl9QVEo=', 'Content-Length': '0'} 2023-10-12T01:45:54.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:44:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1655a22846f6ece-0065274fd2-26e3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:45:54.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.296 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.393 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:45:54.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.395 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.447 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.448 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.494 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:45:54.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.495 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.590 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:45:54.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.593 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.642 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.642 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.693 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:54.693 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:45:54.694 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 19, 2023-10-12T01:45:54.694 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:45:54.694 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:45:54.694 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:45:54.695 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:45:54.695 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:45:54.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.695 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:45:54.695 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:45:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:45:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:45:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:45:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.697 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.698 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:45:54.698 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.698 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:45:54.698 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.699 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:45:54.699 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:45:54.699 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:45:54.699 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:45:54.699 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:45:54.700 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:45:54.700 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:45:54.700 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:45:54.700 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:45:54.700 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:45:54.701 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.701 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.701 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:45:54.702 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:45:54.702 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.702 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:45:54.702 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.703 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:45:54.703 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:45:54.703 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:45:54.703 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:45:54.703 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:45:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:45:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:45:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:45:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:45:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:45:54.705 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.705 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.706 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:45:54.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.706 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:45:54.706 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:45:54.706 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:45:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:45:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:45:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:45:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:45:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:45:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:45:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:45:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:45:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:45:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:45:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:45:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:45:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:45:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:45:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:45:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:45:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:45:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:45:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:45:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:45:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:45:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.715 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:45:54.715 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:45:54.715 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:45:54.715 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:45:54.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.716 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:45:54.716 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:45:54.716 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:45:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:45:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-10-12T01:45:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:45:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:45:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:45:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:45:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:45:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:45:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:45:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:45:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:45:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:45:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:45:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:45:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:45:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:45:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:45:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:45:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:45:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:45:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:45:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:45:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:45:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:45:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:45:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.726 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:45:54.726 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:45:54.726 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:45:54.727 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:45:54.727 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.727 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:45:54.727 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:45:54.727 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:45:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:45:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:45:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:45:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:45:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:45:54.729 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:45:54.729 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:45:54.729 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.729 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.730 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.730 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:45:54.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.730 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:45:54.731 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:45:54.731 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:45:54.731 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:45:54.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.732 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.732 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:45:54.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:45:54.733 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:45:54.733 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:45:54.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.733 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:45:54.734 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:45:54.734 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.734 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:45:54.735 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:45:54.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:45:54.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:45:54.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.736 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.736 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:45:54.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.738 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:45:54.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:45:54.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:54.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:45:54.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:45:54.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.739 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:54.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.740 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:45:54.740 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:45:54.740 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:45:54.740 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:45:54.741 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:45:54.741 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:45:54.741 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:45:54.741 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:45:54.742 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:45:54.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.742 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:45:54.742 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:45:54.743 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:45:54.743 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:45:54.743 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:45:54.743 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:45:54.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.744 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:45:54.744 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:45:54.744 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:45:54.744 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:45:54.745 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:45:54.745 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:45:54.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.745 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:45:54.745 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:45:54.746 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:45:54.746 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:45:54.747 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:45:54.747 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:45:54.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.748 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:45:54.748 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:45:54.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:45:54.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:45:54.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:45:54.750 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:45:54.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:54.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:54.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:45:54.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:45:54.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:54.751 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:54.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.754 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.803 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:54.862 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:45:54.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:54.863 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:54.931 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:54.932 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.058 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:45:55.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.060 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.139 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.287 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:45:55.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-70', '--source-bucket=*', '--dest-bucket=wafmoa-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-70', '--source-bucket=*', '--dest-bucket=wafmoa-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.289 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=wafmoa-70 '--source-bucket=*' --dest-bucket=wafmoa-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.459 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:45:55.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.460 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:55.526 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.527 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.599 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:45:55.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:45:55.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:45:55.651 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.651 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.696 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"wafmoa-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:45:55.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:55.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:55.697 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:55.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.776 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:45:55.789 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:45:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:45:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:45:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.790 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:45:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:45:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.791 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:45:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:45:55.582010Z" 2023-10-12T01:45:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:45:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:45:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.793 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:45:55.795 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:45:55.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:45:55.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:55.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:45:55.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:45:55.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:45:55.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:45:55.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:45:55.876+0000 7f5621de0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:45:55.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:45:55.876+0000 7f5621de0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:45:55.890 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:45:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:45:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:45:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:45:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:45:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:45:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:45:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:45:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:45:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:45:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:45:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:45:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:45:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:45:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:45:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:45:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:45:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:45:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:45:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:45:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:45:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:45:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:45:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:45:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:45:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:45:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:45:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:45:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:45:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:45:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:45:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:45:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:45:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:45:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:45:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:45:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:45:55Z" 2023-10-12T01:45:55.916 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:45:55.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:45:55Z" } 2023-10-12T01:45:55.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:45:55.919 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:45:55.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:45:55.920 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:00.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:00.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:00.923 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:00.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:00.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:01.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:01.012+0000 7fbc3331f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:01.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:01.013+0000 7fbc3331f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:01.028 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:01.028 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:01.029 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:01.030 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:01.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:01.033 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:01.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:01.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:01.036 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:46:01.037 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:46:01.038 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:01.039 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:01.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:01.041 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:01.042 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:01.043 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:01Z" 2023-10-12T01:46:01.043 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:01.044 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:01Z" } 2023-10-12T01:46:01.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:01.045 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:01.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:01.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:06.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:06.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:06.050 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:06.106 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:06.106 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:06.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:06.137+0000 7fb037df6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:06.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:06.138+0000 7fb037df6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:06.153 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:06.153 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:06.153 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:06.154 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:06.154 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:06.154 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:06.154 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:06.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.155 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:06.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:06.155 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:06.155 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:06.157 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:06.157 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:06.157 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:06.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:06.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:06.158 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:06.159 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:06.159 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:06.159 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:06.159 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:06.160 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:06.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:06.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:06.160 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:46:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:46:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:06.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:06.163 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:06.163 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:06.163 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:06.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:06.164 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:06.164 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:06.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:06.164 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:06.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:06.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:06.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:06.165 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:06.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.166 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:06.166 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:06.166 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:06.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:06.167 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:06Z" 2023-10-12T01:46:06.167 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:06.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:06Z" } 2023-10-12T01:46:06.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:06.169 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:06.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:06.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:11.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:11.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:11.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:11.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:11.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:11.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:11.261+0000 7f436802e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:11.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:11.262+0000 7f436802e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:11.277 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:11.277 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:11.277 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:11.278 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:11.278 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:11.278 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:11.278 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:11.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.279 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:11.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:11.280 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:11.280 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:11.281 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:11.281 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:11.282 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:11.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:11.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:11.283 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:11.283 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:11.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:11.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:11.285 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:11.285 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:11.286 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:11.286 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:11.287 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:11.287 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:11.287 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:11.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:11.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:11.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:11.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:11.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:11.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:11.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:11.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:46:11.290 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:11.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:11.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:11.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:46:11.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:11.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:11.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:11.293 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:11.293 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:11.294 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:11.294 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:11.294 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:11.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:11.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:11.295 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:11.296 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:11.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:11.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:11.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:11.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.298 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:11.298 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:11.298 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:11.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:11.299 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:11Z" 2023-10-12T01:46:11.299 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:11.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:11Z" } 2023-10-12T01:46:11.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:11.301 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:11.302 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:11.302 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:16.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:16.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:16.304 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:16.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:16.393+0000 7f996ea50780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:16.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:16.394+0000 7f996ea50780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:16.408 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:46:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:46:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:16Z" 2023-10-12T01:46:16.422 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:16.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:16Z" } 2023-10-12T01:46:16.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:16.424 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:16.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:16.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:21.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:21.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:21.427 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:21.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:21.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:21.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:21.518+0000 7f7269261780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:21.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:21.519+0000 7f7269261780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:21.532 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:21.533 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:21.533 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:21.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:21.533 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:21.533 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:21.534 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:21.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.534 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:21.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:21.535 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:21.535 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:21.535 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:21.536 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:21.536 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:21.536 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:21.536 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:21.537 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:21.537 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:21.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:21.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:21.538 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:21.538 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:21.538 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:21.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:21.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:21.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:21.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:21.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:21.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:21.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:21.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:21.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:21.541 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:21.541 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:21.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075054.939913_914.1", 2023-10-12T01:46:21.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:21.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:21.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:21.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:14.939913Z", 2023-10-12T01:46:21.543 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:21.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:21.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:21.544 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:21.544 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:21.545 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:21.545 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:21.545 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:21.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:21.545 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:21.546 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:21.546 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:21.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:21.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:21.547 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:21.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.547 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:21.547 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:21.547 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:21.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:21.548 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:21Z" 2023-10-12T01:46:21.548 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:21.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075054.939913_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:14.939913Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:21Z" } 2023-10-12T01:46:21.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075054.939913_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:14.939913Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:21.551 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:21.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075054.939913_914.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:21.551 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075155.582010_965.1 target=1_1697075054.939913_914.1 2023-10-12T01:46:26.121 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:46:26.121+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:46:26.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.554 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:26.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:26.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:26.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:26.644+0000 7f443213e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:26.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:26.645+0000 7f443213e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:26.658 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:46:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:45:55.582010Z", 2023-10-12T01:46:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:26Z" 2023-10-12T01:46:26.673 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:26.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075155.582010_965.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:45:55.582010Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:26Z" } 2023-10-12T01:46:26.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075155.582010_965.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:45:55.582010Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:26.676 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:26.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:26.676 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:46:26.677 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:46:26.677 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:46:26.678 DEBUG:boto:Method: HEAD 2023-10-12T01:46:26.678 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:46:26.679 DEBUG:boto:Data: 2023-10-12T01:46:26.679 DEBUG:boto:Headers: {} 2023-10-12T01:46:26.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:46:26.680 DEBUG:boto:Port: 8000 2023-10-12T01:46:26.680 DEBUG:boto:Params: {} 2023-10-12T01:46:26.681 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:46:26.681 DEBUG:boto:Token: None 2023-10-12T01:46:26.682 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:46:26 GMT /wafmoa-70/ 2023-10-12T01:46:26.683 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hiaOXajI8JUnr5nKYvG0J1f+L/Y= 2023-10-12T01:46:26.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:46:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hiaOXajI8JUnr5nKYvG0J1f+L/Y=', 'Content-Length': '0'} 2023-10-12T01:46:26.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013c5ddf0f7c4d098-0065274ff2-27c1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:46:26.689 DEBUG:boto:path=/wafmoa-70/obj5 2023-10-12T01:46:26.690 DEBUG:boto:auth_path=/wafmoa-70/obj5 2023-10-12T01:46:26.690 DEBUG:boto:Method: PUT 2023-10-12T01:46:26.691 DEBUG:boto:Path: /wafmoa-70/obj5 2023-10-12T01:46:26.691 DEBUG:boto:Data: 2023-10-12T01:46:26.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:46:26.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:46:26.692 DEBUG:boto:Port: 8000 2023-10-12T01:46:26.693 DEBUG:boto:Params: {} 2023-10-12T01:46:26.693 DEBUG:boto:Token: None 2023-10-12T01:46:26.694 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:46:26 GMT /wafmoa-70/obj5 2023-10-12T01:46:26.694 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:p5zR3o4aylmQ2OWPJmMGUDD0hLE= 2023-10-12T01:46:26.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:46:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:p5zR3o4aylmQ2OWPJmMGUDD0hLE='} 2023-10-12T01:46:26.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e934eb2afda6bcc6-0065274ff2-27c1-a1'), ('Date', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:46:26.703 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:46:26.704 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:46:26.704 DEBUG:boto:Method: HEAD 2023-10-12T01:46:26.705 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:46:26.705 DEBUG:boto:Data: 2023-10-12T01:46:26.705 DEBUG:boto:Headers: {} 2023-10-12T01:46:26.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:46:26.706 DEBUG:boto:Port: 8000 2023-10-12T01:46:26.707 DEBUG:boto:Params: {} 2023-10-12T01:46:26.707 DEBUG:boto:Token: None 2023-10-12T01:46:26.708 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:46:26 GMT /wafmoa-70/ 2023-10-12T01:46:26.708 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hiaOXajI8JUnr5nKYvG0J1f+L/Y= 2023-10-12T01:46:26.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:46:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hiaOXajI8JUnr5nKYvG0J1f+L/Y=', 'Content-Length': '0'} 2023-10-12T01:46:26.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af57dd50a5c1486-0065274ff2-27c1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:46:26.711 DEBUG:boto:path=/wafmoa-70/obj6 2023-10-12T01:46:26.712 DEBUG:boto:auth_path=/wafmoa-70/obj6 2023-10-12T01:46:26.713 DEBUG:boto:Method: PUT 2023-10-12T01:46:26.713 DEBUG:boto:Path: /wafmoa-70/obj6 2023-10-12T01:46:26.714 DEBUG:boto:Data: 2023-10-12T01:46:26.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:46:26.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:46:26.715 DEBUG:boto:Port: 8000 2023-10-12T01:46:26.715 DEBUG:boto:Params: {} 2023-10-12T01:46:26.716 DEBUG:boto:Token: None 2023-10-12T01:46:26.716 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:46:26 GMT /wafmoa-70/obj6 2023-10-12T01:46:26.717 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qDC6i8HOTpHIDdMJr86rFcPd2yU= 2023-10-12T01:46:26.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:46:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qDC6i8HOTpHIDdMJr86rFcPd2yU='} 2023-10-12T01:46:26.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000667d68034c7b4b3c-0065274ff2-27c1-a1'), ('Date', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:46:26.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.764 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:26.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:46:26.858 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:46:26.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:26.859 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:46:26.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:26.859 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:46:26.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.860 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:46:26.860 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:45:55.582010Z" 2023-10-12T01:46:26.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.861 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:26.861 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:46:26.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.862 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:46:26.863 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:26.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:46:26.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.864 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:26.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:26.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:26.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:26.943+0000 7fc66263b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:26.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:46:26.944+0000 7fc66263b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:46:26.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:46:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:46:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:46:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:46:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:46:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:46:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:46:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:45:55.582010Z", 2023-10-12T01:46:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:46:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:46:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:46:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:46:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:46:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:46:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:26Z" 2023-10-12T01:46:26.970 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:26.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075155.582010_965.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:45:55.582010Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:46:26Z" } 2023-10-12T01:46:26.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075155.582010_965.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:45:55.582010Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:46:26.972 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:26.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:46:26.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:46:26.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:26.974 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:27.073 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:46:27.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:27.073 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:27.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:46:27.074 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:46:27.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:27.075 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:27.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.076 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:46:27.076 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:46:27.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.076 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:46:27.077 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:27.078 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:46:27.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:27.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:27.079 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:27.177 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:27.177 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:27.177 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:27.177 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:27.178 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:27.178 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:27.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.178 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:27.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.179 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:27.179 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:27.179 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:27.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:27.180 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:27.180 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:27.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:27.180 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:27.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:27.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:27.181 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:27.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:27.182 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:27.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:27.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:27.183 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:27.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.183 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:27.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:27.184 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:27.184 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:27.184 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:27.185 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:27.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:27.185 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:27.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:27Z" 2023-10-12T01:46:27.189 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:27.191 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:27Z" } 2023-10-12T01:46:27.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:27.192 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:27.192 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:27.193 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:32.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:32.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:32.195 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:32.254 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:32.254 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:32.301 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:32.303 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:32.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.303 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:32.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:32.303 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:32.304 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:32.304 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:32.304 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:32.304 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:32.304 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:32.305 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:32.305 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:32.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:32.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:32.306 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:32.306 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:32.306 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:32.306 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:32.307 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:32.307 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:32.307 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:32.307 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:32.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:32.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:32.308 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:32.308 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:32.309 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:32.309 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:32.309 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:32.309 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:32.309 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:32.310 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:32.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:32.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:32Z" 2023-10-12T01:46:32.314 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:32.315 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:32Z" } 2023-10-12T01:46:32.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:32.316 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:32.317 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:32.317 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:37.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:37.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:37.320 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:37.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:37.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:37.431 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:37Z" 2023-10-12T01:46:37.444 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:37.445 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:37Z" } 2023-10-12T01:46:37.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:37.446 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:37.447 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:37.447 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:42.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:42.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:42.450 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:42.508 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:42.509 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:42.558 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:42.559 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:42.559 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:42.559 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:42.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:42.560 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:42.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.560 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:42.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:42.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:42.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:42.561 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:42.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:42.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:42.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:42.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:42.562 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:42.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:42.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:42.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:42.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:42Z" 2023-10-12T01:46:42.571 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:42.572 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:42Z" } 2023-10-12T01:46:42.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:42.573 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:42.574 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:42.574 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:47.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:47.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:47.577 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:47.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:47.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:47.686 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:47.686 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:47Z" 2023-10-12T01:46:47.700 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:47.701 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:47Z" } 2023-10-12T01:46:47.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:47.702 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:47.702 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:47.703 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:52.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:52.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:52.705 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:52.812 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:52.814 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:52.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.814 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:52.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:52.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:52.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:52.815 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:52.815 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:52.815 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:52.816 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:52.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:52.816 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:52.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:52.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:52.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:52.817 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:52.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:52.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:52.818 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:52.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:52.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:52.818 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:52.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:52.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:52.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:52.819 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:52.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:52.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:52.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:52.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:52.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:52.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:52.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:52.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:52.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:52.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:52.822 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:52.823 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:52.823 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:52.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:52.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:52.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:52.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:52.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:52.824 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:52.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.824 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:52.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:52.825 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:52.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:52.825 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:52Z" 2023-10-12T01:46:52.825 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:52.826 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:52Z" } 2023-10-12T01:46:52.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:52.827 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:52.828 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:52.828 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:46:57.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:57.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:46:57.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:46:57.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:46:57.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:46:57.937 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:46:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:46:57.937 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:46:57.938 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:46:57.938 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:46:57.938 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:46:57.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.938 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:46:57.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:57.939 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:46:57.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:57.939 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:57.939 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:46:57.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:57.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:46:57.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:57.940 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:46:57.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:57.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:57.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:46:57.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:57.941 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:57.942 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:46:57.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:57.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:46:57.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:57.943 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:46:57.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:57.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:57.943 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:46:57.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:57.944 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:57.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:46:57.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:57.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:57.945 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:57.945 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:46:57.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:57.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:46:57.946 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:46:57.946 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:46:57.946 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:46:57.946 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:46:57.947 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:46:57.947 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:46:57.947 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:46:57.947 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:46:57.948 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:46:57.949 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:46:57.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:46:57.949 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:46:57Z" 2023-10-12T01:46:57.949 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:46:57.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:46:57Z" } 2023-10-12T01:46:57.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:46:57.951 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:46:57.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:46:57.952 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:02.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:02.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:02.955 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:03.064 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:03.067 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:03.068 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:03.069 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:03.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:03.071 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:03.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:03.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:03.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:03.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:03.076 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:03.077 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:03Z" 2023-10-12T01:47:03.077 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:03.078 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:03Z" } 2023-10-12T01:47:03.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:03.079 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:03.079 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:03.080 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:08.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:08.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:08.082 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:08.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:08.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:08.192 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:08.193 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:08.193 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:08.193 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:08.194 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:08.194 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:08.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.194 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:08.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:08.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:08.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:08.195 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:08.195 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:08.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:08.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:08.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:08.196 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:08.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:08.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:08.197 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:08.198 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:08.198 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:08.198 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:08.198 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:08.198 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:08.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:08.199 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:08.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:08.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:08.200 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:08.200 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:08.200 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:08.201 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:08.201 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:08.201 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:08.201 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:08.201 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:08.202 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:08.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:08.202 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:08.203 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:08.203 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:08.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:08.203 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:08.203 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:08.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:08.204 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:08.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:08.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:08.205 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:08.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.205 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:08.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:08.206 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:08.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:08.206 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:08Z" 2023-10-12T01:47:08.206 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:08.207 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:08Z" } 2023-10-12T01:47:08.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:08.208 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:08.209 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:08.209 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:13.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:13.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:13.212 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:13.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:13.272 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:13.321 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:13.322 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:13.322 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:13.322 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.323 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:13.324 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:13.325 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:13.327 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:13.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:13.329 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:13.330 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:13.332 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:13.333 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:13.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:13.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:13.334 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:13.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.335 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:13.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:13.335 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:13.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:13.335 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:13Z" 2023-10-12T01:47:13.336 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:13.337 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:13Z" } 2023-10-12T01:47:13.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:13.338 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:13.339 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:13.339 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:18.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:18.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:18.343 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:18.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:18.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:18.450 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:18.456 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:18.456 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:18.456 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:18.456 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:18Z" 2023-10-12T01:47:18.464 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:18.465 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:18Z" } 2023-10-12T01:47:18.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:18.466 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:18.466 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:18.467 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:23.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:23.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:23.469 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:23.577 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:23.579 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:23.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.579 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:23.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:23.579 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:23.580 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:23.580 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:23.580 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:23.580 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:23.580 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:23.581 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:23.581 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:23.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:23.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:23.582 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:23.582 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:23.582 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:23.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:23.583 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:23.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:23.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:23.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:23.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:23.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:23.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:23.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:23.584 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:23.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:23.585 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:23.585 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:23.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:23.586 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:23.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:23.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:23.586 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:23.588 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:23.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:23.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:23.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:23.588 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:23.590 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:23Z" 2023-10-12T01:47:23.590 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:23.591 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:23Z" } 2023-10-12T01:47:23.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:23.592 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:23.592 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:23.593 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:28.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:28.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:28.596 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:28.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:28.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:28.708 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:28.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:28.708 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:28.708 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:28.709 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:28.709 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:28.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.709 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:28.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:28.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:28.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:28.710 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:28.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:28.711 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:28.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:28.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:28.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:28.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:28.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:28.712 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:28.712 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:28.713 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:28.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:28.713 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:28.713 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:28.716 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:28.716 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:28.716 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:28.716 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:28.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:28.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:28.717 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:28.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:28.718 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:28.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:28.718 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:28.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:28.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:28.719 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:28.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:28.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:28.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:28.721 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:28Z" 2023-10-12T01:47:28.721 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:28.722 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:28Z" } 2023-10-12T01:47:28.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:28.723 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:28.724 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:28.724 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:33.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:33.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:33.727 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:33.786 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:33.786 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:33.838 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:33.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:33.839 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:33.839 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:33.839 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:33.840 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:33.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.840 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:33.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:33.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:33.841 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:33.841 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:33.841 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:33.841 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:33.842 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:33.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:33.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:33.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:33.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:33.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:33.844 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:33.844 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:33.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:33.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:33.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:33.845 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:33.845 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:33.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:33.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:33.846 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:33.846 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:33.846 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:33.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:33.847 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:33.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:33.847 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:33.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:33.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:33.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:33.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:33.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:33.849 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:33.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:33.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:33.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:33.850 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:33.850 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:33.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:33.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:33.850 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:33.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.851 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:33.851 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:33.851 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:33.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:33.852 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:33Z" 2023-10-12T01:47:33.852 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:33.853 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:33Z" } 2023-10-12T01:47:33.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:33.854 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:33.855 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:33.855 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:38.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:38.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:38.857 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:38.917 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:38.917 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:38.968 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:38.968 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:38.969 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:38.969 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:38.969 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:38.969 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:38.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.970 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:38.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:38.970 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:38.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:38.971 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:38.971 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:38.971 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:38.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:38.972 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:38.972 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:38.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:38.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:38.973 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:38.973 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:38.973 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:38.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:38.973 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:38.974 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:38.974 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:38.974 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:38.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:38.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:38.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:38.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:38.975 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:38.975 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:38.976 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:38.976 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:38.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:38.976 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:38.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:38.980 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:38Z" 2023-10-12T01:47:38.980 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:38.981 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:38Z" } 2023-10-12T01:47:38.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:38.983 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:38.983 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:38.983 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:43.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:43.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:43.986 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:44.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:44.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:44.094 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:44.094 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:44.094 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:44.095 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:44.095 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:44.095 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:44.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:44.097 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:44.097 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:44.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:44.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:44.098 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:44.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:44.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:44.098 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:44.099 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:44.099 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:44.099 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:44.099 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:44.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:44.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:44.100 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:44.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:44.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:44.101 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:44.101 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:44.101 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:44.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:44.102 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:44.102 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:44.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:44.102 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:44.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:44.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:44.103 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:44.103 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:44.103 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:44.104 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:44.104 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:44.104 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:44.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:44.105 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:44.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:44.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:44.105 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:44.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.106 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:44.106 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:44.106 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:44.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:44.106 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:44Z" 2023-10-12T01:47:44.107 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:44.108 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:44Z" } 2023-10-12T01:47:44.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:44.109 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:44.109 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:44.109 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:49.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:49.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:49.112 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:49.220 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:49.221 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:49.221 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:49.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:49.221 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:49.222 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:49.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.222 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:49.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:49.223 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:49.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:49.223 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:49.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:49.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:49.224 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:49.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:49.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:49.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:49.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:49.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:49.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:49.225 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:49.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:49.226 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:49.226 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:49.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:49.226 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:49.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:49.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:49.228 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:49.228 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:49.228 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:49.228 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:49.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:49.229 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:49.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:49.229 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:49.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:49.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:49.230 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:49.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:49.230 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:49.231 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:49.231 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:49.231 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:49.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:49.232 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:49.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:49.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:49.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:49.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.233 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:49.233 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:49.233 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:49.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:49.233 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:49Z" 2023-10-12T01:47:49.234 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:49.235 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:49Z" } 2023-10-12T01:47:49.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:49.236 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:49.236 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:49.236 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:54.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:54.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:54.239 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:54.298 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:54.298 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:54.346 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:54.346 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:54.346 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:54.347 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:54.347 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:54.347 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:54.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.347 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:54.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:54.348 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:54.348 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:54.348 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:54.349 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:54.349 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:54.349 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:54.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:54.349 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:54.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:54.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:54.350 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:54.350 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:54.351 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:54.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051670", 2023-10-12T01:47:54.351 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:54.351 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:54.352 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:54.352 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.080163Z" 2023-10-12T01:47:54.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:54.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:54.353 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:54.353 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:54.353 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:54.353 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:54.353 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:54.354 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:54.354 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:54.354 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:54.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:54.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:54.355 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:54.355 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:54.355 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:54.356 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-12T01:47:54.356 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:54.356 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:54.356 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:54.356 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:44:21.109740Z" 2023-10-12T01:47:54.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:54.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:54.357 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:54.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.357 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:54.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:54.358 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:54.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:54.358 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:54Z" 2023-10-12T01:47:54.358 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:54.360 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051670", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:44:21.080163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:44:21.109740Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:54Z" } 2023-10-12T01:47:54.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051670', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.080163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:44:21.109740Z'}}] 2023-10-12T01:47:54.361 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:54.361 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051670', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000045939'} 2023-10-12T01:47:54.361 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051670, shard=3 master=00000000000000000000:00000000000000046079 target=00000000000000000000:00000000000000045939 2023-10-12T01:47:59.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.364 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:59.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:59.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:59.475 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:47:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032339", 2023-10-12T01:47:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:47:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:31:00.531428Z" 2023-10-12T01:47:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:47:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:47:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:47:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044465", 2023-10-12T01:47:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:39:45.196178Z" 2023-10-12T01:47:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:47:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:47:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:47:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:47:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:47:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:59Z" 2023-10-12T01:47:59.488 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:59.489 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032339", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:31:00.531428Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:39:45.196178Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:47:59Z" } 2023-10-12T01:47:59.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032339', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:31:00.531428Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:39:45.196178Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:47:59.490 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:59.491 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032339', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044465', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:47:59.491 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:47:59.491 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:47:59.492 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:47:59.492 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.493 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:47:59.493 DEBUG:boto:Data: 2023-10-12T01:47:59.494 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.495 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.496 DEBUG:boto:Params: {} 2023-10-12T01:47:59.497 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:47:59.497 DEBUG:boto:Token: None 2023-10-12T01:47:59.498 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-70/ 2023-10-12T01:47:59.498 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:j4Uq4+hibKbogAFG/6pCG/0lBYY= 2023-10-12T01:47:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j4Uq4+hibKbogAFG/6pCG/0lBYY=', 'Content-Length': '0'} 2023-10-12T01:47:59.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dff69fdb711afe2a-006527504f-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.505 DEBUG:boto:path=/wafmoa-70/obj5 2023-10-12T01:47:59.506 DEBUG:boto:auth_path=/wafmoa-70/obj5 2023-10-12T01:47:59.506 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.507 DEBUG:boto:Path: /wafmoa-70/obj5 2023-10-12T01:47:59.507 DEBUG:boto:Data: 2023-10-12T01:47:59.508 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.508 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.509 DEBUG:boto:Params: {} 2023-10-12T01:47:59.509 DEBUG:boto:Token: None 2023-10-12T01:47:59.510 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-70/obj5 2023-10-12T01:47:59.510 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dEAcqTNXEda1FoB5zHiax1ZS20A= 2023-10-12T01:47:59.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dEAcqTNXEda1FoB5zHiax1ZS20A=', 'Content-Length': '0'} 2023-10-12T01:47:59.512 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f5c5caaf41e83377-006527504f-27b6-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.513 DEBUG:boto:path=/wafmoa-70/obj6 2023-10-12T01:47:59.513 DEBUG:boto:auth_path=/wafmoa-70/obj6 2023-10-12T01:47:59.514 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.514 DEBUG:boto:Path: /wafmoa-70/obj6 2023-10-12T01:47:59.515 DEBUG:boto:Data: 2023-10-12T01:47:59.515 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.516 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.516 DEBUG:boto:Params: {} 2023-10-12T01:47:59.517 DEBUG:boto:Token: None 2023-10-12T01:47:59.517 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-70/obj6 2023-10-12T01:47:59.518 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:2L+BmpUpRQAJGos3MN03ivY9wEo= 2023-10-12T01:47:59.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2L+BmpUpRQAJGos3MN03ivY9wEo=', 'Content-Length': '0'} 2023-10-12T01:47:59.520 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000037b5015bba053508-006527504f-27b6-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.520 DEBUG:boto:path=/wafmoa-71/ 2023-10-12T01:47:59.521 DEBUG:boto:auth_path=/wafmoa-71/ 2023-10-12T01:47:59.521 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.522 DEBUG:boto:Path: /wafmoa-71/ 2023-10-12T01:47:59.522 DEBUG:boto:Data: 2023-10-12T01:47:59.522 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.523 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.524 DEBUG:boto:Params: {} 2023-10-12T01:47:59.524 DEBUG:boto:Token: None 2023-10-12T01:47:59.525 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-71/ 2023-10-12T01:47:59.525 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:T92tkk9ynvQOtumINHsxbIPaoBo= 2023-10-12T01:47:59.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:T92tkk9ynvQOtumINHsxbIPaoBo=', 'Content-Length': '0'} 2023-10-12T01:47:59.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd5babed257e112b-006527504f-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.528 DEBUG:boto:path=/wafmoa-71/obj5 2023-10-12T01:47:59.528 DEBUG:boto:auth_path=/wafmoa-71/obj5 2023-10-12T01:47:59.529 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.529 DEBUG:boto:Path: /wafmoa-71/obj5 2023-10-12T01:47:59.530 DEBUG:boto:Data: 2023-10-12T01:47:59.530 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.531 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.531 DEBUG:boto:Params: {} 2023-10-12T01:47:59.532 DEBUG:boto:Token: None 2023-10-12T01:47:59.532 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-71/obj5 2023-10-12T01:47:59.532 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:lrOSVTjLM95gck7IyN8b+M8G9oM= 2023-10-12T01:47:59.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lrOSVTjLM95gck7IyN8b+M8G9oM=', 'Content-Length': '0'} 2023-10-12T01:47:59.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e59cb6c390db3a5-006527504f-27b6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.536 DEBUG:boto:path=/wafmoa-71/obj5 2023-10-12T01:47:59.536 DEBUG:boto:auth_path=/wafmoa-71/obj5 2023-10-12T01:47:59.536 DEBUG:boto:Method: GET 2023-10-12T01:47:59.537 DEBUG:boto:Path: /wafmoa-71/obj5 2023-10-12T01:47:59.537 DEBUG:boto:Data: 2023-10-12T01:47:59.538 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.539 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.539 DEBUG:boto:Params: {} 2023-10-12T01:47:59.539 DEBUG:boto:Token: None 2023-10-12T01:47:59.540 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-71/obj5 2023-10-12T01:47:59.540 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yYZOrdx0TScd5yHw94uMlBWmuZA= 2023-10-12T01:47:59.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yYZOrdx0TScd5yHw94uMlBWmuZA=', 'Content-Length': '0'} 2023-10-12T01:47:59.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077c376c3a44863bf-006527504f-27b6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.544 DEBUG:boto:path=/wafmoa-71/obj6 2023-10-12T01:47:59.544 DEBUG:boto:auth_path=/wafmoa-71/obj6 2023-10-12T01:47:59.545 DEBUG:boto:Method: HEAD 2023-10-12T01:47:59.545 DEBUG:boto:Path: /wafmoa-71/obj6 2023-10-12T01:47:59.545 DEBUG:boto:Data: 2023-10-12T01:47:59.546 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.547 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.547 DEBUG:boto:Params: {} 2023-10-12T01:47:59.547 DEBUG:boto:Token: None 2023-10-12T01:47:59.548 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-71/obj6 2023-10-12T01:47:59.548 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AMReiiyj6LmJy0idF87whsrEYGA= 2023-10-12T01:47:59.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AMReiiyj6LmJy0idF87whsrEYGA=', 'Content-Length': '0'} 2023-10-12T01:47:59.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061eb43a5b1a0eb05-006527504f-27b6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.551 DEBUG:boto:path=/wafmoa-71/obj6 2023-10-12T01:47:59.552 DEBUG:boto:auth_path=/wafmoa-71/obj6 2023-10-12T01:47:59.552 DEBUG:boto:Method: GET 2023-10-12T01:47:59.552 DEBUG:boto:Path: /wafmoa-71/obj6 2023-10-12T01:47:59.553 DEBUG:boto:Data: 2023-10-12T01:47:59.553 DEBUG:boto:Headers: {} 2023-10-12T01:47:59.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:47:59.554 DEBUG:boto:Port: 8001 2023-10-12T01:47:59.555 DEBUG:boto:Params: {} 2023-10-12T01:47:59.555 DEBUG:boto:Token: None 2023-10-12T01:47:59.556 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:47:59 GMT /wafmoa-71/obj6 2023-10-12T01:47:59.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:tg38BVUcZ0wc5/uQK5dR+OTxh8Y= 2023-10-12T01:47:59.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:47:59 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tg38BVUcZ0wc5/uQK5dR+OTxh8Y=', 'Content-Length': '0'} 2023-10-12T01:47:59.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:46:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-70:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d07444b9c0e07fd-006527504f-27b6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:47:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:47:59.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:47:59.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:47:59.604 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:47:59.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:59.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:59.711 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:47:59.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.712 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:59.791 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:47:59.805 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:47:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:47:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:47:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:47:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:47:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075279.692765_967.1", 2023-10-12T01:47:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:47:59.692765Z" 2023-10-12T01:47:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:47:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:47:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.810 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:47:59.811 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:47:59.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:47:59.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:47:59.814 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:47:59.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:47:59.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:47:59.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:47:59.891+0000 7fc562d32780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:47:59.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:47:59.892+0000 7fc562d32780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:47:59.906 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:47:59.907 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:47:59.907 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:47:59.907 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:47:59.907 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:47:59.908 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:47:59.908 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:47:59.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.908 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:47:59.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:47:59.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:47:59.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:47:59.910 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:47:59.910 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.910 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:47:59.911 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:47:59.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.911 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:47:59.912 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.912 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:47:59.912 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:47:59.912 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.912 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:59.913 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.913 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:47:59.913 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:47:59.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.914 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:47:59.914 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.914 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:47:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:47:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.915 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:45:55.582010Z", 2023-10-12T01:47:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:47:59.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:47:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:47:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:47:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:47:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:47:59.918 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:47:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:47:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:47:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:47:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:47:59.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:47:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:47:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:47:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:47:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:47:59.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:47:59.922 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:47:59Z" 2023-10-12T01:47:59.922 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:47:59.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075155.582010_965.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:45:55.582010Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:47:59Z" } 2023-10-12T01:47:59.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075155.582010_965.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:45:55.582010Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:47:59.924 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:47:59.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:47:59.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075279.692765_967.1 target=1_1697075155.582010_965.1 2023-10-12T01:48:04.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:04.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:04.927 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:04.986 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:05.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:05.017+0000 7f1985473780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:05.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:05.018+0000 7f1985473780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:05.033 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:05.033 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:05.034 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:05.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:05.036 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:05.037 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:05.038 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:05.039 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:05.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:05.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075155.582010_965.1", 2023-10-12T01:48:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:05.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:45:55.582010Z", 2023-10-12T01:48:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:05.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:05.044 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:05.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:05.047 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:05.048 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:05Z" 2023-10-12T01:48:05.048 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:05.049 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075155.582010_965.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:45:55.582010Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:05Z" } 2023-10-12T01:48:05.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075155.582010_965.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:45:55.582010Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:05.050 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:05.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075155.582010_965.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:05.051 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075279.692765_967.1 target=1_1697075155.582010_965.1 2023-10-12T01:48:10.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:10.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:10.053 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:10.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.111 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:10.142+0000 7fe0a3b16780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:10.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:10.143+0000 7fe0a3b16780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:10.159 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:10.159 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:10.159 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:10.160 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:10.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:10.160 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:10.160 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:10.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.161 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:10.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:10.161 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:10.161 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:10.162 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:10.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:10.162 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:10.162 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:10.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:10.163 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:10.163 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:10.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:10.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:10.164 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:10.164 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:10.164 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:10.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:10.165 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:10.165 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:10.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:10.165 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:10.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:10.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:10.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:10.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:10.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:10.167 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:10.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075279.692765_967.1", 2023-10-12T01:48:10.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:10.169 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:10.169 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:10.169 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:47:59.692765Z", 2023-10-12T01:48:10.170 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:10.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:10.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:10.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:10.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:10.171 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:10.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:10.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:10.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:10.172 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:10.172 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:10.172 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:10.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:10.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:10.173 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:10.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.173 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:10.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:10.174 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:10.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:10.174 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:10Z" 2023-10-12T01:48:10.174 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:10.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075279.692765_967.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:47:59.692765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:10Z" } 2023-10-12T01:48:10.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075279.692765_967.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:47:59.692765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:10.177 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:10.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:10.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:48:10.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.179 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.228 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.274 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:48:10.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.276 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.377 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:48:10.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.378 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.484 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:48:10.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-70', '--source-bucket=wafmoa-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=wafmoa-70', '--source-bucket=wafmoa-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.485 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=wafmoa-70 --source-bucket=wafmoa-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.542 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.599 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:48:10.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.600 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.653 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.701 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:48:10.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:10.703 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:10.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.796 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"wafmoa-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:48:10.796 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:48:10.797 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:48:10.797 DEBUG:boto:Method: HEAD 2023-10-12T01:48:10.798 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:48:10.798 DEBUG:boto:Data: 2023-10-12T01:48:10.798 DEBUG:boto:Headers: {} 2023-10-12T01:48:10.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.799 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.800 DEBUG:boto:Params: {} 2023-10-12T01:48:10.800 DEBUG:boto:Token: None 2023-10-12T01:48:10.801 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-70/ 2023-10-12T01:48:10.801 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:61MHymDbJe1kmP36i7AFPwTTFBI= 2023-10-12T01:48:10.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:61MHymDbJe1kmP36i7AFPwTTFBI=', 'Content-Length': '0'} 2023-10-12T01:48:10.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008e1ec0d2d192d84-006527505a-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.805 DEBUG:boto:path=/wafmoa-70/obj7 2023-10-12T01:48:10.805 DEBUG:boto:auth_path=/wafmoa-70/obj7 2023-10-12T01:48:10.805 DEBUG:boto:Method: PUT 2023-10-12T01:48:10.806 DEBUG:boto:Path: /wafmoa-70/obj7 2023-10-12T01:48:10.806 DEBUG:boto:Data: 2023-10-12T01:48:10.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:10.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.808 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.808 DEBUG:boto:Params: {} 2023-10-12T01:48:10.808 DEBUG:boto:Token: None 2023-10-12T01:48:10.809 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-70/obj7 2023-10-12T01:48:10.809 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nphBF1UdtH62NPzq61WotGkZqwI= 2023-10-12T01:48:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nphBF1UdtH62NPzq61WotGkZqwI='} 2023-10-12T01:48:10.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042f51f62c9ebc53f-006527505a-27b6-a2'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.816 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:48:10.817 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:48:10.817 DEBUG:boto:Method: HEAD 2023-10-12T01:48:10.818 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:48:10.818 DEBUG:boto:Data: 2023-10-12T01:48:10.818 DEBUG:boto:Headers: {} 2023-10-12T01:48:10.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.819 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.820 DEBUG:boto:Params: {} 2023-10-12T01:48:10.820 DEBUG:boto:Token: None 2023-10-12T01:48:10.821 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-70/ 2023-10-12T01:48:10.821 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:61MHymDbJe1kmP36i7AFPwTTFBI= 2023-10-12T01:48:10.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:61MHymDbJe1kmP36i7AFPwTTFBI=', 'Content-Length': '0'} 2023-10-12T01:48:10.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093b392bc16010163-006527505a-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.824 DEBUG:boto:path=/wafmoa-70/obj8 2023-10-12T01:48:10.824 DEBUG:boto:auth_path=/wafmoa-70/obj8 2023-10-12T01:48:10.825 DEBUG:boto:Method: PUT 2023-10-12T01:48:10.825 DEBUG:boto:Path: /wafmoa-70/obj8 2023-10-12T01:48:10.826 DEBUG:boto:Data: 2023-10-12T01:48:10.826 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:10.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.827 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.827 DEBUG:boto:Params: {} 2023-10-12T01:48:10.828 DEBUG:boto:Token: None 2023-10-12T01:48:10.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-70/obj8 2023-10-12T01:48:10.829 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:W/yHNlOuauq4ksDbci3RvXTHews= 2023-10-12T01:48:10.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:W/yHNlOuauq4ksDbci3RvXTHews='} 2023-10-12T01:48:10.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e48c0540b72d42b5-006527505a-27b6-a2'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.835 DEBUG:boto:path=/wafmoa-71/ 2023-10-12T01:48:10.835 DEBUG:boto:auth_path=/wafmoa-71/ 2023-10-12T01:48:10.836 DEBUG:boto:Method: HEAD 2023-10-12T01:48:10.836 DEBUG:boto:Path: /wafmoa-71/ 2023-10-12T01:48:10.836 DEBUG:boto:Data: 2023-10-12T01:48:10.837 DEBUG:boto:Headers: {} 2023-10-12T01:48:10.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.838 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.838 DEBUG:boto:Params: {} 2023-10-12T01:48:10.839 DEBUG:boto:Token: None 2023-10-12T01:48:10.839 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-71/ 2023-10-12T01:48:10.839 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GYpos/xPhjzlmf/xWLzMUneS8Vs= 2023-10-12T01:48:10.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GYpos/xPhjzlmf/xWLzMUneS8Vs=', 'Content-Length': '0'} 2023-10-12T01:48:10.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a907c3c1d56d5500-006527505a-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.845 DEBUG:boto:path=/wafmoa-71/obj9 2023-10-12T01:48:10.845 DEBUG:boto:auth_path=/wafmoa-71/obj9 2023-10-12T01:48:10.846 DEBUG:boto:Method: PUT 2023-10-12T01:48:10.846 DEBUG:boto:Path: /wafmoa-71/obj9 2023-10-12T01:48:10.847 DEBUG:boto:Data: 2023-10-12T01:48:10.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:10.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.848 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.848 DEBUG:boto:Params: {} 2023-10-12T01:48:10.849 DEBUG:boto:Token: None 2023-10-12T01:48:10.849 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-71/obj9 2023-10-12T01:48:10.850 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GBb4P606oYiPfHi49yhfT0QIR0o= 2023-10-12T01:48:10.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GBb4P606oYiPfHi49yhfT0QIR0o='} 2023-10-12T01:48:10.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d80b44ac3779320-006527505a-27b6-a2'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.856 DEBUG:boto:path=/wafmoa-71/ 2023-10-12T01:48:10.856 DEBUG:boto:auth_path=/wafmoa-71/ 2023-10-12T01:48:10.857 DEBUG:boto:Method: HEAD 2023-10-12T01:48:10.857 DEBUG:boto:Path: /wafmoa-71/ 2023-10-12T01:48:10.857 DEBUG:boto:Data: 2023-10-12T01:48:10.858 DEBUG:boto:Headers: {} 2023-10-12T01:48:10.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.859 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.859 DEBUG:boto:Params: {} 2023-10-12T01:48:10.860 DEBUG:boto:Token: None 2023-10-12T01:48:10.860 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-71/ 2023-10-12T01:48:10.860 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:GYpos/xPhjzlmf/xWLzMUneS8Vs= 2023-10-12T01:48:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GYpos/xPhjzlmf/xWLzMUneS8Vs=', 'Content-Length': '0'} 2023-10-12T01:48:10.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000331213d2835f2a07-006527505a-27b6-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.864 DEBUG:boto:path=/wafmoa-71/obj10 2023-10-12T01:48:10.864 DEBUG:boto:auth_path=/wafmoa-71/obj10 2023-10-12T01:48:10.864 DEBUG:boto:Method: PUT 2023-10-12T01:48:10.865 DEBUG:boto:Path: /wafmoa-71/obj10 2023-10-12T01:48:10.865 DEBUG:boto:Data: 2023-10-12T01:48:10.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:10.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:48:10.867 DEBUG:boto:Port: 8001 2023-10-12T01:48:10.867 DEBUG:boto:Params: {} 2023-10-12T01:48:10.868 DEBUG:boto:Token: None 2023-10-12T01:48:10.868 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:10 GMT /wafmoa-71/obj10 2023-10-12T01:48:10.868 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ffWD3g77MpQbLrUBtvExrSst0C8= 2023-10-12T01:48:10.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:10 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ffWD3g77MpQbLrUBtvExrSst0C8='} 2023-10-12T01:48:10.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c25667c5e417676a-006527505a-27b6-a2'), ('Date', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:10.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:10.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:10.913 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:10.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:10.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:10.998 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:48:11.011 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:11.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.012 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:11.012 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:48:11.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:11.013 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:48:11.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:10.687891Z" 2023-10-12T01:48:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:11.014 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:48:11.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.015 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:11.016 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:11.017 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:48:11.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:11.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:11.018 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:11.068 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:11.069 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:11.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:11.097+0000 7f17fcb29780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:11.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:11.098+0000 7f17fcb29780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:11.112 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:11.112 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:11.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:11.114 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:11.115 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:11.116 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:11.116 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.117 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:11.117 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:11.117 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:11.117 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:11.117 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:11.118 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:11.118 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:11.118 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:11.118 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:11.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.119 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:11.119 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:11.119 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:11.120 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075279.692765_967.1", 2023-10-12T01:48:11.120 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:11.120 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:11.120 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:11.121 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:47:59.692765Z", 2023-10-12T01:48:11.121 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:11.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:11.122 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:11.123 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:11.123 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:11.123 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:11.123 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:11.124 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:11.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:11.124 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:11.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:11.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:11.125 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:11.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.125 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:11.125 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:11.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:11.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:11.126 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:11Z" 2023-10-12T01:48:11.126 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:11.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075279.692765_967.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:47:59.692765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:11Z" } 2023-10-12T01:48:11.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075279.692765_967.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:47:59.692765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:11.128 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:11.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:11.129 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075290.687891_975.1 target=1_1697075279.692765_967.1 2023-10-12T01:48:16.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:16.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:16.132 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:16.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:16.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:16.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:16.223+0000 7f3bff5e8780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:16.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:16.224+0000 7f3bff5e8780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:16.237 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:16.238 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:16.239 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:16.239 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:16.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.239 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:16.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:16.240 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:16.240 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:16.240 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:16.240 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:16.241 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:16.241 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:16.241 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:16.241 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:16.241 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:16.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:16.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:16.242 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:16.242 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:16.243 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:16.243 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:16.243 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:16.243 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:16.244 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:16.244 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:16.244 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:16.244 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:16.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:16.245 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:16.245 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:16.245 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:16.245 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075279.692765_967.1", 2023-10-12T01:48:16.246 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:16.246 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:16.246 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:16.246 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:47:59.692765Z", 2023-10-12T01:48:16.246 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:16.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:16.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:16.247 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:16.247 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:16.248 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:16.248 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:16.248 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:16.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:16.249 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:16.249 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:16.249 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:16.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:16.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:16.250 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:16.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.251 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:16.251 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:16.251 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:16.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:16.252 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:16Z" 2023-10-12T01:48:16.252 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:16.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075279.692765_967.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:47:59.692765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:16Z" } 2023-10-12T01:48:16.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075279.692765_967.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:47:59.692765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:16.254 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:16.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:16.255 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075290.687891_975.1 target=1_1697075279.692765_967.1 2023-10-12T01:48:21.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:21.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:21.257 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:21.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:21.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:21.348+0000 7fe3cebf3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:21.349+0000 7fe3cebf3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:21.362 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075279.692765_967.1", 2023-10-12T01:48:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:47:59.692765Z", 2023-10-12T01:48:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:21.374 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:21.374 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:21.374 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:21.374 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:21.375 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:21.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:21.375 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:21.375 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:21.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:21.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:21.376 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:21.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.376 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:21.377 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:21.377 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:21.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:21.377 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:21Z" 2023-10-12T01:48:21.377 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:21.379 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075279.692765_967.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:47:59.692765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:21Z" } 2023-10-12T01:48:21.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075279.692765_967.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:47:59.692765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:21.380 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:21.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075279.692765_967.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:21.381 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075290.687891_975.1 target=1_1697075279.692765_967.1 2023-10-12T01:48:26.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.383 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:26.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:26.442 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:26.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:26.471+0000 7ff239891780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:26.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:26.472+0000 7ff239891780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:26.485 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:26.486 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:26.486 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:26.486 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:26.487 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:26.487 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:26.487 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:26.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.488 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:26.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.488 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:26.489 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.489 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:26.489 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:26.490 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.490 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:26.490 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.491 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:26.491 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:26.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.492 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:26.492 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.492 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:26.492 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:26.492 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.493 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.493 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:26.493 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:26.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.494 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:26.494 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.495 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:26.495 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:26.495 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.495 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.496 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.687891Z", 2023-10-12T01:48:26.496 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:26.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.497 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:26.497 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.497 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:26.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:26.498 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.498 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.499 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:26.499 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:26.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.500 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:26.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.500 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:26.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:26.501 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:26.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.501 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:26Z" 2023-10-12T01:48:26.501 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:26.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075290.687891_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.687891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:26Z" } 2023-10-12T01:48:26.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075290.687891_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.687891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:26.504 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:26.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:26.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:48:26.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.506 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:26.556 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:26.557 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:26.608 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006427", 2023-10-12T01:48:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:10.874028Z" 2023-10-12T01:48:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016852", 2023-10-12T01:48:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:10.816043Z" 2023-10-12T01:48:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015736", 2023-10-12T01:48:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:10.855398Z" 2023-10-12T01:48:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013372", 2023-10-12T01:48:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:47:56.210022Z" 2023-10-12T01:48:26.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.612 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:26.613 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006427', 1: '00000000000000000000:00000000000000016852', 2: '00000000000000000000:00000000000000015736', 3: '00000000000000000000:00000000000000013372'} 2023-10-12T01:48:26.614 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:48:26.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.615 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:26.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:26.713 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:26.713 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:26.714 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:26.714 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:26.714 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:26.714 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 11585258768050342181 2023-10-12T01:48:26.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.715 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:26.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.715 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:26.715 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006427", 2023-10-12T01:48:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.874028Z" 2023-10-12T01:48:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016852", 2023-10-12T01:48:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.816043Z" 2023-10-12T01:48:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015736", 2023-10-12T01:48:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.855398Z" 2023-10-12T01:48:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013372", 2023-10-12T01:48:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:47:56.210022Z" 2023-10-12T01:48:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-10-12T01:48:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-10-12T01:48:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:26Z" 2023-10-12T01:48:26.728 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:26.728 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11585258768050342181 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006427", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.874028Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016852", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.816043Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.855398Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013372", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:47:56.210022Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-12T01:48:26Z" } 2023-10-12T01:48:26.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006427', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.874028Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016852', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.816043Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.855398Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013372', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:47:56.210022Z'}}] 2023-10-12T01:48:26.729 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006427', 1: '00000000000000000000:00000000000000016852', 2: '00000000000000000000:00000000000000015736', 3: '00000000000000000000:00000000000000013372'} 2023-10-12T01:48:26.730 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006427', 1: '00000000000000000000:00000000000000016852', 2: '00000000000000000000:00000000000000015736', 3: '00000000000000000000:00000000000000013372'} 2023-10-12T01:48:26.730 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-12T01:48:26.731 DEBUG:boto:path=/wafmoa-70/ 2023-10-12T01:48:26.731 DEBUG:boto:auth_path=/wafmoa-70/ 2023-10-12T01:48:26.732 DEBUG:boto:Method: HEAD 2023-10-12T01:48:26.732 DEBUG:boto:Path: /wafmoa-70/ 2023-10-12T01:48:26.733 DEBUG:boto:Data: 2023-10-12T01:48:26.733 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.734 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.734 DEBUG:boto:Params: {} 2023-10-12T01:48:26.735 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:48:26.735 DEBUG:boto:Token: None 2023-10-12T01:48:26.736 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/ 2023-10-12T01:48:26.736 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UfYgi1ASTCODn4qVGSj2LaxSbmk= 2023-10-12T01:48:26.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UfYgi1ASTCODn4qVGSj2LaxSbmk=', 'Content-Length': '0'} 2023-10-12T01:48:26.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000732ac6452dd45d51-006527506a-27c1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.741 DEBUG:boto:path=/wafmoa-70/obj7 2023-10-12T01:48:26.742 DEBUG:boto:auth_path=/wafmoa-70/obj7 2023-10-12T01:48:26.742 DEBUG:boto:Method: HEAD 2023-10-12T01:48:26.743 DEBUG:boto:Path: /wafmoa-70/obj7 2023-10-12T01:48:26.743 DEBUG:boto:Data: 2023-10-12T01:48:26.743 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.744 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.745 DEBUG:boto:Params: {} 2023-10-12T01:48:26.745 DEBUG:boto:Token: None 2023-10-12T01:48:26.746 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj7 2023-10-12T01:48:26.746 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xle+IVcMgBqIzbO9ysCojzUNKzw= 2023-10-12T01:48:26.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xle+IVcMgBqIzbO9ysCojzUNKzw=', 'Content-Length': '0'} 2023-10-12T01:48:26.748 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000da37bf0a1ae308dc-006527506a-27c1-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.749 DEBUG:boto:path=/wafmoa-70/obj8 2023-10-12T01:48:26.750 DEBUG:boto:auth_path=/wafmoa-70/obj8 2023-10-12T01:48:26.751 DEBUG:boto:Method: HEAD 2023-10-12T01:48:26.752 DEBUG:boto:Path: /wafmoa-70/obj8 2023-10-12T01:48:26.752 DEBUG:boto:Data: 2023-10-12T01:48:26.753 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.754 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.754 DEBUG:boto:Params: {} 2023-10-12T01:48:26.755 DEBUG:boto:Token: None 2023-10-12T01:48:26.755 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj8 2023-10-12T01:48:26.756 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:QVCSmkDI4EbzrKpCXP9Lfk9vZeQ= 2023-10-12T01:48:26.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QVCSmkDI4EbzrKpCXP9Lfk9vZeQ=', 'Content-Length': '0'} 2023-10-12T01:48:26.758 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000dc5a4e24e81af835-006527506a-27c1-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.758 DEBUG:boto:path=/wafmoa-70/obj9 2023-10-12T01:48:26.759 DEBUG:boto:auth_path=/wafmoa-70/obj9 2023-10-12T01:48:26.759 DEBUG:boto:Method: HEAD 2023-10-12T01:48:26.760 DEBUG:boto:Path: /wafmoa-70/obj9 2023-10-12T01:48:26.760 DEBUG:boto:Data: 2023-10-12T01:48:26.761 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.762 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.763 DEBUG:boto:Params: {} 2023-10-12T01:48:26.763 DEBUG:boto:Token: None 2023-10-12T01:48:26.764 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj9 2023-10-12T01:48:26.764 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:80cw7CvOSMkHiPF36jEcOjRZico= 2023-10-12T01:48:26.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:80cw7CvOSMkHiPF36jEcOjRZico=', 'Content-Length': '0'} 2023-10-12T01:48:26.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-71:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ce1529b669b7ee-006527506a-27c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.767 DEBUG:boto:path=/wafmoa-70/obj9 2023-10-12T01:48:26.767 DEBUG:boto:auth_path=/wafmoa-70/obj9 2023-10-12T01:48:26.768 DEBUG:boto:Method: GET 2023-10-12T01:48:26.768 DEBUG:boto:Path: /wafmoa-70/obj9 2023-10-12T01:48:26.769 DEBUG:boto:Data: 2023-10-12T01:48:26.769 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.770 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.770 DEBUG:boto:Params: {} 2023-10-12T01:48:26.771 DEBUG:boto:Token: None 2023-10-12T01:48:26.771 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj9 2023-10-12T01:48:26.772 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:rZ47C6q/hvILcU2L/Ncj0nS9Aic= 2023-10-12T01:48:26.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rZ47C6q/hvILcU2L/Ncj0nS9Aic=', 'Content-Length': '0'} 2023-10-12T01:48:26.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-71:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fb5a97060395742-006527506a-27c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.776 DEBUG:boto:path=/wafmoa-70/obj10 2023-10-12T01:48:26.776 DEBUG:boto:auth_path=/wafmoa-70/obj10 2023-10-12T01:48:26.777 DEBUG:boto:Method: HEAD 2023-10-12T01:48:26.777 DEBUG:boto:Path: /wafmoa-70/obj10 2023-10-12T01:48:26.777 DEBUG:boto:Data: 2023-10-12T01:48:26.778 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.780 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.780 DEBUG:boto:Params: {} 2023-10-12T01:48:26.781 DEBUG:boto:Token: None 2023-10-12T01:48:26.782 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj10 2023-10-12T01:48:26.783 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:5EH1q0d2VzgfACf5QfUlWnMorH0= 2023-10-12T01:48:26.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5EH1q0d2VzgfACf5QfUlWnMorH0=', 'Content-Length': '0'} 2023-10-12T01:48:26.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-71:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015b2e74c52135985-006527506a-27c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.786 DEBUG:boto:path=/wafmoa-70/obj10 2023-10-12T01:48:26.786 DEBUG:boto:auth_path=/wafmoa-70/obj10 2023-10-12T01:48:26.787 DEBUG:boto:Method: GET 2023-10-12T01:48:26.787 DEBUG:boto:Path: /wafmoa-70/obj10 2023-10-12T01:48:26.787 DEBUG:boto:Data: 2023-10-12T01:48:26.788 DEBUG:boto:Headers: {} 2023-10-12T01:48:26.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:26.789 DEBUG:boto:Port: 8000 2023-10-12T01:48:26.789 DEBUG:boto:Params: {} 2023-10-12T01:48:26.790 DEBUG:boto:Token: None 2023-10-12T01:48:26.790 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:48:26 GMT /wafmoa-70/obj10 2023-10-12T01:48:26.791 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:uIOQSXR3JiXX2YYRGC1xzocU4OE= 2023-10-12T01:48:26.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:26 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uIOQSXR3JiXX2YYRGC1xzocU4OE=', 'Content-Length': '0'} 2023-10-12T01:48:26.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'c93c7bc5-2183-452b-b2b7-6a812a26f746:wafmoa-71:677f5a1d-3ab4-4b5f-af15-9890c2c79648.9542.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a224af07242de087-006527506a-27c1-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:48:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:26.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:26.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:26.795 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket wafmoa-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:26.896 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:48:26.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:26.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:26.898 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:26.945 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:26.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:26.991 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:48:26.992 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-10-12T01:48:26.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:26.995 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:27.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:27.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:27.074 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:48:27.086 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:27.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.087 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:27.087 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:48:27.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.088 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:27.088 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:48:27.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.089 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075306.879663_977.1", 2023-10-12T01:48:27.089 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:26.879663Z" 2023-10-12T01:48:27.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.090 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:27.090 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:48:27.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.090 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:27.091 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:27.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:48:27.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:27.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:27.093 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:27.143 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:27.172+0000 7efd41163780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:27.173+0000 7efd41163780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:27.186 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:27.186 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:27.187 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:27.188 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:27.189 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:27.190 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:27.190 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:27.190 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:27.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.191 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:27.191 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:27.191 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:27.192 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:27.192 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:27.192 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:27.192 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:27.192 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:27.193 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:27.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:27.194 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.687891Z", 2023-10-12T01:48:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:27.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:27.196 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:27.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:27.198 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:27.199 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:27.200 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:27Z" 2023-10-12T01:48:27.200 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:27.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075290.687891_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.687891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:27Z" } 2023-10-12T01:48:27.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075290.687891_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.687891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:27.202 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:27.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:27.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075306.879663_977.1 target=1_1697075290.687891_975.1 2023-10-12T01:48:32.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:32.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:32.206 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:32.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:32.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:32.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:32.295+0000 7efdc62b5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:32.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:32.296+0000 7efdc62b5780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:32.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.687891Z", 2023-10-12T01:48:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:32Z" 2023-10-12T01:48:32.325 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:32.325 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075290.687891_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.687891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:32Z" } 2023-10-12T01:48:32.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075290.687891_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.687891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:32.327 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:32.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:32.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075306.879663_977.1 target=1_1697075290.687891_975.1 2023-10-12T01:48:37.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:37.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:37.330 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:37.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:37.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:37.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:37.418+0000 7f197a370780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:37.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:37.420+0000 7f197a370780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:37.439 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.687891Z", 2023-10-12T01:48:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:37Z" 2023-10-12T01:48:37.453 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:37.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075290.687891_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.687891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:37Z" } 2023-10-12T01:48:37.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075290.687891_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.687891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:37.455 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:37.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:37.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075306.879663_977.1 target=1_1697075290.687891_975.1 2023-10-12T01:48:42.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:42.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:42.458 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:42.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:42.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:42.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:42.548+0000 7fc069ffa780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:42.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:42.549+0000 7fc069ffa780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:42.564 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:42.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:42.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:42.572 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:42.572 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075290.687891_975.1", 2023-10-12T01:48:42.573 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:42.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:42.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:42.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:10.687891Z", 2023-10-12T01:48:42.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:42.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:42.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:42.574 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:42.575 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:42.575 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:42.575 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:42.575 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:42.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:42.576 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:42.576 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:42.576 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:42.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:42.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:42.577 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:42.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.577 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:42.577 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:42.577 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:42.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:42.578 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:42Z" 2023-10-12T01:48:42.578 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:42.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075290.687891_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:10.687891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:42Z" } 2023-10-12T01:48:42.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075290.687891_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:10.687891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:42.580 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:42.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075290.687891_975.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:42.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075306.879663_977.1 target=1_1697075290.687891_975.1 2023-10-12T01:48:47.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.584 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:47.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:47.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:47.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:47.671+0000 7f3631172780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:47.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:47.672+0000 7f3631172780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:47.686 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:47.686 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:47.687 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.688 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:47.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:47.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:47.691 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:47.692 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.693 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075306.879663_977.1", 2023-10-12T01:48:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:47.694 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:26.879663Z", 2023-10-12T01:48:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:47.696 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:47.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:47.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.699 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:47Z" 2023-10-12T01:48:47.700 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:47.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075306.879663_977.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:26.879663Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:47Z" } 2023-10-12T01:48:47.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075306.879663_977.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:26.879663Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:47.702 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:47.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075306.879663_977.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:47.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:48:47.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:47.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:47.704 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:47.796 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:48:47.796 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-72 2023-10-12T01:48:47.797 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:48:47.798 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:48:47.798 DEBUG:boto:Method: PUT 2023-10-12T01:48:47.799 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:48:47.799 DEBUG:boto:Data: 2023-10-12T01:48:47.800 DEBUG:boto:Headers: {} 2023-10-12T01:48:47.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:47.801 DEBUG:boto:Port: 8000 2023-10-12T01:48:47.801 DEBUG:boto:Params: {} 2023-10-12T01:48:47.802 DEBUG:boto:Token: None 2023-10-12T01:48:47.802 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:48:47 GMT /wafmoa-72/ 2023-10-12T01:48:47.803 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Ql+Vgk6kEDj9B6YlgBLQow22BpQ= 2023-10-12T01:48:47.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Ql+Vgk6kEDj9B6YlgBLQow22BpQ=', 'Content-Length': '0'} 2023-10-12T01:48:47.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005064220f7a612009-006527507f-27c1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:47.820 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-73 2023-10-12T01:48:47.820 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:48:47.821 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:48:47.821 DEBUG:boto:Method: PUT 2023-10-12T01:48:47.822 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:48:47.822 DEBUG:boto:Data: 2023-10-12T01:48:47.822 DEBUG:boto:Headers: {} 2023-10-12T01:48:47.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:47.823 DEBUG:boto:Port: 8000 2023-10-12T01:48:47.824 DEBUG:boto:Params: {} 2023-10-12T01:48:47.824 DEBUG:boto:Token: None 2023-10-12T01:48:47.825 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:48:47 GMT /wafmoa-73/ 2023-10-12T01:48:47.825 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:hvyRKGhO3ceOZ5kHuwvb1i+WlyE= 2023-10-12T01:48:47.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:47 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hvyRKGhO3ceOZ5kHuwvb1i+WlyE=', 'Content-Length': '0'} 2023-10-12T01:48:47.840 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd34fe6b0f8600f7-006527507f-27c1-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:47.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.842 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:47.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:48:47.933 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:48:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:48:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:48:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:47.839417Z" 2023-10-12T01:48:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:47.937 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:48:47.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:47.937 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:47.938 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:47.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:48:47.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:47.940 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:47.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:47.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:48.021+0000 7fbe3a4a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:48.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:48.022+0000 7fbe3a4a0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:48.040 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:48.040 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:48.040 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:48.041 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:48.041 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:48.041 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:48.041 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:48.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.042 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:48.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:48.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:48.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:48.043 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:48.043 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:48.043 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:48.043 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:48.043 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:48.044 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:48.044 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:48.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:48.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:48.045 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:48.045 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:48.045 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:48.045 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:48.046 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:48.046 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:48.046 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:48.046 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:48.046 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:48.047 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:48.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:48.047 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:48.047 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:48.048 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:48.048 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.812320_978.1", 2023-10-12T01:48:48.048 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:48.048 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:48.048 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:48.049 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.812320Z", 2023-10-12T01:48:48.049 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:48.049 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:48.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:48.050 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:48.050 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:48.050 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:48.050 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:48.051 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:48.051 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:48.051 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:48.051 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:48.051 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:48.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:48.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:48.052 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:48.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.053 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:48.053 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:48.053 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:48.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:48.053 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:48Z" 2023-10-12T01:48:48.054 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:48.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.812320_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.812320Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:48Z" } 2023-10-12T01:48:48.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.812320_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.812320Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:48.056 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:48.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.812320_978.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:48.056 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075327.839417_985.1 target=1_1697075327.812320_978.1 2023-10-12T01:48:53.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:53.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:53.059 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:53.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:53.150+0000 7fb01e27f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:53.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:53.151+0000 7fb01e27f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:53.164 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:53.165 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:53.165 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:53.165 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:53.166 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:53.166 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:53.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.167 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:53.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:53.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:53.167 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:53.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:53.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:53.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:53.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.169 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:53.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:53.170 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:53.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:53.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:53.173 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:53.173 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:48:53.173 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:53.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:53.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:53.174 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:48:53.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:53.175 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:53.175 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:53.175 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:53.176 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:53.176 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:53.176 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:53.176 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:53.177 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.178 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:53.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:53.178 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:53.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.179 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:53Z" 2023-10-12T01:48:53.179 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:53.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:53Z" } 2023-10-12T01:48:53.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:53.181 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:53.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:53.182 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:48:53.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.183 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.234 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.234 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.279 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:48:53.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.281 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.338 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:48:53.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-72', '--dest-bucket=wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-72', '--dest-bucket=wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.386 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-72 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-72 --dest-bucket=wafmoa-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.493 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:48:53.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-73', '--dest-bucket=wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-73', '--dest-bucket=wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.496 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-73 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-73 --dest-bucket=wafmoa-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.551 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.595 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:48:53.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.691 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:48:53.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.693 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.791 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:53.791 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:53.792 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20, 2023-10-12T01:48:53.792 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:48:53.792 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:48:53.793 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:48:53.793 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:53.793 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:48:53.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.794 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:48:53.794 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:48:53.795 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:48:53.796 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:48:53.796 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.796 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:48:53.797 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.797 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:48:53.797 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:48:53.797 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:48:53.798 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.798 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:48:53.799 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:48:53.799 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.799 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:48:53.799 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.800 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:48:53.800 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:48:53.800 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:48:53.800 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:48:53.800 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:48:53.801 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:48:53.801 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:48:53.801 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:48:53.801 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:48:53.802 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:48:53.802 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.802 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.803 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:48:53.803 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:48:53.803 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.803 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:48:53.804 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.804 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:48:53.804 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:48:53.804 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:48:53.805 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:48:53.805 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:48:53.805 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:48:53.805 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:48:53.806 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:48:53.806 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:48:53.806 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:48:53.807 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.807 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.807 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:48:53.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.808 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:48:53.808 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:48:53.808 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:48:53.808 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:48:53.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.809 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.809 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:48:53.810 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:48:53.810 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:48:53.810 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:48:53.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.811 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:48:53.811 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:48:53.811 INFO:teuthology.orchestra.run.smithi112.stdout: "directional": [ 2023-10-12T01:48:53.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.811 INFO:teuthology.orchestra.run.smithi112.stdout: "source_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:48:53.812 INFO:teuthology.orchestra.run.smithi112.stdout: "dest_zone": "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:48:53.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.812 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:48:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-72", 2023-10-12T01:48:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:48:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-72", 2023-10-12T01:48:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:48:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:48:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-73", 2023-10-12T01:48:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:48:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:48:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:48:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:48:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:48:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:48:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:48:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:48:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:48:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.821 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-73", 2023-10-12T01:48:53.821 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:48:53.822 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-73", 2023-10-12T01:48:53.822 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.822 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:48:53.822 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.823 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:48:53.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-73", 2023-10-12T01:48:53.823 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.823 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:48:53.824 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.824 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:48:53.824 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:48:53.825 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:48:53.825 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:48:53.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.826 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:48:53.826 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:48:53.826 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:48:53.826 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:48:53.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.827 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.827 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:48:53.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.828 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.829 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:48:53.829 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.830 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:48:53.830 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:48:53.830 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:48:53.830 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:48:53.831 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.831 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:48:53.831 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.831 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:48:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:48:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:48:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:48:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.832 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:48:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:48:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.833 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:48:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:48:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:48:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:48:53.834 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:48:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:48:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:48:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:48:53.835 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:48:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:48:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:48:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:48:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:48:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:48:53.837 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:48:53.838 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.838 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:48:53.838 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:48:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:48:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:48:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:48:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:48:53.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:48:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:48:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:48:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:48:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.841 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.842 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:48:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:48:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:48:53.843 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:48:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:48:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.844 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:48:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:48:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:48:53.845 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:48:53.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.846 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:48:53.846 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:48:53.846 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.847 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:48:53.847 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:48:53.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:48:53.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:48:53.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:48:53.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:48:53.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:48:53.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:48:53.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:53.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:48:53.851 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:48:53.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:53.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.851 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:48:53.852 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:48:53.852 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:48:53.852 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:48:53.852 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:48:53.852 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:48:53.853 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:48:53.853 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:48:53.853 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:48:53.855 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:48:53.855 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:48:53.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.856 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:48:53.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:48:53.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:48:53.856 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:48:53.857 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:48:53.857 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:48:53.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.857 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:48:53.857 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:48:53.858 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:48:53.858 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:48:53.858 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:48:53.858 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:48:53.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.859 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:48:53.859 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:48:53.859 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:48:53.859 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:48:53.860 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:48:53.860 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:48:53.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:53.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:53.860 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:48:53.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:48:53.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:53.861 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:53.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:48:53.863 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:48:53.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:53.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:53.954 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:48:53.954 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:48:53.955 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:48:53.955 DEBUG:boto:Method: HEAD 2023-10-12T01:48:53.956 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:48:53.956 DEBUG:boto:Data: 2023-10-12T01:48:53.957 DEBUG:boto:Headers: {} 2023-10-12T01:48:53.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:53.958 DEBUG:boto:Port: 8000 2023-10-12T01:48:53.958 DEBUG:boto:Params: {} 2023-10-12T01:48:53.958 DEBUG:boto:Token: None 2023-10-12T01:48:53.959 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:53 GMT /wafmoa-72/ 2023-10-12T01:48:53.959 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0KBjbqtZExdMIxq3Kv15avl9GK8= 2023-10-12T01:48:53.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0KBjbqtZExdMIxq3Kv15avl9GK8=', 'Content-Length': '0'} 2023-10-12T01:48:54.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d2c4f9d5bcaa80-0065275086-2a06-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.103 DEBUG:boto:path=/wafmoa-72/obj1 2023-10-12T01:48:54.103 DEBUG:boto:auth_path=/wafmoa-72/obj1 2023-10-12T01:48:54.104 DEBUG:boto:Method: PUT 2023-10-12T01:48:54.104 DEBUG:boto:Path: /wafmoa-72/obj1 2023-10-12T01:48:54.105 DEBUG:boto:Data: 2023-10-12T01:48:54.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:54.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.106 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.107 DEBUG:boto:Params: {} 2023-10-12T01:48:54.107 DEBUG:boto:Token: None 2023-10-12T01:48:54.108 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-72/obj1 2023-10-12T01:48:54.109 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:C23KFVlVmgJotE6yDBW6gSBc9SM= 2023-10-12T01:48:54.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C23KFVlVmgJotE6yDBW6gSBc9SM='} 2023-10-12T01:48:54.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd898a88aba9fccd-0065275086-2a06-a1'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.145 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:48:54.145 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:48:54.146 DEBUG:boto:Method: HEAD 2023-10-12T01:48:54.146 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:48:54.147 DEBUG:boto:Data: 2023-10-12T01:48:54.147 DEBUG:boto:Headers: {} 2023-10-12T01:48:54.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.148 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.148 DEBUG:boto:Params: {} 2023-10-12T01:48:54.149 DEBUG:boto:Token: None 2023-10-12T01:48:54.150 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-72/ 2023-10-12T01:48:54.151 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r/lqzMFbumwuX1ZHAtljZ+XNNOQ= 2023-10-12T01:48:54.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r/lqzMFbumwuX1ZHAtljZ+XNNOQ=', 'Content-Length': '0'} 2023-10-12T01:48:54.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013037cbac7d61435-0065275086-2a06-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.158 DEBUG:boto:path=/wafmoa-72/obj2 2023-10-12T01:48:54.158 DEBUG:boto:auth_path=/wafmoa-72/obj2 2023-10-12T01:48:54.159 DEBUG:boto:Method: PUT 2023-10-12T01:48:54.159 DEBUG:boto:Path: /wafmoa-72/obj2 2023-10-12T01:48:54.159 DEBUG:boto:Data: 2023-10-12T01:48:54.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:54.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.161 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.161 DEBUG:boto:Params: {} 2023-10-12T01:48:54.161 DEBUG:boto:Token: None 2023-10-12T01:48:54.162 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-72/obj2 2023-10-12T01:48:54.163 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RrK8gwGRL6mEYf8+eF28qvRSGS8= 2023-10-12T01:48:54.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RrK8gwGRL6mEYf8+eF28qvRSGS8='} 2023-10-12T01:48:54.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9cabe93f424a523-0065275086-2a06-a1'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.207 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:48:54.208 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:48:54.208 DEBUG:boto:Method: HEAD 2023-10-12T01:48:54.208 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:48:54.209 DEBUG:boto:Data: 2023-10-12T01:48:54.209 DEBUG:boto:Headers: {} 2023-10-12T01:48:54.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.210 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.211 DEBUG:boto:Params: {} 2023-10-12T01:48:54.211 DEBUG:boto:Token: None 2023-10-12T01:48:54.212 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-73/ 2023-10-12T01:48:54.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jGHfSVvtPgE2f3LvKXQpZQExZ8M= 2023-10-12T01:48:54.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jGHfSVvtPgE2f3LvKXQpZQExZ8M=', 'Content-Length': '0'} 2023-10-12T01:48:54.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1fe9d1f5e039f6f-0065275086-2a06-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.224 DEBUG:boto:path=/wafmoa-73/obj3 2023-10-12T01:48:54.224 DEBUG:boto:auth_path=/wafmoa-73/obj3 2023-10-12T01:48:54.225 DEBUG:boto:Method: PUT 2023-10-12T01:48:54.225 DEBUG:boto:Path: /wafmoa-73/obj3 2023-10-12T01:48:54.225 DEBUG:boto:Data: 2023-10-12T01:48:54.226 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:54.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.227 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.227 DEBUG:boto:Params: {} 2023-10-12T01:48:54.227 DEBUG:boto:Token: None 2023-10-12T01:48:54.228 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-73/obj3 2023-10-12T01:48:54.228 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:esYP3S4O1NDf+pihpmz9OOiWClY= 2023-10-12T01:48:54.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:esYP3S4O1NDf+pihpmz9OOiWClY='} 2023-10-12T01:48:54.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bededfef391ffa9-0065275086-2a06-a1'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.250 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:48:54.250 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:48:54.251 DEBUG:boto:Method: HEAD 2023-10-12T01:48:54.251 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:48:54.251 DEBUG:boto:Data: 2023-10-12T01:48:54.252 DEBUG:boto:Headers: {} 2023-10-12T01:48:54.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.253 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.253 DEBUG:boto:Params: {} 2023-10-12T01:48:54.254 DEBUG:boto:Token: None 2023-10-12T01:48:54.254 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-73/ 2023-10-12T01:48:54.254 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:jGHfSVvtPgE2f3LvKXQpZQExZ8M= 2023-10-12T01:48:54.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jGHfSVvtPgE2f3LvKXQpZQExZ8M=', 'Content-Length': '0'} 2023-10-12T01:48:54.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6390c893dea724a-0065275086-2a06-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.260 DEBUG:boto:path=/wafmoa-73/obj4 2023-10-12T01:48:54.261 DEBUG:boto:auth_path=/wafmoa-73/obj4 2023-10-12T01:48:54.261 DEBUG:boto:Method: PUT 2023-10-12T01:48:54.262 DEBUG:boto:Path: /wafmoa-73/obj4 2023-10-12T01:48:54.262 DEBUG:boto:Data: 2023-10-12T01:48:54.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:48:54.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:48:54.263 DEBUG:boto:Port: 8000 2023-10-12T01:48:54.264 DEBUG:boto:Params: {} 2023-10-12T01:48:54.264 DEBUG:boto:Token: None 2023-10-12T01:48:54.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:48:54 GMT /wafmoa-73/obj4 2023-10-12T01:48:54.265 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:LBQRgbqV2YTFm/aGQZzecDhtbCs= 2023-10-12T01:48:54.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:48:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:LBQRgbqV2YTFm/aGQZzecDhtbCs='} 2023-10-12T01:48:54.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de97645cd0220eb1-0065275086-2a06-a1'), ('Date', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:48:54.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.312 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:54.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:54.510 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:48:54.554 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:48:54.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:48:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:48:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:47.839417Z" 2023-10-12T01:48:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:48:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:54.559 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:54.560 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:48:54.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.561 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:54.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:54.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:54.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:54.689+0000 7f6b9fb63780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:54.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:48:54.691+0000 7f6b9fb63780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:48:54.707 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:54.707 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:48:54.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.709 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:48:54.710 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:48:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:48:54.711 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.712 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:48:54.713 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:48:54.714 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:54.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.717 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:48:54.718 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:48:54.719 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.720 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:48:54.721 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:48:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:54.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:48:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:48:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:54.724 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:48:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:48:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.725 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:54Z" 2023-10-12T01:48:54.725 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:54.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:48:54Z" } 2023-10-12T01:48:54.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:48:54.728 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:54.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:48:54.729 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:48:54.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.731 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:54.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:54.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:54.853 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:48:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:48:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:48:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:48:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:48:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:48:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:48:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:48:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:48:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.858 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:48:54.860 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:48:54.860 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:48:54.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:54.864 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:48:54.915 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:48:54.916 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:48:54.958 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:48:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:48:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:48:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:48:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:48:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:48:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:48:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:48:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:48:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:48:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:48:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:48:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:48:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:48:54.966 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.966 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:48:54.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.967 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:48:54.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.967 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:48:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:48:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:48:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:48:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:48:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:48:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:48:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:48:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:48:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:48:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:48:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:48:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:48:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:48:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:48:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:48:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:48:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:48:54Z" 2023-10-12T01:48:54.973 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:48:54.975 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:48:54Z" } 2023-10-12T01:48:54.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:48:54.976 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:48:54.976 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:48:54.977 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:48:59.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:59.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:48:59.979 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:00.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:00.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:00.086 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:00.087 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:00.087 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:00.087 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:00.087 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:00.087 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:00.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.088 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:00.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:00.088 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:00.089 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:00.089 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:00.089 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:00.089 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:00.089 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:00.090 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:00.090 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:00.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:00.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:00.091 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:00.092 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:00.092 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:00.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:00.092 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:00.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:00.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:00.093 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:00.094 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:00.094 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:00.094 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:00.094 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:00.095 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:00.095 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:00.095 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:00.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:00.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:00.096 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:00.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:00.096 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:00.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:00.097 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:00.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:00.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:00.097 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:00.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:00.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:00.098 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:00.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.098 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:00.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:00.099 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:00.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:00.099 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:00Z" 2023-10-12T01:49:00.099 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:00.100 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:00Z" } 2023-10-12T01:49:00.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:00.101 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:00.102 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:00.102 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:05.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:05.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:05.105 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:05.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:05.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:05.216 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:05Z" 2023-10-12T01:49:05.229 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:05.229 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:05Z" } 2023-10-12T01:49:05.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:05.230 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:05.231 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:05.231 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:10.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:10.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:10.234 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:10.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:10.296 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:10.347 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:10.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:10.358 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:10.358 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:10.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:10.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:10.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:10.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.359 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:10.359 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:10.360 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:10.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:10.360 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:10Z" 2023-10-12T01:49:10.360 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:10.361 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:10Z" } 2023-10-12T01:49:10.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:10.362 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:10.363 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:10.363 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:15.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:15.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:15.365 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:15.425 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:15.425 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:15.476 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:15.476 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:15.476 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:15.477 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:15.477 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:15.477 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:15.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.477 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:15.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:15.478 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:15.478 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:15.478 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:15.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:15.479 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:15.479 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:15.479 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:15.479 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:15.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:15.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:15.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:15.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:15.481 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:15.481 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:15.481 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:15.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:15.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:15.482 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:15.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:15.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:15.483 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:15.483 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:15.483 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:15.483 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:15.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:15.484 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:15.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:15.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:15.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:15.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:15.485 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:15.485 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:15.486 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:15.486 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:15.486 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:15.486 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:15.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:15.487 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:15.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:15.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:15.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:15.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.488 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:15.488 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:15.488 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:15.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:15.489 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:15Z" 2023-10-12T01:49:15.489 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:15.490 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:15Z" } 2023-10-12T01:49:15.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:15.491 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:15.491 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:15.492 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:20.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:20.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:20.494 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:20.554 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:20.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:20.601 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:20.602 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:20.602 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:20.602 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:20.602 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:20.603 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:20.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.603 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:20.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:20.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:20.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:20.605 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:20.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:20.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:20.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:20.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:20.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:20.607 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:20.607 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:20.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:20.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:20.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:20.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:20.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:20.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:20.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:20.609 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:20.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:20.609 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:20.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:20.611 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:20.612 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.613 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:20.614 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:20Z" 2023-10-12T01:49:20.614 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:20.616 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:20Z" } 2023-10-12T01:49:20.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:20.617 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:20.617 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:20.617 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:25.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:25.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:25.619 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:25.727 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:25.727 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:25.727 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:25.727 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:25.728 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:25.728 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:25.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.728 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:25.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:25.729 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:25.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:25.729 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:25.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:25.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:25.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:25.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:25.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:25.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:25.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:25.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:25.731 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:25.731 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:25Z" 2023-10-12T01:49:25.740 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:25.741 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:25Z" } 2023-10-12T01:49:25.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:25.742 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:25.742 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:25.742 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:30.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:30.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:30.745 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:30.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:30.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:30.852 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:30.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:30.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:30.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:30.856 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:30.856 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:30.856 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:30.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:30.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:30.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:30.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:30.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:30.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:30.860 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:30.860 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:30.860 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:30.861 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:30.861 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:30.861 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:30.862 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:30.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:30.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:30.862 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:30.863 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:30.863 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:30.863 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:30.863 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:30.864 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:30.864 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:30.864 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:30.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:30.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:30.865 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:30.865 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:30.865 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:30.865 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:30.866 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:30.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:30.866 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:30.866 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:30.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:30.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:30.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:30.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.867 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:30.868 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:30.868 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:30.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:30.868 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:30Z" 2023-10-12T01:49:30.869 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:30.870 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:30Z" } 2023-10-12T01:49:30.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:30.871 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:30.872 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:30.872 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:35.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:35.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:35.874 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:35.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:35.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:35.982 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:35.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:35.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:35.990 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:35.990 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:35.990 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:35.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:35.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:35.991 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:35.991 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:35.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:35.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:35Z" 2023-10-12T01:49:35.995 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:35.996 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:35Z" } 2023-10-12T01:49:35.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:35.997 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:35.998 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:35.998 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:40.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:41.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:41.000 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:41.108 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:41.113 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:41.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:41.115 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:41Z" 2023-10-12T01:49:41.120 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:41.121 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:41Z" } 2023-10-12T01:49:41.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:41.122 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:41.123 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:41.123 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:46.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:46.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:46.125 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:46.235 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:46.235 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:46.235 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:46.235 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:46.236 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:46.236 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:46.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.236 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:46.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:46.237 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:46.237 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:46.237 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:46.237 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:46.238 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:46.238 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:46.238 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:46.238 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:46.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:46.239 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:46.240 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:46.240 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:46.240 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:46.240 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:46.240 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:46.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:46.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:46.241 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:46.242 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:46.242 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:46.242 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:46.242 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:46.242 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:46.243 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:46.243 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:46.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:46.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:46.244 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:46.244 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:46.244 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:46.244 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:46.245 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:46.245 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:46.245 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:46.245 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:46.245 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:46.246 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:46.246 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:46.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.246 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:46.246 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:46.247 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:46.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:46.247 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:46Z" 2023-10-12T01:49:46.247 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:46.248 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:46Z" } 2023-10-12T01:49:46.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:46.249 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:46.250 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:46.250 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:51.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:51.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:51.252 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:51.313 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:51.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:51.362 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:51.363 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:51.363 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:51.363 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:51.364 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:51.364 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:51.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.364 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:51.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:51.365 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:51.365 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:51.365 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:51.365 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:51.366 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:51.366 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:51.366 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:51.366 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:51.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:51.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:51.367 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:51.367 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:51.368 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:51.368 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:51.368 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:51.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:51.368 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:51.370 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:51.370 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:51.370 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:51.370 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:51.370 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:51.371 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:51.371 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:51.371 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:51.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:51.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:51.372 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:51.372 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:51.372 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:51.373 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:51.373 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:51.373 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:51.373 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:51.374 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:51.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:51.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:51.374 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:51.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.375 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:51.375 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:51.375 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:51.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:51.376 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:51Z" 2023-10-12T01:49:51.376 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:51.377 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:51Z" } 2023-10-12T01:49:51.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:51.379 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:51.379 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:51.380 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:49:56.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:56.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:49:56.383 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:49:56.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:49:56.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:49:56.491 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:49:56.492 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:49:56.492 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:49:56.492 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:49:56.492 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:49:56.492 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:49:56.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.493 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:49:56.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:56.493 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:49:56.493 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:56.494 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:56.494 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:49:56.494 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:56.494 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:49:56.495 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:56.495 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:49:56.495 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:56.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:56.496 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:49:56.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:56.496 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:56.496 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:49:56.496 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:56.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:49:56.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:56.497 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:49:56.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:56.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:56.498 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:49:56.498 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:56.498 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:56.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:49:56.499 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:56.499 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:56.499 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:56.499 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:49:56.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:56.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:49:56.500 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:49:56.500 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:49:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:49:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:49:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:49:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:49:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:49:56.502 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:49:56.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:56.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:49:56.502 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:49:56.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.503 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:49:56.503 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:49:56.503 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:49:56.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:49:56.503 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:49:56Z" 2023-10-12T01:49:56.504 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:49:56.505 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:49:56Z" } 2023-10-12T01:49:56.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:49:56.506 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:49:56.506 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:49:56.506 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:01.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:01.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:01.509 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:01.568 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:01.569 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:01.616 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:01.617 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:01.617 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:01.617 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:01.617 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:01.618 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:01.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.618 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:01.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:01.618 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:01.619 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:01.619 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:01.619 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:50:01.619 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:01.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:01.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:01.620 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:50:01.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:01.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:01.621 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:01.622 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:01.622 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:01.623 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:50:01.623 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:01.623 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:01.623 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:01.623 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:50:01.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:01.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:01.624 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:01.625 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:01.625 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:01.625 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:01.625 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:01.625 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:01.626 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:01.626 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:01.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:01.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:01.627 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:01.627 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:01.627 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:01.628 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:50:01.628 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:01.628 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:01.628 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:01.628 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:50:01.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:01.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:01.629 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:01.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.630 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:01.630 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:01.630 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:01.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:01.631 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:01Z" 2023-10-12T01:50:01.631 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:01.632 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:01Z" } 2023-10-12T01:50:01.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:50:01.633 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:01.634 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:50:01.634 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:06.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:06.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:06.637 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:06.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:06.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:06.744 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:06.745 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:06.745 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:06.745 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:06.745 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:06.746 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:06.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.746 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:06.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:06.747 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:06.747 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:06.747 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:06.747 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:50:06.748 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:06.748 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:06.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:06.748 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:50:06.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:06.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:06.749 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:06.749 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:06.750 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:06.750 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:50:06.750 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:06.750 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:06.753 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:06.754 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:50:06.754 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:06.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:06.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:06.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:06.756 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:06.756 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:06.757 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:06.757 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:06.758 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:06.758 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:06.759 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:06.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:06.759 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:06.760 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:06.760 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:06.760 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:50:06.760 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:06.761 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:06.761 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:06.761 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:50:06.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:06.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:06.762 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:06.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.762 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:06.763 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:06.763 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:06.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:06.763 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:06Z" 2023-10-12T01:50:06.764 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:06.765 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:06Z" } 2023-10-12T01:50:06.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:50:06.766 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:06.766 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:50:06.767 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:11.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:11.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:11.770 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:11.831 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:11.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:11.879 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:11.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:11.880 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:11.880 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:11.880 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:11.881 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:11.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.881 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:11.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:11.882 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:11.882 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:11.883 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:11.883 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:50:11.883 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:11.884 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:11.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:11.884 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:50:11.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:11.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:11.886 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:11.887 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:11.887 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:11.887 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:50:11.888 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:11.888 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:11.888 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:11.888 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:50:11.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:11.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:11.890 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:11.891 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:11.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:11.893 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:11.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:50:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:11.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:50:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:11.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:11.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:11.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:11.899 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:11Z" 2023-10-12T01:50:11.900 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:11.901 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:11Z" } 2023-10-12T01:50:11.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:50:11.903 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:11.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:50:11.904 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:16.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:16.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:16.906 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:17.013 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:17.014 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:17.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:50:17.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:50:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:17.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:17.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:50:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:17.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:50:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:17.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:17.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:17.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:50:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:50:17.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.025 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:17.026 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:17Z" 2023-10-12T01:50:17.026 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:17.027 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:17Z" } 2023-10-12T01:50:17.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:50:17.028 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:17.029 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:50:17.029 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:22.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:22.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:22.032 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:22.093 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:22.093 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:22.141 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:22.141 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:22.142 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:22.143 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032480", 2023-10-12T01:50:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:22.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.218185Z" 2023-10-12T01:50:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-12T01:50:22.146 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.702906Z" 2023-10-12T01:50:22.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:22.148 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:22.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:22.150 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046079", 2023-10-12T01:50:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:22.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:46:26.724889Z" 2023-10-12T01:50:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:22.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:22.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.153 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:22.153 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:22.153 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:22.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:22.154 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:22Z" 2023-10-12T01:50:22.154 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:22.155 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032480", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:15.218185Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:46:26.702906Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046079", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:46:26.724889Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:22Z" } 2023-10-12T01:50:22.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032480', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.218185Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.702906Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046079', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:46:26.724889Z'}}] 2023-10-12T01:50:22.156 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:22.156 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032480', 1: '00000000000000000000:00000000000000051810', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046079'} 2023-10-12T01:50:22.157 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032620 target=00000000000000000000:00000000000000032480, shard=1 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051810, shard=3 master=00000000000000000000:00000000000000046219 target=00000000000000000000:00000000000000046079 2023-10-12T01:50:27.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:27.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:27.159 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.266 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:27.266 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:27.267 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:50:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:27.268 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:50:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:27.269 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:50:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:50:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:27.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:27.271 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:50:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:50:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:27.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:50:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:27.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:50:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:27.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:50:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:50:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:50:27.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:50:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:27.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:50:27.278 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:50:27.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:27.279 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:27Z" 2023-10-12T01:50:27.279 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:27.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:50:27Z" } 2023-10-12T01:50:27.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:50:27.281 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:27.281 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:50:27.282 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:50:27.282 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:50:27.283 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:50:27.283 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.284 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:50:27.284 DEBUG:boto:Data: 2023-10-12T01:50:27.284 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.285 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.286 DEBUG:boto:Params: {} 2023-10-12T01:50:27.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:50:27.287 DEBUG:boto:Token: None 2023-10-12T01:50:27.287 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-72/ 2023-10-12T01:50:27.288 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:NhW/qCevYz1yZod2LbaCvUeMaIc= 2023-10-12T01:50:27.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NhW/qCevYz1yZod2LbaCvUeMaIc=', 'Content-Length': '0'} 2023-10-12T01:50:27.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c780adbe041dd00c-00652750e3-2a04-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.295 DEBUG:boto:path=/wafmoa-72/obj1 2023-10-12T01:50:27.295 DEBUG:boto:auth_path=/wafmoa-72/obj1 2023-10-12T01:50:27.296 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.296 DEBUG:boto:Path: /wafmoa-72/obj1 2023-10-12T01:50:27.297 DEBUG:boto:Data: 2023-10-12T01:50:27.297 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.298 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.298 DEBUG:boto:Params: {} 2023-10-12T01:50:27.299 DEBUG:boto:Token: None 2023-10-12T01:50:27.299 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-72/obj1 2023-10-12T01:50:27.300 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UX4cXrSLk617osQo7WjuRNk1Y+A= 2023-10-12T01:50:27.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UX4cXrSLk617osQo7WjuRNk1Y+A=', 'Content-Length': '0'} 2023-10-12T01:50:27.302 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000891f3faca0a0058e-00652750e3-2a04-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.302 DEBUG:boto:path=/wafmoa-72/obj2 2023-10-12T01:50:27.303 DEBUG:boto:auth_path=/wafmoa-72/obj2 2023-10-12T01:50:27.303 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.304 DEBUG:boto:Path: /wafmoa-72/obj2 2023-10-12T01:50:27.304 DEBUG:boto:Data: 2023-10-12T01:50:27.304 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.305 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.306 DEBUG:boto:Params: {} 2023-10-12T01:50:27.306 DEBUG:boto:Token: None 2023-10-12T01:50:27.307 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-72/obj2 2023-10-12T01:50:27.307 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:O9CMQWsOi+lKJWEQAQRdSwJF9S4= 2023-10-12T01:50:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:O9CMQWsOi+lKJWEQAQRdSwJF9S4=', 'Content-Length': '0'} 2023-10-12T01:50:27.309 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c6a7a7cb3756bdef-00652750e3-2a04-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.310 DEBUG:boto:path=/wafmoa-72/obj3 2023-10-12T01:50:27.310 DEBUG:boto:auth_path=/wafmoa-72/obj3 2023-10-12T01:50:27.310 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.311 DEBUG:boto:Path: /wafmoa-72/obj3 2023-10-12T01:50:27.311 DEBUG:boto:Data: 2023-10-12T01:50:27.312 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.312 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.313 DEBUG:boto:Params: {} 2023-10-12T01:50:27.313 DEBUG:boto:Token: None 2023-10-12T01:50:27.314 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-72/obj3 2023-10-12T01:50:27.314 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VVqCmKoScR4H5hK28Af65kjtd7o= 2023-10-12T01:50:27.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VVqCmKoScR4H5hK28Af65kjtd7o=', 'Content-Length': '0'} 2023-10-12T01:50:27.316 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b83773d3204f7fc9-00652750e3-2a04-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.317 DEBUG:boto:path=/wafmoa-72/obj4 2023-10-12T01:50:27.317 DEBUG:boto:auth_path=/wafmoa-72/obj4 2023-10-12T01:50:27.318 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.318 DEBUG:boto:Path: /wafmoa-72/obj4 2023-10-12T01:50:27.319 DEBUG:boto:Data: 2023-10-12T01:50:27.319 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.320 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.320 DEBUG:boto:Params: {} 2023-10-12T01:50:27.320 DEBUG:boto:Token: None 2023-10-12T01:50:27.321 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-72/obj4 2023-10-12T01:50:27.321 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:6/YrGnzgi0A67oI+AzIn9RQS2aM= 2023-10-12T01:50:27.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6/YrGnzgi0A67oI+AzIn9RQS2aM=', 'Content-Length': '0'} 2023-10-12T01:50:27.323 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f3fab2d8d0008a1f-00652750e3-2a04-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.324 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:50:27.324 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:50:27.325 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.325 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:50:27.326 DEBUG:boto:Data: 2023-10-12T01:50:27.326 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.327 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.327 DEBUG:boto:Params: {} 2023-10-12T01:50:27.328 DEBUG:boto:Token: None 2023-10-12T01:50:27.328 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/ 2023-10-12T01:50:27.328 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ekGzSLH4v7DIM3yYyCKYBQY66iA= 2023-10-12T01:50:27.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ekGzSLH4v7DIM3yYyCKYBQY66iA=', 'Content-Length': '0'} 2023-10-12T01:50:27.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033efa0ac227d2e49-00652750e3-2a04-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.332 DEBUG:boto:path=/wafmoa-73/obj1 2023-10-12T01:50:27.332 DEBUG:boto:auth_path=/wafmoa-73/obj1 2023-10-12T01:50:27.332 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.333 DEBUG:boto:Path: /wafmoa-73/obj1 2023-10-12T01:50:27.333 DEBUG:boto:Data: 2023-10-12T01:50:27.334 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.334 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.335 DEBUG:boto:Params: {} 2023-10-12T01:50:27.335 DEBUG:boto:Token: None 2023-10-12T01:50:27.336 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj1 2023-10-12T01:50:27.336 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VfLyr5i2TXpnujRYnpCvxe5tGDY= 2023-10-12T01:50:27.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VfLyr5i2TXpnujRYnpCvxe5tGDY=', 'Content-Length': '0'} 2023-10-12T01:50:27.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-72:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d8f1bcb0698b2b6-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.339 DEBUG:boto:path=/wafmoa-73/obj1 2023-10-12T01:50:27.340 DEBUG:boto:auth_path=/wafmoa-73/obj1 2023-10-12T01:50:27.340 DEBUG:boto:Method: GET 2023-10-12T01:50:27.341 DEBUG:boto:Path: /wafmoa-73/obj1 2023-10-12T01:50:27.341 DEBUG:boto:Data: 2023-10-12T01:50:27.341 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.342 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.343 DEBUG:boto:Params: {} 2023-10-12T01:50:27.343 DEBUG:boto:Token: None 2023-10-12T01:50:27.343 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj1 2023-10-12T01:50:27.344 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:zVXCAKNAtDyyD+ooC9vF30NkKQM= 2023-10-12T01:50:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zVXCAKNAtDyyD+ooC9vF30NkKQM=', 'Content-Length': '0'} 2023-10-12T01:50:27.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-72:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e2e343c7e16c1a7-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.347 DEBUG:boto:path=/wafmoa-73/obj2 2023-10-12T01:50:27.347 DEBUG:boto:auth_path=/wafmoa-73/obj2 2023-10-12T01:50:27.348 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.348 DEBUG:boto:Path: /wafmoa-73/obj2 2023-10-12T01:50:27.349 DEBUG:boto:Data: 2023-10-12T01:50:27.349 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.350 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.350 DEBUG:boto:Params: {} 2023-10-12T01:50:27.351 DEBUG:boto:Token: None 2023-10-12T01:50:27.351 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj2 2023-10-12T01:50:27.351 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:nlT/+EG485G7VSpAlLsBA/z+Uvg= 2023-10-12T01:50:27.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:nlT/+EG485G7VSpAlLsBA/z+Uvg=', 'Content-Length': '0'} 2023-10-12T01:50:27.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-72:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec22df839a78aadc-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.354 DEBUG:boto:path=/wafmoa-73/obj2 2023-10-12T01:50:27.354 DEBUG:boto:auth_path=/wafmoa-73/obj2 2023-10-12T01:50:27.355 DEBUG:boto:Method: GET 2023-10-12T01:50:27.355 DEBUG:boto:Path: /wafmoa-73/obj2 2023-10-12T01:50:27.356 DEBUG:boto:Data: 2023-10-12T01:50:27.356 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.357 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.357 DEBUG:boto:Params: {} 2023-10-12T01:50:27.358 DEBUG:boto:Token: None 2023-10-12T01:50:27.358 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj2 2023-10-12T01:50:27.359 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:J9zMW7Ri9xjKEeGM8o/K+9/0RWM= 2023-10-12T01:50:27.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:J9zMW7Ri9xjKEeGM8o/K+9/0RWM=', 'Content-Length': '0'} 2023-10-12T01:50:27.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-72:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a5a7ff12da3560e-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.406 DEBUG:boto:path=/wafmoa-73/obj3 2023-10-12T01:50:27.407 DEBUG:boto:auth_path=/wafmoa-73/obj3 2023-10-12T01:50:27.407 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.408 DEBUG:boto:Path: /wafmoa-73/obj3 2023-10-12T01:50:27.408 DEBUG:boto:Data: 2023-10-12T01:50:27.408 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.409 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.410 DEBUG:boto:Params: {} 2023-10-12T01:50:27.410 DEBUG:boto:Token: None 2023-10-12T01:50:27.411 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj3 2023-10-12T01:50:27.411 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3iRAg0F3Q1D8bOMt1rMv9LGmBSM= 2023-10-12T01:50:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3iRAg0F3Q1D8bOMt1rMv9LGmBSM=', 'Content-Length': '0'} 2023-10-12T01:50:27.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b356c5fc3e6835a2-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.414 DEBUG:boto:path=/wafmoa-73/obj3 2023-10-12T01:50:27.415 DEBUG:boto:auth_path=/wafmoa-73/obj3 2023-10-12T01:50:27.415 DEBUG:boto:Method: GET 2023-10-12T01:50:27.415 DEBUG:boto:Path: /wafmoa-73/obj3 2023-10-12T01:50:27.416 DEBUG:boto:Data: 2023-10-12T01:50:27.416 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.417 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.418 DEBUG:boto:Params: {} 2023-10-12T01:50:27.418 DEBUG:boto:Token: None 2023-10-12T01:50:27.418 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj3 2023-10-12T01:50:27.419 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:HaBOPEHpuBOBwoahEltTNRaqoic= 2023-10-12T01:50:27.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:HaBOPEHpuBOBwoahEltTNRaqoic=', 'Content-Length': '0'} 2023-10-12T01:50:27.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e11ee1406034958c-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.422 DEBUG:boto:path=/wafmoa-73/obj4 2023-10-12T01:50:27.423 DEBUG:boto:auth_path=/wafmoa-73/obj4 2023-10-12T01:50:27.423 DEBUG:boto:Method: HEAD 2023-10-12T01:50:27.423 DEBUG:boto:Path: /wafmoa-73/obj4 2023-10-12T01:50:27.424 DEBUG:boto:Data: 2023-10-12T01:50:27.424 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.425 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.426 DEBUG:boto:Params: {} 2023-10-12T01:50:27.426 DEBUG:boto:Token: None 2023-10-12T01:50:27.427 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj4 2023-10-12T01:50:27.427 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:UgxzJ64jacOoC7yJ3PxESD3NMbc= 2023-10-12T01:50:27.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:UgxzJ64jacOoC7yJ3PxESD3NMbc=', 'Content-Length': '0'} 2023-10-12T01:50:27.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078f62eb7f1136d81-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.430 DEBUG:boto:path=/wafmoa-73/obj4 2023-10-12T01:50:27.431 DEBUG:boto:auth_path=/wafmoa-73/obj4 2023-10-12T01:50:27.431 DEBUG:boto:Method: GET 2023-10-12T01:50:27.431 DEBUG:boto:Path: /wafmoa-73/obj4 2023-10-12T01:50:27.432 DEBUG:boto:Data: 2023-10-12T01:50:27.432 DEBUG:boto:Headers: {} 2023-10-12T01:50:27.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:50:27.433 DEBUG:boto:Port: 8001 2023-10-12T01:50:27.433 DEBUG:boto:Params: {} 2023-10-12T01:50:27.434 DEBUG:boto:Token: None 2023-10-12T01:50:27.434 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:50:27 GMT /wafmoa-73/obj4 2023-10-12T01:50:27.435 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F6IJWpyOhTW/JpxKqUhRC6nGmRQ= 2023-10-12T01:50:27.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:50:27 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F6IJWpyOhTW/JpxKqUhRC6nGmRQ=', 'Content-Length': '0'} 2023-10-12T01:50:27.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:48:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069ef44158f02639e-00652750e3-2a04-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:50:27.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-wafmoa-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.541 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"wafmoa-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:50:27.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.542 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-wafmoa-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.638 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:50:27.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.640 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.739 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-12T01:50:27.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.740 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.794 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.794 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.841 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-12T01:50:27.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.843 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:27.940 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:50:27.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:27.942 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.039 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:50:28.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.041 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.141 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:28.142 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21, 2023-10-12T01:50:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:50:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:50:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:50:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:50:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.144 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:50:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:50:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:50:28.145 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:50:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:50:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.146 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:50:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:50:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:50:28.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:50:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:50:28.148 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:50:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:50:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:50:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:50:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:50:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:50:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:50:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:50:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:50:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:50:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:50:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:50:28.153 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:50:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:50:28.154 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:50:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:50:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:50:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:50:28.155 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:50:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:50:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:50:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:50:28.156 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:50:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.157 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:50:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:50:28.158 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:50:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:50:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:50:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:50:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:50:28.160 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:50:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:50:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:50:28.161 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:50:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:50:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:50:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:50:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:50:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:50:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:50:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:50:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:50:28.165 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:50:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:50:28.166 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:50:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:50:28.167 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:50:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:50:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:50:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:50:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:50:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:50:28.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:50:28.170 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:50:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:50:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:50:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.171 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-10-12T01:50:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:50:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:50:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:50:28.174 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:50:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:50:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.177 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:50:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:50:28.178 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:50:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:50:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:50:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:50:28.179 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:50:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:50:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:50:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:50:28.180 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:50:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:50:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:50:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:50:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:50:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:50:28.183 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:50:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:50:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:50:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:50:28.184 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:50:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:50:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:50:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:50:28.185 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:50:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.186 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:50:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:50:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:50:28.187 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:50:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:50:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:50:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:50:28.189 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:50:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:50:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:50:28.190 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:50:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:50:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:50:28.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:50:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:50:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.193 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.193 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:50:28.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:50:28.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:50:28.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:28.196 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:50:28.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:50:28.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:28.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.198 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:50:28.198 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:50:28.198 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:50:28.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:50:28.199 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:50:28.199 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:50:28.199 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:50:28.199 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:50:28.199 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:50:28.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.200 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:50:28.200 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:50:28.200 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:50:28.200 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:50:28.201 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:50:28.201 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:50:28.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.201 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:50:28.202 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:50:28.202 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:50:28.202 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:50:28.202 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:50:28.203 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:50:28.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.203 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:50:28.203 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:50:28.203 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:50:28.204 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:50:28.204 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:50:28.204 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:50:28.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.205 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:50:28.205 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:50:28.205 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:50:28.205 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:50:28.205 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:50:28.206 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:50:28.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:28.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:28.206 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:50:28.207 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:50:28.207 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:28.207 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:28.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.209 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.273 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.302 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:50:28.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.303 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.366 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.367 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.473 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:50:28.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.475 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.566 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.719 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:50:28.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-72', '--source-bucket=wafmoa-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-72', '--source-bucket=wafmoa-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.721 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-72 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-72 --source-bucket=wafmoa-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.808 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:28.835 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2023-10-12T01:50:28.743+0000 7fd50519e700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:50:28.902 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:50:28.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-72', '--source-bucket=wafmoa-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-72', '--source-bucket=wafmoa-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:28.904 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-73 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-72 --source-bucket=wafmoa-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:28.977 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:28.977 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:29.061 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:50:29.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:29.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:29.062 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:29.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:29.125 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:29.174 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-72","source":{"bucket":"wafmoa-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-73","source":{"bucket":"wafmoa-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:50:29.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:29.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:50:29.176 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:50:29.221 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:29.221 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:29.270 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:50:29.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:29.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:29.272 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:29.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:29.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:29.352 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:50:29.365 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:50:29.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:50:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:50:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:50:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:50:29.160408Z" 2023-10-12T01:50:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:50:29.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.369 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:50:29.371 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:29.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:50:29.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:29.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:29.372 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:29.421 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:29.422 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:29.456+0000 7efffc0c0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:29.457+0000 7efffc0c0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:29.471 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:29.471 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:29.471 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:29.472 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:29.472 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:29.472 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:29.472 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:29.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.473 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:29.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.473 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:29.473 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:29.474 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:29.474 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:29.474 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:29.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:29.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:29.475 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:29.475 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:29.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.476 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:29.476 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:29.476 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:29.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:29.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:29.477 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:29.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:29.477 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:29.478 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:29.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.479 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:29.479 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:29.479 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:29.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:29.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:29.480 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:29.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:29.480 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:29.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:29.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:29.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:29.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:29.482 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:29.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:29.483 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:29.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:29.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:29.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:29.483 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:29.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:29.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:29.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.484 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:29.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:29.485 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:29.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:29.486 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:29Z" 2023-10-12T01:50:29.486 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:29.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:29Z" } 2023-10-12T01:50:29.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:29.489 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:29.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:29.490 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:50:34.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:34.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:34.492 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:34.549 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:34.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:34.582+0000 7f797f509780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:34.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:34.583+0000 7f797f509780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:34.597 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:34.598 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.599 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:34.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:34.601 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:34.602 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:34.603 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:34.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:34.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:34.604 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:34.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:34.605 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:34.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:34.606 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:34.606 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:34.606 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:34Z" 2023-10-12T01:50:34.612 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:34.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:34Z" } 2023-10-12T01:50:34.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:34.614 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:34.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:34.615 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:50:39.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:39.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:39.617 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:39.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:39.676 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:39.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:39.708+0000 7f840d4ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:39.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:39.709+0000 7f840d4ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:39.724 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:39.724 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:39.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:39.726 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:39.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:39.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:39.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:39.731 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:39.732 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:39.734 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:39.735 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:39.737 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:39Z" 2023-10-12T01:50:39.738 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:39.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:39Z" } 2023-10-12T01:50:39.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:39.740 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:39.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:39.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:50:44.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:44.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:44.743 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:44.801 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:44.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:44.834+0000 7fe89c77b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:44.835+0000 7fe89c77b780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:44.849 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:44Z" 2023-10-12T01:50:44.864 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:44.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:44Z" } 2023-10-12T01:50:44.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:44.866 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:44.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:44.867 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:50:49.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:49.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:49.869 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:49.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:49.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:49.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:49.959+0000 7f9caa829780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:49.960+0000 7f9caa829780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:49.973 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:49.974 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:49.974 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:49.975 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:49.975 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:49.975 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:49.975 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:49.976 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:49.977 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:49.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:49.979 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:49.980 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:49.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:49.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:49.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:49.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:49.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:49.983 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:49.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:49.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:49.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:49.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:49.988 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:49Z" 2023-10-12T01:50:49.988 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:49.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:49Z" } 2023-10-12T01:50:49.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:49.990 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:49.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:49.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:50:54.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:54.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:50:54.993 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:50:55.050 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:50:55.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:50:55.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:55.082+0000 7f38620d6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:55.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:50:55.083+0000 7f38620d6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:50:55.097 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:50:55.097 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:50:55.097 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:50:55.098 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:50:55.098 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:50:55.098 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:50:55.098 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:55.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.099 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:50:55.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:55.099 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:50:55.099 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:55.100 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:55.100 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:50:55.100 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:55.100 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:50:55.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:55.101 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:50:55.101 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:55.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:55.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:55.102 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:50:55.102 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:55.102 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:55.102 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:50:55.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:55.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:55.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:55.103 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:50:55.103 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:50:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:55.105 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:55.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075327.839417_985.1", 2023-10-12T01:50:55.105 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:55.105 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:55.105 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:47.839417Z", 2023-10-12T01:50:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:55.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:50:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:50:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:50:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:50:55.107 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:50:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:50:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:50:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:50:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:50:55.108 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:50:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:50:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:50:55.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:50:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:50:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:50:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:50:55.110 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:50:55Z" 2023-10-12T01:50:55.111 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:50:55.112 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075327.839417_985.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:47.839417Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:50:55Z" } 2023-10-12T01:50:55.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075327.839417_985.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:47.839417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:50:55.113 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:55.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075327.839417_985.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:50:55.113 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075429.160408_1037.1 target=1_1697075327.839417_985.1 2023-10-12T01:51:00.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.116 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:00.172 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:00.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:00.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:51:00.204+0000 7fb53ab1e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:51:00.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:51:00.205+0000 7fb53ab1e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:51:00.220 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:00.220 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:51:00.221 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.222 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:51:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:51:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:51:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:51:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:51:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:51:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:51:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:51:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:51:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:51:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:51:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:00Z" 2023-10-12T01:51:00.235 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:00.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075429.160408_1037.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:50:29.160408Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:51:00Z" } 2023-10-12T01:51:00.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:51:00.238 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:51:00.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:51:00.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:51:00.239 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:51:00.239 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:51:00.240 DEBUG:boto:Method: HEAD 2023-10-12T01:51:00.240 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:51:00.241 DEBUG:boto:Data: 2023-10-12T01:51:00.241 DEBUG:boto:Headers: {} 2023-10-12T01:51:00.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.242 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.243 DEBUG:boto:Params: {} 2023-10-12T01:51:00.243 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:51:00.244 DEBUG:boto:Token: None 2023-10-12T01:51:00.245 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-72/ 2023-10-12T01:51:00.245 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r3vTg+Xo1mNrco9iuj5Et48vFRw= 2023-10-12T01:51:00.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r3vTg+Xo1mNrco9iuj5Et48vFRw=', 'Content-Length': '0'} 2023-10-12T01:51:00.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b28b685cf2b68636-0065275104-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.252 DEBUG:boto:path=/wafmoa-72/obj5 2023-10-12T01:51:00.253 DEBUG:boto:auth_path=/wafmoa-72/obj5 2023-10-12T01:51:00.253 DEBUG:boto:Method: PUT 2023-10-12T01:51:00.254 DEBUG:boto:Path: /wafmoa-72/obj5 2023-10-12T01:51:00.254 DEBUG:boto:Data: 2023-10-12T01:51:00.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:51:00.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.256 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.256 DEBUG:boto:Params: {} 2023-10-12T01:51:00.257 DEBUG:boto:Token: None 2023-10-12T01:51:00.258 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-72/obj5 2023-10-12T01:51:00.258 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:yEBsFJGvokzFZVJg39L8n83+zL0= 2023-10-12T01:51:00.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yEBsFJGvokzFZVJg39L8n83+zL0='} 2023-10-12T01:51:00.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1785c6ea41aaaca-0065275104-2afb-a1'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.269 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:51:00.270 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:51:00.270 DEBUG:boto:Method: HEAD 2023-10-12T01:51:00.273 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:51:00.274 DEBUG:boto:Data: 2023-10-12T01:51:00.275 DEBUG:boto:Headers: {} 2023-10-12T01:51:00.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.275 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.276 DEBUG:boto:Params: {} 2023-10-12T01:51:00.276 DEBUG:boto:Token: None 2023-10-12T01:51:00.277 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-72/ 2023-10-12T01:51:00.277 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:r3vTg+Xo1mNrco9iuj5Et48vFRw= 2023-10-12T01:51:00.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:r3vTg+Xo1mNrco9iuj5Et48vFRw=', 'Content-Length': '0'} 2023-10-12T01:51:00.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041bab3b089ec8c89-0065275104-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.281 DEBUG:boto:path=/wafmoa-72/obj6 2023-10-12T01:51:00.282 DEBUG:boto:auth_path=/wafmoa-72/obj6 2023-10-12T01:51:00.282 DEBUG:boto:Method: PUT 2023-10-12T01:51:00.283 DEBUG:boto:Path: /wafmoa-72/obj6 2023-10-12T01:51:00.283 DEBUG:boto:Data: 2023-10-12T01:51:00.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:51:00.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.284 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.285 DEBUG:boto:Params: {} 2023-10-12T01:51:00.285 DEBUG:boto:Token: None 2023-10-12T01:51:00.286 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-72/obj6 2023-10-12T01:51:00.286 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:DOQZwP1VZ9RTMCDrjEPKndePCpw= 2023-10-12T01:51:00.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DOQZwP1VZ9RTMCDrjEPKndePCpw='} 2023-10-12T01:51:00.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000534efcaa00216330-0065275104-2afb-a1'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.299 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:51:00.299 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:51:00.300 DEBUG:boto:Method: HEAD 2023-10-12T01:51:00.300 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:51:00.300 DEBUG:boto:Data: 2023-10-12T01:51:00.301 DEBUG:boto:Headers: {} 2023-10-12T01:51:00.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.302 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.302 DEBUG:boto:Params: {} 2023-10-12T01:51:00.303 DEBUG:boto:Token: None 2023-10-12T01:51:00.303 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-73/ 2023-10-12T01:51:00.303 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AaA3aHgSYtR4YIJE8q4c2G7bGCw= 2023-10-12T01:51:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AaA3aHgSYtR4YIJE8q4c2G7bGCw=', 'Content-Length': '0'} 2023-10-12T01:51:00.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007775da1bd6af22df-0065275104-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.307 DEBUG:boto:path=/wafmoa-73/obj7 2023-10-12T01:51:00.307 DEBUG:boto:auth_path=/wafmoa-73/obj7 2023-10-12T01:51:00.308 DEBUG:boto:Method: PUT 2023-10-12T01:51:00.308 DEBUG:boto:Path: /wafmoa-73/obj7 2023-10-12T01:51:00.309 DEBUG:boto:Data: 2023-10-12T01:51:00.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:51:00.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.311 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.311 DEBUG:boto:Params: {} 2023-10-12T01:51:00.312 DEBUG:boto:Token: None 2023-10-12T01:51:00.312 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-73/obj7 2023-10-12T01:51:00.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:49tntU2oRWbgsM6Z1P5bAovexeA= 2023-10-12T01:51:00.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:49tntU2oRWbgsM6Z1P5bAovexeA='} 2023-10-12T01:51:00.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8689ea3576e501b-0065275104-2afb-a1'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.359 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:51:00.360 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:51:00.360 DEBUG:boto:Method: HEAD 2023-10-12T01:51:00.361 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:51:00.361 DEBUG:boto:Data: 2023-10-12T01:51:00.361 DEBUG:boto:Headers: {} 2023-10-12T01:51:00.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.363 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.363 DEBUG:boto:Params: {} 2023-10-12T01:51:00.364 DEBUG:boto:Token: None 2023-10-12T01:51:00.365 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-73/ 2023-10-12T01:51:00.366 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:AaA3aHgSYtR4YIJE8q4c2G7bGCw= 2023-10-12T01:51:00.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AaA3aHgSYtR4YIJE8q4c2G7bGCw=', 'Content-Length': '0'} 2023-10-12T01:51:00.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb5d5f9e50ef413a-0065275104-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.370 DEBUG:boto:path=/wafmoa-73/obj8 2023-10-12T01:51:00.370 DEBUG:boto:auth_path=/wafmoa-73/obj8 2023-10-12T01:51:00.371 DEBUG:boto:Method: PUT 2023-10-12T01:51:00.371 DEBUG:boto:Path: /wafmoa-73/obj8 2023-10-12T01:51:00.372 DEBUG:boto:Data: 2023-10-12T01:51:00.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:51:00.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:51:00.373 DEBUG:boto:Port: 8000 2023-10-12T01:51:00.374 DEBUG:boto:Params: {} 2023-10-12T01:51:00.374 DEBUG:boto:Token: None 2023-10-12T01:51:00.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:51:00 GMT /wafmoa-73/obj8 2023-10-12T01:51:00.375 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:K+y1gbsErbzwVUpc5dupe+zAmhs= 2023-10-12T01:51:00.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:51:00 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K+y1gbsErbzwVUpc5dupe+zAmhs='} 2023-10-12T01:51:00.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004312b54e7630df96-0065275104-2afb-a1'), ('Date', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:51:00.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.384 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:00.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:00.466 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:51:00.479 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:51:00.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:51:00.479 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:51:00.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:51:00.480 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:51:00.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.481 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:51:00.481 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:50:29.160408Z" 2023-10-12T01:51:00.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:51:00.482 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:51:00.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.482 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:51:00.484 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:51:00.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:51:00.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.485 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:00.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:00.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:00.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:51:00.562+0000 7f4302671780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:51:00.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:51:00.563+0000 7f4302671780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:51:00.577 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:51:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:51:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:51:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:51:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:51:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:51:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.585 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:51:00.585 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.585 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.586 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:51:00.586 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.587 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:00.587 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.587 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:51:00.587 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:51:00.588 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.588 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.588 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:51:00.588 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:51:00.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.589 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:00.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.590 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:00.590 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:51:00.590 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:51:00.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.590 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:00Z" 2023-10-12T01:51:00.591 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:00.592 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075429.160408_1037.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:50:29.160408Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:51:00Z" } 2023-10-12T01:51:00.592 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:51:00.593 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:51:00.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:51:00.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:51:00.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:00.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:00.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:00.694 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:51:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:51:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:51:00.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:51:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:51:00.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:51:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:51:00.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.698 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:51:00.699 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:00.700 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:51:00.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:00.701 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:00.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:00.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:00.796 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:00.796 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:00.798 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.799 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:00.801 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.802 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:00.803 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:00.805 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:00.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:00.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:00.809 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:00Z" 2023-10-12T01:51:00.809 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:00.810 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:00Z" } 2023-10-12T01:51:00.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:00.811 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:00.811 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:00.812 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:05.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:05.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:05.814 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:05.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:05.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:05.921 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:05.926 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:05Z" 2023-10-12T01:51:05.934 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:05.935 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:05Z" } 2023-10-12T01:51:05.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:05.936 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:05.936 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:05.936 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:10.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:10.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:10.939 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:10.998 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:10.999 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:11.048 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:11.048 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:11.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:11.050 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:11.051 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:11.052 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:11.053 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:11.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:11.057 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:11.058 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.059 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:11.060 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:11.060 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:11.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:11.060 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:11Z" 2023-10-12T01:51:11.060 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:11.061 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:11Z" } 2023-10-12T01:51:11.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:11.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:11.063 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:11.063 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:16.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:16.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:16.066 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:16.125 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:16.126 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:16.175 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:16.175 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:16.175 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:16.175 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:16.176 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:16.176 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:16.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.176 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:16.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:16.177 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:16.177 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:16.177 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:16.177 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:16.178 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:16.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:16.178 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:16.178 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:16.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:16.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:16.179 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:16.179 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:16.180 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:16.180 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:16.180 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:16.180 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:16.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:16.181 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:16.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:16.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:16.182 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:16.182 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:16.182 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:16.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:16.182 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:16.183 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:16.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:16.183 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:16.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:16.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:16.184 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:16.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:16.184 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:16.184 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:16.185 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:16.185 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:16.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:16.185 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:16.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:16.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:16.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:16.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.186 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:16.187 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:16.187 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:16.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:16.187 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:16Z" 2023-10-12T01:51:16.188 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:16.189 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:16Z" } 2023-10-12T01:51:16.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:16.190 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:16.190 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:16.191 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:21.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:21.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:21.194 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:21.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:21.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:21.303 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:21.304 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:21.304 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:21.304 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:21.305 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:21.305 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:21.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.305 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:21.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:21.314 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:21.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:21.314 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:21.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:21.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:21.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:21.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.315 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:21.315 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:21.316 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:21.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:21.316 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:21Z" 2023-10-12T01:51:21.316 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:21.317 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:21Z" } 2023-10-12T01:51:21.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:21.318 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:21.319 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:21.319 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:26.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:26.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:26.321 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:26.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:26.380 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:26.428 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:26.428 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:26.429 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:26.429 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:26.429 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:26.429 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:26.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.430 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:26.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:26.430 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:26.430 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:26.431 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:26.431 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:26.431 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:26.431 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:26.432 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:26.432 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:26.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:26.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:26.433 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:26.433 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:26.433 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:26Z" 2023-10-12T01:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:26.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:26Z" } 2023-10-12T01:51:26.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:26.444 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:26.444 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:26.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:31.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:31.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:31.447 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:31.507 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:31.507 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:31.557 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:31Z" 2023-10-12T01:51:31.570 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:31.571 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:31Z" } 2023-10-12T01:51:31.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:31.572 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:31.572 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:31.573 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:36.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:36.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:36.575 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:36.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:36.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:36.682 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:36.683 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:36.683 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:36.683 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:36.683 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:36.683 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:36.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.684 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:36.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:36.684 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:36.685 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:36.685 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:36.685 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:36.685 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:36.685 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:36.686 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:36.686 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:36.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:36.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:36.687 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:36.687 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:36.687 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:36.687 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:36.687 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:36.688 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:36.688 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:36.688 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:36.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:36.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:36.689 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:36.689 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:36.689 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:36.690 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:36.690 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:36.690 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:36.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:36.691 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:36.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:36.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:36.691 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:36.692 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:36.692 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:36.692 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:36.692 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:36.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:36.693 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:36.693 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:36.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:36.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:36.693 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:36.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.694 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:36.694 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:36.694 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:36.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:36.695 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:36Z" 2023-10-12T01:51:36.695 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:36.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:36Z" } 2023-10-12T01:51:36.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:36.697 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:36.697 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:36.698 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:41.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:41.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:41.700 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:41.810 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:41Z" 2023-10-12T01:51:41.823 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:41.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:41Z" } 2023-10-12T01:51:41.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:41.826 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:41.826 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:41.826 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:46.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:46.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:46.829 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:46.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:46.888 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:46.937 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:46.937 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:46.938 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:46.938 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:46.938 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:46.938 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:46.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.939 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:46.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:46.939 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:46.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:46.940 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:46.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:46.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:46.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:46.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:46.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:46.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:46.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:46.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:46.942 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:46.942 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:46.942 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:46.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:46.943 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:46.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:46.943 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:46.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:46.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:46.944 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:46.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:46.944 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:46.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:46.945 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:46.945 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:46.945 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:46.945 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:46.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:46.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:46.946 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:46.947 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:46.947 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:46.947 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:46.947 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:46.948 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:46.948 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:46.948 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:46.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:46.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:46.949 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:46.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.949 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:46.949 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:46.950 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:46.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:46.950 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:46Z" 2023-10-12T01:51:46.950 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:46.954 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:46Z" } 2023-10-12T01:51:46.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:46.955 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:46.955 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:46.956 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:51.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:51.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:51.958 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:52.068 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:52.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:52.069 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:52.069 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:52.069 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.070 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:52.071 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:52.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:52.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:52.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:52.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:52.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:52.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:52.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:52.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:52.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:52.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:52.075 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:52.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:52.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:52.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:52.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:52.076 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:52.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:52.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:52.077 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:52.077 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:52.077 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:52.078 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:52.078 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:52.079 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:52.079 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:52.079 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:52.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:52.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:52.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:52.080 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:52.080 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:52.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:52.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:52.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:52.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:52.081 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:52.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:52.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:52.083 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:52.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.083 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:52.083 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:52.083 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:52.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:52.084 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:52Z" 2023-10-12T01:51:52.084 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:52.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:52Z" } 2023-10-12T01:51:52.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:52.088 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:52.088 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:52.089 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:51:57.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:57.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:51:57.091 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:51:57.150 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:51:57.150 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:51:57.199 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:51:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:51:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:51:57.200 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:51:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:51:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:51:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:51:57.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:51:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:51:57.202 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:51:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:51:57.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:51:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:57.204 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:51:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:51:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:57.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:51:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:51:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:51:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:51:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:51:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:51:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:51:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:51:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:51:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:51:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:51:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:51:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:51:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:51:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:51:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:51:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:51:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:51:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:51:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:51:57Z" 2023-10-12T01:51:57.212 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:51:57.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:51:57Z" } 2023-10-12T01:51:57.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:51:57.214 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:51:57.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:51:57.215 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:02.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:02.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:02.217 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:02.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:02.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:02.326 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:02.326 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:02.327 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:02.327 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:02.327 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:02.327 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:02.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.328 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:02.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:02.328 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:02.328 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:02.329 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:02.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:02.331 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:02.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:02.333 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:02.334 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:02.335 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:02.336 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:02.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:02.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:02.339 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:02Z" 2023-10-12T01:52:02.339 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:02.340 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:02Z" } 2023-10-12T01:52:02.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:02.341 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:02.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:02.342 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:07.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:07.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:07.345 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:07.405 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:07.406 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:07.456 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:07.457 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:07.457 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:07.457 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:07.457 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:07.458 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:07.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.458 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:07.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:07.459 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:07.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:07.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:07.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:07.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:07.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:07.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:07.466 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:07.467 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:07.469 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:07Z" 2023-10-12T01:52:07.470 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:07.471 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:07Z" } 2023-10-12T01:52:07.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:07.472 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:07.472 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:07.472 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:12.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:12.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:12.475 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:12.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:12.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:12.582 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:12.583 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:12.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:12.586 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:12.587 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:12.588 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:12.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:12.590 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:12.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:12.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:12.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:12.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:12.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:12.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.597 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:12.598 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:12Z" 2023-10-12T01:52:12.598 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:12.600 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:12Z" } 2023-10-12T01:52:12.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:12.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:12.601 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:12.602 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:17.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:17.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:17.604 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:17.713 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:17.714 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.715 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:17.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:17.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:17.718 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:17.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:17.720 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:17.721 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:17.722 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:17.722 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:17.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:17.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:17.723 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:17.723 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:17.723 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:17.723 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:17.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:17.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:17.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.725 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:17.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:17.726 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:17.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:17.726 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:17Z" 2023-10-12T01:52:17.726 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:17.727 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:17Z" } 2023-10-12T01:52:17.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:17.728 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:17.729 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:17.729 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:22.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:22.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:22.731 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:22.788 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:22.789 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:22.836 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:22.836 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.837 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:22.838 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:22.839 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:22.840 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:22.841 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:22.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:22.843 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:22.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:22.845 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.848 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:22.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:22.848 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:22.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:22.848 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:22Z" 2023-10-12T01:52:22.849 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:22.850 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:22Z" } 2023-10-12T01:52:22.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:22.851 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:22.851 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:22.852 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:27.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:27.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:27.854 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:27.912 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:27.913 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:27.963 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:27.963 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.964 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:27.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032620", 2023-10-12T01:52:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.282204Z" 2023-10-12T01:52:27.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:27.967 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-10-12T01:52:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:27.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.247120Z" 2023-10-12T01:52:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:27.970 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:27.970 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:27.970 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:27.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:27.970 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:27.971 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:27.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:27.971 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:27.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:27.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:27.972 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:27.972 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:27.972 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:27.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046219", 2023-10-12T01:52:27.973 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:27.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:27.973 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:27.973 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:54.182384Z" 2023-10-12T01:52:27.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:27.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:27.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:27.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.974 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:27.975 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:27.975 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:27.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:27.975 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:27Z" 2023-10-12T01:52:27.975 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:27.976 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032620", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:48:54.282204Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:48:54.247120Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046219", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:54.182384Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:27Z" } 2023-10-12T01:52:27.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032620', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.282204Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.247120Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046219', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:54.182384Z'}}] 2023-10-12T01:52:27.977 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:27.978 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032620', 1: '00000000000000000000:00000000000000052091', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046219'} 2023-10-12T01:52:27.978 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032761 target=00000000000000000000:00000000000000032620, shard=1 master=00000000000000000000:00000000000000052373 target=00000000000000000000:00000000000000052091, shard=3 master=00000000000000000000:00000000000000046360 target=00000000000000000000:00000000000000046219 2023-10-12T01:52:32.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:32.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:32.981 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:33.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:33.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:33.089 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:33.089 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:33.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.091 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:52:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:33.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:52:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.093 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:33.094 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.094 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:33.094 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:52:33.094 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.095 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:33.095 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.095 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:52:33.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.096 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:33.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.096 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:33.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:33.097 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:33.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.097 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:33.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.098 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:33.098 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.098 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:33.099 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:52:33.099 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:33.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.100 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:52:33.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.100 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:33.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.101 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:33.101 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:33.101 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:33.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.101 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:33Z" 2023-10-12T01:52:33.102 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:33.103 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:33Z" } 2023-10-12T01:52:33.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:52:33.104 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:33.104 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:33.104 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:52:33.105 DEBUG:boto:path=/wafmoa-73/ 2023-10-12T01:52:33.105 DEBUG:boto:auth_path=/wafmoa-73/ 2023-10-12T01:52:33.106 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.106 DEBUG:boto:Path: /wafmoa-73/ 2023-10-12T01:52:33.107 DEBUG:boto:Data: 2023-10-12T01:52:33.107 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.108 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.108 DEBUG:boto:Params: {} 2023-10-12T01:52:33.109 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:52:33.110 DEBUG:boto:Token: None 2023-10-12T01:52:33.110 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-73/ 2023-10-12T01:52:33.111 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:h/v8rDKgrT+PQwBATEH43R6ZddE= 2023-10-12T01:52:33.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:h/v8rDKgrT+PQwBATEH43R6ZddE=', 'Content-Length': '0'} 2023-10-12T01:52:33.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007337f0aeee56de03-0065275161-2afd-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.117 DEBUG:boto:path=/wafmoa-73/obj5 2023-10-12T01:52:33.117 DEBUG:boto:auth_path=/wafmoa-73/obj5 2023-10-12T01:52:33.118 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.118 DEBUG:boto:Path: /wafmoa-73/obj5 2023-10-12T01:52:33.118 DEBUG:boto:Data: 2023-10-12T01:52:33.119 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.120 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.120 DEBUG:boto:Params: {} 2023-10-12T01:52:33.120 DEBUG:boto:Token: None 2023-10-12T01:52:33.121 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-73/obj5 2023-10-12T01:52:33.121 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:RZp7wnTPNIH03N2k+RP/Mq7Zx/A= 2023-10-12T01:52:33.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RZp7wnTPNIH03N2k+RP/Mq7Zx/A=', 'Content-Length': '0'} 2023-10-12T01:52:33.123 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000979abc5a811f28b5-0065275161-2afd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.124 DEBUG:boto:path=/wafmoa-73/obj6 2023-10-12T01:52:33.124 DEBUG:boto:auth_path=/wafmoa-73/obj6 2023-10-12T01:52:33.125 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.125 DEBUG:boto:Path: /wafmoa-73/obj6 2023-10-12T01:52:33.126 DEBUG:boto:Data: 2023-10-12T01:52:33.126 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.127 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.127 DEBUG:boto:Params: {} 2023-10-12T01:52:33.128 DEBUG:boto:Token: None 2023-10-12T01:52:33.128 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-73/obj6 2023-10-12T01:52:33.129 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:4RXoQzxMG0e6mTH+HwnRYe8AYrA= 2023-10-12T01:52:33.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4RXoQzxMG0e6mTH+HwnRYe8AYrA=', 'Content-Length': '0'} 2023-10-12T01:52:33.131 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000001fec6a3734b17cf-0065275161-2afd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.131 DEBUG:boto:path=/wafmoa-73/obj7 2023-10-12T01:52:33.132 DEBUG:boto:auth_path=/wafmoa-73/obj7 2023-10-12T01:52:33.132 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.132 DEBUG:boto:Path: /wafmoa-73/obj7 2023-10-12T01:52:33.133 DEBUG:boto:Data: 2023-10-12T01:52:33.133 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.134 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.134 DEBUG:boto:Params: {} 2023-10-12T01:52:33.135 DEBUG:boto:Token: None 2023-10-12T01:52:33.135 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-73/obj7 2023-10-12T01:52:33.136 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ieSB6AfJyzRZby9iZxiyVvb+72Y= 2023-10-12T01:52:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ieSB6AfJyzRZby9iZxiyVvb+72Y=', 'Content-Length': '0'} 2023-10-12T01:52:33.138 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000338a9bf553e06da1-0065275161-2afd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.138 DEBUG:boto:path=/wafmoa-73/obj8 2023-10-12T01:52:33.139 DEBUG:boto:auth_path=/wafmoa-73/obj8 2023-10-12T01:52:33.139 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.139 DEBUG:boto:Path: /wafmoa-73/obj8 2023-10-12T01:52:33.140 DEBUG:boto:Data: 2023-10-12T01:52:33.140 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.141 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.141 DEBUG:boto:Params: {} 2023-10-12T01:52:33.142 DEBUG:boto:Token: None 2023-10-12T01:52:33.142 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-73/obj8 2023-10-12T01:52:33.143 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8JGkdmaWu9+//4qMcGNg0ec7ZVM= 2023-10-12T01:52:33.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8JGkdmaWu9+//4qMcGNg0ec7ZVM=', 'Content-Length': '0'} 2023-10-12T01:52:33.145 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b70c62373fd9937e-0065275161-2afd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.145 DEBUG:boto:path=/wafmoa-72/ 2023-10-12T01:52:33.145 DEBUG:boto:auth_path=/wafmoa-72/ 2023-10-12T01:52:33.146 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.146 DEBUG:boto:Path: /wafmoa-72/ 2023-10-12T01:52:33.147 DEBUG:boto:Data: 2023-10-12T01:52:33.147 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.148 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.148 DEBUG:boto:Params: {} 2023-10-12T01:52:33.149 DEBUG:boto:Token: None 2023-10-12T01:52:33.149 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/ 2023-10-12T01:52:33.150 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:riH/wiuk8AVXFQx0euMptbQVn9g= 2023-10-12T01:52:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:riH/wiuk8AVXFQx0euMptbQVn9g=', 'Content-Length': '0'} 2023-10-12T01:52:33.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001587cbc8b6891ad6-0065275161-2afd-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.153 DEBUG:boto:path=/wafmoa-72/obj7 2023-10-12T01:52:33.153 DEBUG:boto:auth_path=/wafmoa-72/obj7 2023-10-12T01:52:33.154 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.154 DEBUG:boto:Path: /wafmoa-72/obj7 2023-10-12T01:52:33.155 DEBUG:boto:Data: 2023-10-12T01:52:33.155 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.156 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.156 DEBUG:boto:Params: {} 2023-10-12T01:52:33.157 DEBUG:boto:Token: None 2023-10-12T01:52:33.157 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj7 2023-10-12T01:52:33.158 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qVZ8XWBX+fwpb5R0pnFaVSwPJOc= 2023-10-12T01:52:33.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qVZ8XWBX+fwpb5R0pnFaVSwPJOc=', 'Content-Length': '0'} 2023-10-12T01:52:33.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e77041f9d8397e26-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.161 DEBUG:boto:path=/wafmoa-72/obj7 2023-10-12T01:52:33.161 DEBUG:boto:auth_path=/wafmoa-72/obj7 2023-10-12T01:52:33.162 DEBUG:boto:Method: GET 2023-10-12T01:52:33.162 DEBUG:boto:Path: /wafmoa-72/obj7 2023-10-12T01:52:33.163 DEBUG:boto:Data: 2023-10-12T01:52:33.163 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.164 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.164 DEBUG:boto:Params: {} 2023-10-12T01:52:33.165 DEBUG:boto:Token: None 2023-10-12T01:52:33.165 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj7 2023-10-12T01:52:33.166 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ccKBJXOCu0zTT1NVGBJKNe8j6FU= 2023-10-12T01:52:33.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ccKBJXOCu0zTT1NVGBJKNe8j6FU=', 'Content-Length': '0'} 2023-10-12T01:52:33.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be2e6ba03a810e29-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.169 DEBUG:boto:path=/wafmoa-72/obj8 2023-10-12T01:52:33.169 DEBUG:boto:auth_path=/wafmoa-72/obj8 2023-10-12T01:52:33.170 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.170 DEBUG:boto:Path: /wafmoa-72/obj8 2023-10-12T01:52:33.171 DEBUG:boto:Data: 2023-10-12T01:52:33.171 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.172 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.172 DEBUG:boto:Params: {} 2023-10-12T01:52:33.173 DEBUG:boto:Token: None 2023-10-12T01:52:33.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj8 2023-10-12T01:52:33.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/+1n9BJt6b5NXFs9E2D93RQkAW8= 2023-10-12T01:52:33.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/+1n9BJt6b5NXFs9E2D93RQkAW8=', 'Content-Length': '0'} 2023-10-12T01:52:33.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c983fab44973f338-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.176 DEBUG:boto:path=/wafmoa-72/obj8 2023-10-12T01:52:33.177 DEBUG:boto:auth_path=/wafmoa-72/obj8 2023-10-12T01:52:33.177 DEBUG:boto:Method: GET 2023-10-12T01:52:33.178 DEBUG:boto:Path: /wafmoa-72/obj8 2023-10-12T01:52:33.178 DEBUG:boto:Data: 2023-10-12T01:52:33.179 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.179 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.180 DEBUG:boto:Params: {} 2023-10-12T01:52:33.180 DEBUG:boto:Token: None 2023-10-12T01:52:33.181 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj8 2023-10-12T01:52:33.181 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bt+N6K3AAPmQ7vOCFTeHR9+IMnw= 2023-10-12T01:52:33.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bt+N6K3AAPmQ7vOCFTeHR9+IMnw=', 'Content-Length': '0'} 2023-10-12T01:52:33.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a65a0c002a40a68-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.184 DEBUG:boto:path=/wafmoa-72/obj7 2023-10-12T01:52:33.185 DEBUG:boto:auth_path=/wafmoa-72/obj7 2023-10-12T01:52:33.185 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.186 DEBUG:boto:Path: /wafmoa-72/obj7 2023-10-12T01:52:33.186 DEBUG:boto:Data: 2023-10-12T01:52:33.187 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.187 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.188 DEBUG:boto:Params: {} 2023-10-12T01:52:33.188 DEBUG:boto:Token: None 2023-10-12T01:52:33.189 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj7 2023-10-12T01:52:33.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:qVZ8XWBX+fwpb5R0pnFaVSwPJOc= 2023-10-12T01:52:33.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qVZ8XWBX+fwpb5R0pnFaVSwPJOc=', 'Content-Length': '0'} 2023-10-12T01:52:33.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7fdf91465e2eb77-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.192 DEBUG:boto:path=/wafmoa-72/obj7 2023-10-12T01:52:33.192 DEBUG:boto:auth_path=/wafmoa-72/obj7 2023-10-12T01:52:33.193 DEBUG:boto:Method: GET 2023-10-12T01:52:33.193 DEBUG:boto:Path: /wafmoa-72/obj7 2023-10-12T01:52:33.193 DEBUG:boto:Data: 2023-10-12T01:52:33.194 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.195 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.195 DEBUG:boto:Params: {} 2023-10-12T01:52:33.196 DEBUG:boto:Token: None 2023-10-12T01:52:33.196 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj7 2023-10-12T01:52:33.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ccKBJXOCu0zTT1NVGBJKNe8j6FU= 2023-10-12T01:52:33.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ccKBJXOCu0zTT1NVGBJKNe8j6FU=', 'Content-Length': '0'} 2023-10-12T01:52:33.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006624487a1796ddd-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.199 DEBUG:boto:path=/wafmoa-72/obj8 2023-10-12T01:52:33.200 DEBUG:boto:auth_path=/wafmoa-72/obj8 2023-10-12T01:52:33.200 DEBUG:boto:Method: HEAD 2023-10-12T01:52:33.201 DEBUG:boto:Path: /wafmoa-72/obj8 2023-10-12T01:52:33.201 DEBUG:boto:Data: 2023-10-12T01:52:33.201 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.202 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.203 DEBUG:boto:Params: {} 2023-10-12T01:52:33.203 DEBUG:boto:Token: None 2023-10-12T01:52:33.204 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj8 2023-10-12T01:52:33.204 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/+1n9BJt6b5NXFs9E2D93RQkAW8= 2023-10-12T01:52:33.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/+1n9BJt6b5NXFs9E2D93RQkAW8=', 'Content-Length': '0'} 2023-10-12T01:52:33.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d0dee73670d5d85-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.207 DEBUG:boto:path=/wafmoa-72/obj8 2023-10-12T01:52:33.207 DEBUG:boto:auth_path=/wafmoa-72/obj8 2023-10-12T01:52:33.208 DEBUG:boto:Method: GET 2023-10-12T01:52:33.208 DEBUG:boto:Path: /wafmoa-72/obj8 2023-10-12T01:52:33.209 DEBUG:boto:Data: 2023-10-12T01:52:33.209 DEBUG:boto:Headers: {} 2023-10-12T01:52:33.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:52:33.210 DEBUG:boto:Port: 8001 2023-10-12T01:52:33.211 DEBUG:boto:Params: {} 2023-10-12T01:52:33.211 DEBUG:boto:Token: None 2023-10-12T01:52:33.211 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:52:33 GMT /wafmoa-72/obj8 2023-10-12T01:52:33.212 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Bt+N6K3AAPmQ7vOCFTeHR9+IMnw= 2023-10-12T01:52:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:33 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Bt+N6K3AAPmQ7vOCFTeHR9+IMnw=', 'Content-Length': '0'} 2023-10-12T01:52:33.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:51:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-73:677f5a1d-3ab4-4b5f-af15-9890c2c79648.10185.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8eb54d5a07376da-0065275161-2afd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:52:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:33.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:33.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'wafmoa-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:33.255 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket wafmoa-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:33.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:33.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:33.365 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:52:33.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:33.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:33.367 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:33.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:33.415 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:33.459 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-10-12T01:52:33.459 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2023-10-12T01:52:33.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:33.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:33.462 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:33.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:33.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:33.544 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:52:33.558 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:52:33.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.558 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:33.559 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:52:33.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.559 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:33.560 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:52:33.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075553.345544_1039.1", 2023-10-12T01:52:33.560 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:33.345544Z" 2023-10-12T01:52:33.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:33.561 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:52:33.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.562 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:52:33.563 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:33.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:52:33.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:33.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:33.565 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:33.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:33.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:33.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:33.642+0000 7ff20bed6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:33.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:33.643+0000 7ff20bed6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:33.657 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:33.657 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:33.657 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:33.658 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:33.658 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:33.658 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:33.658 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:33.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.659 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:33.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.659 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:33.659 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.660 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:33.660 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:33.660 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.660 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:52:33.661 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.661 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:52:33.661 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:33.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.662 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:33.662 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.662 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:33.663 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:33.663 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.663 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:33.663 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.663 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:52:33.664 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:33.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.665 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:33.665 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.665 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:33.665 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:52:33.665 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.666 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:33.666 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.666 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:52:33.666 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:33.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:33.667 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:33.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:33.668 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:33.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:33.668 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:33.668 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:33.669 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:33.669 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:52:33.669 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:33.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:33.670 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:33.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.670 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:33.670 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:52:33.671 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:52:33.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:33.671 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:33Z" 2023-10-12T01:52:33.671 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:33.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075429.160408_1037.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:50:29.160408Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:52:33Z" } 2023-10-12T01:52:33.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:52:33.673 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:33.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:33.674 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075553.345544_1039.1 target=1_1697075429.160408_1037.1 2023-10-12T01:52:38.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:38.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:38.676 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:38.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:38.766+0000 7f73b0122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:38.768+0000 7f73b0122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:38.782 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:38.782 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:38.782 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:38.782 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:38.783 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:38.783 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:38.783 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:38.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.785 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:38.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:38.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:38.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:38.786 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:38.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:38.787 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:38.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:52:38.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:38.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:52:38.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:52:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:52:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:52:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:52:38.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:38.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:38.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:38.796 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:38.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.797 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:38.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:52:38.797 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:52:38.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:38.797 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:38Z" 2023-10-12T01:52:38.798 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:38.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075429.160408_1037.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:50:29.160408Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:52:38Z" } 2023-10-12T01:52:38.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:52:38.800 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:38.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:38.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075553.345544_1039.1 target=1_1697075429.160408_1037.1 2023-10-12T01:52:43.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:43.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:43.803 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:43.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:43.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:43.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:43.892+0000 7faea2411780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:43.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:43.893+0000 7faea2411780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:43.906 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:43.907 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:43.907 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:43.907 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:43.907 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:43.907 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:43.908 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:43.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.908 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:43.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:43.908 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:43.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:43.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:43.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:43.909 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:43.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:52:43.910 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:43.910 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:52:43.910 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:43.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:43.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:43.911 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:43.911 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:43.912 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:43.912 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:43.912 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:43.912 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:43.912 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:43.913 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:52:43.913 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:43.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:43.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:43.914 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:43.914 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:43.914 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:43.914 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075553.345544_1039.1", 2023-10-12T01:52:43.915 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:43.915 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:43.915 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:43.915 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:33.345544Z", 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:43.916 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:43.917 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:43.917 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:43.917 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:43.917 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:43.917 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:43.918 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:52:43.918 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:43.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:43.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:43.918 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:43.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.919 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:43.919 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:52:43.919 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:52:43.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:43.920 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:43Z" 2023-10-12T01:52:43.920 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:43.921 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075553.345544_1039.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:33.345544Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:52:43Z" } 2023-10-12T01:52:43.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075553.345544_1039.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:33.345544Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:52:43.922 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:43.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:43.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:52:43.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:43.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:43.924 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:43.970 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.016 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-10-12T01:52:44.017 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-74 2023-10-12T01:52:44.018 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:52:44.018 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:52:44.018 DEBUG:boto:Method: PUT 2023-10-12T01:52:44.019 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:52:44.019 DEBUG:boto:Data: 2023-10-12T01:52:44.020 DEBUG:boto:Headers: {} 2023-10-12T01:52:44.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:44.020 DEBUG:boto:Port: 8000 2023-10-12T01:52:44.021 DEBUG:boto:Params: {} 2023-10-12T01:52:44.021 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:52:44.022 DEBUG:boto:Token: None 2023-10-12T01:52:44.022 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:52:44 GMT /wafmoa-74/ 2023-10-12T01:52:44.023 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:9QsSXpup9CUoG1SMJeQ7jJz2KrI= 2023-10-12T01:52:44.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9QsSXpup9CUoG1SMJeQ7jJz2KrI=', 'Content-Length': '0'} 2023-10-12T01:52:44.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c2b72d4e05395b5-006527516c-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:44.041 INFO:rgw_multi.tests:create bucket zone=a1 name=wafmoa-75 2023-10-12T01:52:44.042 DEBUG:boto:path=/wafmoa-75/ 2023-10-12T01:52:44.042 DEBUG:boto:auth_path=/wafmoa-75/ 2023-10-12T01:52:44.043 DEBUG:boto:Method: PUT 2023-10-12T01:52:44.043 DEBUG:boto:Path: /wafmoa-75/ 2023-10-12T01:52:44.043 DEBUG:boto:Data: 2023-10-12T01:52:44.044 DEBUG:boto:Headers: {} 2023-10-12T01:52:44.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:44.044 DEBUG:boto:Port: 8000 2023-10-12T01:52:44.045 DEBUG:boto:Params: {} 2023-10-12T01:52:44.045 DEBUG:boto:Token: None 2023-10-12T01:52:44.046 DEBUG:boto:StringToSign: PUT Thu, 12 Oct 2023 01:52:44 GMT /wafmoa-75/ 2023-10-12T01:52:44.046 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:S0dp9zxkBo7dNpAn8w0X3d8O1DY= 2023-10-12T01:52:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S0dp9zxkBo7dNpAn8w0X3d8O1DY=', 'Content-Length': '0'} 2023-10-12T01:52:44.061 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000719e264dc552b955-006527516c-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:44.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:44.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:44.063 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:44.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.148 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:52:44.165 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:52:44.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:44.166 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:52:44.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:52:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:52:44.168 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:44.060671Z" 2023-10-12T01:52:44.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:44.168 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:52:44.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.169 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:52:44.170 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:44.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:52:44.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:44.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:44.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:44.221 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.221 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:44.249+0000 7f6e838ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:44.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:44.250+0000 7f6e838ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:44.262 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:44.262 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:44.263 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:44.263 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:44.263 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:44.263 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:52:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:52:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:52:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:52:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:52:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:52:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:52:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:52:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:44Z" 2023-10-12T01:52:44.276 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:44.277 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:52:44Z" } 2023-10-12T01:52:44.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:52:44.278 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:44.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:44.279 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:52:44.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.280 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.376 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:52:44.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.378 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.480 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:52:44.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.481 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-74 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.583 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:52:44.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.585 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-75 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.639 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.687 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:52:44.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.689 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.744 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.789 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:52:44.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.791 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.887 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:52:44.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:44.888 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:44.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:44.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:44.987 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 22, 2023-10-12T01:52:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:52:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:52:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:52:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:52:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:52:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:52:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:52:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:52:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:52:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:52:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:52:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:52:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:52:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:52:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:52:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:52:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:52:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:52:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:52:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:52:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:52:44.994 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:52:44.994 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:52:44.994 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:52:44.994 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:52:44.994 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:44.995 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:44.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:44.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:44.996 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:52:44.996 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:52:44.996 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:44.996 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:52:44.996 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:44.997 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:52:44.997 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:52:44.997 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:52:44.997 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:52:44.997 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:52:44.998 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:52:44.998 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:52:44.998 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:52:44.998 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:52:44.999 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:52:44.999 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:44.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:44.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:44.999 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.000 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:52:45.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.000 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:52:45.000 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:52:45.000 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:52:45.001 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:52:45.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.001 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.001 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:52:45.002 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:52:45.002 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:52:45.002 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:52:45.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.002 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:52:45.003 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:52:45.003 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:52:45.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.003 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:52:45.003 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.004 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:52:45.004 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:52:45.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.005 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.005 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:52:45.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.005 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-74", 2023-10-12T01:52:45.006 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.006 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:52:45.006 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.006 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:52:45.006 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.007 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:52:45.007 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:52:45.007 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.008 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:52:45.008 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.008 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:52:45.008 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.009 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:52:45.009 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:52:45.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.009 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:52:45.010 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:52:45.010 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:52:45.010 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:52:45.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.011 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-75", 2023-10-12T01:52:45.011 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.011 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:52:45.011 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.012 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:52:45.012 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.012 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:52:45.013 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-75", 2023-10-12T01:52:45.013 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.013 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:52:45.013 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.014 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:52:45.014 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.014 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:52:45.014 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:52:45.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.015 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:52:45.015 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:52:45.015 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:52:45.015 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:52:45.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.016 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:52:45.016 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.017 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:52:45.017 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.017 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:52:45.017 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.018 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:52:45.018 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-75", 2023-10-12T01:52:45.018 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.018 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:52:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:52:45.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.020 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:52:45.020 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:52:45.021 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:52:45.021 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:52:45.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.021 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.022 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-10-12T01:52:45.022 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.022 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.022 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:52:45.023 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:45.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.023 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:52:45.024 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:52:45.024 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:52:45.024 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:52:45.024 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:45.025 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:52:45.025 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.025 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:52:45.025 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:52:45.025 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:52:45.026 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:52:45.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.027 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:52:45.027 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:52:45.027 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:45.028 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:52:45.028 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.028 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:52:45.028 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:52:45.028 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:52:45.029 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:52:45.029 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:52:45.029 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:52:45.029 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:52:45.030 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:52:45.030 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:52:45.030 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:52:45.030 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:45.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.031 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:52:45.031 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:52:45.032 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:52:45.032 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:52:45.032 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.032 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:52:45.032 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:52:45.033 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:52:45.033 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:52:45.033 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:52:45.033 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:52:45.033 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:52:45.034 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:52:45.034 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:52:45.034 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:52:45.034 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:45.035 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.035 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.035 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:52:45.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.036 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:52:45.036 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:52:45.036 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:52:45.036 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:52:45.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.037 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.037 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.037 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:52:45.037 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:52:45.038 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:52:45.038 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:52:45.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.038 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:52:45.038 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:52:45.039 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.039 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:52:45.039 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:52:45.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.040 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:52:45.040 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:52:45.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:52:45.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:52:45.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:52:45.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:52:45.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:52:45.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:52:45.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.044 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:52:45.044 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:52:45.044 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:52:45.044 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:52:45.044 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:52:45.045 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:52:45.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:52:45.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:52:45.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:52:45.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.046 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:52:45.046 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:52:45.046 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:52:45.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:52:45.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:52:45.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:52:45.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.047 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:52:45.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:52:45.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:52:45.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:52:45.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:52:45.048 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:52:45.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.049 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:52:45.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:52:45.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:52:45.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:52:45.049 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:52:45.050 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:52:45.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.050 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:52:45.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:52:45.050 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:52:45.051 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:52:45.051 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:52:45.051 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:52:45.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:52:45.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:52:45.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.052 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:45.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:45.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:52:45.054 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:52:45.099 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:45.100 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:45.158 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:52:45.159 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:52:45.159 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:52:45.159 DEBUG:boto:Method: HEAD 2023-10-12T01:52:45.160 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:52:45.160 DEBUG:boto:Data: 2023-10-12T01:52:45.161 DEBUG:boto:Headers: {} 2023-10-12T01:52:45.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:45.161 DEBUG:boto:Port: 8000 2023-10-12T01:52:45.162 DEBUG:boto:Params: {} 2023-10-12T01:52:45.162 DEBUG:boto:Token: None 2023-10-12T01:52:45.163 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:45 GMT /wafmoa-74/ 2023-10-12T01:52:45.163 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0= 2023-10-12T01:52:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0=', 'Content-Length': '0'} 2023-10-12T01:52:45.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000777211d26619a11c-006527516d-2ca6-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:45.232 DEBUG:boto:path=/wafmoa-74/obj1 2023-10-12T01:52:45.232 DEBUG:boto:auth_path=/wafmoa-74/obj1 2023-10-12T01:52:45.233 DEBUG:boto:Method: PUT 2023-10-12T01:52:45.233 DEBUG:boto:Path: /wafmoa-74/obj1 2023-10-12T01:52:45.234 DEBUG:boto:Data: 2023-10-12T01:52:45.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:52:45.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:45.235 DEBUG:boto:Port: 8000 2023-10-12T01:52:45.236 DEBUG:boto:Params: {} 2023-10-12T01:52:45.236 DEBUG:boto:Token: None 2023-10-12T01:52:45.237 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:52:45 GMT /wafmoa-74/obj1 2023-10-12T01:52:45.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:YVXUbTkAh8rBtdQbJBPs3n2ddZs= 2023-10-12T01:52:45.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YVXUbTkAh8rBtdQbJBPs3n2ddZs='} 2023-10-12T01:52:45.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a03177079e3a3f0-006527516d-2ca6-a1'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:45.258 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:52:45.259 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:52:45.259 DEBUG:boto:Method: HEAD 2023-10-12T01:52:45.260 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:52:45.260 DEBUG:boto:Data: 2023-10-12T01:52:45.260 DEBUG:boto:Headers: {} 2023-10-12T01:52:45.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:45.261 DEBUG:boto:Port: 8000 2023-10-12T01:52:45.262 DEBUG:boto:Params: {} 2023-10-12T01:52:45.262 DEBUG:boto:Token: None 2023-10-12T01:52:45.263 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:52:45 GMT /wafmoa-74/ 2023-10-12T01:52:45.263 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0= 2023-10-12T01:52:45.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0=', 'Content-Length': '0'} 2023-10-12T01:52:45.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007518ec431a1c54b8-006527516d-2ca6-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:45.268 DEBUG:boto:path=/wafmoa-74/obj2 2023-10-12T01:52:45.269 DEBUG:boto:auth_path=/wafmoa-74/obj2 2023-10-12T01:52:45.269 DEBUG:boto:Method: PUT 2023-10-12T01:52:45.270 DEBUG:boto:Path: /wafmoa-74/obj2 2023-10-12T01:52:45.270 DEBUG:boto:Data: 2023-10-12T01:52:45.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:52:45.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:52:45.271 DEBUG:boto:Port: 8000 2023-10-12T01:52:45.272 DEBUG:boto:Params: {} 2023-10-12T01:52:45.272 DEBUG:boto:Token: None 2023-10-12T01:52:45.273 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:52:45 GMT /wafmoa-74/obj2 2023-10-12T01:52:45.273 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:F1AiG+oY96278K6TuF6ZUUnkBpg= 2023-10-12T01:52:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F1AiG+oY96278K6TuF6ZUUnkBpg='} 2023-10-12T01:52:45.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055bee7e779dbddf7-006527516d-2ca6-a1'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:52:45.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.320 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:45.396 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:45.397 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:45.533 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:52:45.589 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:52:45.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:45.589 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:52:45.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.590 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:45.590 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:52:45.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:52:45.591 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:44.060671Z" 2023-10-12T01:52:45.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:45.592 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:52:45.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.592 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:52:45.593 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:45.594 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:52:45.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:45.677 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:45.678 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:45.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:45.762+0000 7f7616cdf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:45.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:52:45.764+0000 7f7616cdf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:52:45.799 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:45.799 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:45.799 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:45.799 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:45.800 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:45.800 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:52:45.800 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.800 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:52:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:52:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:52:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:52:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:52:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:52:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:52:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:52:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:52:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:52:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:52:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:52:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:52:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:45Z" 2023-10-12T01:52:45.813 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:45.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:52:45Z" } 2023-10-12T01:52:45.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:52:45.815 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:45.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:52:45.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:52:45.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.817 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:45.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:45.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:45.969 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:52:45.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:52:45.970 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:52:45.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.971 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:52:45.971 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:52:45.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.972 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:45.972 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:45.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:45.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:45.973 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:52:45.973 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:52:45.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:45.974 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:52:45.975 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:52:45.976 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:52:45.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:45.977 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:46.033 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:46.034 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:46.077 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:46.078 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:46.078 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:46.078 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:46.078 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:46.079 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:46.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.079 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:46.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:46.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:46.080 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:46.080 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:46.080 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:52:46.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:46.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:46.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:46.081 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:52:46.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:46.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:46.082 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:46.082 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:46.083 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:46.083 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:52:46.083 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:46.083 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:46.083 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:46.084 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:52:46.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:46.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:46.085 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:46.085 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:46.085 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:46.085 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:46.086 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:46.086 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:46.086 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:46.086 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:46.086 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:46.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:46.087 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:46.087 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:46.087 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:46.088 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:52:46.088 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:46.088 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:46.088 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:46.088 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:52:46.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:46.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:46.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:46.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.089 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:46.090 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:46.090 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:46.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:46.090 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:46Z" 2023-10-12T01:52:46.091 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:46.092 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:46Z" } 2023-10-12T01:52:46.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:52:46.093 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:52:46.093 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:46.094 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:52:51.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:51.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:51.097 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:51.158 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:51.158 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:51.208 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:51.208 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:51.208 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:51.208 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:51.209 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:51.209 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:51.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.209 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:52:51.211 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:51.211 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:51.211 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:51.211 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:52:51.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:51.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:51.212 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:51.212 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:51.213 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:51.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:52:51.213 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:51.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:51.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:51.214 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:52:51.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:51.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:51.215 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:51.215 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:51.215 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:51.215 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:51.215 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:51.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:51.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:51.216 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:51.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:51.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:51.217 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:51.217 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:51.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:51.218 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:52:51.218 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:51.218 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:51.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:51.218 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:52:51.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:51.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:51.219 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:51.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.219 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:51.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:51.220 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:51.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:51.220 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:51Z" 2023-10-12T01:52:51.220 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:51.221 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:51Z" } 2023-10-12T01:52:51.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:52:51.222 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:52:51.223 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:51.223 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:52:56.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:56.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:52:56.226 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:52:56.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:52:56.285 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:52:56.333 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:52:56.333 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:52:56.334 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:52:56.334 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:52:56.334 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:52:56.334 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:52:56.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.335 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:52:56.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:56.335 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:52:56.335 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:56.336 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:56.336 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:52:56.336 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:56.336 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:52:56.336 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:56.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:52:56.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:56.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:56.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:52:56.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:56.338 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:56.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:52:56.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:56.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:52:56.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:56.339 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:52:56.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:56.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:56.340 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:52:56.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:56.340 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:56.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:52:56.341 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:56.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:56.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:56.341 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:52:56.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:56.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:52:56.342 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:52:56.342 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:52:56.343 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:52:56.343 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:52:56.343 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:52:56.343 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:52:56.344 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:52:56.344 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:52:56.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:56.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:52:56.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:52:56.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.345 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:52:56.345 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:52:56.345 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:52:56.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:52:56.346 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:52:56Z" 2023-10-12T01:52:56.346 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:52:56.347 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:52:56Z" } 2023-10-12T01:52:56.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:52:56.348 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:52:56.348 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:52:56.349 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:01.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:01.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:01.351 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:01.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:01.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:01.465 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:01.465 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:01.465 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:01.466 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:01.466 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:01.466 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:01.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.467 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:01.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:01.467 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:01.467 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:01.467 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:01.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:01.468 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:01.468 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:01.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:01.469 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:01.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:01.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:01.469 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:01.470 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:01.470 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:01.470 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:01.470 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:01.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:01.471 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:01.471 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:01.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:01.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:01.472 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:01.472 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:01.472 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:01.473 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:01.474 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:01.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:01.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:01.474 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:01.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:01.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:01.475 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:01.475 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:01.476 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:01.476 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:01.476 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:01.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:01.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:01.477 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:01.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:01.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:01.477 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:01.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.478 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:01.478 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:01.478 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:01.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:01.479 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:01Z" 2023-10-12T01:53:01.479 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:01.480 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:01Z" } 2023-10-12T01:53:01.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:01.481 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:01.481 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:01.482 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:06.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:06.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:06.484 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:06.543 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:06.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:06.593 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:06.593 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:06.593 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:06.594 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:06.594 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:06.594 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:06.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.595 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:06.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:06.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:06.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:06.596 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:06.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:06.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:06.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:06.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:06.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:06.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:06.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:06.598 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:06.599 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:06.599 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:06.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:06.599 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:06.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:06.600 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:06.600 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:06.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:06.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:06.601 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:06.601 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:06.601 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:06.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:06.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:06Z" 2023-10-12T01:53:06.607 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:06.608 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:06Z" } 2023-10-12T01:53:06.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:06.609 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:06.610 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:06.610 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:11.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:11.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:11.613 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:11.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:11.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:11.721 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:11.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:11.722 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:11.722 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:11.723 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:11.723 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:11.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.723 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:11.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:11.724 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:11.724 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:11.724 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:11.724 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:11.725 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:11.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:11.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:11.725 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:11.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:11.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:11.726 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:11.726 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:11.727 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:11.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:11.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:11.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:11.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:11.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:11.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:11.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:11.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:11.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:11.729 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:11.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:11.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:11.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:11.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:11.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:11.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:11.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:11.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:11.731 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:11.731 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:11.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:11.732 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:11.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:11.732 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:11.732 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:11.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:11.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:11.733 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:11.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.733 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:11.733 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:11.734 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:11.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:11.734 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:11Z" 2023-10-12T01:53:11.734 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:11.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:11Z" } 2023-10-12T01:53:11.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:11.736 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:11.737 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:11.737 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:16.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:16.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:16.740 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:16.848 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:16Z" 2023-10-12T01:53:16.862 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:16.863 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:16Z" } 2023-10-12T01:53:16.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:16.864 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:16.864 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:16.865 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:21.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:21.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:21.868 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:21.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:21.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:21.975 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:21.975 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:21.975 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:21.975 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:21.976 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:21.976 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:21.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.976 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:21.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:21.977 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:21.977 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:21.977 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:21.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:21.978 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:21.978 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:21.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:21.978 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:21.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:21.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:21.979 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:21.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:21.980 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:21.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:21.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:21.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:21.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:21.981 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:21.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:21.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:21.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:21.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:21.982 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:21.982 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:21.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:21.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:21.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:21.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:21.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:21.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:21.984 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:21.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:21.984 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:21.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:21.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:21.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:21.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:21.985 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:21.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:21.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:21.986 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:21.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.986 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:21.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:21.987 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:21.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:21.987 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:21Z" 2023-10-12T01:53:21.987 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:21.989 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:21Z" } 2023-10-12T01:53:21.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:21.990 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:21.990 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:21.990 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:26.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:26.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:26.993 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:27.053 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:27.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:27.102 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:27.103 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:27.103 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:27.103 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:27.103 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:27.104 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:27.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.104 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:27.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:27.104 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:27.105 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:27.105 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:27.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:27.105 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:27.106 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:27.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:27.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:27.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:27.107 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:27.108 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:27.108 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:27.108 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:27.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:27.109 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:27.109 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:27.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:27.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:27.111 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:27.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:27.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:27.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:27.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:27.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:27.112 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:27.112 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:27.112 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:27.113 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:27.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:27.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:27.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:27.114 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:27.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:27.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:27.115 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:27.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.115 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:27.115 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:27.115 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:27.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:27.116 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:27Z" 2023-10-12T01:53:27.116 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:27.117 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:27Z" } 2023-10-12T01:53:27.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:27.118 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:27.119 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:27.119 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:32.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:32.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:32.122 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:32.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:32.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:32.230 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:32Z" 2023-10-12T01:53:32.243 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:32.244 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:32Z" } 2023-10-12T01:53:32.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:32.245 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:32.246 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:32.246 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:37.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:37.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:37.248 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:37.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:37.309 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:37.359 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:37.359 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:37.359 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:37.359 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:37.360 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:37.360 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:37.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.360 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:37.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:37.361 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:37.361 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:37.361 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:37.361 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:37.362 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:37.362 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:37.362 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:37.362 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:37.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:37.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:37.363 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:37.364 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:37.364 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:37.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:37.364 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:37.364 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:37.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:37.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:37.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:37.369 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:37.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:37.369 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:37.369 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:37.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:37.370 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:37.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:37.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:37.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:37.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.371 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:37.371 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:37.371 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:37.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:37.372 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:37Z" 2023-10-12T01:53:37.372 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:37.373 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:37Z" } 2023-10-12T01:53:37.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:37.374 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:37.375 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:37.375 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:42.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:42.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:42.377 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:42.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:42.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:42.489 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:42.489 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:42.489 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:42.490 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:42.490 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:42.490 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:42.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.491 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:42.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:42.491 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:42.491 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:42.491 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:42.492 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:42.492 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:42.492 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:42.492 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:42.493 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:42.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:42.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:42.493 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:42.494 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:42.494 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:42.494 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:42.494 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:42.494 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:42.495 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:42.495 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:42.495 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:42.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:42.496 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:42.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:42.496 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:42.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:42.497 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:42.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:42.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:42.498 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:42.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:42.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:42.498 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:42.499 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:42.499 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:42.499 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:42.499 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:42.500 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:42.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:42.500 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:42.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:42.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:42.501 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:42.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.501 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:42.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:42.501 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:42.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:42.502 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:42Z" 2023-10-12T01:53:42.502 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:42.503 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:42Z" } 2023-10-12T01:53:42.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:42.504 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:42.505 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:42.505 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:47.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:47.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:47.507 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:47.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:47.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:47.615 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:47.616 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:47.616 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:47.616 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:47.617 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:47.617 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:47.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.617 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:47.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:47.618 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:47.618 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:47.618 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:47.618 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:47.619 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:47.619 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:47.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:47.619 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:47.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:47.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:47.620 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:47.620 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:47.620 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:47.621 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:47.621 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:47.621 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:47.621 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:47.622 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:47.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:47.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:47.622 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:47.623 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:47.623 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:47.623 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:47.623 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:47.623 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:47.624 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:47.624 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:47.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:47.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:47.625 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:47.625 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:47.625 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:47.625 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:47.626 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:47.626 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:47.626 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:47.626 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:47.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:47.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:47.627 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:47.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.627 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:47.627 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:47.628 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:47.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:47.628 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:47Z" 2023-10-12T01:53:47.628 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:47.629 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:47Z" } 2023-10-12T01:53:47.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:47.630 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:47.631 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:47.631 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:52.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:52.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:52.634 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:52.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:52.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:52.742 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:52.742 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.743 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:52.744 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:52.745 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:52.746 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:52.747 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:52.748 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:52.750 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:52.751 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:52.752 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:52.754 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:52.755 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:52.757 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:52.758 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:52Z" 2023-10-12T01:53:52.758 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:52.760 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:52Z" } 2023-10-12T01:53:52.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:52.761 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:52.762 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:52.763 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:53:57.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:57.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:53:57.765 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:53:57.823 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:53:57.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:53:57.872 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:53:57.872 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:53:57.873 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:53:57.873 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:53:57.873 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:53:57.873 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:53:57.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.874 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:53:57.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:57.874 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:53:57.874 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:57.875 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:57.875 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:53:57.875 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:57.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:53:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:53:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:57.877 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:53:57.877 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:57.877 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:57.877 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:53:57.877 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:57.878 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:53:57.878 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:57.878 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:53:57.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:57.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:57.879 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:53:57.879 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:57.879 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:57.880 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:53:57.880 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:57.880 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:57.880 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:57.880 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:53:57.881 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:57.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:53:57.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:53:57.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:53:57.882 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:53:57.882 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:53:57.882 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:53:57.882 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:53:57.882 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:53:57.883 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:53:57.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:57.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:53:57.883 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:53:57.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.884 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:53:57.884 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:53:57.884 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:53:57.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:53:57.884 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:53:57Z" 2023-10-12T01:53:57.885 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:53:57.886 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:53:57Z" } 2023-10-12T01:53:57.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:53:57.887 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:53:57.887 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:53:57.887 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:02.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:02.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:02.890 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:02.949 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:02.949 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:02.998 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:03.000 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:03.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:03.002 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:03.005 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:03.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:03.007 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:03.008 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.009 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:03.010 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:02Z" 2023-10-12T01:54:03.011 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:03.012 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:02Z" } 2023-10-12T01:54:03.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:03.013 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:03.013 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:03.013 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:08.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:08.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:08.016 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:08.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:08.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:08.124 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:08.124 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.125 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:08.126 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:08.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:08.128 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:08.129 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:08.130 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:08.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:08.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:08.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:08.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:08.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:08.136 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:08Z" 2023-10-12T01:54:08.136 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:08.137 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:08Z" } 2023-10-12T01:54:08.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:08.138 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:08.139 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:08.139 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:13.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:13.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:13.141 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:13.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:13.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:13.250 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:13.251 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:13.251 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:13.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:13.251 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:13.252 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:13.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.252 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:13.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:13.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:13.253 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:13.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:13.253 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:13.253 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:13.253 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:13.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:13.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:13.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:13.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:13.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:13.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:13.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:13.256 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:13.256 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:13.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:13.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:13.257 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:13.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:13.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:13.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:13.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:13.258 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:13.258 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:13.258 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:13.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:13.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:13.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:13.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:13.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:13.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:13.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:13.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:13.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:13.261 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:13.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:13.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:13.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:13.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:13.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:13.262 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:13.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.262 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:13.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:13.263 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:13.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:13.263 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:13Z" 2023-10-12T01:54:13.263 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:13.264 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:13Z" } 2023-10-12T01:54:13.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:13.265 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:13.266 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:13.266 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:18.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:18.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:18.269 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:18.377 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:18Z" 2023-10-12T01:54:18.390 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:18.391 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:18Z" } 2023-10-12T01:54:18.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:18.392 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:18.392 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:18.393 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:23.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:23.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:23.395 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:23.504 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:23.505 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:23.505 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:23.505 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:23.506 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:23.506 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:23.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.506 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:23.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:23.507 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:23.507 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:23.507 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:23.507 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:23.507 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:23.508 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:23.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:23.508 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:23.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:23.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:23.509 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:23.509 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:23.509 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:23.510 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:23.510 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:23.510 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:23.510 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:23.510 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:23.512 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:23.512 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:23.512 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:23.512 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:23.512 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:23.513 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:23.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:23.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:23.514 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:23.514 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:23.514 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:23.514 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:23.514 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:23.515 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:23.515 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:23.515 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:23.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:23.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:23.516 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:23.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:23.517 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:23Z" 2023-10-12T01:54:23.517 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:23.518 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:23Z" } 2023-10-12T01:54:23.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:23.519 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:23.519 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:23.520 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:28.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:28.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:28.523 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:28.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:28.582 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:28.630 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:28.631 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:28.631 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:28.631 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:28.632 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:28.632 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:28.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.632 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:28.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:28.634 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:28.634 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:28.634 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:28.634 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:28.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:28.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:28.635 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:28.635 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:28.635 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:28.636 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:28.636 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:28.636 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:28.636 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:28.637 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:28.638 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:28.638 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:28.638 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:28.638 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:28.639 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:28.639 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:28.639 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:28.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:28.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:28.640 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:28.640 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:28.641 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:28.641 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:28.641 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:28.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:28.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:28.642 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:28.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:28.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:28.643 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:28.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.644 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:28.644 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:28.644 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:28.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:28.645 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:28Z" 2023-10-12T01:54:28.645 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:28.646 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:28Z" } 2023-10-12T01:54:28.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:28.647 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:28.648 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:28.648 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:33.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:33.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:33.650 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:33.709 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:33.709 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:33.758 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:33.758 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:33.758 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:33.759 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:33.759 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:33.759 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:33.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.759 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:33.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:33.760 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:33.760 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:33.760 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:33.761 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:33.761 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:33.761 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:33.761 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:33.761 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:33.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:33.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:33.762 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:33.763 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:33.763 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:33.763 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:33.763 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:33.763 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:33.764 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:33.764 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:33.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:33.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:33.765 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:33.765 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:33.765 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:33.765 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:33.766 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:33.766 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:33.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:33.766 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:33.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:33.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:33.767 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:33.767 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:33.768 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:33.768 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:33.768 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:33.768 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:33.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:33.769 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:33.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:33.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:33.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:33.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.770 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:33.770 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:33.770 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:33.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:33.771 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:33Z" 2023-10-12T01:54:33.771 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:33.772 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:33Z" } 2023-10-12T01:54:33.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:33.773 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:33.774 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:33.774 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:38.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:38.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:38.776 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:38.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:38.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:38.883 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:38.884 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:38.884 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:38.884 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:38.884 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:38.885 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:38.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.885 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:38.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:38.886 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:38.886 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:38.886 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:38.886 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:38.887 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:38.887 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:38.887 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:38.887 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:38.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:38.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:38.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:38.889 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:38.889 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:38.889 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:38.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:38.889 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:38.890 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:38.890 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:38.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:38.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:38.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:38.891 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:38.891 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:38.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:38.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:38.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:38.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:38.893 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:38.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:38.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:38.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:38.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:38.894 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:38.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:38.895 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:38.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:38.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:38.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:38.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:38.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:38.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:38.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.896 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:38.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:38.897 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:38.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:38.897 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:38Z" 2023-10-12T01:54:38.897 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:38.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:38Z" } 2023-10-12T01:54:38.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:38.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:38.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:38.901 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:43.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:43.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:43.903 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:44.011 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:44.012 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:44.012 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:44.012 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:44.013 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:44.013 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:44.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.013 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:44.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:44.014 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:44.014 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:44.014 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:44.015 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:44.015 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:44.015 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:44.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:44.016 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:44.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:44.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:44.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:44.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:44.017 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:44.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:54:44.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:44.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:44.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:44.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:44.020 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:44.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:44.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:44.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:44.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:44.021 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:44.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:44.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:44.022 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:44.022 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:44.023 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:44.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:54:44.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:44.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:44.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:44.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:54:44.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:44.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:44.025 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:44.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.025 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:44.025 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:44.026 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:44.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:44.026 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:43Z" 2023-10-12T01:54:44.026 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:44.028 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052373", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:51:00.319820Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046360", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:51:00.298028Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:43Z" } 2023-10-12T01:54:44.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:54:44.029 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:44.029 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:54:44.029 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:54:49.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:49.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:49.032 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:49.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.140 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:49.140 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:49.140 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:49.141 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:49.141 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:49.141 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:54:49.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.142 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:49.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.142 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:49.142 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:49.143 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:49.143 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:54:49.143 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:49.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:54:49.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:49.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:54:49.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.145 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:49.145 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:49.145 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:49.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:54:49.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:49.146 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:54:49.146 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:49.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:54:49.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:49.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:49.147 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:49.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:54:49.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:49.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:49.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:49.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:54:49.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.149 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:49.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:49.150 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:54:49.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:54:49.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:49.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:49.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:49.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:54:49.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.152 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:49.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:54:49.152 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:54:49.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.153 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:49Z" 2023-10-12T01:54:49.153 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:49.154 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:54:49Z" } 2023-10-12T01:54:49.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:54:49.155 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:49.155 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:54:49.156 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:54:49.156 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:54:49.157 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:54:49.157 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.158 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:54:49.158 DEBUG:boto:Data: 2023-10-12T01:54:49.159 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.159 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.160 DEBUG:boto:Params: {} 2023-10-12T01:54:49.160 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:54:49.161 DEBUG:boto:Token: None 2023-10-12T01:54:49.161 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-74/ 2023-10-12T01:54:49.162 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:vMW3oHiq8TTQ+WnWjRNb5UFkQtc= 2023-10-12T01:54:49.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vMW3oHiq8TTQ+WnWjRNb5UFkQtc=', 'Content-Length': '0'} 2023-10-12T01:54:49.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008014fdd5c3f7ef22-00652751e9-2cac-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.169 DEBUG:boto:path=/wafmoa-74/obj1 2023-10-12T01:54:49.169 DEBUG:boto:auth_path=/wafmoa-74/obj1 2023-10-12T01:54:49.170 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.170 DEBUG:boto:Path: /wafmoa-74/obj1 2023-10-12T01:54:49.171 DEBUG:boto:Data: 2023-10-12T01:54:49.171 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.172 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.172 DEBUG:boto:Params: {} 2023-10-12T01:54:49.173 DEBUG:boto:Token: None 2023-10-12T01:54:49.173 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-74/obj1 2023-10-12T01:54:49.174 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:29w0Ng6QI4cEFr2CPLgBozgL2m0= 2023-10-12T01:54:49.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29w0Ng6QI4cEFr2CPLgBozgL2m0=', 'Content-Length': '0'} 2023-10-12T01:54:49.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5a49af7bd17c4e-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.177 DEBUG:boto:path=/wafmoa-74/obj1 2023-10-12T01:54:49.177 DEBUG:boto:auth_path=/wafmoa-74/obj1 2023-10-12T01:54:49.178 DEBUG:boto:Method: GET 2023-10-12T01:54:49.178 DEBUG:boto:Path: /wafmoa-74/obj1 2023-10-12T01:54:49.179 DEBUG:boto:Data: 2023-10-12T01:54:49.179 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.180 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.180 DEBUG:boto:Params: {} 2023-10-12T01:54:49.181 DEBUG:boto:Token: None 2023-10-12T01:54:49.181 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-74/obj1 2023-10-12T01:54:49.182 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:8EkpEbKizObKnJM/4T4lSxDOanE= 2023-10-12T01:54:49.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8EkpEbKizObKnJM/4T4lSxDOanE=', 'Content-Length': '0'} 2023-10-12T01:54:49.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4fdc4238cbb9330-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.185 DEBUG:boto:path=/wafmoa-74/obj2 2023-10-12T01:54:49.185 DEBUG:boto:auth_path=/wafmoa-74/obj2 2023-10-12T01:54:49.186 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.186 DEBUG:boto:Path: /wafmoa-74/obj2 2023-10-12T01:54:49.186 DEBUG:boto:Data: 2023-10-12T01:54:49.187 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.188 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.188 DEBUG:boto:Params: {} 2023-10-12T01:54:49.188 DEBUG:boto:Token: None 2023-10-12T01:54:49.189 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-74/obj2 2023-10-12T01:54:49.189 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:0K2MvtnSswDK1yLm15Jc4nqc1Ms= 2023-10-12T01:54:49.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0K2MvtnSswDK1yLm15Jc4nqc1Ms=', 'Content-Length': '0'} 2023-10-12T01:54:49.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e44731fa6b2b1c8d-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.192 DEBUG:boto:path=/wafmoa-74/obj2 2023-10-12T01:54:49.193 DEBUG:boto:auth_path=/wafmoa-74/obj2 2023-10-12T01:54:49.193 DEBUG:boto:Method: GET 2023-10-12T01:54:49.193 DEBUG:boto:Path: /wafmoa-74/obj2 2023-10-12T01:54:49.194 DEBUG:boto:Data: 2023-10-12T01:54:49.194 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.195 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.196 DEBUG:boto:Params: {} 2023-10-12T01:54:49.196 DEBUG:boto:Token: None 2023-10-12T01:54:49.197 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-74/obj2 2023-10-12T01:54:49.197 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:JDmACmILcnb1oVVL4ELl8DjCNUs= 2023-10-12T01:54:49.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDmACmILcnb1oVVL4ELl8DjCNUs=', 'Content-Length': '0'} 2023-10-12T01:54:49.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f54a1e31d8d8cb-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.200 DEBUG:boto:path=/wafmoa-75/ 2023-10-12T01:54:49.201 DEBUG:boto:auth_path=/wafmoa-75/ 2023-10-12T01:54:49.201 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.201 DEBUG:boto:Path: /wafmoa-75/ 2023-10-12T01:54:49.202 DEBUG:boto:Data: 2023-10-12T01:54:49.202 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.203 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.203 DEBUG:boto:Params: {} 2023-10-12T01:54:49.204 DEBUG:boto:Token: None 2023-10-12T01:54:49.204 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-75/ 2023-10-12T01:54:49.205 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3j3FOPItytQnMkwssCgRR9Aj7Fk= 2023-10-12T01:54:49.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3j3FOPItytQnMkwssCgRR9Aj7Fk=', 'Content-Length': '0'} 2023-10-12T01:54:49.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bbee016d1da4269-00652751e9-2cac-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.208 DEBUG:boto:path=/wafmoa-75/obj1 2023-10-12T01:54:49.209 DEBUG:boto:auth_path=/wafmoa-75/obj1 2023-10-12T01:54:49.209 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.210 DEBUG:boto:Path: /wafmoa-75/obj1 2023-10-12T01:54:49.210 DEBUG:boto:Data: 2023-10-12T01:54:49.210 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.211 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.212 DEBUG:boto:Params: {} 2023-10-12T01:54:49.212 DEBUG:boto:Token: None 2023-10-12T01:54:49.213 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-75/obj1 2023-10-12T01:54:49.213 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:3LyohQCiiXPzXk62mSSy7hMPdxk= 2023-10-12T01:54:49.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3LyohQCiiXPzXk62mSSy7hMPdxk=', 'Content-Length': '0'} 2023-10-12T01:54:49.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f9b628f86752ab-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.216 DEBUG:boto:path=/wafmoa-75/obj1 2023-10-12T01:54:49.216 DEBUG:boto:auth_path=/wafmoa-75/obj1 2023-10-12T01:54:49.217 DEBUG:boto:Method: GET 2023-10-12T01:54:49.217 DEBUG:boto:Path: /wafmoa-75/obj1 2023-10-12T01:54:49.218 DEBUG:boto:Data: 2023-10-12T01:54:49.218 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.219 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.219 DEBUG:boto:Params: {} 2023-10-12T01:54:49.220 DEBUG:boto:Token: None 2023-10-12T01:54:49.220 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-75/obj1 2023-10-12T01:54:49.221 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:VU5lPwadi66/1DlV5jiZhurgRv4= 2023-10-12T01:54:49.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VU5lPwadi66/1DlV5jiZhurgRv4=', 'Content-Length': '0'} 2023-10-12T01:54:49.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db06a495b627c683-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.224 DEBUG:boto:path=/wafmoa-75/obj2 2023-10-12T01:54:49.224 DEBUG:boto:auth_path=/wafmoa-75/obj2 2023-10-12T01:54:49.225 DEBUG:boto:Method: HEAD 2023-10-12T01:54:49.225 DEBUG:boto:Path: /wafmoa-75/obj2 2023-10-12T01:54:49.226 DEBUG:boto:Data: 2023-10-12T01:54:49.226 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.227 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.227 DEBUG:boto:Params: {} 2023-10-12T01:54:49.228 DEBUG:boto:Token: None 2023-10-12T01:54:49.228 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-75/obj2 2023-10-12T01:54:49.229 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:WCLxnwFhlhAcbxBB611Wv5IN2io= 2023-10-12T01:54:49.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WCLxnwFhlhAcbxBB611Wv5IN2io=', 'Content-Length': '0'} 2023-10-12T01:54:49.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d64ee582d243cd2e-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.232 DEBUG:boto:path=/wafmoa-75/obj2 2023-10-12T01:54:49.232 DEBUG:boto:auth_path=/wafmoa-75/obj2 2023-10-12T01:54:49.232 DEBUG:boto:Method: GET 2023-10-12T01:54:49.233 DEBUG:boto:Path: /wafmoa-75/obj2 2023-10-12T01:54:49.233 DEBUG:boto:Data: 2023-10-12T01:54:49.234 DEBUG:boto:Headers: {} 2023-10-12T01:54:49.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:54:49.235 DEBUG:boto:Port: 8001 2023-10-12T01:54:49.235 DEBUG:boto:Params: {} 2023-10-12T01:54:49.236 DEBUG:boto:Token: None 2023-10-12T01:54:49.236 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:54:49 GMT /wafmoa-75/obj2 2023-10-12T01:54:49.237 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:iseyMEg1ZjYO+0WppQ+lyndDzU4= 2023-10-12T01:54:49.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iseyMEg1ZjYO+0WppQ+lyndDzU4=', 'Content-Length': '0'} 2023-10-12T01:54:49.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002a11054adea3a45-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:54:49.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.335 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.335 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.381 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:54:49.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.383 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.464 INFO:teuthology.orchestra.run.smithi112.stderr:WARNING: cannot find source zone id for name=* 2023-10-12T01:54:49.465 INFO:teuthology.orchestra.run.smithi112.stderr:WARNING: cannot find source zone id for name=* 2023-10-12T01:54:49.484 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:54:49.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.487 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.538 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.538 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.587 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:54:49.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.589 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23, 2023-10-12T01:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-10-12T01:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-10-12T01:54:49.689 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-10-12T01:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-10-12T01:54:49.690 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-10-12T01:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:54:49.691 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-10-12T01:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:54:49.693 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:54:49.694 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-10-12T01:54:49.695 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.696 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:54:49.697 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:54:49.698 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:54:49.699 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-10-12T01:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:54:49.700 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:54:49.701 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:54:49.702 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:54:49.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:54:49.704 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:54:49.705 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-10-12T01:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.706 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-10-12T01:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-10-12T01:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-10-12T01:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-10-12T01:54:49.707 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.708 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-10-12T01:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.709 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-74", 2023-10-12T01:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.710 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:54:49.711 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:54:49.712 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:54:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-wafmoa-75", 2023-10-12T01:54:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-74", 2023-10-12T01:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "wafmoa-75", 2023-10-12T01:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:54:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:54:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-10-12T01:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.721 INFO:teuthology.orchestra.run.smithi112.stdout: "*" 2023-10-12T01:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-10-12T01:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-10-12T01:54:49.722 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "*" 2023-10-12T01:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-10-12T01:54:49.723 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-10-12T01:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-10-12T01:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-10-12T01:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-10-12T01:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-10-12T01:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-10-12T01:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-10-12T01:54:49.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-10-12T01:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-10-12T01:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-10-12T01:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-10-12T01:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-10-12T01:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-10-12T01:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-10-12T01:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-10-12T01:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-10-12T01:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:54:49.734 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.735 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:54:49.736 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-10-12T01:54:49.736 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-10-12T01:54:49.736 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:54:49.736 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.736 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-10-12T01:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-10-12T01:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-10-12T01:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-10-12T01:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-10-12T01:54:49.737 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-10-12T01:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-10-12T01:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-10-12T01:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-10-12T01:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-10-12T01:54:49.738 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.739 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.739 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.739 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-10-12T01:54:49.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.740 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-10-12T01:54:49.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-10-12T01:54:49.740 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-10-12T01:54:49.740 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-10-12T01:54:49.741 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.741 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.741 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-10-12T01:54:49.741 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:54:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-10-12T01:54:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-10-12T01:54:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-10-12T01:54:49.742 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-10-12T01:54:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-10-12T01:54:49.743 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-10-12T01:54:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:54:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3962893999 2023-10-12T01:54:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.745 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:54:49.745 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3435037168 2023-10-12T01:54:49.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.745 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:54:49.746 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 2465750364 2023-10-12T01:54:49.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:49.746 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:54:49.746 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1412291983 2023-10-12T01:54:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-10-12T01:54:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:54:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-10-12T01:54:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-10-12T01:54:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:54:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:54:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:54:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:54:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:54:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-10-12T01:54:49.750 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-10-12T01:54:49.750 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-10-12T01:54:49.750 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-10-12T01:54:49.750 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-10-12T01:54:49.750 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-10-12T01:54:49.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.751 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-10-12T01:54:49.751 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:54:49.751 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:54:49.751 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:54:49.752 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:54:49.752 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:54:49.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.752 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-10-12T01:54:49.752 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:54:49.753 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:54:49.753 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:54:49.753 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:54:49.753 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:54:49.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.754 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-10-12T01:54:49.754 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-10-12T01:54:49.754 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-10-12T01:54:49.754 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-10-12T01:54:49.755 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-10-12T01:54:49.755 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-10-12T01:54:49.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:49.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:49.755 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:54:49.756 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-10-12T01:54:49.756 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:49.756 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:49.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.758 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.810 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.810 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:49.893 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:54:49.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:49.894 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.081 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:54:50.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.083 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:50.102 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:54:50.100+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:54:50.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.306 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:54:50.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.309 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-74 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-74 '--source-bucket=*' --dest-bucket=wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:50.421 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.422 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.501 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:54:50.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.502 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-75 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-74 '--source-bucket=*' --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:50.564 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.565 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.625 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:54:50.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.626 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:50.678 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.678 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.727 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:54:50.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:54:50.728 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:54:50.773 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.774 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.819 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:54:50.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:50.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:50.820 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:50.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.898 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:54:50.910 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:54:50.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:50.911 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:54:50.911 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:54:50.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:50.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:50.912 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:54:50.912 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:54:50.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:50.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:50.913 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:54:50.913 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:54:50.713117Z" 2023-10-12T01:54:50.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:50.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:50.913 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:54:50.914 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:54:50.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:50.914 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:54:50.915 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:54:50.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:54:50.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:50.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:50.917 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:50.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:50.966 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:54:50.994+0000 7f653fadc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:54:50.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:54:50.995+0000 7f653fadc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:54:51.009 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:51.009 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:54:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.010 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:51.011 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:54:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:54:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:51.012 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:54:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:51.013 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:54:51.014 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:54:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:51.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:51.016 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:54:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:51.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:54:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:51.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:54:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:54:51.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.021 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:54:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:54:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:51.022 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:50Z" 2023-10-12T01:54:51.023 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:51.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:54:50Z" } 2023-10-12T01:54:51.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:54:51.025 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:54:51.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:54:51.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:54:56.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:56.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:54:56.028 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:54:56.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:54:56.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:54:56.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:54:56.117+0000 7f6bab12d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:54:56.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:54:56.118+0000 7f6bab12d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:54:56.135 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:54:56.135 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:54:56.136 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:54:56.136 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:54:56.136 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:54:56.137 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:54:56.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:56.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.138 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:54:56.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:56.139 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:54:56.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:56.140 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:56.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:54:56.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:56.141 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:54:56.141 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:56.141 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:54:56.142 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:56.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:56.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:56.143 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:54:56.144 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:56.144 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:56.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:54:56.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:56.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:56.146 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:56.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:54:56.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:56.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:56.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:56.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:54:56.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:56.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:56.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:54:56.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:56.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:56.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:56.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:54:56.151 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:56.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:56.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:54:56.153 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:54:56.153 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:54:56.154 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:54:56.154 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:54:56.155 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:54:56.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:54:56.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:54:56.156 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:54:56.156 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:54:56.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:56.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:54:56.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:54:56.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.158 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:54:56.158 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:54:56.159 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:54:56.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:54:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:54:56Z" 2023-10-12T01:54:56.160 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:54:56.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:54:56Z" } 2023-10-12T01:54:56.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:54:56.163 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:54:56.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:54:56.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:01.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:01.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:01.168 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:01.221 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:01.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:01.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:01.251+0000 7f6fed622780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:01.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:01.252+0000 7f6fed622780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:01.266 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:01.267 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:01.267 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:01.267 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:01.267 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:01.267 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:01.268 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:01.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.268 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:01.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:01.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:01.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:01.269 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:01.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:01.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:01.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:01.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:01.270 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:01.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:01.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:01.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:01.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:01.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:01.272 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:01.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:01.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:01.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:01.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:01.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:01.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:01.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:01.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:01.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:01.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:01.276 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:01.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:01.277 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:01.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:01.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:01.278 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:01.278 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:01.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:01.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:01.279 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:01.279 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:01.279 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:01.279 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:01.280 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:01.280 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:01.280 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:01.280 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:01.280 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:01.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:01.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:01.281 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:01.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.282 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:01.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:01.283 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:01.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:01.283 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:01Z" 2023-10-12T01:55:01.283 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:01.285 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:01Z" } 2023-10-12T01:55:01.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:01.286 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:01.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:01.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:06.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:06.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:06.290 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:06.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:06.347 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:06.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:06.377+0000 7ff9fe8a1780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:06.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:06.378+0000 7ff9fe8a1780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:06.393 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:06Z" 2023-10-12T01:55:06.406 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:06.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:06Z" } 2023-10-12T01:55:06.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:06.409 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:06.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:06.409 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:11.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:11.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:11.412 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:11.471 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:11.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:11.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:11.504+0000 7f8bbc9e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:11.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:11.505+0000 7f8bbc9e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:11.519 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:11.519 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:11.519 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:11.520 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:11.520 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:11.520 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:11.521 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:11.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.521 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:11.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:11.522 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:11.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:11.523 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:11.523 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:11.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:11.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:11.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:11.524 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:11.524 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:11.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:11.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:11.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:11.525 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:11.525 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:11.526 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:11.526 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:11.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:11.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:11.527 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:11.527 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:11.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:11.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:11.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:11.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:11.528 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:11.528 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:11.529 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:11.529 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:11.529 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:11.530 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:11.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:11.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:11.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:11.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:11.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:11.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:11.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:11.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:11.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:11.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:11.532 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:11.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:11.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:11.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:11.533 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:11.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.534 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:11.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:11.534 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:11.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:11.534 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:11Z" 2023-10-12T01:55:11.535 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:11.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:11Z" } 2023-10-12T01:55:11.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:11.537 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:11.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:11.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:16.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:16.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:16.540 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:16.630+0000 7fae97d23780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:16.631+0000 7fae97d23780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:16.644 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:16.645 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:16.645 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:16.645 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:16.646 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:16.646 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:16.646 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:16.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.647 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:16.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:16.647 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:16.647 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:16.648 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:16.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:16.648 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:16.648 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:16.648 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:16.649 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:16.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:16.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:16.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:16.653 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:16.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:16.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:16.655 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:16.655 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:16.655 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:16.656 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:16.656 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:16.656 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:16.656 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:16.657 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:16.657 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:16.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:16.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:16.657 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:16.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.658 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:16.658 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:16.658 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:16.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:16.659 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:16Z" 2023-10-12T01:55:16.659 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:16.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:16Z" } 2023-10-12T01:55:16.660 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:16.661 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:16.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:16.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:21.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:21.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:21.664 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:21.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:21.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:21.754+0000 7fe7ae210780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:21.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:21.755+0000 7fe7ae210780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:21.769 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:21.769 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:21.769 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:21.770 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:21.770 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:21.770 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:21.770 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:21.772 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:21.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:21.774 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:21.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:21.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:21.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:21.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:21.775 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:21.776 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:21.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:21.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:21.776 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:21.777 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:21.777 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:21.777 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:21.777 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:21.777 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:21.778 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:21.778 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:21.778 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:21.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:21.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:21.779 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:21.779 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:21.779 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:21.780 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:21.780 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:21.780 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:21.780 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:21.780 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.781 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:21.782 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:21.782 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:21.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:21.782 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:21Z" 2023-10-12T01:55:21.783 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:21.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:21Z" } 2023-10-12T01:55:21.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:21.785 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:21.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:21.785 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:26.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:26.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:26.788 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:26.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:26.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:26.876+0000 7f555b2f2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:26.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:26.877+0000 7f555b2f2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:26.891 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:26Z" 2023-10-12T01:55:26.906 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:26.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:26Z" } 2023-10-12T01:55:26.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:26.908 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:26.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:26.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:31.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:31.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:31.911 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:31.968 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:31.968 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:32.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:31.999+0000 7fe5b5304780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:32.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:32.000+0000 7fe5b5304780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:32.015 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:32.015 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:32.015 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:32.015 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:32.016 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:32.016 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:32.016 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:32.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.017 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:32.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:32.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:32.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:32.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:32.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:32.018 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:32.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:32.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:32.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:32.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:32.021 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:32.021 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:32.022 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:32.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:32.022 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:32.022 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:32.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:32.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:32.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:32.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:32.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:32.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:32.024 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:32.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:32.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:32.025 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:32.025 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:32.025 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:32.025 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:32.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:32.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:32.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:32.026 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:32.026 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:32.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:32.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:32.027 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:32.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.027 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:32.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:32.028 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:32.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:32.028 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:32Z" 2023-10-12T01:55:32.028 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:32.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:32Z" } 2023-10-12T01:55:32.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:32.031 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:32.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:32.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:37.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:37.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:37.034 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:37.090 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:37.091 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:37.122+0000 7f7a52ab0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:37.123+0000 7f7a52ab0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:37.137 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:37.138 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:37.138 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:37.138 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:37.138 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:37.139 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:37.139 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:37.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.139 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:37.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:37.140 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:37.140 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:37.141 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:37.141 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:37.141 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:37.141 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:37.141 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:37.142 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:37.142 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:37.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:37.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:37.143 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:37.143 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:37.143 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:37.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:37.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:37.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:37.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:37.145 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:37.145 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:37.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:37.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:37.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:37.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:37.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:37.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:37.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:37.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:37.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:37.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:37.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:37.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:37.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:37.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:37.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:37.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:37.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:37.149 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:37.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:37.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:37.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:37.150 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:37.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:37.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:37.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:37.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.152 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:37.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:37.153 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:37.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:37.153 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:37Z" 2023-10-12T01:55:37.154 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:37.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:37Z" } 2023-10-12T01:55:37.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:37.156 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:37.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:37.157 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:42.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:42.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:42.159 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:42.215 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:42.216 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:42.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:42.248+0000 7ff0f9f53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:42.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:42.249+0000 7ff0f9f53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:42.264 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:42.264 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:42.264 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:42.264 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:42.265 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:42.265 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:42.265 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:42.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.266 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:42.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:42.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:42.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:42.266 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:42.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:42.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:42.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:42.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:42.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:42.268 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:42.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:42.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:42.268 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:42.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:42.269 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:42.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:42.269 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:42.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:42.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:42.270 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:42.270 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:42.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:42.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:42.271 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:42.271 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:42.271 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:42.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:42.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:42.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:42.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:42.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:42.273 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:42.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:42.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:42.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:42.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:42.274 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:42.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:42.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:42.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:42.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:42.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:42.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:42.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:42.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:42.276 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:42.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.276 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:42.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:42.277 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:42.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:42.277 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:42Z" 2023-10-12T01:55:42.277 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:42.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:42Z" } 2023-10-12T01:55:42.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:42.280 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:42.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:42.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:47.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:47.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:47.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:47.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:47.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:47.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:47.374+0000 7f142f118780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:47.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:47.375+0000 7f142f118780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:47.389 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:47.390 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:47.390 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:47.390 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:47.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:47.391 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:47.391 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:47.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.391 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:47.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:47.392 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:47.392 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:47.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:47.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:47.393 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:47.393 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:47.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:47.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:47.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:47.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:47.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:47.395 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:47.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:47.395 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:47.395 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:47.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:47.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:47.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:47.396 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:47.396 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:47.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:47.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:47.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:47.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:47.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:47.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:47.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:47.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:47.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:47.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:47.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:47.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:47.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:47.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:47.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:47.400 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:47.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:47.400 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:47.401 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:47.401 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:47.401 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:47.401 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:47.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:47.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:47.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:47.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.402 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:47.403 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:47.403 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:47.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:47.403 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:47Z" 2023-10-12T01:55:47.403 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:47.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:47Z" } 2023-10-12T01:55:47.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:47.405 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:47.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:47.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:52.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:52.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:52.409 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:52.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:52.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:52.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:52.499+0000 7f6d6d18f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:52.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:52.500+0000 7f6d6d18f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:52.514 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:52.514 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:52.515 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:52.515 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:52.515 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:52.515 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:52.515 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:52.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.516 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:52.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:52.516 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:52.517 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:52.517 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:52.517 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:52.517 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:52.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:52.518 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:52.518 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:52.518 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:52.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:52.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:52.528 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:52Z" 2023-10-12T01:55:52.528 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:52.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:52Z" } 2023-10-12T01:55:52.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:52.530 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:52.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:52.531 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:55:57.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:57.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:55:57.534 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:55:57.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:55:57.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:55:57.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:57.623+0000 7f0c163f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:57.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:55:57.624+0000 7f0c163f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:55:57.638 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:55:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:55:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:55:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:55:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:55:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:55:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:55:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:55:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:55:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:55:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:55:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:55:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:55:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:57.645 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:57.645 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:57.645 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:55:57.645 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:57.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:57.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:57.646 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:55:57.646 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:57.647 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:57.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:55:57.647 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:57.647 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:57.647 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:57.648 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:55:57.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:57.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:57.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:55:57.649 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:55:57.649 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:55:57.649 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:55:57.649 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:55:57.650 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:55:57.650 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:55:57.650 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:55:57.650 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:55:57.651 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:55:57.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:57.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:55:57.651 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:55:57.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.652 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:55:57.652 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:55:57.652 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:55:57.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:55:57.653 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:55:57Z" 2023-10-12T01:55:57.653 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:55:57.654 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:55:57Z" } 2023-10-12T01:55:57.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:55:57.656 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:57.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:55:57.656 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:02.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:02.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:02.659 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:02.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:02.716 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:02.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:02.746+0000 7f850e06f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:02.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:02.747+0000 7f850e06f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:02.762 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:02.763 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:02.763 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:02.763 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:02.763 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:02.764 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:02.764 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:02.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.764 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:02.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:02.765 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:02.765 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:02.765 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:02.765 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:02.766 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:02.766 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:02.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:02.766 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:02.766 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:02.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:02.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:02.768 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:02.768 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:02.768 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:02.768 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:02.768 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:02.769 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:02.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:02.769 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:02.769 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:02.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:02.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:02.770 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:02.770 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:02.771 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:02.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:02.771 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:02.771 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:02.772 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:02.772 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:02.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:02.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:02.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:02.773 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:02.773 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:02.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:02.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:02.774 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:02.774 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:02.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:02.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:02.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:02.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:02.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:02.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:02.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.776 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:02.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:02.776 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:02.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:02.776 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:02Z" 2023-10-12T01:56:02.777 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:02.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:02Z" } 2023-10-12T01:56:02.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:02.779 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:02.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:02.779 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:07.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:07.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:07.782 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:07.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:07.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:07.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:07.871+0000 7f4c95e53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:07.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:07.872+0000 7f4c95e53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:07.886 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:07.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:07.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:07.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:07.892 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:07.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:07.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:07.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:07.893 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:07.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:07.894 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:07.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:07.894 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:07.894 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:07.895 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:07.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:07.895 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:07.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:07.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:07.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:07.896 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:07.896 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:07.896 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:07.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:07.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:07.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:07.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:07.897 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:07.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:07.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:07.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:07.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.898 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:07.899 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:07.899 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:07.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:07.899 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:07Z" 2023-10-12T01:56:07.899 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:07.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:07Z" } 2023-10-12T01:56:07.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:07.902 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:07.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:07.902 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:12.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:12.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:12.905 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:12.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:12.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:12.994+0000 7fbdf14f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:12.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:12.995+0000 7fbdf14f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:13.010 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:13.011 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:13.013 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:13.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:13.015 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:13.016 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:12Z" 2023-10-12T01:56:13.025 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:13.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:12Z" } 2023-10-12T01:56:13.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:13.027 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:13.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:13.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:18.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:18.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:18.030 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:18.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:18.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:18.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:18.120+0000 7f7b43624780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:18.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:18.121+0000 7f7b43624780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:18.136 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:18Z" 2023-10-12T01:56:18.150 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:18.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075564.060671_1047.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:44.060671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:18Z" } 2023-10-12T01:56:18.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:18.152 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:18.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:18.153 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:23.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.156 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:23.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:23.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:23.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:23.244+0000 7faecc2f0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:23.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:23.246+0000 7faecc2f0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:23.260 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:23.261 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.262 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.263 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.264 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:23.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:23.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.264 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:23.265 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.265 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:23.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.266 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.266 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:23.266 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:23.267 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.268 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:23.268 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.269 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:23.269 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.269 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.269 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.270 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:23.270 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.271 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:23.271 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.271 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:23.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:23.272 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.273 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:23.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.273 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:23.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:23.274 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:23.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.274 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:23.274 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:23.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075690.713117_1099.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:54:50.713117Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:23Z" } 2023-10-12T01:56:23.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075690.713117_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:54:50.713117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:23.277 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:23.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:23.278 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:56:23.278 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:56:23.279 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:56:23.279 DEBUG:boto:Method: HEAD 2023-10-12T01:56:23.280 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:56:23.280 DEBUG:boto:Data: 2023-10-12T01:56:23.281 DEBUG:boto:Headers: {} 2023-10-12T01:56:23.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:23.281 DEBUG:boto:Port: 8000 2023-10-12T01:56:23.282 DEBUG:boto:Params: {} 2023-10-12T01:56:23.282 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:56:23.283 DEBUG:boto:Token: None 2023-10-12T01:56:23.284 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:56:23 GMT /wafmoa-74/ 2023-10-12T01:56:23.284 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y= 2023-10-12T01:56:23.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y=', 'Content-Length': '0'} 2023-10-12T01:56:23.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c04f8b404b65b32-0065275247-2dbc-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:23.291 DEBUG:boto:path=/wafmoa-74/obj3 2023-10-12T01:56:23.292 DEBUG:boto:auth_path=/wafmoa-74/obj3 2023-10-12T01:56:23.292 DEBUG:boto:Method: PUT 2023-10-12T01:56:23.293 DEBUG:boto:Path: /wafmoa-74/obj3 2023-10-12T01:56:23.293 DEBUG:boto:Data: 2023-10-12T01:56:23.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:23.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:23.295 DEBUG:boto:Port: 8000 2023-10-12T01:56:23.295 DEBUG:boto:Params: {} 2023-10-12T01:56:23.295 DEBUG:boto:Token: None 2023-10-12T01:56:23.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:56:23 GMT /wafmoa-74/obj3 2023-10-12T01:56:23.296 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:f+j/5b2A1O9qI9OLn9AAQxyqVAg= 2023-10-12T01:56:23.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f+j/5b2A1O9qI9OLn9AAQxyqVAg='} 2023-10-12T01:56:23.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7db5aee452e8c3-0065275247-2dbc-a1'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:23.307 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:56:23.308 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:56:23.308 DEBUG:boto:Method: HEAD 2023-10-12T01:56:23.309 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:56:23.309 DEBUG:boto:Data: 2023-10-12T01:56:23.309 DEBUG:boto:Headers: {} 2023-10-12T01:56:23.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:23.311 DEBUG:boto:Port: 8000 2023-10-12T01:56:23.312 DEBUG:boto:Params: {} 2023-10-12T01:56:23.312 DEBUG:boto:Token: None 2023-10-12T01:56:23.313 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:56:23 GMT /wafmoa-74/ 2023-10-12T01:56:23.313 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y= 2023-10-12T01:56:23.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y=', 'Content-Length': '0'} 2023-10-12T01:56:23.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094cd1041729a9b9b-0065275247-2dbc-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:23.317 DEBUG:boto:path=/wafmoa-74/obj4 2023-10-12T01:56:23.317 DEBUG:boto:auth_path=/wafmoa-74/obj4 2023-10-12T01:56:23.318 DEBUG:boto:Method: PUT 2023-10-12T01:56:23.318 DEBUG:boto:Path: /wafmoa-74/obj4 2023-10-12T01:56:23.319 DEBUG:boto:Data: 2023-10-12T01:56:23.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:23.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:23.320 DEBUG:boto:Port: 8000 2023-10-12T01:56:23.320 DEBUG:boto:Params: {} 2023-10-12T01:56:23.321 DEBUG:boto:Token: None 2023-10-12T01:56:23.321 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Thu, 12 Oct 2023 01:56:23 GMT /wafmoa-74/obj4 2023-10-12T01:56:23.322 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:xU30QjPSaZnfxVCndE1/64hSxSs= 2023-10-12T01:56:23.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xU30QjPSaZnfxVCndE1/64hSxSs='} 2023-10-12T01:56:23.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077df1f2bd37449eb-0065275247-2dbc-a1'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:23.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.332 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:23.381 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:23.382 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:23.413 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:23.425 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:56:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:23.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.427 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:23.427 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:23.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.428 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:23.428 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:54:50.713117Z" 2023-10-12T01:56:23.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.430 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:56:23.431 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:23.432 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-12T01:56:23.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.433 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:23.484 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:23.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:23.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:23.513+0000 7fb3fd8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:23.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-10-12T01:56:23.514+0000 7fb3fd8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:23.529 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:23.530 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:23.530 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:23.530 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:23.530 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:23.531 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:23.531 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.531 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:23.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.532 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:23.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.532 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-10-12T01:56:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:23.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.535 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:23.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.536 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:23.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:23.538 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:23.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:23.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-10-12T01:56:23.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:23.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:23.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-10-12T01:56:23.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.541 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:23.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.542 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:23.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-10-12T01:56:23.542 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-10-12T01:56:23.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.543 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:23.543 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:23.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697073131.659356_324.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-12T01:12:11.659356Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697073131.679864_578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:12:11.679864Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697075690.713117_1099.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:54:50.713117Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697070188.940549_124.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T00:23:08.940549Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-12T01:56:23Z" } 2023-10-12T01:56:23.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075690.713117_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:54:50.713117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:23.545 DEBUG:rgw_multi.tests:log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:23.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:23.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-12T01:56:23.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.547 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:23.645 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-10-12T01:56:23.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.646 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032902", 2023-10-12T01:56:23.646 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:56:23.329656Z" 2023-10-12T01:56:23.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-10-12T01:56:23.647 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:56:23.306608Z" 2023-10-12T01:56:23.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:23.648 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:23.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:23.649 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:23.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.649 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-10-12T01:56:23.650 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:23.651 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:23.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:23.652 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:23.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:23.703 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:23.749 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:23.749 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:23.749 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:23.750 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:23.750 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:23.750 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:23.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.750 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:23.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.751 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:23.751 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.751 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:23.752 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:23.752 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:23.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.752 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:23.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.753 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:23.753 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.754 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:23.754 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:23.754 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.754 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.756 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:23.756 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.756 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:23.756 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:23.756 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:23.762 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:23.763 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:23Z" } 2023-10-12T01:56:23.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:23.764 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:23.764 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:23.764 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:28.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:28.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:28.767 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:28.826 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:28.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:28.878 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:28.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:28.879 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:28.879 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:28.880 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:28.880 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:28.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.880 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:28.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:28.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:28.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:28.881 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:28.881 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:28.882 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:28.882 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:28.882 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:28.882 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:28.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:28.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:28.883 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:28.883 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:28.884 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:28.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:28.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:28.885 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:28.885 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:28.886 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:28.886 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:28.886 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:28.886 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:28.887 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:28.887 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:28.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:28.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:28.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:28.888 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:28.888 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:28.888 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:28.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:28.889 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:28.889 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:28.889 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:28.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:28.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:28.890 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:28.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.891 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:28.891 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:28.891 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:28.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:28.892 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:28Z" 2023-10-12T01:56:28.892 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:28.892 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:28Z" } 2023-10-12T01:56:28.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:28.893 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:28.893 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:28.894 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:33.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:33.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:33.897 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:33.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:34.005 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:34.006 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:34.006 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:34.006 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:34.007 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:34.007 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:34.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.007 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:34.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:34.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:34.008 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:34.008 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:34.008 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:34.009 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:34.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:34.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:34.009 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:34.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:34.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:34.010 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:34.010 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:34.011 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:34.011 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:34.011 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:34.011 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:34.011 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:34.012 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:34.012 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:34.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:34.012 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:34.013 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:34.013 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:34.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:34.013 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:34.013 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:34.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:34.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:34.015 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:34.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:34.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:34.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:34.016 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:34.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:34.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:34.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:34.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:34.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:34.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:34.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:34.017 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:34.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.018 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:34.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:34.018 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:34.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:34.019 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:33Z" 2023-10-12T01:56:34.019 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:34.020 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:33Z" } 2023-10-12T01:56:34.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:34.021 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:34.022 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:34.022 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:39.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:39.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:39.024 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:39.132 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:39.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:39.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:39.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:39.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:39.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:39.139 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:39.140 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:39.141 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:39.142 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:39.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:39.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:39.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:39.146 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:39Z" 2023-10-12T01:56:39.146 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:39.148 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:39Z" } 2023-10-12T01:56:39.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:39.149 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:39.149 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:39.150 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:44.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:44.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:44.153 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:44.215 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:44.215 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:44.263 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:44.264 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.265 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:44.266 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:44.269 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:44.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:44.272 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:44.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:44.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:44.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:44.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.277 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:44.278 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:44.278 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:44.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:44.278 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:44Z" 2023-10-12T01:56:44.279 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:44.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:44Z" } 2023-10-12T01:56:44.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:44.281 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:44.281 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:44.282 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:49.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:49.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:49.284 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:49.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:49.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:49.394 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:49.395 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:49.395 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:49.395 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:49.395 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:49.396 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:49.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.396 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:49.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:49.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:49.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:49.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:49.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:49.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:49.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:49.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:49.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:49.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:49.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:49.399 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:49.399 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:49.399 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:49.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:49.400 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:49.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:49.400 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:49.400 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:49.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:49.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:49.401 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:49.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:49.401 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:49.402 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:49.402 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:49.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:49.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:49.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:49.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:49.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:49.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:49.403 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:49.404 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:49.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:49.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:49.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:49.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:49.405 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:49.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:49.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:49.405 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:49.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.406 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:49.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:49.406 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:49.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:49.407 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:49Z" 2023-10-12T01:56:49.407 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:49.408 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032761", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:51:00.382943Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:52:45.256140Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:49Z" } 2023-10-12T01:56:49.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:49.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:49.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:49.410 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:54.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.412 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:54.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.473 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.523 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-10-12T01:56:54.523 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-10-12T01:56:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-10-12T01:56:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-10-12T01:56:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-10-12T01:56:54.524 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 12355881351314922323 2023-10-12T01:56:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-10-12T01:56:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-10-12T01:56:54.525 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032902", 2023-10-12T01:56:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-10-12T01:56:54.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:56:23.329656Z" 2023-10-12T01:56:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-10-12T01:56:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-10-12T01:56:54.528 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-10-12T01:56:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:56:23.306608Z" 2023-10-12T01:56:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:54.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-10-12T01:56:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:54.530 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:54.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-10-12T01:56:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-10-12T01:56:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-10-12T01:56:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-10-12T01:56:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-10-12T01:56:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-10-12T01:56:54.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-10-12T01:56:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-10-12T01:56:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-10-12T01:56:54.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-10-12T01:56:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-10-12T01:56:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-10-12T01:56:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-10-12T01:56:54.535 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-10-12T01:56:54Z" 2023-10-12T01:56:54.536 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-10-12T01:56:54.537 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12355881351314922323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032902", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-12T01:56:23.329656Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-12T01:56:23.306608Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044605", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:48:15.266091Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046501", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-12T01:52:45.296985Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-12T01:56:54Z" } 2023-10-12T01:56:54.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032902', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:56:23.329656Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:56:23.306608Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:54.538 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:54.538 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:54.538 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:54.539 DEBUG:boto:path=/wafmoa-74/ 2023-10-12T01:56:54.539 DEBUG:boto:auth_path=/wafmoa-74/ 2023-10-12T01:56:54.540 DEBUG:boto:Method: HEAD 2023-10-12T01:56:54.540 DEBUG:boto:Path: /wafmoa-74/ 2023-10-12T01:56:54.541 DEBUG:boto:Data: 2023-10-12T01:56:54.541 DEBUG:boto:Headers: {} 2023-10-12T01:56:54.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:54.542 DEBUG:boto:Port: 8001 2023-10-12T01:56:54.542 DEBUG:boto:Params: {} 2023-10-12T01:56:54.543 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:56:54.543 DEBUG:boto:Token: None 2023-10-12T01:56:54.544 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:56:54 GMT /wafmoa-74/ 2023-10-12T01:56:54.544 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:B4+zqU5aims+9f5mvqQzjQrXXLg= 2023-10-12T01:56:54.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4+zqU5aims+9f5mvqQzjQrXXLg=', 'Content-Length': '0'} 2023-10-12T01:56:54.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000526bf317909aa66e-0065275266-2db0-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.551 DEBUG:boto:path=/wafmoa-74/obj3 2023-10-12T01:56:54.551 DEBUG:boto:auth_path=/wafmoa-74/obj3 2023-10-12T01:56:54.552 DEBUG:boto:Method: HEAD 2023-10-12T01:56:54.552 DEBUG:boto:Path: /wafmoa-74/obj3 2023-10-12T01:56:54.553 DEBUG:boto:Data: 2023-10-12T01:56:54.553 DEBUG:boto:Headers: {} 2023-10-12T01:56:54.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:54.554 DEBUG:boto:Port: 8001 2023-10-12T01:56:54.554 DEBUG:boto:Params: {} 2023-10-12T01:56:54.555 DEBUG:boto:Token: None 2023-10-12T01:56:54.555 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:56:54 GMT /wafmoa-74/obj3 2023-10-12T01:56:54.556 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:Mf9+RigBQgICzMRUXBrnsG5yND8= 2023-10-12T01:56:54.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mf9+RigBQgICzMRUXBrnsG5yND8=', 'Content-Length': '0'} 2023-10-12T01:56:54.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a819f2665e8cfd19-0065275266-2db0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.559 DEBUG:boto:path=/wafmoa-74/obj3 2023-10-12T01:56:54.559 DEBUG:boto:auth_path=/wafmoa-74/obj3 2023-10-12T01:56:54.560 DEBUG:boto:Method: GET 2023-10-12T01:56:54.560 DEBUG:boto:Path: /wafmoa-74/obj3 2023-10-12T01:56:54.560 DEBUG:boto:Data: 2023-10-12T01:56:54.561 DEBUG:boto:Headers: {} 2023-10-12T01:56:54.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:54.562 DEBUG:boto:Port: 8001 2023-10-12T01:56:54.562 DEBUG:boto:Params: {} 2023-10-12T01:56:54.563 DEBUG:boto:Token: None 2023-10-12T01:56:54.563 DEBUG:boto:StringToSign: GET Thu, 12 Oct 2023 01:56:54 GMT /wafmoa-74/obj3 2023-10-12T01:56:54.564 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:dyN2nWGNEaTetDcQ4Fp5H6oazKY= 2023-10-12T01:56:54.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dyN2nWGNEaTetDcQ4Fp5H6oazKY=', 'Content-Length': '0'} 2023-10-12T01:56:54.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000527beb72414f01cd-0065275266-2db0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.567 DEBUG:boto:path=/wafmoa-74/obj4 2023-10-12T01:56:54.567 DEBUG:boto:auth_path=/wafmoa-74/obj4 2023-10-12T01:56:54.568 DEBUG:boto:Method: HEAD 2023-10-12T01:56:54.568 DEBUG:boto:Path: /wafmoa-74/obj4 2023-10-12T01:56:54.568 DEBUG:boto:Data: 2023-10-12T01:56:54.569 DEBUG:boto:Headers: {} 2023-10-12T01:56:54.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:54.570 DEBUG:boto:Port: 8001 2023-10-12T01:56:54.570 DEBUG:boto:Params: {} 2023-10-12T01:56:54.570 DEBUG:boto:Token: None 2023-10-12T01:56:54.571 DEBUG:boto:StringToSign: HEAD Thu, 12 Oct 2023 01:56:54 GMT /wafmoa-74/obj4 2023-10-12T01:56:54.571 DEBUG:boto:Signature: AWS L12R7HDAL0U7FKTOBSDN:ORoyy0Eyt6fZsyTMJHba1URL9rc= 2023-10-12T01:56:54.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ORoyy0Eyt6fZsyTMJHba1URL9rc=', 'Content-Length': '0'} 2023-10-12T01:56:54.573 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000098659b738bed71bf-0065275266-2db0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.578 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2023-10-12T01:56:54.578 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.579 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T01:56:54.579 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-12T01:56:54.579 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T01:56:54.580 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T01:56:54.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T01:56:54.580 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T01:56:54.581 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-12T01:56:54.581 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-12T01:56:54.581 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-12T01:56:54.582 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-12T01:56:54.582 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-12T01:56:54.582 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-12T01:56:54.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-12T01:56:54.583 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-12T01:56:54.583 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-12T01:56:54.583 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-12T01:56:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.587 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.587 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.588 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.588 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:07 GMT 2023-10-12T01:56:54.588 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.589 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:4cTpPqbe4rTNgE6yo/fL3yMXG1Q= 2023-10-12T01:56:54.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:07 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:4cTpPqbe4rTNgE6yo/fL3yMXG1Q=', 'Content-Length': '0'} 2023-10-12T01:56:54.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000764e5bb5cb966634-0065274af3-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=wafmoa-63 2023-10-12T01:56:54.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:54.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:54.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:54.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:54.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073907.272991_694.1", 2023-10-12T01:56:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:25:07.272991Z" 2023-10-12T01:56:54.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:54.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:54.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:54.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:54.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:54.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:54.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:07.457+0000 7f4d81570780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:54.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:07.458+0000 7f4d81570780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:54.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:54.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:54.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:54.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:54.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:54.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:54.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073641.852711_690.1", 2023-10-12T01:56:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:20:41.852711Z", 2023-10-12T01:56:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:54.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:54.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:54.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:54.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:54.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:25:07Z" 2023-10-12T01:56:54.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:54.624 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:54.624 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:54.625 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:54.625 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:54.625 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:54.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.626 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.626 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:54.627 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.627 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:54.627 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:54.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.628 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:54.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:54.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.630 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.630 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.630 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.630 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:54.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.631 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:54.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:54.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.633 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.634 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.634 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.634 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:54.635 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073641.852711_690.1", 2023-10-12T01:56:54.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:20:41.852711Z", 2023-10-12T01:56:54.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.637 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.637 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.638 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.638 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:54.638 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.639 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:54.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:54.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.641 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.641 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.641 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:54.642 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.642 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:54.642 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:54.643 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:54.643 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.643 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:25:07Z" 2023-10-12T01:56:54.643 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:54.644 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073641.852711_690.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:20:41.852711Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:54.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:54.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073641.852711_690.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:54.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697073907.272991_694.1 target=1_1697073641.852711_690.1 2023-10-12T01:56:54.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:54.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:54.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:12.579+0000 7f500174e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:54.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:12.580+0000 7f500174e780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:54.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:54.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:54.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:54.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073907.272991_694.1", 2023-10-12T01:56:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:25:07.272991Z", 2023-10-12T01:56:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:54.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:54.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:54.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:54.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:25:12Z" 2023-10-12T01:56:54.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:54.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:54.667 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:54.667 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:54.668 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:54.668 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:54.668 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:54.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.669 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.669 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:54.670 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.670 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:54.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:54.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.671 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:54.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:54.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.673 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.673 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.673 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.674 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:54.674 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:54.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:54.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.676 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.677 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.677 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.677 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:54.677 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073907.272991_694.1", 2023-10-12T01:56:54.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:25:07.272991Z", 2023-10-12T01:56:54.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.680 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.680 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.680 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:54.681 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:54.681 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:54.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:54.682 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:54.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:54.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:54.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:54.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:54.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:54.683 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.684 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:54.684 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:54.684 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.685 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:54.685 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:54.685 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:54.685 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:54.686 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:25:12Z" 2023-10-12T01:56:54.686 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:54.686 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697073907.272991_694.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:25:07.272991Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:54.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:54.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697073907.272991_694.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:54.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.691 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.691 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.692 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.692 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.692 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.693 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000477a20c628141b25-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj11 2023-10-12T01:56:54.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj11 2023-10-12T01:56:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj11 2023-10-12T01:56:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.697 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.697 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.698 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.698 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.698 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj11 2023-10-12T01:56:54.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.699 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:C6wxCu0gqiry7gFYy+QKDhrBsPM= 2023-10-12T01:56:54.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:C6wxCu0gqiry7gFYy+QKDhrBsPM='} 2023-10-12T01:56:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2e54c30ec478bcb-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.703 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.703 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.704 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.704 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.704 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.705 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008141fe71a249f16f-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj22 2023-10-12T01:56:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj22 2023-10-12T01:56:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj22 2023-10-12T01:56:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.709 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.709 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.710 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.710 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.710 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj22 2023-10-12T01:56:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.711 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8FULwpja/oK0HeQU9p8AsJjNtiI= 2023-10-12T01:56:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8FULwpja/oK0HeQU9p8AsJjNtiI='} 2023-10-12T01:56:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daf9ed7830c6bf6f-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.715 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.715 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.716 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.717 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.718 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.718 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000330255d270983df7-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj33 2023-10-12T01:56:54.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj33 2023-10-12T01:56:54.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj33 2023-10-12T01:56:54.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.723 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.723 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.723 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.723 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.724 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj33 2023-10-12T01:56:54.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.724 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:GHYAsjxj1pnsiABrakIvCUDILnM= 2023-10-12T01:56:54.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GHYAsjxj1pnsiABrakIvCUDILnM='} 2023-10-12T01:56:54.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a2acdb477a53d5a-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.729 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.729 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.729 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.729 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.730 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.730 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065373d4f0658d5e6-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj44 2023-10-12T01:56:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj44 2023-10-12T01:56:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj44 2023-10-12T01:56:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.734 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.735 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.735 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.735 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.736 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj44 2023-10-12T01:56:54.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.736 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:lvyzW1H7M6qVemD+wtU3RjqiEvg= 2023-10-12T01:56:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lvyzW1H7M6qVemD+wtU3RjqiEvg='} 2023-10-12T01:56:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd228e0b5ab0c79c-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.740 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.741 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.742 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.742 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001944a7f56fcc8fa-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj55 2023-10-12T01:56:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj55 2023-10-12T01:56:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj55 2023-10-12T01:56:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.746 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.747 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.747 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.747 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.748 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj55 2023-10-12T01:56:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.748 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:xma0COUtyNWBbfy2d2pREhtegh8= 2023-10-12T01:56:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xma0COUtyNWBbfy2d2pREhtegh8='} 2023-10-12T01:56:54.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060276106f20a0a3f-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:54.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 5 2023-10-12T01:56:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.755 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.755 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.756 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.756 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.756 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.757 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c1791a02e592f7-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj17 2023-10-12T01:56:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj17 2023-10-12T01:56:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj17 2023-10-12T01:56:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.761 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.761 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.762 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.762 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj17 2023-10-12T01:56:54.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.763 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:xv1EDN5c70xQKxwH6kNmYjRD/WY= 2023-10-12T01:56:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xv1EDN5c70xQKxwH6kNmYjRD/WY='} 2023-10-12T01:56:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf6f47f7e8021af3-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.767 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.767 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.768 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.768 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.768 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.769 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U= 2023-10-12T01:56:54.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:P51viYSADqhygE994pftWMXL62U=', 'Content-Length': '0'} 2023-10-12T01:56:54.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d18291a360482b27-0065274af8-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj34 2023-10-12T01:56:54.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj34 2023-10-12T01:56:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj34 2023-10-12T01:56:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.773 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.774 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.774 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.774 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:12 GMT 2023-10-12T01:56:54.774 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj34 2023-10-12T01:56:54.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.775 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:wjnLKy8auPBzkwzXzEh/51RCmF0= 2023-10-12T01:56:54.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:12 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wjnLKy8auPBzkwzXzEh/51RCmF0='} 2023-10-12T01:56:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f04a63ad869a61d-0065274af8-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.779 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.779 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.780 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.780 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.780 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.781 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfc87b1726f3bf44-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj51 2023-10-12T01:56:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj51 2023-10-12T01:56:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj51 2023-10-12T01:56:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.785 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.786 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.786 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.787 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj51 2023-10-12T01:56:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.787 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:0KD6wRT6tsOdt1Qe/Q8ecJLYLBw= 2023-10-12T01:56:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0KD6wRT6tsOdt1Qe/Q8ecJLYLBw='} 2023-10-12T01:56:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091be1681e679689d-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.792 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.792 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.792 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.792 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.793 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.793 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052ed63fa72c97191-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj68 2023-10-12T01:56:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj68 2023-10-12T01:56:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj68 2023-10-12T01:56:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.798 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.798 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.799 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj68 2023-10-12T01:56:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.799 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:21TxuEV9Ng1Q7srYtwpGkqrR7mM= 2023-10-12T01:56:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:21TxuEV9Ng1Q7srYtwpGkqrR7mM='} 2023-10-12T01:56:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5466cc15005f019-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.804 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.804 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.804 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.805 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.805 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.805 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2697b764caebc12-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj85 2023-10-12T01:56:54.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj85 2023-10-12T01:56:54.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj85 2023-10-12T01:56:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.810 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.810 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.811 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.811 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.811 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj85 2023-10-12T01:56:54.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.812 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:SxwQRri9Jbc7v2042kDR3oULtTw= 2023-10-12T01:56:54.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SxwQRri9Jbc7v2042kDR3oULtTw='} 2023-10-12T01:56:54.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005effddd8e47d93c4-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:54.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:54.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 10 2023-10-12T01:56:54.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.819 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.819 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.819 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.820 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.820 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.821 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000712e4f22e9e38574-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj19 2023-10-12T01:56:54.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj19 2023-10-12T01:56:54.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj19 2023-10-12T01:56:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.825 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.825 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.826 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.826 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.827 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj19 2023-10-12T01:56:54.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.828 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:iXm827ttWxhZxBRmzYLMOudecsE= 2023-10-12T01:56:54.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iXm827ttWxhZxBRmzYLMOudecsE='} 2023-10-12T01:56:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e9fe7522d9a8399-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.832 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.833 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.833 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.833 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.834 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.834 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4d4ee4c3b81d4e2-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj38 2023-10-12T01:56:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj38 2023-10-12T01:56:54.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj38 2023-10-12T01:56:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.839 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.839 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.840 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.840 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.840 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj38 2023-10-12T01:56:54.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.841 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:mSbSzOK2GYfFkQkz5hhYf6+Wpq4= 2023-10-12T01:56:54.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mSbSzOK2GYfFkQkz5hhYf6+Wpq4='} 2023-10-12T01:56:54.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a0e9e9943642a50-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.847 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.847 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.847 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.848 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.848 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.849 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049060741b175eeee-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj57 2023-10-12T01:56:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj57 2023-10-12T01:56:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj57 2023-10-12T01:56:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.853 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.853 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.854 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.854 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.854 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj57 2023-10-12T01:56:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.855 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:7tuFEBepqrXQuE8ke5ranXqRR0A= 2023-10-12T01:56:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7tuFEBepqrXQuE8ke5ranXqRR0A='} 2023-10-12T01:56:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa1b4265bedf362a-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.859 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.859 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.860 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.860 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.860 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.861 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000607e0095deba56ee-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj76 2023-10-12T01:56:54.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj76 2023-10-12T01:56:54.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj76 2023-10-12T01:56:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.867 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.868 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.868 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.868 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.868 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj76 2023-10-12T01:56:54.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.869 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:2SEOW2CpapHhZrUNdy9oc8jf/D0= 2023-10-12T01:56:54.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2SEOW2CpapHhZrUNdy9oc8jf/D0='} 2023-10-12T01:56:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c270bdbfe99e1e-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.873 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.873 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.874 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.874 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.874 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.875 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000069a06b439f45473-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj95 2023-10-12T01:56:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj95 2023-10-12T01:56:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj95 2023-10-12T01:56:54.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.879 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.879 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.880 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.880 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.880 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj95 2023-10-12T01:56:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.881 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Hqf98kNnJG12G40H56glYwWGkfo= 2023-10-12T01:56:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Hqf98kNnJG12G40H56glYwWGkfo='} 2023-10-12T01:56:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009667689a646eb89-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:54.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:54.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:54.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 15 2023-10-12T01:56:54.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.888 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.888 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.888 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.889 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.889 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.890 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004539ae95070403ac-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj23 2023-10-12T01:56:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj23 2023-10-12T01:56:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj23 2023-10-12T01:56:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.894 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.894 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.894 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.895 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.895 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj23 2023-10-12T01:56:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.896 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:d9aTojXsW75gZhZ3b9ukzOpU64o= 2023-10-12T01:56:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:d9aTojXsW75gZhZ3b9ukzOpU64o='} 2023-10-12T01:56:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000096167a90e33bd1f-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.900 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.900 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.900 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.901 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.901 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.902 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031abb046b6d5be1f-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj46 2023-10-12T01:56:54.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj46 2023-10-12T01:56:54.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj46 2023-10-12T01:56:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.906 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.906 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.906 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.907 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.907 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj46 2023-10-12T01:56:54.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.908 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yD83Nlgem3kO4gbCHmhWwbXfm6g= 2023-10-12T01:56:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yD83Nlgem3kO4gbCHmhWwbXfm6g='} 2023-10-12T01:56:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a027c83aa6f881e0-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.912 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.912 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.912 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.912 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.913 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.913 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f4c51d61a0c7f4-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj69 2023-10-12T01:56:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj69 2023-10-12T01:56:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj69 2023-10-12T01:56:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.917 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.918 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.918 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.918 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.919 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj69 2023-10-12T01:56:54.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.919 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sm4VpdGGPGQMd/Nc9ZTlu8xdRlk= 2023-10-12T01:56:54.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sm4VpdGGPGQMd/Nc9ZTlu8xdRlk='} 2023-10-12T01:56:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001eeb24404385dd82-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.923 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.924 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.924 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.924 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.925 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.925 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd75d3658f3f1d03-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj92 2023-10-12T01:56:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj92 2023-10-12T01:56:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj92 2023-10-12T01:56:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.929 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.929 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.930 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.930 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.930 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj92 2023-10-12T01:56:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.931 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ZLm0vqJXX8gaieDI0s+KArCH92A= 2023-10-12T01:56:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZLm0vqJXX8gaieDI0s+KArCH92A='} 2023-10-12T01:56:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084fc3a3c82b81ef0-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.935 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.935 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.936 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.936 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.936 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.937 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc= 2023-10-12T01:56:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:hu39LV8QQmOaCJ9EVYCqljWfmxc=', 'Content-Length': '0'} 2023-10-12T01:56:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000409a3a499869be9b-0065274af9-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj115 2023-10-12T01:56:54.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj115 2023-10-12T01:56:54.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj115 2023-10-12T01:56:54.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.941 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.941 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.942 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.942 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:13 GMT 2023-10-12T01:56:54.942 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj115 2023-10-12T01:56:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.943 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:xE5duTDERiOFpfg8UDtn2gT5dWA= 2023-10-12T01:56:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:13 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xE5duTDERiOFpfg8UDtn2gT5dWA='} 2023-10-12T01:56:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b8b72c8120d8e6d-0065274af9-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:54.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:54.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:54.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 20 2023-10-12T01:56:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.949 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.950 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.950 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.950 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.951 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.951 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee88bf8e6d911180-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj29 2023-10-12T01:56:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj29 2023-10-12T01:56:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj29 2023-10-12T01:56:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.955 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.956 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.956 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.956 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.957 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj29 2023-10-12T01:56:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.957 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:IlgiGIujuh7KFp7hz23A83hVG6s= 2023-10-12T01:56:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:IlgiGIujuh7KFp7hz23A83hVG6s='} 2023-10-12T01:56:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087806eed63e5efaf-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.961 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.962 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.962 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.962 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.963 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.963 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:54.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd93f728c45f993-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj58 2023-10-12T01:56:54.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj58 2023-10-12T01:56:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj58 2023-10-12T01:56:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.967 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.968 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.968 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.968 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.969 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj58 2023-10-12T01:56:54.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.969 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:lzdBz5TpMXXW+PKvUp2DI0IXN0w= 2023-10-12T01:56:54.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:lzdBz5TpMXXW+PKvUp2DI0IXN0w='} 2023-10-12T01:56:54.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0af9cfe501c0d6e-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.973 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.974 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.974 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.974 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.975 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.975 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095ea9bcbe38cd249-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj87 2023-10-12T01:56:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj87 2023-10-12T01:56:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj87 2023-10-12T01:56:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.980 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.980 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.980 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.981 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.981 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj87 2023-10-12T01:56:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.981 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:I4HMB/9WdA9GQp5RWMQv/TGy564= 2023-10-12T01:56:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:I4HMB/9WdA9GQp5RWMQv/TGy564='} 2023-10-12T01:56:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a7c0a1981064446-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.986 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.986 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.986 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.987 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.987 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.987 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:54.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:54.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023d891402eee6490-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj116 2023-10-12T01:56:54.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj116 2023-10-12T01:56:54.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:54.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj116 2023-10-12T01:56:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.992 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:54.992 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:54.992 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:54.993 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.993 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj116 2023-10-12T01:56:54.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:54.994 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:tG+EdQo21ImmTrgU4/PiV2XXQRs= 2023-10-12T01:56:54.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tG+EdQo21ImmTrgU4/PiV2XXQRs='} 2023-10-12T01:56:54.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000883b6129c3e4a9c6-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:54.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:54.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:54.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:54.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:54.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:54.998 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:54.998 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.998 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:54.999 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:54.999 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:54.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.000 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038d7e57c68d7c917-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj145 2023-10-12T01:56:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj145 2023-10-12T01:56:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj145 2023-10-12T01:56:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.004 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.004 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.005 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.005 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.005 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj145 2023-10-12T01:56:55.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.006 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EVhhuFslffKQwaWBZVt6LEW+heA= 2023-10-12T01:56:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVhhuFslffKQwaWBZVt6LEW+heA='} 2023-10-12T01:56:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed1c3d47fda2340-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:55.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:55.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 25 2023-10-12T01:56:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.012 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.013 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.013 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.013 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.013 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.014 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daf09a286872ba45-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj31 2023-10-12T01:56:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj31 2023-10-12T01:56:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj31 2023-10-12T01:56:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.019 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.019 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.019 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.019 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.020 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj31 2023-10-12T01:56:55.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.020 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:GFWhFD9cLoWZMh+XnFX+RSB7ruc= 2023-10-12T01:56:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GFWhFD9cLoWZMh+XnFX+RSB7ruc='} 2023-10-12T01:56:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c0af0c1145ca8ca-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.025 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.025 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.025 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.025 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.026 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.026 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3d8106cece000b4-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj62 2023-10-12T01:56:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj62 2023-10-12T01:56:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj62 2023-10-12T01:56:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.031 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.031 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.031 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.032 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.032 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj62 2023-10-12T01:56:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.033 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:znP6iipYkJ4XwG4ojp4FjS8Xvv0= 2023-10-12T01:56:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:znP6iipYkJ4XwG4ojp4FjS8Xvv0='} 2023-10-12T01:56:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000addc2f286dca1abb-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.037 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.037 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.038 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.038 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.038 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.039 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6c5de11d5d8d3fe-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj93 2023-10-12T01:56:55.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj93 2023-10-12T01:56:55.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj93 2023-10-12T01:56:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.043 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.043 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.044 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.044 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.044 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj93 2023-10-12T01:56:55.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.045 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:/XB5zulF3bTtLNAtqUq2qtLAFMA= 2023-10-12T01:56:55.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/XB5zulF3bTtLNAtqUq2qtLAFMA='} 2023-10-12T01:56:55.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0448fc2e964c74d-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.049 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.049 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.050 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.050 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.050 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.051 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d24e2d1acd4b90-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj124 2023-10-12T01:56:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj124 2023-10-12T01:56:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj124 2023-10-12T01:56:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.055 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.055 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.056 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.056 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.056 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj124 2023-10-12T01:56:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.057 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8cJU3DjEnWK58XslCHBbYVYyH20= 2023-10-12T01:56:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8cJU3DjEnWK58XslCHBbYVYyH20='} 2023-10-12T01:56:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e96b664b6e67f5e3-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.061 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.061 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.062 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.062 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.062 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.063 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f23f3c046225583-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj155 2023-10-12T01:56:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj155 2023-10-12T01:56:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj155 2023-10-12T01:56:55.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.067 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.068 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.068 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.068 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj155 2023-10-12T01:56:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.069 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:7fYxoerX0Ps6YRdc3J/DyWsIAMw= 2023-10-12T01:56:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7fYxoerX0Ps6YRdc3J/DyWsIAMw='} 2023-10-12T01:56:55.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b7e5cd91c0a8139-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:55.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:55.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-63 2023-10-12T01:56:55.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 30 2023-10-12T01:56:55.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.076 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.076 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.076 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.077 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.077 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.077 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d1dc8c5485fb11-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj37 2023-10-12T01:56:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj37 2023-10-12T01:56:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj37 2023-10-12T01:56:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.082 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.082 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.082 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.083 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.083 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj37 2023-10-12T01:56:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.084 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fshpsWlmaC/Fei8FkluWyGV22e4= 2023-10-12T01:56:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fshpsWlmaC/Fei8FkluWyGV22e4='} 2023-10-12T01:56:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb0311eaa1113bf2-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.088 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.088 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.088 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.089 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.089 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.090 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3ce7f8724f5b7a5-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj74 2023-10-12T01:56:55.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj74 2023-10-12T01:56:55.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj74 2023-10-12T01:56:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.094 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.094 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.094 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.095 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.095 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj74 2023-10-12T01:56:55.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.096 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:+SQYFAZOsYgf7aXyF7Fc4jZxh2s= 2023-10-12T01:56:55.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:+SQYFAZOsYgf7aXyF7Fc4jZxh2s='} 2023-10-12T01:56:55.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb688eb02d416489-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.100 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.100 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.100 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.101 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.101 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.102 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026a90689ab34eb32-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj111 2023-10-12T01:56:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj111 2023-10-12T01:56:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj111 2023-10-12T01:56:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.106 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.106 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.107 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.107 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.107 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj111 2023-10-12T01:56:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.108 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:wsKDnnf/2F+WewbxqOSGUbsH5UA= 2023-10-12T01:56:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wsKDnnf/2F+WewbxqOSGUbsH5UA='} 2023-10-12T01:56:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023786c046015ecd5-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.112 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.113 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.113 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.113 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.114 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.114 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd450ef88ca394e7-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj148 2023-10-12T01:56:55.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj148 2023-10-12T01:56:55.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj148 2023-10-12T01:56:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.120 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.120 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.121 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.121 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.121 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj148 2023-10-12T01:56:55.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.122 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:AGghmVGrUASOKD9z+8p4UeHKNTY= 2023-10-12T01:56:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:AGghmVGrUASOKD9z+8p4UeHKNTY='} 2023-10-12T01:56:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074b7a90db89ac803-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/ 2023-10-12T01:56:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/ 2023-10-12T01:56:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/ 2023-10-12T01:56:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.126 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.126 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.130 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.130 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.131 INFO:tasks.rgw_multisite_tests:/wafmoa-63/ 2023-10-12T01:56:55.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.131 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c= 2023-10-12T01:56:55.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Sa0r0FZt6j68AVatutDdcLTz4/c=', 'Content-Length': '0'} 2023-10-12T01:56:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c64fab5317b65ae5-0065274afa-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-63/obj185 2023-10-12T01:56:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-63/obj185 2023-10-12T01:56:55.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-63/obj185 2023-10-12T01:56:55.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.136 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.137 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.137 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.138 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:25:14 GMT 2023-10-12T01:56:55.138 INFO:tasks.rgw_multisite_tests:/wafmoa-63/obj185 2023-10-12T01:56:55.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.139 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:GMb8M6GM3ps8MeCNn7vdEdKwj7k= 2023-10-12T01:56:55.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:25:14 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:GMb8M6GM3ps8MeCNn7vdEdKwj7k='} 2023-10-12T01:56:55.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7ac61e2cdee3e35-0065274afa-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=wafmoa-63 2023-10-12T01:56:55.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:56:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:15.052+0000 7f9f7b8ff780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:20.053+0000 7f9f7b8ff780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:25.055+0000 7f9f7b8ff780 1 bucket sync caught up with source: 2023-10-12T01:56:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , 00000000002.88.6, , , , , , 00000000002.120.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.90.6, , , , 00000000001.128.6] 2023-10-12T01:56:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , 00000000002.88.6, , , , , , 00000000002.120.6, , , , , 00000000002.102.6, , , , , , , , , , , , , , , 00000000001.90.6, , , , 00000000001.128.6] 2023-10-12T01:56:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:25.055+0000 7f9f7b8ff780 0 bucket checkpoint complete 2023-10-12T01:56:55.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=wafmoa-63 2023-10-12T01:56:55.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:56:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:25.166+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:30.168+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:35.169+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:40.168+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:45.169+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:50.170+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:25:55.171+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:00.171+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:05.172+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:10.173+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:15.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:20.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:25.174+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:30.175+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:35.176+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:40.176+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:45.178+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:50.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:26:55.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:00.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:05.179+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:10.180+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:15.181+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:20.183+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:25.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:30.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:35.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:40.184+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:45.185+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:50.185+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:27:55.187+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:00.188+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:05.188+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.157 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi112.stdout: INFO: 2023-10-12T01:28:05.272+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:56:55.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:10.189+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:15.189+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:20.190+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:25.190+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:30.191+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:35.192+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:40.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:45.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:50.193+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:28:55.194+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:00.194+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:05.195+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:10.196+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:15.197+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:20.198+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:25.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:30.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:35.199+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:40.200+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:45.200+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:50.201+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:29:55.202+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:00.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:05.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:10.203+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:15.204+0000 7f48f099e780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.205+0000 7f48f099e780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-12T01:56:55.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.205+0000 7f48f099e780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:56:55.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.205+0000 7f48f099e780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:56:55.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-12T01:56:55.166 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-12T01:56:55.167 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.167 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T01:56:55.167 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-12T01:56:55.168 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T01:56:55.168 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T01:56:55.169 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T01:56:55.169 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T01:56:55.169 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-12T01:56:55.170 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-12T01:56:55.170 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-12T01:56:55.170 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-12T01:56:55.171 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-12T01:56:55.171 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-12T01:56:55.171 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-12T01:56:55.172 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-12T01:56:55.172 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-12T01:56:55.172 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-12T01:56:55.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:56:55.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.176 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.176 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.177 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.177 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.177 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.178 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EprK931kEfLwdn4fhLNmw0HHAEQ= 2023-10-12T01:56:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EprK931kEfLwdn4fhLNmw0HHAEQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000c5c17e232d7dba7-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=wafmoa-64 2023-10-12T01:56:55.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:55.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:55.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:55.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:55.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:55.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:55.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697074220.255567_728.1", 2023-10-12T01:56:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:30:20.255567Z" 2023-10-12T01:56:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:55.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:55.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:55.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:55.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:55.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:55.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.448+0000 7f0bf8eeb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:55.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.448+0000 7f0bf8eeb780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:55.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:55.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:55.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:55.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:55.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:55.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:55.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:55.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:55.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:55.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:55.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:55.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:55.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:55.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:55.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:55.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:55.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:55.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:55.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697074220.255567_728.1", 2023-10-12T01:56:55.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:55.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:55.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:55.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:30:20.255567Z", 2023-10-12T01:56:55.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:55.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:55.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:55.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:55.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:55.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:55.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:55.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:55.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:55.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:55.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:55.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:55.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:55.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:55.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:55.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:55.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:30:20Z" 2023-10-12T01:56:55.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:55.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:55.212 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:55.212 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:55.212 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:55.212 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:55.213 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:55.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:55.213 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:55.214 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:55.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:55.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:55.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:55.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:55.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:55.216 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:55.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:55.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:55.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:55.217 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:55.217 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.218 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:55.218 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:55.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:55.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:55.219 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:55.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:55.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:55.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:55.220 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:55.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:55.220 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:55.221 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.221 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:55.221 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:55.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:55.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:55.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1697074220.255567_728.1", 2023-10-12T01:56:55.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:55.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:55.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:55.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:30:20.255567Z", 2023-10-12T01:56:55.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:55.224 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:55.224 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.225 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:55.225 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:55.225 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:55.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:55.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:55.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:55.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:55.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:55.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:55.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:55.228 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:55.228 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:55.228 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:55.228 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:55.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:55.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:55.230 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:55.230 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:30:20Z" 2023-10-12T01:56:55.230 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:55.231 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697074220.255567_728.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:30:20.255567Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:55.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:55.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697074220.255567_728.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:55.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.235 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.236 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.236 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.236 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.237 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.237 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:56:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:56:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ef27b0ee737079-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11 2023-10-12T01:56:55.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.241 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.242 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.242 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.242 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.243 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11 2023-10-12T01:56:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.243 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:NCUlPQFyDU1TE5Fe5zKe+ld6SzA= 2023-10-12T01:56:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:NCUlPQFyDU1TE5Fe5zKe+ld6SzA='} 2023-10-12T01:56:55.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087de8b05277a155a-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.247 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.248 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.248 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.248 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.249 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.249 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:56:55.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:56:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b0fedd5f950a5dc-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22 2023-10-12T01:56:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.253 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.254 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.254 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.254 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.255 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22 2023-10-12T01:56:55.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.255 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:87oPWsLqTMEej8BVCDz3tuVA/8M= 2023-10-12T01:56:55.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:87oPWsLqTMEej8BVCDz3tuVA/8M='} 2023-10-12T01:56:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d91d09c89e6a1e4b-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.259 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.260 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.260 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.260 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.261 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.261 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:56:55.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:56:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d9be4e2b531672c-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33 2023-10-12T01:56:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.265 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.266 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.266 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.266 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.267 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33 2023-10-12T01:56:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.267 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:wLmEm+f7riW5E1a2foZucljf6wU= 2023-10-12T01:56:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:wLmEm+f7riW5E1a2foZucljf6wU='} 2023-10-12T01:56:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006323aef16ad79298-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.271 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.272 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.272 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.272 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.272 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.273 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:56:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:56:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a94e30c473916127-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44 2023-10-12T01:56:55.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.277 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.277 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.278 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.278 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.278 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44 2023-10-12T01:56:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.279 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:kBbN+pbJI86/ff9n5J0wop+6iDM= 2023-10-12T01:56:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:kBbN+pbJI86/ff9n5J0wop+6iDM='} 2023-10-12T01:56:55.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000748e219bec4a4ae4-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.283 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.284 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.284 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.284 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.284 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.285 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc= 2023-10-12T01:56:55.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yUWaGVlZJP9sPtuR/4eUbozxzbc=', 'Content-Length': '0'} 2023-10-12T01:56:55.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca3d6cb9e49edc7-0065274c2c-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55 2023-10-12T01:56:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.289 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.289 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.290 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.290 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:20 GMT 2023-10-12T01:56:55.290 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55 2023-10-12T01:56:55.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.291 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:jayQ9u/JPcxUVz2loRgGb4ycmj8= 2023-10-12T01:56:55.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:20 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:jayQ9u/JPcxUVz2loRgGb4ycmj8='} 2023-10-12T01:56:55.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8c0da041751dbda-0065274c2c-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=wafmoa-64 2023-10-12T01:56:55.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:56:55.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:20.705+0000 7f2ea8280780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:25.706+0000 7f2ea8280780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:30.709+0000 7f2ea8280780 1 bucket sync caught up with source: 2023-10-12T01:56:55.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , , , 00000000001.107.6, 00000000002.126.6, 00000000002.109.6] 2023-10-12T01:56:55.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.126.6, 00000000002.109.6] 2023-10-12T01:56:55.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:30.709+0000 7f2ea8280780 0 bucket checkpoint complete 2023-10-12T01:56:55.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=wafmoa-64 2023-10-12T01:56:55.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:55.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:56:55.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:30.829+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:35.830+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:40.830+0000 7f5040497780 1 waiting to reach incremental sync.. 2023-10-12T01:56:55.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:45.830+0000 7f5040497780 1 bucket sync caught up with source: 2023-10-12T01:56:55.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , , , 00000000001.362.6, 00000000002.147.6, 00000000002.378.6] 2023-10-12T01:56:55.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , , , 00000000001.362.6, 00000000002.147.6, 00000000002.378.6] 2023-10-12T01:56:55.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:45.830+0000 7f5040497780 0 bucket checkpoint complete 2023-10-12T01:56:55.299 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=wafmoa-64 zones={a2, a1} 2023-10-12T01:56:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:56:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.303 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.304 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.304 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.304 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.305 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.305 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY= 2023-10-12T01:56:55.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY=', 'Content-Length': '0'} 2023-10-12T01:56:55.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008252213f32c32a6d-0065274c45-1fd8-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.309 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.310 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.310 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.310 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.311 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.311 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY= 2023-10-12T01:56:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:5QyL0rT/4SrLUtAMXA8MiK3W+tY=', 'Content-Length': '0'} 2023-10-12T01:56:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc5155fd8e8768b9-0065274c45-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.312 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-12T01:56:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.316 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.317 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.317 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.317 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.318 INFO:tasks.rgw_multisite_tests:/wafmoa-64/?versions 2023-10-12T01:56:55.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.318 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:56:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:56:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c96d658d38488c-0065274c45-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-12T01:56:55.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-12T01:56:55.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-12T01:56:55.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-12T01:56:55.321 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-12T01:56:55.321 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-12T01:56:55.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.325 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.325 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.326 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.326 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.326 INFO:tasks.rgw_multisite_tests:/wafmoa-64/?versions 2023-10-12T01:56:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.327 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:56:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:56:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783cd5e8e30dd3ed-0065274c45-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.328 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-12T01:56:55.328 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-12T01:56:55.329 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-12T01:56:55.329 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-12T01:56:55.329 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-12T01:56:55.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.333 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.334 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.334 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.334 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.335 INFO:tasks.rgw_multisite_tests:/wafmoa-64/?versions 2023-10-12T01:56:55.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.335 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:56:55.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:56:55.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5c9bdb5488be8b-0065274c45-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?versions&max-keys=999 2023-10-12T01:56:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.340 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.341 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.341 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.341 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.342 INFO:tasks.rgw_multisite_tests:/wafmoa-64/?versions 2023-10-12T01:56:55.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.342 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c= 2023-10-12T01:56:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Jquby3+tox//PmETkA5tUVdCY2c=', 'Content-Length': '0'} 2023-10-12T01:56:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c24dc747f77b5b1-0065274c45-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64999falseobj11nulltrue2023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-12T01:56:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.348 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.348 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.348 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.349 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.349 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.350 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:56:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:56:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f9202f6b7dae85-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.355 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.355 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.355 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.355 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.356 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.356 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:56:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:56:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa8e0f03fed8b05-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.361 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.361 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.362 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.362 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.362 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.363 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY= 2023-10-12T01:56:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY=', 'Content-Length': '0'} 2023-10-12T01:56:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c350ce304ee182b4-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.368 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.368 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.369 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.369 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.370 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY= 2023-10-12T01:56:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:2MQHq/m+HC5Mop/Mc+4HE0ZtsLY=', 'Content-Length': '0'} 2023-10-12T01:56:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1428ed02c557151-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.371 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-12T01:56:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.375 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.375 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.375 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.376 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.376 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.377 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:56:55.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:56:55.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac620c7a80bd08eb-0065274c45-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.381 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.382 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.382 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.382 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:45 GMT 2023-10-12T01:56:55.383 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11?versionId=null 2023-10-12T01:56:55.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.383 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0= 2023-10-12T01:56:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8UOZbxYmnghdaqEDeDBhjXcnXg0=', 'Content-Length': '0'} 2023-10-12T01:56:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc79641a99699d1-0065274c45-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.388 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.388 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.389 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.389 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.389 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.390 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f40d50a3601ef0-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.395 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.395 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.396 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.396 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.396 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.397 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6795281344c943c-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11 2023-10-12T01:56:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.401 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.402 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.402 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.402 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.403 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11 2023-10-12T01:56:55.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.403 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU= 2023-10-12T01:56:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU=', 'Content-Length': '0'} 2023-10-12T01:56:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85933b2af40f926-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11 2023-10-12T01:56:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.408 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.408 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.408 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.409 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.409 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11 2023-10-12T01:56:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.409 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU= 2023-10-12T01:56:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FvkIcEJb99jQBsuos7prhsrtZPU=', 'Content-Length': '0'} 2023-10-12T01:56:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009605c67a523e7456-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.410 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-12T01:56:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11 2023-10-12T01:56:55.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.414 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.414 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.415 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.415 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.415 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11 2023-10-12T01:56:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.416 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8= 2023-10-12T01:56:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8=', 'Content-Length': '0'} 2023-10-12T01:56:55.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb6d39d5126985b-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj11 2023-10-12T01:56:55.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj11 2023-10-12T01:56:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj11 2023-10-12T01:56:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.420 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.421 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.421 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.421 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.421 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj11 2023-10-12T01:56:55.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.422 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8= 2023-10-12T01:56:55.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:BjJZ55gvePpnbqsM7UPDxfA5qT8=', 'Content-Length': '0'} 2023-10-12T01:56:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346fd2af61236e50-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.427 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.427 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.427 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.428 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.428 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.429 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141941e23c04493d-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj11 2023-10-12T01:56:55.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.434 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.434 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.434 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.435 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.435 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.436 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007adb96c1e1a26be-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj111000falseobj112023-10-12T01:30:20.497Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.437 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-12T01:56:55.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.441 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.441 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.442 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.442 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.442 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.443 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:56:55.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:56:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf0ee7f5db19442-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.448 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.448 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.449 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.449 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.449 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.450 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:56:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:56:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2d53b20ceea4fa-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.455 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.455 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.455 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.456 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.456 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.457 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY= 2023-10-12T01:56:55.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY=', 'Content-Length': '0'} 2023-10-12T01:56:55.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096a228288570c87-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.461 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.462 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.462 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.462 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.463 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.463 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY= 2023-10-12T01:56:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0mYOxMfceHjxo7r6K1c+OwP+JRY=', 'Content-Length': '0'} 2023-10-12T01:56:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81ba5929dbb4201-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.464 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-12T01:56:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.469 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.469 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.469 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.470 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.470 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.471 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:56:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:56:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e9d517f5a2e2e68-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.475 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.476 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.476 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.476 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.477 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22?versionId=null 2023-10-12T01:56:55.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.477 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88= 2023-10-12T01:56:55.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:gHrDyeDtpTGiroAT003UOGyGv88=', 'Content-Length': '0'} 2023-10-12T01:56:55.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b4d07e0cf785fe-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.482 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.482 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.483 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.483 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.483 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.484 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be81250c274cebb6-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.490 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.490 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.490 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.490 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.491 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.491 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b73a68c5da8ca64-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22 2023-10-12T01:56:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.496 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.496 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.496 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.496 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.497 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22 2023-10-12T01:56:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.497 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo= 2023-10-12T01:56:55.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo=', 'Content-Length': '0'} 2023-10-12T01:56:55.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2327a02570a56ca-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22 2023-10-12T01:56:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.502 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.502 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.502 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.502 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.503 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22 2023-10-12T01:56:55.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.503 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo= 2023-10-12T01:56:55.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:K3/+2yfzWujZ7vxTWbi6mCRLCeo=', 'Content-Length': '0'} 2023-10-12T01:56:55.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9ce54aa2b31d83-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.504 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-12T01:56:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22 2023-10-12T01:56:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.508 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.508 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.508 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.509 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.509 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22 2023-10-12T01:56:55.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.510 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM= 2023-10-12T01:56:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM=', 'Content-Length': '0'} 2023-10-12T01:56:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045b45c3f715bcc6-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj22 2023-10-12T01:56:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj22 2023-10-12T01:56:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj22 2023-10-12T01:56:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.514 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.514 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.514 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.515 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.515 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj22 2023-10-12T01:56:55.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.516 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM= 2023-10-12T01:56:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZfkUa2qzHQeI5Y6+aRBPHpMCzwM=', 'Content-Length': '0'} 2023-10-12T01:56:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6d5789956a6feb-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.520 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.520 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.521 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.521 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.521 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.522 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cf78ffcee407245-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj22 2023-10-12T01:56:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.527 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.527 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.528 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.528 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.528 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.529 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003058291796839c2d-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj221000falseobj222023-10-12T01:30:20.516Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-12T01:56:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.535 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.535 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.535 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.536 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.537 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.538 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:56:55.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:56:55.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055d22dbb5c2ad19-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.542 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.543 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.543 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.543 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.544 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.544 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:56:55.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:56:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005657c0fd5d20e0f0-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.549 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.549 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.550 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.550 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.550 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.551 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs= 2023-10-12T01:56:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs=', 'Content-Length': '0'} 2023-10-12T01:56:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038e7a5b6fcc3499-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.556 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.556 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.556 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.557 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.557 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.557 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs= 2023-10-12T01:56:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S5Att3hI6tK5bRLUCSA1k5cKyUs=', 'Content-Length': '0'} 2023-10-12T01:56:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098edff317e648137-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.558 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-12T01:56:55.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.563 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.563 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.563 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.563 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.564 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.564 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:56:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:56:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d6b1e70f195937-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.569 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.569 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.570 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.570 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.570 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33?versionId=null 2023-10-12T01:56:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.571 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM= 2023-10-12T01:56:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:pir0gGTZv4hTPt4m2qoID5Nd5qM=', 'Content-Length': '0'} 2023-10-12T01:56:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2cdbaec3652ee2-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.576 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.576 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.576 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.577 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.577 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.578 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ce9167101e0fa9-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.583 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.583 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.584 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.584 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.584 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.585 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed52c923f6df55f9-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33 2023-10-12T01:56:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.589 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.590 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.590 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.590 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.590 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33 2023-10-12T01:56:55.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.591 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM= 2023-10-12T01:56:55.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM=', 'Content-Length': '0'} 2023-10-12T01:56:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ab83ffb423f414-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33 2023-10-12T01:56:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.595 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.595 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.596 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.596 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.596 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33 2023-10-12T01:56:55.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.597 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM= 2023-10-12T01:56:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zYRWZ6Hb9W8pFxxJw3wPTKw/BwM=', 'Content-Length': '0'} 2023-10-12T01:56:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfce61710d88113e-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.598 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-12T01:56:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33 2023-10-12T01:56:55.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.601 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.602 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.602 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.602 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.603 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33 2023-10-12T01:56:55.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.603 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0= 2023-10-12T01:56:55.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0=', 'Content-Length': '0'} 2023-10-12T01:56:55.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c975552d468d407-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj33 2023-10-12T01:56:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj33 2023-10-12T01:56:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj33 2023-10-12T01:56:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.608 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.608 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.608 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.609 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.609 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj33 2023-10-12T01:56:55.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.609 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0= 2023-10-12T01:56:55.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EVGzi71qBpcKTmaW+QLZiawo0D0=', 'Content-Length': '0'} 2023-10-12T01:56:55.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf56e3556589180-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.615 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.615 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.615 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.615 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.616 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.616 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3003ac0dc2c350-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj33 2023-10-12T01:56:55.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.622 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.622 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.622 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.623 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.623 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.624 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2bc581debb6dd5-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj331000falseobj332023-10-12T01:30:20.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.625 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-12T01:56:55.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.629 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.629 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.630 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.630 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.630 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.631 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:56:55.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:56:55.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db77de1f4753c536-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.636 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.636 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.636 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.636 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.637 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.637 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:56:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:56:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfaab50f0215f68-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.642 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.642 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.643 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.643 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.643 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.644 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8= 2023-10-12T01:56:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8=', 'Content-Length': '0'} 2023-10-12T01:56:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af25cf56fa23568-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.649 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.649 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.649 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.650 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.650 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.650 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8= 2023-10-12T01:56:55.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:DIfEhEI1emgWTts1BT8fEhAKzW8=', 'Content-Length': '0'} 2023-10-12T01:56:55.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bd68a7dd51a865-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.651 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-12T01:56:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.656 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.656 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.656 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.657 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.657 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.658 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:56:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:56:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7b8232c55429fe-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.662 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.663 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.663 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.663 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.663 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44?versionId=null 2023-10-12T01:56:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.664 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA= 2023-10-12T01:56:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:eAxqe3JDAEl9DpuJpSimLNr4ENA=', 'Content-Length': '0'} 2023-10-12T01:56:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d12c1b5808d4507-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.669 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.669 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.669 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.670 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.670 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.671 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc36c2a06d72044-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.676 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.676 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.676 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.677 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.677 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.677 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b2581414a266e1-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44 2023-10-12T01:56:55.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.682 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.682 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.682 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.683 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.683 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44 2023-10-12T01:56:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.684 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA= 2023-10-12T01:56:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA=', 'Content-Length': '0'} 2023-10-12T01:56:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b482ee95adf23e0f-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44 2023-10-12T01:56:55.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.688 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.688 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.688 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.689 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.689 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44 2023-10-12T01:56:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.689 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA= 2023-10-12T01:56:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iOXQ5g1e8oahMLCyP8e3BN1HixA=', 'Content-Length': '0'} 2023-10-12T01:56:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9b22d467aaf8941-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.690 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-12T01:56:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44 2023-10-12T01:56:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.694 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.694 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.694 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.695 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.695 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44 2023-10-12T01:56:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.696 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os= 2023-10-12T01:56:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os=', 'Content-Length': '0'} 2023-10-12T01:56:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c47c7ee4825327-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj44 2023-10-12T01:56:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj44 2023-10-12T01:56:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj44 2023-10-12T01:56:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.700 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.700 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.700 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.701 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.701 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj44 2023-10-12T01:56:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.702 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os= 2023-10-12T01:56:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:mZIgoUb0OPjrBuebVBAz5EHe6os=', 'Content-Length': '0'} 2023-10-12T01:56:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9db33dfe4eb6a94-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.706 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.707 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.707 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.707 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.707 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.708 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c687a102cd4ec5-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj44 2023-10-12T01:56:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.713 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.713 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.713 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.714 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.714 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.715 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9ce3c0ade60833-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj441000falseobj442023-10-12T01:30:20.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.716 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-12T01:56:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.720 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.720 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.721 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.721 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.721 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.722 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:56:55.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:56:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009255e70e42139756-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.727 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.727 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.728 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.728 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.728 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.729 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:56:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:56:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80334c494f13c5c-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.734 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.734 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.734 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.735 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.735 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.736 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no= 2023-10-12T01:56:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no=', 'Content-Length': '0'} 2023-10-12T01:56:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e64fbf0f6211e9-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.741 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.742 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.742 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.743 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no= 2023-10-12T01:56:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:N1lf6Wvc6cZ4l+0ruLEOMbdI5no=', 'Content-Length': '0'} 2023-10-12T01:56:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9ab6f75c5cfac3-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.746 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-12T01:56:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.750 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.751 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.751 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.751 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.751 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.752 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:56:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:56:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a184c47a20edec21-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.757 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.758 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.758 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.758 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.759 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55?versionId=null 2023-10-12T01:56:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.759 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU= 2023-10-12T01:56:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VUF46bH/PsJCiWxefV94AIlWXkU=', 'Content-Length': '0'} 2023-10-12T01:56:55.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818ecb80056ba36f-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.768 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.768 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.769 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.769 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.769 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.770 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb5d1a5e0d95bc4-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.775 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.775 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.775 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.776 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.776 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.777 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8abe559d8168dd-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55 2023-10-12T01:56:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.781 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.781 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.782 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.782 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.782 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55 2023-10-12T01:56:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.783 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw= 2023-10-12T01:56:55.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw=', 'Content-Length': '0'} 2023-10-12T01:56:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619807bc81e95e56-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55 2023-10-12T01:56:55.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.788 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.788 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.788 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.789 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.789 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55 2023-10-12T01:56:55.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.790 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw= 2023-10-12T01:56:55.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:uJsNcwaNukwpWKetoduL5AtuCvw=', 'Content-Length': '0'} 2023-10-12T01:56:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3d9f913e780189-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-12T01:56:55.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55 2023-10-12T01:56:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.795 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.795 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.796 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.796 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.796 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55 2023-10-12T01:56:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.797 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8= 2023-10-12T01:56:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8=', 'Content-Length': '0'} 2023-10-12T01:56:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-64:677f5a1d-3ab4-4b5f-af15-9890c2c79648.5775.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb97b347ba9b2cf5-0065274c46-1fd8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj55 2023-10-12T01:56:55.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj55 2023-10-12T01:56:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj55 2023-10-12T01:56:55.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.803 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.803 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.804 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.804 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.805 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj55 2023-10-12T01:56:55.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.806 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8= 2023-10-12T01:56:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ycU7rymL7zYjFg+ZHfpU43pgtd8=', 'Content-Length': '0'} 2023-10-12T01:56:55.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:30:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b790f1a83b99d0c-0065274c46-168d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.812 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.812 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.813 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.813 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.813 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.814 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097005e3ed657e1a4-0065274c46-1fd8-a2'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:55.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/?prefix=obj55 2023-10-12T01:56:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.819 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:55.819 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.820 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.820 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:46 GMT 2023-10-12T01:56:55.820 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.821 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ= 2023-10-12T01:56:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:46 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:M3zqrqNKFx+OXVAYwXSsTRAfVXQ=', 'Content-Length': '0'} 2023-10-12T01:56:55.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bda87f5e0c09a84-0065274c46-168d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'wafmoa-64obj551000falseobj552023-10-12T01:30:20.611Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-12T01:56:55.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=wafmoa-64 zones={a2, a1} 2023-10-12T01:56:55.822 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-10-12T01:56:55.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-10-12T01:56:55.823 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-10-12T01:30:46.809+0000 7f2e18867700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 178629) UID: 0 2023-10-12T01:56:55.823 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-10-12T01:30:46.809+0000 7f2e1cad07c0 -1 shutting down 2023-10-12T01:56:55.823 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:56:55.824 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-10-12T01:56:55.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:55.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:55.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:55.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 5 2023-10-12T01:56:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.832 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.833 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.833 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.834 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.834 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.835 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd33e2c1be0ba157-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj17 2023-10-12T01:56:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj17 2023-10-12T01:56:55.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj17 2023-10-12T01:56:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.840 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.841 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.841 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.842 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.843 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj17 2023-10-12T01:56:55.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.844 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:y+9gxoADktU1EOFmsLgoplKXBDo= 2023-10-12T01:56:55.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y+9gxoADktU1EOFmsLgoplKXBDo='} 2023-10-12T01:56:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f8b8bb5e6d4d634-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.849 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.850 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.850 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.850 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.851 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.851 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000984494561bbd4a96-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj34 2023-10-12T01:56:55.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj34 2023-10-12T01:56:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj34 2023-10-12T01:56:55.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.856 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.857 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.857 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.857 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.858 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj34 2023-10-12T01:56:55.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.859 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8VivdHw/npnYSuaxPHpWxA/PmRk= 2023-10-12T01:56:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8VivdHw/npnYSuaxPHpWxA/PmRk='} 2023-10-12T01:56:55.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c5cb222a1f5d30c-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.863 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.863 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.864 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.864 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.864 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.865 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093eadfde7275f515-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj51 2023-10-12T01:56:55.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj51 2023-10-12T01:56:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj51 2023-10-12T01:56:55.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.869 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.870 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.870 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.870 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.871 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj51 2023-10-12T01:56:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.871 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:RiTEWW98Durl/yBJfVDlRtEIH7k= 2023-10-12T01:56:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:RiTEWW98Durl/yBJfVDlRtEIH7k='} 2023-10-12T01:56:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085356dc858a167fd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.877 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.877 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.877 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.878 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.878 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1c1a27b78578cd-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj68 2023-10-12T01:56:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj68 2023-10-12T01:56:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj68 2023-10-12T01:56:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.882 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.883 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.883 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.883 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.884 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj68 2023-10-12T01:56:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.884 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:OXjWTh+n1faeeNGyQnzjOLde/Ws= 2023-10-12T01:56:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:OXjWTh+n1faeeNGyQnzjOLde/Ws='} 2023-10-12T01:56:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e272704c848f5606-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.889 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.889 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.890 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.890 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.890 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.891 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055653d3dfd9bcc1c-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj85 2023-10-12T01:56:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj85 2023-10-12T01:56:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj85 2023-10-12T01:56:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.896 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.896 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.896 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.897 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.897 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj85 2023-10-12T01:56:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.898 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:55vB6yN5bPn6CYwf+GOhInx5GZI= 2023-10-12T01:56:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:55vB6yN5bPn6CYwf+GOhInx5GZI='} 2023-10-12T01:56:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f81f7333d4bb99-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:55.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:55.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:55.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 10 2023-10-12T01:56:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.904 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.905 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.905 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.905 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.906 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.906 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000006f891ab3667341-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj19 2023-10-12T01:56:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj19 2023-10-12T01:56:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj19 2023-10-12T01:56:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.911 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.911 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.912 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.912 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.912 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj19 2023-10-12T01:56:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.913 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:j7KKiwxZwWs7guuzoePfECfbF1s= 2023-10-12T01:56:55.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:j7KKiwxZwWs7guuzoePfECfbF1s='} 2023-10-12T01:56:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000658133ee74edbecd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.917 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.918 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.918 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.919 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.919 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.920 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009964b636605cf823-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj38 2023-10-12T01:56:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj38 2023-10-12T01:56:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj38 2023-10-12T01:56:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.928 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.928 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.929 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.929 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.930 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj38 2023-10-12T01:56:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.931 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:y0fNnsQhdaN/56pBTFkreeQigy0= 2023-10-12T01:56:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:y0fNnsQhdaN/56pBTFkreeQigy0='} 2023-10-12T01:56:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6249e4d63c482e-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.935 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.935 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.935 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.936 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.936 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.937 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d8b1b3e59278d94-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj57 2023-10-12T01:56:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj57 2023-10-12T01:56:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj57 2023-10-12T01:56:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.941 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.941 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.941 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.942 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.942 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj57 2023-10-12T01:56:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.946 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ZeL5KijJWbL9eX7mROeZGT6bmRI= 2023-10-12T01:56:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ZeL5KijJWbL9eX7mROeZGT6bmRI='} 2023-10-12T01:56:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006540aa415bd229e3-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.950 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.951 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.951 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.951 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.952 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.952 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c445f9b58b0568c7-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj76 2023-10-12T01:56:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj76 2023-10-12T01:56:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj76 2023-10-12T01:56:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.956 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.957 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.957 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.957 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.958 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj76 2023-10-12T01:56:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.958 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:3znAqQfSX9UmV9vdQwaRpWtF6QQ= 2023-10-12T01:56:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3znAqQfSX9UmV9vdQwaRpWtF6QQ='} 2023-10-12T01:56:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f4f36e375fea6cd-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.963 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.963 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.963 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.964 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.964 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.964 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001ef78a619f6159e-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj95 2023-10-12T01:56:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj95 2023-10-12T01:56:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj95 2023-10-12T01:56:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.969 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.969 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.969 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.970 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.970 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj95 2023-10-12T01:56:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.971 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:yf12Uz5NugW4dr984cIm0Ro3njk= 2023-10-12T01:56:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:yf12Uz5NugW4dr984cIm0Ro3njk='} 2023-10-12T01:56:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f106f042b8f74416-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:55.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:55.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:55.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:55.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:55.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:55.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 15 2023-10-12T01:56:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.977 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.978 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.978 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.978 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.979 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.979 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002284e3f17b0f1f03-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj23 2023-10-12T01:56:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj23 2023-10-12T01:56:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj23 2023-10-12T01:56:55.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.983 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.984 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.984 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.984 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.985 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj23 2023-10-12T01:56:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.985 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:7PC279C39He0WlgtSmXb/8M2FgM= 2023-10-12T01:56:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:7PC279C39He0WlgtSmXb/8M2FgM='} 2023-10-12T01:56:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a571d47392a13ac7-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.990 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:55.990 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.990 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:55.990 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.991 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.991 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078662b189597c252-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj46 2023-10-12T01:56:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj46 2023-10-12T01:56:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj46 2023-10-12T01:56:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:55.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:55.996 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:55.996 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:55.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:55.997 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:55.997 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj46 2023-10-12T01:56:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:55.997 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:itUlQ96VUE/eezMgE72iGKlAm2M= 2023-10-12T01:56:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:itUlQ96VUE/eezMgE72iGKlAm2M='} 2023-10-12T01:56:55.998 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-10-12T01:56:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bd44ade2dafeb33-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.002 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.002 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.003 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.003 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.003 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.004 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e2d77ecbd83b495-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj69 2023-10-12T01:56:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj69 2023-10-12T01:56:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj69 2023-10-12T01:56:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.008 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.008 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.009 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.009 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.010 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj69 2023-10-12T01:56:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.010 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:6zwjVPJstomIW1B1nWI8vMtDAew= 2023-10-12T01:56:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:6zwjVPJstomIW1B1nWI8vMtDAew='} 2023-10-12T01:56:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002497179e64c3567f-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.015 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.015 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.016 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.016 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.016 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b61745888b51baec-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj92 2023-10-12T01:56:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj92 2023-10-12T01:56:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj92 2023-10-12T01:56:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.021 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.021 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.022 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.022 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj92 2023-10-12T01:56:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.023 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:KLwTrV0Ek4W29RfzsW8NTHf2/Vw= 2023-10-12T01:56:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:KLwTrV0Ek4W29RfzsW8NTHf2/Vw='} 2023-10-12T01:56:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001304fe6e0da42af4-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.027 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.027 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.027 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.028 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.028 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.029 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU= 2023-10-12T01:56:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:fRnEdkmS+WgCWkhDoo6tPXQ3yCU=', 'Content-Length': '0'} 2023-10-12T01:56:56.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c16a49ed4c90fc63-0065274c4d-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj115 2023-10-12T01:56:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj115 2023-10-12T01:56:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj115 2023-10-12T01:56:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.033 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.033 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.034 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.034 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:53 GMT 2023-10-12T01:56:56.034 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj115 2023-10-12T01:56:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.035 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:QI7aq3+GfGKJjx7XV97dTGh5UAE= 2023-10-12T01:56:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:53 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:QI7aq3+GfGKJjx7XV97dTGh5UAE='} 2023-10-12T01:56:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63c6f779a862dc0-0065274c4d-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:56.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:56.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 20 2023-10-12T01:56:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.042 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.042 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.042 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.043 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.043 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.043 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d252b3b11f6ffe-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj29 2023-10-12T01:56:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj29 2023-10-12T01:56:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj29 2023-10-12T01:56:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.048 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.048 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.048 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.049 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.049 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj29 2023-10-12T01:56:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.049 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:zK373+42CpGPQr/eytzAls5ELM4= 2023-10-12T01:56:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:zK373+42CpGPQr/eytzAls5ELM4='} 2023-10-12T01:56:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfe84279932b1b6d-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.054 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.054 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.054 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.055 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.055 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.056 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000900b9ebc159875a3-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj58 2023-10-12T01:56:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj58 2023-10-12T01:56:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj58 2023-10-12T01:56:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.060 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.060 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.060 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.061 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.061 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj58 2023-10-12T01:56:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.062 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:L4b5/NvjQJqIAvd6w7DywHXC7uQ= 2023-10-12T01:56:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:L4b5/NvjQJqIAvd6w7DywHXC7uQ='} 2023-10-12T01:56:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c14eced8b4a3eada-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.066 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.066 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.067 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.067 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.067 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.068 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021cbac25fdf7daf2-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj87 2023-10-12T01:56:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj87 2023-10-12T01:56:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj87 2023-10-12T01:56:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.072 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.072 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.073 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.073 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.073 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj87 2023-10-12T01:56:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.074 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:WAh6CmTZsUqo4I2QWx8Xu92XZgY= 2023-10-12T01:56:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WAh6CmTZsUqo4I2QWx8Xu92XZgY='} 2023-10-12T01:56:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a01ce57d91cc4033-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.078 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.078 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.079 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.079 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.079 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.080 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc10d140381c332-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj116 2023-10-12T01:56:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj116 2023-10-12T01:56:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj116 2023-10-12T01:56:56.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.084 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.084 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.085 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.085 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.085 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj116 2023-10-12T01:56:56.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.086 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:paK3Nb4KIax5TKlGfOg7lY5z7co= 2023-10-12T01:56:56.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:paK3Nb4KIax5TKlGfOg7lY5z7co='} 2023-10-12T01:56:56.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e58d52e3ac4cbb8-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.090 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.091 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.091 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.091 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.091 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.092 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000796e5bbb12d8f9da-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj145 2023-10-12T01:56:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj145 2023-10-12T01:56:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj145 2023-10-12T01:56:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.096 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.097 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.097 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.097 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.098 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj145 2023-10-12T01:56:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.098 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:FAjCk59qfuEi15RKsHE8BIUw1/M= 2023-10-12T01:56:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:FAjCk59qfuEi15RKsHE8BIUw1/M='} 2023-10-12T01:56:56.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d28a495c9f86f2af-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:56.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 25 2023-10-12T01:56:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.105 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.105 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.105 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.106 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.106 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.107 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ef247736c992347-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj31 2023-10-12T01:56:56.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj31 2023-10-12T01:56:56.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj31 2023-10-12T01:56:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.111 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.112 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.112 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.112 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.112 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj31 2023-10-12T01:56:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.113 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:3EcHhEgPjPz7HNAEk934d/M8ADU= 2023-10-12T01:56:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3EcHhEgPjPz7HNAEk934d/M8ADU='} 2023-10-12T01:56:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000674024992b469160-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.117 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.118 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.118 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.118 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.119 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.119 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058025e962717bbe0-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj62 2023-10-12T01:56:56.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj62 2023-10-12T01:56:56.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj62 2023-10-12T01:56:56.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.123 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.124 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.124 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.124 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.125 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj62 2023-10-12T01:56:56.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.125 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:3iWbGe82NWz8QfIR/4lEgENFD8E= 2023-10-12T01:56:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3iWbGe82NWz8QfIR/4lEgENFD8E='} 2023-10-12T01:56:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d084f0a64d962a01-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.129 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.130 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.130 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.130 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.131 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.131 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b5d1053e8440ec6-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj93 2023-10-12T01:56:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj93 2023-10-12T01:56:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj93 2023-10-12T01:56:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.136 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.136 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.136 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.137 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.137 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj93 2023-10-12T01:56:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.137 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VH4dBKt95Vhzs0qhHC1A/DHbBqY= 2023-10-12T01:56:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VH4dBKt95Vhzs0qhHC1A/DHbBqY='} 2023-10-12T01:56:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091de2933594fa48d-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.142 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.142 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.142 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.143 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.143 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.144 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c897b46f1d325354-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj124 2023-10-12T01:56:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj124 2023-10-12T01:56:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj124 2023-10-12T01:56:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.148 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.148 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.148 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.149 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.149 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj124 2023-10-12T01:56:56.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.150 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Fx1Iy6tadhlgC+d0wKv56s9G0ac= 2023-10-12T01:56:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Fx1Iy6tadhlgC+d0wKv56s9G0ac='} 2023-10-12T01:56:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095a8fe5f3d06cb29-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.154 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.154 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.155 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.155 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.155 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.156 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e681b5bd019c67f-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj155 2023-10-12T01:56:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj155 2023-10-12T01:56:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj155 2023-10-12T01:56:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.160 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.160 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.161 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.161 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.161 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj155 2023-10-12T01:56:56.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.162 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:tg5d0RJ5b4xF63VTQZgzCnybL6g= 2023-10-12T01:56:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:tg5d0RJ5b4xF63VTQZgzCnybL6g='} 2023-10-12T01:56:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046b960fd4e38cf45-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'wafmoa-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket wafmoa-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-10-12T01:56:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: wafmoa-64 2023-10-12T01:56:56.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 30 2023-10-12T01:56:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.168 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.169 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.169 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.169 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.170 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.170 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a39f25b0bc4ceb8a-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj37 2023-10-12T01:56:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj37 2023-10-12T01:56:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj37 2023-10-12T01:56:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.175 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.175 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.175 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.175 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.176 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj37 2023-10-12T01:56:56.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.176 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:rx217jRTLSvA+gn6REt7Xveno88= 2023-10-12T01:56:56.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:rx217jRTLSvA+gn6REt7Xveno88='} 2023-10-12T01:56:56.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e63993f1035cd4f3-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.181 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.181 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.181 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.181 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.182 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.182 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7e214a4f7edc436-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj74 2023-10-12T01:56:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj74 2023-10-12T01:56:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj74 2023-10-12T01:56:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.187 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.187 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.187 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.188 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.188 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj74 2023-10-12T01:56:56.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.188 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ymrqEYbc012h1z/+Yre9jMQNJWk= 2023-10-12T01:56:56.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ymrqEYbc012h1z/+Yre9jMQNJWk='} 2023-10-12T01:56:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f8f02964dab2b9e-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.193 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.194 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.194 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.194 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.195 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.195 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a8427edcab942b-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj111 2023-10-12T01:56:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj111 2023-10-12T01:56:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj111 2023-10-12T01:56:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.200 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.200 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.200 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.200 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.201 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj111 2023-10-12T01:56:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.201 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:S4y335bPAcSVe+wWrIQsyqbOsG0= 2023-10-12T01:56:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S4y335bPAcSVe+wWrIQsyqbOsG0='} 2023-10-12T01:56:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a962c2e5804a81c6-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.206 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.206 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.206 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.206 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.207 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.207 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1c0245834264cf3-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj148 2023-10-12T01:56:56.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj148 2023-10-12T01:56:56.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj148 2023-10-12T01:56:56.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.212 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.212 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.212 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.213 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.213 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj148 2023-10-12T01:56:56.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.213 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:qkt9NkTiJb/0TqOYbR1EQ+nOdDc= 2023-10-12T01:56:56.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:qkt9NkTiJb/0TqOYbR1EQ+nOdDc='} 2023-10-12T01:56:56.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049cd0b4ec20fc27c-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/ 2023-10-12T01:56:56.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/ 2023-10-12T01:56:56.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/ 2023-10-12T01:56:56.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.218 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.218 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.218 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.219 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.219 INFO:tasks.rgw_multisite_tests:/wafmoa-64/ 2023-10-12T01:56:56.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.220 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A= 2023-10-12T01:56:56.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mnn5HHwyjzj/YlPwBFAY92FEP2A=', 'Content-Length': '0'} 2023-10-12T01:56:56.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000830c18989217778b-0065274c4e-168d-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-64/obj185 2023-10-12T01:56:56.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-64/obj185 2023-10-12T01:56:56.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-64/obj185 2023-10-12T01:56:56.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-12T01:56:56.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.224 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.224 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-12T01:56:56.224 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.225 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:30:54 GMT 2023-10-12T01:56:56.225 INFO:tasks.rgw_multisite_tests:/wafmoa-64/obj185 2023-10-12T01:56:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.226 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:SGacSw9bzgkhlRb5uMkDvXhe6p4= 2023-10-12T01:56:56.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:30:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:SGacSw9bzgkhlRb5uMkDvXhe6p4='} 2023-10-12T01:56:56.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc4060a51c1412c0-0065274c4e-168d-a1'), ('Date', 'Thu, 12 Oct 2023 01:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:55.020+0000 7fc4490d2780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-12T01:56:56.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:55.021+0000 7fc4490d2780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-12T01:56:56.229 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-10-12T01:56:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-12T01:56:56.229 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-10-12T01:56:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:56:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-12T01:56:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-10-12T01:56:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-10-12T01:56:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > curl http://smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-12T01:56:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-12T01:56:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-12T01:56:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: anonymous 2023-10-12T01:56:56.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=wafmoa-64 2023-10-12T01:56:56.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:56.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wafmoa-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-12T01:56:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wafmoa-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-12T01:56:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:30:56.235+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:01.235+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:06.236+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:11.238+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:16.238+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:21.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:26.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:31.239+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:36.240+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:41.240+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:46.242+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:51.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:31:56.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 0.. 2023-10-12T01:56:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:01.243+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:06.244+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:11.245+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:16.245+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:21.246+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:26.248+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:31.248+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:36.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:41.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:46.249+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:51.250+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:32:56.250+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:01.252+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.242 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi112.stdout: INFO: 2023-10-12T01:33:05.282+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:56:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:06.253+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:11.253+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:16.254+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:21.254+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:26.255+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:31.256+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:36.256+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:41.258+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:46.258+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:51.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:33:56.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:01.259+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:06.260+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:11.261+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:16.261+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:21.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:26.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:31.264+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:36.263+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:41.264+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:46.265+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:51.265+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:34:56.267+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:01.267+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:06.268+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:11.268+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:16.269+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:21.269+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:26.270+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:31.271+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:36.272+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:41.273+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:46.273+0000 7efd02122780 1 waiting to reach latest gen 6, on 1.. 2023-10-12T01:56:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:51.273+0000 7efd02122780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-12T01:56:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:51.273+0000 7efd02122780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:56:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:35:51.273+0000 7efd02122780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-12T01:56:56.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-12T01:56:56.257 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-12T01:56:56.257 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.258 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-12T01:56:56.258 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2023-10-12T01:56:56.258 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T01:56:56.260 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-12T01:56:56.260 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-12T01:56:56.260 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-12T01:56:56.261 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2854, in test_sync_single_bucket_to_multiple 2023-10-12T01:56:56.261 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2023-10-12T01:56:56.261 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-10-12T01:56:56.262 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-10-12T01:56:56.262 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-12T01:56:56.262 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-12T01:56:56.263 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-12T01:56:56.263 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-12T01:56:56.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:56.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:56.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075553.345544_1039.1", 2023-10-12T01:56:56.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:33.345544Z" 2023-10-12T01:56:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:56.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:33.642+0000 7ff20bed6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:33.643+0000 7ff20bed6780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:56.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:56:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:56:56.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:56.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:33Z" 2023-10-12T01:56:56.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:56.293 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.294 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.294 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.294 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.295 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.295 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.295 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.296 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.296 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.297 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.298 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:56.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.299 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.299 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.300 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.300 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.300 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.303 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.303 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.303 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.303 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.304 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:56:56.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.305 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:56:56.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.306 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.306 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.307 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.307 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.308 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.310 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.310 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.310 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.311 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.311 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:56.312 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:56.312 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.312 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:33Z" 2023-10-12T01:56:56.313 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.313 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:56.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075553.345544_1039.1 target=1_1697075429.160408_1037.1 2023-10-12T01:56:56.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:38.766+0000 7f73b0122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:38.768+0000 7f73b0122780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:56:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:56:56.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:56.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:38Z" 2023-10-12T01:56:56.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:56.337 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.337 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.337 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.338 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.338 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.338 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.339 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.339 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.339 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.340 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.341 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:56.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.342 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.342 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.343 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.343 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.343 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.346 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.346 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.346 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.347 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.347 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.348 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075429.160408_1037.1", 2023-10-12T01:56:56.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:50:29.160408Z", 2023-10-12T01:56:56.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.349 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.350 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.350 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.350 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.351 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.353 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.353 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.354 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.354 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.354 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.355 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:56.355 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:56.355 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.355 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:38Z" 2023-10-12T01:56:56.356 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.356 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075429.160408_1037.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:50:29.160408Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075429.160408_1037.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075553.345544_1039.1 target=1_1697075429.160408_1037.1 2023-10-12T01:56:56.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:43.892+0000 7faea2411780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:43.893+0000 7faea2411780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075553.345544_1039.1", 2023-10-12T01:56:56.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:33.345544Z", 2023-10-12T01:56:56.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:56.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:43Z" 2023-10-12T01:56:56.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:56.380 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.380 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.380 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.381 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.381 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.382 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.382 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.382 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.382 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.383 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.384 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:56.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.385 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.385 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.386 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.386 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.386 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.388 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.389 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.389 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.389 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.390 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.390 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075553.345544_1039.1", 2023-10-12T01:56:56.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:33.345544Z", 2023-10-12T01:56:56.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.392 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.393 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.393 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.393 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.396 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.396 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.397 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.397 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.398 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:56.398 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:56.398 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.399 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:43Z" 2023-10-12T01:56:56.399 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.399 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075553.345544_1039.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:33.345544Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075553.345544_1039.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[]} 2023-10-12T01:56:56.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=wafmoa-74 2023-10-12T01:56:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:56.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:56:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.406 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.407 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.407 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.407 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:44 GMT 2023-10-12T01:56:56.408 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:56.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.408 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:9QsSXpup9CUoG1SMJeQ7jJz2KrI= 2023-10-12T01:56:56.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:9QsSXpup9CUoG1SMJeQ7jJz2KrI=', 'Content-Length': '0'} 2023-10-12T01:56:56.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005c2b72d4e05395b5-006527516c-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=wafmoa-75 2023-10-12T01:56:56.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/ 2023-10-12T01:56:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/ 2023-10-12T01:56:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/ 2023-10-12T01:56:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.413 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.413 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.413 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.414 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:44 GMT 2023-10-12T01:56:56.414 INFO:tasks.rgw_multisite_tests:/wafmoa-75/ 2023-10-12T01:56:56.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.415 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:S0dp9zxkBo7dNpAn8w0X3d8O1DY= 2023-10-12T01:56:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:44 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:S0dp9zxkBo7dNpAn8w0X3d8O1DY=', 'Content-Length': '0'} 2023-10-12T01:56:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000719e264dc552b955-006527516c-2afb-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:56.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:44.060671Z" 2023-10-12T01:56:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:56.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:56.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:44.249+0000 7f6e838ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:44.250+0000 7f6e838ef780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:56.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:56.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:56.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:56.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:44Z" 2023-10-12T01:56:56.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:56.446 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.446 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.446 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.446 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.447 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.447 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.448 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.448 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.448 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.450 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:56.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.451 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.451 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.452 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.455 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.455 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.455 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.456 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.456 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:56.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.458 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.459 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.459 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.459 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.460 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.462 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.462 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.463 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.463 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.463 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.463 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:56.464 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:56.464 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.464 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:44Z" 2023-10-12T01:56:56.465 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.465 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:56.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:56:56.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:56:56.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-74 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:56.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-wafmoa-75 --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:56.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=wafmoa-74', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=wafmoa-74 --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:56.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:56:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 22, 2023-10-12T01:56:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:56:56.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": [], 2023-10-12T01:56:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_map": { 2023-10-12T01:56:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zonegroups": [ 2023-10-12T01:56:56.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "a", 2023-10-12T01:56:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a", 2023-10-12T01:56:56.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-10-12T01:56:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": true, 2023-10-12T01:56:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:56:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-10-12T01:56:56.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-10-12T01:56:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a1", 2023-10-12T01:56:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:56:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:56.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:56.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:56.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:56:56.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a2", 2023-10-12T01:56:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:56:56.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:56.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:56.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:56.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-10-12T01:56:56.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-10-12T01:56:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-10-12T01:56:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-10-12T01:56:56.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-10-12T01:56:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-10-12T01:56:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:56.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-10-12T01:56:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [ 2023-10-12T01:56:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-group", 2023-10-12T01:56:56.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "data_flow": { 2023-10-12T01:56:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "symmetrical": [ 2023-10-12T01:56:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-flow1", 2023-10-12T01:56:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:56.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pipes": [ 2023-10-12T01:56:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe-wafmoa-74", 2023-10-12T01:56:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:56.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:56:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:56.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:56.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:56.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe-wafmoa-75", 2023-10-12T01:56:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:56.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:56:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-75", 2023-10-12T01:56:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:56.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:56.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:56.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:56.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:56.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:56.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:56.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:56.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe", 2023-10-12T01:56:56.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:56.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:56:56.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:56.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-75", 2023-10-12T01:56:56.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:56.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:56.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:56.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:56.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:56.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:56.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:56.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:56.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:56.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "enabled" 2023-10-12T01:56:56.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-10-12T01:56:56.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "b", 2023-10-12T01:56:56.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b", 2023-10-12T01:56:56.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-10-12T01:56:56.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": false, 2023-10-12T01:56:56.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:56:56.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-10-12T01:56:56.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-10-12T01:56:56.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:56.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:56.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:56:56.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b2", 2023-10-12T01:56:56.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:56:56.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:56.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:56.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:56.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:56.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:56.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:56.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:56.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:56.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:56.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:56.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:56.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b1", 2023-10-12T01:56:56.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:56.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:56:56.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:56.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:56.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:56.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:56.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:56.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:56.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:56.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:56.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:56.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:56.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-10-12T01:56:56.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-10-12T01:56:56.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-10-12T01:56:56.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-10-12T01:56:56.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-10-12T01:56:56.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-10-12T01:56:56.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:56.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-10-12T01:56:56.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [] 2023-10-12T01:56:56.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-10-12T01:56:56.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:56.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "short_zone_ids": [ 2023-10-12T01:56:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:56:56.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3962893999 2023-10-12T01:56:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3435037168 2023-10-12T01:56:56.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 2465750364 2023-10-12T01:56:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:56:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 1412291983 2023-10-12T01:56:56.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zonegroup": "a", 2023-10-12T01:56:56.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_config": { 2023-10-12T01:56:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_quota": { 2023-10-12T01:56:56.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-10-12T01:56:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-10-12T01:56:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-10-12T01:56:56.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-10-12T01:56:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-10-12T01:56:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_quota": { 2023-10-12T01:56:56.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-10-12T01:56:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-10-12T01:56:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-10-12T01:56:56.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-10-12T01:56:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-10-12T01:56:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_ratelimit": { 2023-10-12T01:56:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:56.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_ratelimit": { 2023-10-12T01:56:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:56.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "anonymous_ratelimit": { 2023-10-12T01:56:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:56.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:56.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_name": "test-realm", 2023-10-12T01:56:56.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:56:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.578 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.579 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.579 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.579 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:45 GMT 2023-10-12T01:56:56.579 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.580 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0= 2023-10-12T01:56:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0=', 'Content-Length': '0'} 2023-10-12T01:56:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000777211d26619a11c-006527516d-2ca6-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj1 2023-10-12T01:56:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj1 2023-10-12T01:56:56.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj1 2023-10-12T01:56:56.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.584 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.585 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-12T01:56:56.585 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.585 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:45 GMT 2023-10-12T01:56:56.586 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj1 2023-10-12T01:56:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.586 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:YVXUbTkAh8rBtdQbJBPs3n2ddZs= 2023-10-12T01:56:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:YVXUbTkAh8rBtdQbJBPs3n2ddZs='} 2023-10-12T01:56:56.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a03177079e3a3f0-006527516d-2ca6-a1'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:56.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.590 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:56.591 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.591 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.591 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:45 GMT 2023-10-12T01:56:56.592 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:56.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.592 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0= 2023-10-12T01:56:56.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:EL37iUhfrc/W+fCS0zjgKORJdJ0=', 'Content-Length': '0'} 2023-10-12T01:56:56.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007518ec431a1c54b8-006527516d-2ca6-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj2 2023-10-12T01:56:56.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj2 2023-10-12T01:56:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj2 2023-10-12T01:56:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:56.596 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:56.597 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-12T01:56:56.597 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:56.597 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:52:45 GMT 2023-10-12T01:56:56.598 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj2 2023-10-12T01:56:56.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:56.598 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:F1AiG+oY96278K6TuF6ZUUnkBpg= 2023-10-12T01:56:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:52:45 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:F1AiG+oY96278K6TuF6ZUUnkBpg='} 2023-10-12T01:56:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055bee7e779dbddf7-006527516d-2ca6-a1'), ('Date', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:56.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:56.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:56.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:56.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:44.060671Z" 2023-10-12T01:56:56.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:56.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:56.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:45.762+0000 7f7616cdf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:52:45.764+0000 7f7616cdf780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:56.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:56.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:56.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:56.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:56.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:56.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:45Z" 2023-10-12T01:56:56.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:56.630 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.630 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.630 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.630 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.631 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:56.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.631 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.632 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.632 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.632 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.633 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:56.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.634 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:56.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:56.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.635 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.635 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.636 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.636 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.637 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:56.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:56.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.642 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.642 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.643 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.644 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:56.646 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.647 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.647 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:56.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.649 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.649 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.650 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.650 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.651 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:56.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:56.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:56.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:56.655 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.656 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.656 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.657 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.658 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.658 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:56.659 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:56.659 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.660 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:45Z" 2023-10-12T01:56:56.660 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.661 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:56.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:56.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:56.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:56.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:56.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:56.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:56.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:56.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:46Z" 2023-10-12T01:56:56.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.707 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.707 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.707 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.708 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.708 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.708 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.709 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.709 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.710 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.711 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.711 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.712 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.712 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.712 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.713 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.713 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.714 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.715 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.715 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.715 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.716 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.716 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.717 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.718 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.718 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.719 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.719 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.719 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.720 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.720 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.721 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.722 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.722 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.722 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.723 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.723 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.724 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.724 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:46Z" 2023-10-12T01:56:56.724 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.724 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:51Z" 2023-10-12T01:56:56.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.746 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.746 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.747 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.747 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.747 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.747 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.748 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.750 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.751 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.751 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.751 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.752 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.752 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.752 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.753 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.753 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.754 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.754 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.755 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.755 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.755 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.756 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.756 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.757 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.758 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.758 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.758 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.759 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.759 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.761 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.761 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.761 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.762 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.762 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.762 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.763 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.763 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.763 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:51Z" 2023-10-12T01:56:56.764 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.764 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:52:56Z" 2023-10-12T01:56:56.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.785 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.786 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.786 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.786 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.787 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.787 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.787 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.787 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.788 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.788 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.788 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.789 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.789 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.790 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.791 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.791 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.791 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.791 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.792 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.793 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.794 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.794 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.794 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.795 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.795 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.795 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.797 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.797 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.798 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.798 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.798 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.799 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.800 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.801 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.801 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.801 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.802 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.802 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.802 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.803 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:52:56Z" 2023-10-12T01:56:56.803 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.803 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:01Z" 2023-10-12T01:56:56.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.825 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.825 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.826 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.826 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.826 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.827 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.827 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.827 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.827 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.828 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.828 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.829 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.830 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.830 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.830 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.832 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.833 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.833 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.834 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.835 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.835 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.837 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.837 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.837 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.837 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.838 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.838 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.838 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.840 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.840 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.840 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.841 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.841 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.841 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.842 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.842 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.842 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:01Z" 2023-10-12T01:56:56.843 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.843 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:06Z" 2023-10-12T01:56:56.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.864 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.865 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.865 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.865 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.866 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.866 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.866 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.867 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.867 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.867 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.868 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.868 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.869 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.870 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.870 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.870 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.870 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.872 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.873 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.873 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.873 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.873 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.874 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.874 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.876 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.876 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.876 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.877 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.877 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.877 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.878 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.879 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.880 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.880 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.880 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.880 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.881 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.881 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.881 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.882 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:06Z" 2023-10-12T01:56:56.882 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.882 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:11Z" 2023-10-12T01:56:56.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.905 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.905 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.905 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.906 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.906 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.907 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.907 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.908 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.909 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.909 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.909 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.910 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.911 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.911 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.912 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.912 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.913 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.913 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.914 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.914 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.915 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.916 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.916 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.916 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.917 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.917 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.919 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.919 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.919 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.920 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.920 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.920 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.921 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.921 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:11Z" 2023-10-12T01:56:56.921 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.922 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:16Z" 2023-10-12T01:56:56.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.943 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.944 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.944 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.945 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.945 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.945 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.945 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.946 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.946 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.946 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.947 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.947 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.948 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.948 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.949 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.949 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.950 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.950 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.951 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.951 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.952 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.952 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.952 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.953 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.953 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.954 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.955 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.955 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.955 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.956 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.956 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.956 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.957 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.958 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.958 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.959 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:56.959 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:56.960 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:56.960 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:56.960 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.961 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:16Z" 2023-10-12T01:56:56.961 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:56.961 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:56.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:56.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:56.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:56.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:56.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:56.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:56.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:56.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:56.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:56.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:56.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:21Z" 2023-10-12T01:56:56.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:56.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:56.985 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:56.985 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:56.985 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:56.986 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:56.986 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:56.986 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.987 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:56.987 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.987 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:56.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.988 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.988 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:56.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.989 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:56.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:56.990 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.990 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.990 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.991 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:56.991 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.991 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.992 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:56.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.992 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:56.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:56.993 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.993 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.994 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.994 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:56.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.995 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.995 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:56.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:56.996 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:56.997 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:56.997 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:56.997 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:56.998 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:56.998 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:56.998 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:56.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:56.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:56.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:56.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.000 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.000 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.000 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.001 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.001 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.001 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.002 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.002 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.002 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:21Z" 2023-10-12T01:56:57.002 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.003 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:27Z" 2023-10-12T01:56:57.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.024 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.025 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.025 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.025 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.026 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.026 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.026 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.026 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.027 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.027 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.028 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.028 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.029 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.029 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.030 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.030 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.030 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.031 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.031 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.032 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.033 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.033 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.033 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.033 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.034 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.034 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.034 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.036 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.036 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.036 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.037 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.037 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.038 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.039 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.039 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.040 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.040 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.040 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.041 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.041 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.041 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.042 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:27Z" 2023-10-12T01:56:57.042 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.042 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:32Z" 2023-10-12T01:56:57.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.064 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.064 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.064 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.065 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.065 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.065 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.066 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.066 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.066 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.067 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.067 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.068 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.069 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.069 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.069 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.070 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.070 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.070 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.070 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.071 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.072 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.072 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.073 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.073 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.074 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.074 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.075 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.076 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.076 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.076 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.077 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.077 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.077 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.078 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.078 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.079 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.079 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.079 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.080 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.080 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.080 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.080 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.081 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.081 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:32Z" 2023-10-12T01:56:57.081 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.082 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:37Z" 2023-10-12T01:56:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.103 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.104 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.104 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.104 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.104 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.105 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.105 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.105 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.106 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.106 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.107 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.107 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.108 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.108 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.109 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.109 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.110 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.110 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.110 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.111 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.112 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.112 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.112 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.113 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.113 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.113 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.114 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.115 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.115 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.115 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.116 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.116 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.116 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.116 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.118 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.118 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.119 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.119 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.119 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.120 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.120 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.120 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.120 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:37Z" 2023-10-12T01:56:57.121 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.121 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:42Z" 2023-10-12T01:56:57.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.143 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.144 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.144 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.144 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.145 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.145 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.146 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.146 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.147 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.147 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.148 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.148 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.148 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.149 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.149 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.149 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.150 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.151 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.151 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.151 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.152 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.152 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.153 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.154 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.154 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.155 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.155 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.155 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.156 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.156 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.157 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.157 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.158 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.158 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.158 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.159 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.159 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.159 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.160 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.160 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:42Z" 2023-10-12T01:56:57.160 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.161 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:47Z" 2023-10-12T01:56:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.185 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.185 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.186 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.186 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.187 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.187 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.187 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.188 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.188 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.188 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.189 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.189 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.190 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.191 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.191 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.192 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.192 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.193 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.193 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.193 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.194 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.195 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.195 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.195 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.196 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.196 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.196 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.198 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.199 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.199 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.199 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.200 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.200 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.202 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.202 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.202 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.203 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.203 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.203 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.204 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.204 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.204 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:47Z" 2023-10-12T01:56:57.205 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.205 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:52Z" 2023-10-12T01:56:57.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.230 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.230 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.230 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.231 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.231 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.231 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.232 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.232 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.232 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.233 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.233 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.234 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.235 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.235 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.235 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.236 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.236 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.236 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.237 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.238 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.239 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.239 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.239 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.240 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.240 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.240 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.241 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.241 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.242 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.243 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.243 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.243 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.244 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.244 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.245 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.246 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.247 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.247 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.247 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.248 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.248 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.248 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.249 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.249 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:52Z" 2023-10-12T01:56:57.249 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.250 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:53:57Z" 2023-10-12T01:56:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.272 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.272 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.272 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.273 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.273 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.273 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.274 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.274 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.274 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.275 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.275 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.275 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.276 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.277 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.277 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.278 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.278 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.278 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.279 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.279 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.280 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.281 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.281 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.281 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.282 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.282 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.283 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.284 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.285 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.285 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.285 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.286 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.286 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.286 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.288 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.288 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.289 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.289 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.290 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.290 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.290 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.291 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:53:57Z" 2023-10-12T01:56:57.291 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.291 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:02Z" 2023-10-12T01:56:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.315 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.315 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.315 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.316 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.316 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.316 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.317 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.317 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.317 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.318 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.318 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.319 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.320 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.320 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.321 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.321 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.321 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.322 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.322 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.323 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.324 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.324 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.324 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.325 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.325 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.326 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.327 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.328 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.328 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.328 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.329 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.329 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.329 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.331 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.331 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.332 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.332 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.332 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.333 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.333 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.333 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.333 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:02Z" 2023-10-12T01:56:57.334 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.334 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:08Z" 2023-10-12T01:56:57.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.357 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.357 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.357 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.358 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.358 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.358 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.359 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.359 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.359 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.360 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.360 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.361 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.362 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.362 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.362 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.363 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.363 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.363 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.364 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.364 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.365 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.365 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.366 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.366 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.367 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.367 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.369 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.369 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.370 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.370 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.370 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.371 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.371 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.375 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.376 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.376 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.377 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.377 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.378 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.378 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.378 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.379 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.379 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:08Z" 2023-10-12T01:56:57.379 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.379 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:13Z" 2023-10-12T01:56:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.403 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.403 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.404 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.405 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.405 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.405 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.405 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.406 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.407 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.408 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.408 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.409 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.409 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.409 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.409 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.410 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.410 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.411 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.412 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.412 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.412 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.413 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.413 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.413 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.414 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.414 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.415 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.415 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.416 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.416 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.416 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.417 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.417 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.419 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.419 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.419 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.420 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.420 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.420 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.421 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.421 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.421 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:13Z" 2023-10-12T01:56:57.422 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.422 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:18Z" 2023-10-12T01:56:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.446 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.446 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.446 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.447 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.447 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.447 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.448 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.448 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.448 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.449 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.449 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.450 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.451 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.451 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.452 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.453 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.453 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.454 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.455 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.455 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.455 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.456 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.456 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.456 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.457 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.459 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.459 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.459 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.460 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.460 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.460 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.461 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.462 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.463 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.463 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.463 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.464 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.464 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.464 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.465 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.465 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:18Z" 2023-10-12T01:56:57.465 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.466 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:23Z" 2023-10-12T01:56:57.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.490 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.490 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.490 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.491 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.491 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.491 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.492 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.492 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.492 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.493 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.493 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.493 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.494 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.495 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.495 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.496 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.496 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.497 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.498 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.499 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.499 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.499 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.500 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.500 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.502 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.502 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.502 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.503 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.503 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.503 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.503 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.504 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.505 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.506 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.506 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.507 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.507 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.508 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.508 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.508 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:23Z" 2023-10-12T01:56:57.508 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.509 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:28Z" 2023-10-12T01:56:57.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.532 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.532 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.532 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.533 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.533 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.534 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.535 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.535 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.536 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.537 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.537 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.538 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.538 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.538 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.539 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.539 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.540 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.540 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.541 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.541 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.541 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.542 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.543 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.544 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.544 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.545 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.545 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.545 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.545 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.548 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.548 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.548 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.548 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.549 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.549 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.549 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.550 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.550 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:28Z" 2023-10-12T01:56:57.550 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.551 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:33Z" 2023-10-12T01:56:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.574 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.574 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.574 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.575 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.575 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.575 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.576 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.576 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.576 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.577 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.577 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.577 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.578 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.579 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.580 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.582 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.582 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.582 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.583 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.583 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.584 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.585 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.585 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.585 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.586 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.586 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.586 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.587 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.588 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.588 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.589 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.589 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.590 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.590 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.590 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.592 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.592 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.593 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.593 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.593 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.594 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.594 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.594 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.595 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:33Z" 2023-10-12T01:56:57.595 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.595 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:38Z" 2023-10-12T01:56:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.620 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.621 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.621 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.621 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.622 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.622 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.622 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.623 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.623 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.624 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.624 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.625 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.625 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.626 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.626 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.627 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.628 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.628 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.628 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.629 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.629 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.629 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.630 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.631 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.632 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.632 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.632 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.633 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.633 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.633 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.635 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.635 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.636 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.636 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.637 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.637 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.637 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.637 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:38Z" 2023-10-12T01:56:57.638 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.638 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:43Z" 2023-10-12T01:56:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.661 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.662 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.662 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.662 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.663 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.663 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.663 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.664 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.664 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.664 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.664 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.665 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.665 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.666 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.667 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.667 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.668 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.668 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.668 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052373", 2023-10-12T01:56:57.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.669 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.319820Z" 2023-10-12T01:56:57.670 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.670 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.671 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.671 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.672 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.672 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.673 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.674 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.674 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.674 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.675 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.675 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.676 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046360", 2023-10-12T01:56:57.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.676 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.298028Z" 2023-10-12T01:56:57.677 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.677 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.678 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.678 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.679 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.679 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.679 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.679 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.680 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:43Z" 2023-10-12T01:56:57.680 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.680 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052373', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.319820Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046360', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.298028Z'}}] 2023-10-12T01:56:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052373', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046360'} 2023-10-12T01:56:57.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052373, shard=3 master=00000000000000000000:00000000000000046501 target=00000000000000000000:00000000000000046360 2023-10-12T01:56:57.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:49Z" 2023-10-12T01:56:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:57.704 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.704 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.704 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.705 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.705 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:57.705 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.706 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.706 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.706 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.707 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.707 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:57.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.708 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:57.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:57.709 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.709 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.709 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.710 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.710 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.710 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:57.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.711 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:57.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:57.712 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.712 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.713 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.713 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.713 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.714 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:57.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:57.715 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.716 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.716 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.716 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.717 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:57.717 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:57.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:57.719 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.719 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.719 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.720 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.720 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.720 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:57.721 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:57.721 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.721 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:49Z" 2023-10-12T01:56:57.722 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.722 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:57.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:57.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:57.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:57.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:57.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:56:57.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.728 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.728 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.728 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.729 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.729 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:57.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.730 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:vMW3oHiq8TTQ+WnWjRNb5UFkQtc= 2023-10-12T01:56:57.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:vMW3oHiq8TTQ+WnWjRNb5UFkQtc=', 'Content-Length': '0'} 2023-10-12T01:56:57.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008014fdd5c3f7ef22-00652751e9-2cac-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj1 2023-10-12T01:56:57.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj1 2023-10-12T01:56:57.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj1 2023-10-12T01:56:57.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.734 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.735 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.735 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.735 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.736 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj1 2023-10-12T01:56:57.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.736 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:29w0Ng6QI4cEFr2CPLgBozgL2m0= 2023-10-12T01:56:57.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:29w0Ng6QI4cEFr2CPLgBozgL2m0=', 'Content-Length': '0'} 2023-10-12T01:56:57.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5a49af7bd17c4e-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj1 2023-10-12T01:56:57.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj1 2023-10-12T01:56:57.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:57.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj1 2023-10-12T01:56:57.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.741 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:57.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.741 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.741 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.742 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj1 2023-10-12T01:56:57.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.743 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:8EkpEbKizObKnJM/4T4lSxDOanE= 2023-10-12T01:56:57.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:8EkpEbKizObKnJM/4T4lSxDOanE=', 'Content-Length': '0'} 2023-10-12T01:56:57.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4fdc4238cbb9330-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj2 2023-10-12T01:56:57.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj2 2023-10-12T01:56:57.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj2 2023-10-12T01:56:57.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.747 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.748 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.748 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.748 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.749 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj2 2023-10-12T01:56:57.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.749 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:0K2MvtnSswDK1yLm15Jc4nqc1Ms= 2023-10-12T01:56:57.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:0K2MvtnSswDK1yLm15Jc4nqc1Ms=', 'Content-Length': '0'} 2023-10-12T01:56:57.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e44731fa6b2b1c8d-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj2 2023-10-12T01:56:57.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj2 2023-10-12T01:56:57.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:57.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj2 2023-10-12T01:56:57.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.754 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:57.754 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.754 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.755 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.755 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj2 2023-10-12T01:56:57.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.756 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:JDmACmILcnb1oVVL4ELl8DjCNUs= 2023-10-12T01:56:57.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:JDmACmILcnb1oVVL4ELl8DjCNUs=', 'Content-Length': '0'} 2023-10-12T01:56:57.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f54a1e31d8d8cb-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/ 2023-10-12T01:56:57.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/ 2023-10-12T01:56:57.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/ 2023-10-12T01:56:57.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.760 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.760 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.761 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.761 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.761 INFO:tasks.rgw_multisite_tests:/wafmoa-75/ 2023-10-12T01:56:57.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.762 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:3j3FOPItytQnMkwssCgRR9Aj7Fk= 2023-10-12T01:56:57.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3j3FOPItytQnMkwssCgRR9Aj7Fk=', 'Content-Length': '0'} 2023-10-12T01:56:57.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bbee016d1da4269-00652751e9-2cac-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/obj1 2023-10-12T01:56:57.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/obj1 2023-10-12T01:56:57.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/obj1 2023-10-12T01:56:57.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.766 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.767 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.767 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.767 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.768 INFO:tasks.rgw_multisite_tests:/wafmoa-75/obj1 2023-10-12T01:56:57.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.768 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:3LyohQCiiXPzXk62mSSy7hMPdxk= 2023-10-12T01:56:57.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:3LyohQCiiXPzXk62mSSy7hMPdxk=', 'Content-Length': '0'} 2023-10-12T01:56:57.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f9b628f86752ab-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/obj1 2023-10-12T01:56:57.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/obj1 2023-10-12T01:56:57.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:57.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/obj1 2023-10-12T01:56:57.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.773 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:57.773 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.773 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.774 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.774 INFO:tasks.rgw_multisite_tests:/wafmoa-75/obj1 2023-10-12T01:56:57.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.775 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:VU5lPwadi66/1DlV5jiZhurgRv4= 2023-10-12T01:56:57.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:VU5lPwadi66/1DlV5jiZhurgRv4=', 'Content-Length': '0'} 2023-10-12T01:56:57.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db06a495b627c683-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/obj2 2023-10-12T01:56:57.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/obj2 2023-10-12T01:56:57.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:57.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/obj2 2023-10-12T01:56:57.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.780 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:57.780 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.780 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.780 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.781 INFO:tasks.rgw_multisite_tests:/wafmoa-75/obj2 2023-10-12T01:56:57.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.781 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:WCLxnwFhlhAcbxBB611Wv5IN2io= 2023-10-12T01:56:57.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:WCLxnwFhlhAcbxBB611Wv5IN2io=', 'Content-Length': '0'} 2023-10-12T01:56:57.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d64ee582d243cd2e-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-75/obj2 2023-10-12T01:56:57.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-75/obj2 2023-10-12T01:56:57.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:57.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-75/obj2 2023-10-12T01:56:57.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:57.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:57.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:57.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:57.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:57.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:57.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:57.786 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:57.786 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.787 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.787 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:54:49 GMT 2023-10-12T01:56:57.787 INFO:tasks.rgw_multisite_tests:/wafmoa-75/obj2 2023-10-12T01:56:57.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:57.788 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:iseyMEg1ZjYO+0WppQ+lyndDzU4= 2023-10-12T01:56:57.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:54:49 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:iseyMEg1ZjYO+0WppQ+lyndDzU4=', 'Content-Length': '0'} 2023-10-12T01:56:57.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:52:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002a11054adea3a45-00652751e9-2cac-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:57.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:56:57.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: WARNING: cannot find source zone id for name=* 2023-10-12T01:56:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: WARNING: cannot find source zone id for name=* 2023-10-12T01:56:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-12T01:56:57.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:57.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 23, 2023-10-12T01:56:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "predecessor_uuid": "26ed623f-4dfb-47f6-b4f4-62e562c80d92", 2023-10-12T01:56:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": [], 2023-10-12T01:56:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_map": { 2023-10-12T01:56:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zonegroups": [ 2023-10-12T01:56:57.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "a", 2023-10-12T01:56:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a", 2023-10-12T01:56:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-10-12T01:56:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": true, 2023-10-12T01:56:57.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:56:57.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-10-12T01:56:57.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-10-12T01:56:57.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:57.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:57.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a1", 2023-10-12T01:56:57.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-10-12T01:56:57.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:57.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:57.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:57.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:57.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:57.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:57.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:57.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:57.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:57.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:57.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:56:57.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a2", 2023-10-12T01:56:57.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-10-12T01:56:57.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:57.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:57.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:57.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:57.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:57.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:57.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:57.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:57.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:57.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:57.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-10-12T01:56:57.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-10-12T01:56:57.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-10-12T01:56:57.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-10-12T01:56:57.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-10-12T01:56:57.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-10-12T01:56:57.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:57.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-10-12T01:56:57.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [ 2023-10-12T01:56:57.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-group", 2023-10-12T01:56:57.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "data_flow": { 2023-10-12T01:56:57.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "symmetrical": [ 2023-10-12T01:56:57.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-flow1", 2023-10-12T01:56:57.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:57.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:57.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pipes": [ 2023-10-12T01:56:57.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe-wafmoa-74", 2023-10-12T01:56:57.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:57.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:56:57.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe-wafmoa-75", 2023-10-12T01:56:57.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-74", 2023-10-12T01:56:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "677f5a1d-3ab4-4b5f-af15-9890c2c79648" 2023-10-12T01:56:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "wafmoa-75", 2023-10-12T01:56:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "c93c7bc5-2183-452b-b2b7-6a812a26f746" 2023-10-12T01:56:57.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:57.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:57.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:57.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:57.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:57.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe", 2023-10-12T01:56:57.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "*", 2023-10-12T01:56:57.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "*" 2023-10-12T01:56:57.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-10-12T01:56:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "*", 2023-10-12T01:56:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "*" 2023-10-12T01:56:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-10-12T01:56:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-10-12T01:56:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-10-12T01:56:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-10-12T01:56:57.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-10-12T01:56:57.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-10-12T01:56:57.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-10-12T01:56:57.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-10-12T01:56:57.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "allowed" 2023-10-12T01:56:57.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-10-12T01:56:57.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "b", 2023-10-12T01:56:57.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b", 2023-10-12T01:56:57.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-10-12T01:56:57.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": false, 2023-10-12T01:56:57.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:56:57.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-10-12T01:56:57.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-10-12T01:56:57.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:57.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-10-12T01:56:57.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:56:57.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b2", 2023-10-12T01:56:57.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8001" 2023-10-12T01:56:57.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:57.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:57.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:57.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:57.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:57.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:57.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:57.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:57.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:57.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:57.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:57.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b1", 2023-10-12T01:56:57.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-10-12T01:56:57.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi118.front.sepia.ceph.com:8000" 2023-10-12T01:56:57.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-10-12T01:56:57.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-10-12T01:56:57.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-12T01:56:57.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-10-12T01:56:57.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-10-12T01:56:57.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-10-12T01:56:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-10-12T01:56:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-10-12T01:56:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-10-12T01:56:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-10-12T01:56:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-10-12T01:56:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-10-12T01:56:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-10-12T01:56:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-10-12T01:56:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-10-12T01:56:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-10-12T01:56:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-10-12T01:56:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [] 2023-10-12T01:56:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-10-12T01:56:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-10-12T01:56:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-10-12T01:56:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "short_zone_ids": [ 2023-10-12T01:56:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "3eb85de3-e2b8-4ea2-8138-f2fe8dd1ded6", 2023-10-12T01:56:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3962893999 2023-10-12T01:56:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3435037168 2023-10-12T01:56:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "759fe5ff-3772-4580-8ac2-3b7d6fa2c3a5", 2023-10-12T01:56:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 2465750364 2023-10-12T01:56:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "c93c7bc5-2183-452b-b2b7-6a812a26f746", 2023-10-12T01:56:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 1412291983 2023-10-12T01:56:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zonegroup": "a", 2023-10-12T01:56:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "677f5a1d-3ab4-4b5f-af15-9890c2c79648", 2023-10-12T01:56:57.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_config": { 2023-10-12T01:56:57.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_quota": { 2023-10-12T01:56:57.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-10-12T01:56:57.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-10-12T01:56:57.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-10-12T01:56:57.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-10-12T01:56:57.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-10-12T01:56:57.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_quota": { 2023-10-12T01:56:57.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-10-12T01:56:57.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-10-12T01:56:57.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-10-12T01:56:57.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-10-12T01:56:57.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-10-12T01:56:57.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_ratelimit": { 2023-10-12T01:56:57.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:57.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:57.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:57.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:57.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:57.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_ratelimit": { 2023-10-12T01:56:57.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:57.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:57.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:57.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:57.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:57.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "anonymous_ratelimit": { 2023-10-12T01:56:57.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-10-12T01:56:57.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-10-12T01:56:57.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-10-12T01:56:57.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-10-12T01:56:57.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-10-12T01:56:57.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "a9c32acc-d391-49fc-954e-a55395784dbf", 2023-10-12T01:56:57.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_name": "test-realm", 2023-10-12T01:56:57.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:57.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-12T01:56:57.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.904 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi112.stdout: INFO: 2023-10-12T01:54:50.100+0000 7f58c46f0700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-12T01:56:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-12T01:56:57.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-74 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-74 '--source-bucket=*' --dest-bucket=wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:56:57.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-wafmoa-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=wafmoa-74', '--source-bucket=*', '--dest-bucket=wafmoa-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-wafmoa-75 --source-zones=a1 --dest-zones=a2 --bucket=wafmoa-74 '--source-bucket=*' --dest-bucket=wafmoa-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-12T01:56:57.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'wafmoa-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket wafmoa-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-12T01:56:57.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-12T01:56:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-12T01:56:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-wafmoa-74","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-wafmoa-75","source":{"bucket":"wafmoa-74","zones":["a1"]},"dest":{"bucket":"wafmoa-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-12T01:56:57.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:57.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:54:50.713117Z" 2023-10-12T01:56:57.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:57.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:57.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:57.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:57.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:54:50.994+0000 7f653fadc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:57.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:54:50.995+0000 7f653fadc780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:57.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:57.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:57.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:57.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:57.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:57.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:50Z" 2023-10-12T01:56:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:57.946 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.947 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.947 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.948 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.948 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.949 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.949 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:57.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:57.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.951 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:57.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:57.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.952 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.953 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.953 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.953 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:57.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:57.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:57.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:57.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.956 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.956 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.957 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:57.957 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:57.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:57.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:57.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.960 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.960 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.960 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:57.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:57.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:57.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:57.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:57.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:57.963 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:57.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.964 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.964 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:57.964 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:57.964 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.965 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:57.965 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:57.965 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:57.966 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.966 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:50Z" 2023-10-12T01:56:57.966 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:57.967 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:57.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:57.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:57.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:57.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:57.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:57.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:57.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:54:56.117+0000 7f6bab12d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:54:56.118+0000 7f6bab12d780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:57.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:57.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:57.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:57.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:57.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:57.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:57.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:57.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:57.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:57.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:57.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:57.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:57.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:54:56Z" 2023-10-12T01:56:57.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:57.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:57.996 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:57.996 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:57.996 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:57.997 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:57.997 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:57.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:57.998 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:57.998 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:57.998 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:57.999 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:57.999 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:57.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:57.999 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.001 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.001 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.002 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.002 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.003 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.005 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.005 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.006 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.006 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.006 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.009 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.009 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.010 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.010 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.013 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.013 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.013 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.014 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.014 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.014 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.015 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.015 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.015 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:54:56Z" 2023-10-12T01:56:58.015 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.016 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.018 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:01.251+0000 7f6fed622780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:01.252+0000 7f6fed622780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:01Z" 2023-10-12T01:56:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.040 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.040 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.041 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.041 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.041 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.042 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.042 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.042 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.043 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.043 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.044 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.045 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.046 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.046 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.046 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.047 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.049 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.049 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.050 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.050 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.053 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.053 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.053 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.054 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.054 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.055 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.056 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.056 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.057 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.057 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.057 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.058 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.058 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.058 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.059 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.059 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:01Z" 2023-10-12T01:56:58.059 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.060 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:06.377+0000 7ff9fe8a1780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:06.378+0000 7ff9fe8a1780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:06Z" 2023-10-12T01:56:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.085 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.085 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.086 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.086 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.087 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.087 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.088 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.088 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.088 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.089 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.089 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.090 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.091 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.091 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.092 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.092 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.092 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.095 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.095 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.096 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.096 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.096 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.099 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.099 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.100 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.100 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.100 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.103 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.103 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.104 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.104 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.104 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.104 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.105 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.105 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.105 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:06Z" 2023-10-12T01:56:58.106 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.106 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:11.504+0000 7f8bbc9e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:11.505+0000 7f8bbc9e3780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:11Z" 2023-10-12T01:56:58.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.130 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.131 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.131 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.131 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.132 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.132 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.133 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.133 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.133 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.133 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.135 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.136 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.136 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.137 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.137 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.137 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.138 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.140 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.140 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.141 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.141 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.141 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.142 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.142 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.142 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.143 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.144 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.144 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.144 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.145 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.147 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.148 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.148 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.149 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.149 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.149 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.150 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.150 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.150 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:11Z" 2023-10-12T01:56:58.151 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.151 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:16.630+0000 7fae97d23780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:16.631+0000 7fae97d23780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:16Z" 2023-10-12T01:56:58.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.175 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.175 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.176 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.176 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.176 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.177 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.177 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.177 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.178 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.178 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.181 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.181 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.181 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.182 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.184 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.185 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.185 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.188 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.189 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.189 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.189 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.190 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.195 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.196 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.196 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.196 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.197 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.197 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.198 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.198 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.198 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:16Z" 2023-10-12T01:56:58.199 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.199 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:21.754+0000 7fe7ae210780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:21.755+0000 7fe7ae210780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:21Z" 2023-10-12T01:56:58.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.224 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.224 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.225 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.225 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.226 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.226 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.226 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.227 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.228 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.229 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.230 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.230 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.230 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.231 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.231 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.233 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.233 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.234 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.234 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.234 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.237 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.237 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.237 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.238 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.241 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.241 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.241 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.242 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.242 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.242 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.243 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.243 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.243 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:21Z" 2023-10-12T01:56:58.243 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.244 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:26.876+0000 7f555b2f2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:26.877+0000 7f555b2f2780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:26Z" 2023-10-12T01:56:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.268 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.268 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.269 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.269 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.270 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.270 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.271 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.271 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.271 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.272 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.274 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.274 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.274 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.275 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.275 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.277 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.278 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.278 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.278 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.279 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.281 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.282 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.282 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.282 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.285 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.285 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.285 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.286 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.286 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.286 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.287 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.287 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:26Z" 2023-10-12T01:56:58.288 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.288 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:31.999+0000 7fe5b5304780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:32.000+0000 7fe5b5304780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:32Z" 2023-10-12T01:56:58.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.313 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.314 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.314 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.314 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.315 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.315 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.315 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.316 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.316 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.318 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.319 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.319 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.319 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.320 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.322 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.323 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.323 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.323 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.324 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.325 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.326 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.326 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.327 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.327 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.327 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.328 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.330 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.330 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.330 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.331 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.331 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.331 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.332 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.332 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.332 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:32Z" 2023-10-12T01:56:58.333 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.333 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:37.122+0000 7f7a52ab0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:37.123+0000 7f7a52ab0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:37Z" 2023-10-12T01:56:58.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.358 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.358 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.358 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.359 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.359 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.360 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.360 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.360 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.361 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.361 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.362 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.363 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.363 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.363 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.364 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.364 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.364 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.365 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.365 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.368 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.368 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.368 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.369 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.372 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.372 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.372 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.373 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.373 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.374 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.376 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.376 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.376 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.377 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.377 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.377 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.378 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.378 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.378 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:37Z" 2023-10-12T01:56:58.379 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.379 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:42.248+0000 7ff0f9f53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:42.249+0000 7ff0f9f53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:42Z" 2023-10-12T01:56:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.417 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.417 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.417 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.418 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.418 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.419 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.420 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.420 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.420 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.422 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.424 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.424 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.425 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.425 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.426 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.429 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.429 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.429 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.430 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.430 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.433 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.433 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.433 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.434 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.434 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.434 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.437 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.437 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.438 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.438 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.439 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.439 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.439 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.440 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.440 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:42Z" 2023-10-12T01:56:58.441 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.441 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.444 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:47.374+0000 7f142f118780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:47.375+0000 7f142f118780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:47Z" 2023-10-12T01:56:58.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.470 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.471 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.471 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.471 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.471 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.472 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.472 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.473 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.473 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.473 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.475 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.476 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.476 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.476 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.477 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.477 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.478 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.479 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.480 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.480 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.480 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.481 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.481 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.481 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.482 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.483 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.484 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.484 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.485 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.487 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.487 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.488 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.488 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.488 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.489 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.489 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.489 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.490 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:47Z" 2023-10-12T01:56:58.490 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.490 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:52.499+0000 7f6d6d18f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:52.500+0000 7f6d6d18f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:52Z" 2023-10-12T01:56:58.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.515 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.515 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.516 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.516 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.516 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.517 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.517 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.517 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.518 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.518 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.519 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.521 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.521 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.521 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.521 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.522 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.522 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.524 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.525 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.525 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.525 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.525 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.528 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.528 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.529 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.529 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.529 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.532 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.532 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.532 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.533 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.533 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.533 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.533 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.534 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.534 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:52Z" 2023-10-12T01:56:58.534 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.535 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:57.623+0000 7f0c163f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:55:57.624+0000 7f0c163f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:55:57Z" 2023-10-12T01:56:58.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.559 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.559 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.560 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.560 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.560 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.561 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.561 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.562 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.562 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.562 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.563 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.564 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.565 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.565 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.566 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.566 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.566 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.569 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.569 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.570 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.570 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.570 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.573 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.573 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.573 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.574 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.574 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.576 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.577 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.577 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.577 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.578 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.578 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.578 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.579 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.579 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:55:57Z" 2023-10-12T01:56:58.579 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.580 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:02.746+0000 7f850e06f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:02.747+0000 7f850e06f780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:02Z" 2023-10-12T01:56:58.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.608 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.608 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.608 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.609 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.609 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.609 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.610 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.610 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.610 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.612 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.613 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.613 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.614 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.614 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.615 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.617 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.617 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.618 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.618 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.618 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.621 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.621 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.622 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.622 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.622 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.625 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.625 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.625 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.626 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.626 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.626 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.627 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.627 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.627 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:02Z" 2023-10-12T01:56:58.627 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.628 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.629 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:07.871+0000 7f4c95e53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:07.872+0000 7f4c95e53780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:07Z" 2023-10-12T01:56:58.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.652 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.652 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.653 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.653 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.653 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.654 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.654 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.655 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.655 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.656 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.656 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.658 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.658 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.659 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.659 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.661 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.662 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.662 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.663 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.665 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.665 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.666 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.666 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.669 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.669 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.670 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.670 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.670 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.671 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.671 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.671 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.671 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:07Z" 2023-10-12T01:56:58.672 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.672 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:12.994+0000 7fbdf14f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:12.995+0000 7fbdf14f9780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:12Z" 2023-10-12T01:56:58.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.697 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.697 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.698 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.698 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.698 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.699 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.699 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.700 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.700 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.701 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.703 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.703 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.703 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.703 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.706 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.707 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.707 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.707 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.710 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.711 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.711 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.711 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.712 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.714 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.714 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.715 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.715 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.715 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.716 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.716 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.716 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.716 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:12Z" 2023-10-12T01:56:58.717 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.717 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:18.120+0000 7f7b43624780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:18.121+0000 7f7b43624780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:18Z" 2023-10-12T01:56:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.741 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.741 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.742 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.742 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.743 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.743 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.743 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.743 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.744 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.745 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.746 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.746 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.746 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.747 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.749 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.750 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.750 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.751 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.751 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075564.060671_1047.1", 2023-10-12T01:56:58.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:44.060671Z", 2023-10-12T01:56:58.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.753 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.753 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.753 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.754 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.754 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.755 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.756 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.757 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.757 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.757 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.758 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.758 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.758 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.758 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.759 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:18Z" 2023-10-12T01:56:58.759 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.759 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075564.060671_1047.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:44.060671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075564.060671_1047.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1697075690.713117_1099.1 target=1_1697075564.060671_1047.1 2023-10-12T01:56:58.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:23.244+0000 7faecc2f0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:23.246+0000 7faecc2f0780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.783 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.784 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.784 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.784 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.785 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.785 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.785 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.786 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.787 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.788 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.788 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.789 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.789 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.789 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.789 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.790 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.792 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.792 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.792 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.792 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:58.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:58.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.795 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.795 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.796 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.796 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.799 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.799 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.799 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.800 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.800 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.800 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.801 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.801 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.801 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.801 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075690.713117_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:54:50.713117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:58.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:58.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:58.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-12T01:56:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:58.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:58.810 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:58.810 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.810 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.811 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:23 GMT 2023-10-12T01:56:58.811 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:58.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:58.812 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y= 2023-10-12T01:56:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y=', 'Content-Length': '0'} 2023-10-12T01:56:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c04f8b404b65b32-0065275247-2dbc-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:58.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj3 2023-10-12T01:56:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj3 2023-10-12T01:56:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:58.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj3 2023-10-12T01:56:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:58.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:58.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:58.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:58.816 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:58.816 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-12T01:56:58.816 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:58.817 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:23 GMT 2023-10-12T01:56:58.817 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj3 2023-10-12T01:56:58.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:58.818 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:f+j/5b2A1O9qI9OLn9AAQxyqVAg= 2023-10-12T01:56:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:f+j/5b2A1O9qI9OLn9AAQxyqVAg='} 2023-10-12T01:56:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7db5aee452e8c3-0065275247-2dbc-a1'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:58.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:58.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:58.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:58.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:58.822 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:58.822 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.822 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.823 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:23 GMT 2023-10-12T01:56:58.823 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:58.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:58.824 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y= 2023-10-12T01:56:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:/NJH8tfzc7UDu6XTuV6GwGpEZ9Y=', 'Content-Length': '0'} 2023-10-12T01:56:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094cd1041729a9b9b-0065275247-2dbc-a1'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:58.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj4 2023-10-12T01:56:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj4 2023-10-12T01:56:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-12T01:56:58.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj4 2023-10-12T01:56:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-12T01:56:58.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-10-12T01:56:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-12T01:56:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:58.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:58.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:58.828 INFO:tasks.rgw_multisite_tests:PUT 2023-10-12T01:56:58.828 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-12T01:56:58.829 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-12T01:56:58.829 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:23 GMT 2023-10-12T01:56:58.829 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj4 2023-10-12T01:56:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:58.830 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:xU30QjPSaZnfxVCndE1/64hSxSs= 2023-10-12T01:56:58.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Thu, 12 Oct 2023 01:56:23 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:xU30QjPSaZnfxVCndE1/64hSxSs='} 2023-10-12T01:56:58.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077df1f2bd37449eb-0065275247-2dbc-a1'), ('Date', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:58.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=f8a42833-a8a4-4b21-a670-8605a7b80b21 2023-10-12T01:56:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.659356Z" 2023-10-12T01:56:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:12:11.679864Z" 2023-10-12T01:56:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:54:50.713117Z" 2023-10-12T01:56:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T00:23:08.940549Z" 2023-10-12T01:56:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-12T01:56:58.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:23.513+0000 7fb3fd8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-10-12T01:56:23.514+0000 7fb3fd8d7780 20 RGW-SYNC:meta: read sync status 2023-10-12T01:56:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-10-12T01:56:58.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:58.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:58.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-10-12T01:56:58.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-10-12T01:56:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-10-12T01:56:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-10-12T01:56:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-12T01:56:58.862 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.862 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.862 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.863 INFO:tasks.rgw_multisite_tests: "period": "f8a42833-a8a4-4b21-a670-8605a7b80b21", 2023-10-12T01:56:58.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.864 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.864 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.864 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.865 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.865 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.659356_324.1", 2023-10-12T01:56:58.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.866 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-12T01:56:58.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.659356Z", 2023-10-12T01:56:58.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.868 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.868 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.868 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.869 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1697073131.679864_578.1", 2023-10-12T01:56:58.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:12:11.679864Z", 2023-10-12T01:56:58.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.871 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.872 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.872 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.872 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.872 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.873 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.873 INFO:tasks.rgw_multisite_tests: "marker": "1_1697075690.713117_1099.1", 2023-10-12T01:56:58.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:54:50.713117Z", 2023-10-12T01:56:58.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.875 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.875 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.876 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-12T01:56:58.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1697070188.940549_124.1", 2023-10-12T01:56:58.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T00:23:08.940549Z", 2023-10-12T01:56:58.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-12T01:56:58.879 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.879 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.880 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.880 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.881 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.881 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-12T01:56:58.881 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-12T01:56:58.882 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.882 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.882 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.883 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697073131.659356_324.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.659356Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697073131.679864_578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:12:11.679864Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697075690.713117_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:54:50.713117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697070188.940549_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T00:23:08.940549Z', 'realm_epoch': 2}}] 2023-10-12T01:56:58.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697073131.659356_324.1', 1: '1_1697073131.679864_578.1', 2: '1_1697075690.713117_1099.1', 3: '1_1697070188.940549_124.1'} 2023-10-12T01:56:58.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-12T01:56:58.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-10-12T01:56:58.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032902", 2023-10-12T01:56:58.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:56:23.329656Z" 2023-10-12T01:56:58.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052655", 2023-10-12T01:56:58.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:56:23.306608Z" 2023-10-12T01:56:58.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:58.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:58.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:58.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:58.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:58.912 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.912 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.912 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.913 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.913 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:58.913 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.913 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.914 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.914 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.914 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.915 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.915 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.915 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:58.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.916 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.917 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.917 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.918 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.918 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.919 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:58.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:58.920 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.920 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.920 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.920 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.921 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.922 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.922 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.923 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.923 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.924 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.924 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.925 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.925 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.925 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.927 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.927 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.928 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.928 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.928 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.929 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:58.929 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:58.929 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.930 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:23Z" 2023-10-12T01:56:58.930 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.930 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:58.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:58.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:58.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:58.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:58.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:58.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:58.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:58.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:28Z" 2023-10-12T01:56:58.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:58.952 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.952 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.953 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.953 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.953 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:58.954 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.954 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.954 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.954 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.955 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.955 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.956 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:58.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.957 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.957 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.957 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.958 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.958 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.958 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.959 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.959 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:58.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:58.960 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.961 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.961 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.961 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:58.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.962 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.962 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.963 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.964 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.964 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.964 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.964 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:58.965 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.965 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.965 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:58.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.967 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.967 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.968 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:58.968 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.968 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:58.968 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:58.969 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:58.969 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.969 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:28Z" 2023-10-12T01:56:58.970 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:58.970 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:58.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:58.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:58.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:58.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:58.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:58.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:58.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:58.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:58.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:58.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:58.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:58.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:58.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:58.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:58.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:58.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:58.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:33Z" 2023-10-12T01:56:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:58.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:58.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:58.992 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:58.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:58.993 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:58.993 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:58.993 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:58.994 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.994 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:58.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.995 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.995 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:58.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.996 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:58.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:58.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:58.997 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:58.997 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:58.997 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:58.997 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:58.998 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:58.998 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:58.998 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:58.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:58.999 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:58.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.000 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.001 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.001 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:59.001 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.002 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.002 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.004 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.004 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.004 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:59.005 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.005 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.005 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.006 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.007 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.007 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.007 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.008 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:59.008 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.008 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:59.008 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:59.009 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:59.009 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.009 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:33Z" 2023-10-12T01:56:59.010 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:59.010 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:59.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:59.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:59.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:59.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:59.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:59.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:59.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:59.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:39Z" 2023-10-12T01:56:59.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:59.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:59.034 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:59.034 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:59.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:59.035 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:59.035 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.036 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:59.036 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.036 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:59.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.037 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.037 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.038 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:59.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.039 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.039 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.039 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.040 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:59.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.040 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.041 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.041 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:59.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.042 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.043 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.043 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.043 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:59.043 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.044 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.044 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.046 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.046 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.046 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:59.047 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.047 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.048 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.049 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.049 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.050 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:59.050 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.050 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:59.051 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:59.051 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:59.051 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.052 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:39Z" 2023-10-12T01:56:59.052 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:59.052 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:59.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:59.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:59.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:59.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:59.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:59.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:59.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:59.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:59.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:59.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:59.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:59.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:59.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:59.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:59.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:44Z" 2023-10-12T01:56:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:59.075 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:59.076 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:59.076 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:59.076 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:59.077 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:59.077 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.077 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:59.077 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.078 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:59.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.078 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.079 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.079 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:59.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.080 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.080 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.081 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.081 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:59.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.082 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.082 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.083 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:59.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.084 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.084 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.084 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.084 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:59.085 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.085 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.085 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.087 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.087 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.088 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.088 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:59.088 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.089 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.089 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.089 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.090 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.091 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.091 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:59.091 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.092 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:59.092 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:59.092 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:59.093 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.093 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:44Z" 2023-10-12T01:56:59.093 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:59.093 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:59.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:59.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:59.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:59.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:59.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:59.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:59.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:59.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:59.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:59.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:59.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:59.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:59.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:59.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:59.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:59.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:59.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:59.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:59.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:59.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:59.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:49Z" 2023-10-12T01:56:59.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:59.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:59.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:59.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:59.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:59.116 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:59.116 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:59.117 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:59.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.117 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.118 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032761", 2023-10-12T01:56:59.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.118 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:59.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:51:00.382943Z" 2023-10-12T01:56:59.119 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.120 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.120 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.120 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:59.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.121 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.121 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-10-12T01:56:59.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.122 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:59.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.256140Z" 2023-10-12T01:56:59.123 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.123 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.123 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:59.124 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.124 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.125 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.126 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.126 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.127 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.127 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:59.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.128 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.128 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.130 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.130 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.130 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:59.131 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.131 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:59.131 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:59.132 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:59.132 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.132 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:49Z" 2023-10-12T01:56:59.133 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:59.133 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032761', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:51:00.382943Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.256140Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:59.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032761', 1: '00000000000000000000:00000000000000052514', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032902 target=00000000000000000000:00000000000000032761, shard=1 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514 2023-10-12T01:56:59.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-12T01:56:59.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-12T01:56:59.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-12T01:56:59.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-12T01:56:59.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-10-12T01:56:59.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-10-12T01:56:59.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-10-12T01:56:59.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-10-12T01:56:59.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 12355881351314922323 2023-10-12T01:56:59.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-10-12T01:56:59.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-10-12T01:56:59.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000032902", 2023-10-12T01:56:59.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-10-12T01:56:59.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:56:23.329656Z" 2023-10-12T01:56:59.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-10-12T01:56:59.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000052655", 2023-10-12T01:56:59.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-10-12T01:56:59.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:56:23.306608Z" 2023-10-12T01:56:59.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-10-12T01:56:59.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-10-12T01:56:59.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-10-12T01:56:59.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-10-12T01:56:59.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-10-12T01:56:59.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-10-12T01:56:59.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-10-12T01:56:59.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-10-12T01:56:59.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-10-12T01:56:59.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-10-12T01:56:59.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-10-12T01:56:59.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-10-12T01:56:59.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-10-12T01:56:59.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-10-12T01:56:54Z" 2023-10-12T01:56:59.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-10-12T01:56:59.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-12T01:56:59.155 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-12T01:56:59.155 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-12T01:56:59.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-12T01:56:59.156 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-12T01:56:59.156 INFO:tasks.rgw_multisite_tests: "instance_id": 12355881351314922323 2023-10-12T01:56:59.156 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.156 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-12T01:56:59.157 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.157 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-12T01:56:59.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.158 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.158 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032902", 2023-10-12T01:56:59.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.158 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-12T01:56:59.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:56:23.329656Z" 2023-10-12T01:56:59.159 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.160 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.160 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-12T01:56:59.160 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052655", 2023-10-12T01:56:59.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.162 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-12T01:56:59.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:56:23.306608Z" 2023-10-12T01:56:59.163 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.163 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.163 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.163 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-12T01:56:59.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.164 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.164 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044605", 2023-10-12T01:56:59.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:48:15.266091Z" 2023-10-12T01:56:59.166 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.166 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.166 INFO:tasks.rgw_multisite_tests: { 2023-10-12T01:56:59.167 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-12T01:56:59.167 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-12T01:56:59.167 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-12T01:56:59.168 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046501", 2023-10-12T01:56:59.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-12T01:56:59.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-12T01:56:59.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-12T01:56:59.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-12T01:52:45.296985Z" 2023-10-12T01:56:59.169 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.170 INFO:tasks.rgw_multisite_tests: } 2023-10-12T01:56:59.170 INFO:tasks.rgw_multisite_tests: ] 2023-10-12T01:56:59.170 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.170 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-12T01:56:59.171 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-12T01:56:59.171 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-12T01:56:59.171 INFO:tasks.rgw_multisite_tests: }, 2023-10-12T01:56:59.172 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-12T01:56:54Z" 2023-10-12T01:56:59.172 INFO:tasks.rgw_multisite_tests:} 2023-10-12T01:56:59.172 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032902', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-12T01:56:23.329656Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-12T01:56:23.306608Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044605', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:48:15.266091Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046501', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-12T01:52:45.296985Z'}}] 2023-10-12T01:56:59.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000032902', 1: '00000000000000000000:00000000000000052655', 2: '00000000000000000000:00000000000000044605', 3: '00000000000000000000:00000000000000046501'} 2023-10-12T01:56:59.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-12T01:56:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/ 2023-10-12T01:56:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/ 2023-10-12T01:56:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:59.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/ 2023-10-12T01:56:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:59.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-12T01:56:59.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:59.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:59.177 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:59.177 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.178 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.178 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:54 GMT 2023-10-12T01:56:59.178 INFO:tasks.rgw_multisite_tests:/wafmoa-74/ 2023-10-12T01:56:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:59.179 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:B4+zqU5aims+9f5mvqQzjQrXXLg= 2023-10-12T01:56:59.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:B4+zqU5aims+9f5mvqQzjQrXXLg=', 'Content-Length': '0'} 2023-10-12T01:56:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000526bf317909aa66e-0065275266-2db0-a2'), ('Content-Length', '0'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj3 2023-10-12T01:56:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj3 2023-10-12T01:56:59.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj3 2023-10-12T01:56:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:59.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:59.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:59.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:59.183 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:59.183 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.184 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.184 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:54 GMT 2023-10-12T01:56:59.184 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj3 2023-10-12T01:56:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:59.185 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:Mf9+RigBQgICzMRUXBrnsG5yND8= 2023-10-12T01:56:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:Mf9+RigBQgICzMRUXBrnsG5yND8=', 'Content-Length': '0'} 2023-10-12T01:56:59.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a819f2665e8cfd19-0065275266-2db0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj3 2023-10-12T01:56:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj3 2023-10-12T01:56:59.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-12T01:56:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj3 2023-10-12T01:56:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:59.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:59.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:59.189 INFO:tasks.rgw_multisite_tests:GET 2023-10-12T01:56:59.189 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.189 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.190 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:54 GMT 2023-10-12T01:56:59.190 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj3 2023-10-12T01:56:59.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:59.191 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:dyN2nWGNEaTetDcQ4Fp5H6oazKY= 2023-10-12T01:56:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:dyN2nWGNEaTetDcQ4Fp5H6oazKY=', 'Content-Length': '0'} 2023-10-12T01:56:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Thu, 12 Oct 2023 01:56:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '677f5a1d-3ab4-4b5f-af15-9890c2c79648:wafmoa-74:677f5a1d-3ab4-4b5f-af15-9890c2c79648.11009.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000527beb72414f01cd-0065275266-2db0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:59.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/wafmoa-74/obj4 2023-10-12T01:56:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/wafmoa-74/obj4 2023-10-12T01:56:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-12T01:56:59.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /wafmoa-74/obj4 2023-10-12T01:56:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-12T01:56:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-12T01:56:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-10-12T01:56:59.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-12T01:56:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-12T01:56:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-12T01:56:59.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-12T01:56:59.195 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-12T01:56:59.195 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.195 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.195 INFO:tasks.rgw_multisite_tests:Thu, 12 Oct 2023 01:56:54 GMT 2023-10-12T01:56:59.196 INFO:tasks.rgw_multisite_tests:/wafmoa-74/obj4 2023-10-12T01:56:59.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-12T01:56:59.196 INFO:tasks.rgw_multisite_tests:AWS L12R7HDAL0U7FKTOBSDN:ORoyy0Eyt6fZsyTMJHba1URL9rc= 2023-10-12T01:56:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Thu, 12 Oct 2023 01:56:54 GMT', 'Authorization': 'AWS L12R7HDAL0U7FKTOBSDN:ORoyy0Eyt6fZsyTMJHba1URL9rc=', 'Content-Length': '0'} 2023-10-12T01:56:59.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000098659b738bed71bf-0065275266-2db0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Thu, 12 Oct 2023 01:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-12T01:56:59.197 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-12T01:56:59.198 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.198 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-12T01:56:59.199 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 5786.428s 2023-10-12T01:56:59.199 INFO:tasks.rgw_multisite_tests: 2023-10-12T01:56:59.199 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=3) 2023-10-12T01:56:59.201 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-12T01:56:59.423 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c98563a7a5fa4bb4875205fddcabd2a4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-12T01:56:59.425 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-10-12T01:56:59.437 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-10-12T01:56:59.489 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-12T01:56:59.499 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-12T01:56:59.499 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:56:59.512 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:56:59.498+0000 7f58e2f2d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 121734) UID: 0 2023-10-12T01:56:59.512 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-10-12T01:56:59.498+0000 7f58e71967c0 -1 shutting down 2023-10-12T01:57:00.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-12T01:57:05.601 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-12T01:57:05.603 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-10-12T01:57:05.627 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T01:57:05.700 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-12T01:57:05.700 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:05.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-10-12T01:57:05.741 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:57:05.741+0000 7fa16b063700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 182396) UID: 0 2023-10-12T01:57:05.742 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-10-12T01:57:05.741+0000 7fa16f2cc7c0 -1 shutting down 2023-10-12T01:57:11.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-12T01:57:11.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-12T01:57:11.802 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-12T01:57:11.803 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-10-12T01:57:11.826 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T01:57:11.900 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-12T01:57:11.900 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:11.901 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2023-10-12T01:57:11.899+0000 7fd5243e3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 121069) UID: 0 2023-10-12T01:57:11.901 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2023-10-12T01:57:11.899+0000 7fd52864c7c0 -1 shutting down 2023-10-12T01:57:16.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-12T01:57:17.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-12T01:57:18.002 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-12T01:57:18.003 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-10-12T01:57:18.206 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T01:57:18.282 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-12T01:57:18.283 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:18.284 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T01:57:18.282+0000 7feb92254700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 122364) UID: 0 2023-10-12T01:57:18.293 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2023-10-12T01:57:18.282+0000 7feb964bd7c0 -1 shutting down 2023-10-12T01:57:22.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-12T01:57:22.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-12T01:57:24.385 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-12T01:57:24.386 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-10-12T01:57:24.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/vault-root-token 2023-10-12T01:57:24.483 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-12T01:57:24.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-10-12T01:57:25.387 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-12T01:57:25.387 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-12T01:57:25.388 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-10-12T01:57:25.388 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:25.427 INFO:tasks.ceph.c2.osd.0:Stopped 2023-10-12T01:57:25.428 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-10-12T01:57:25.428 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:28.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2023-10-12T01:57:28.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-12T01:57:31.530 INFO:tasks.ceph.c2.osd.1:Stopped 2023-10-12T01:57:31.531 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-10-12T01:57:31.531 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:31.592 INFO:tasks.ceph.c2.osd.2:Stopped 2023-10-12T01:57:31.593 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-12T01:57:31.593 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-10-12T01:57:31.594 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:31.619 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-10-12T01:57:31.619 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-12T01:57:31.620 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-10-12T01:57:31.620 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:31.679 INFO:tasks.ceph.c2.mon.a:Stopped 2023-10-12T01:57:31.679 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-12T01:57:31.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-12T01:57:31.905 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T01:57:31.905 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-10-12T01:57:32.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T01:57:32.549 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-10-12T01:57:32.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T01:57:32.695 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-10-12T01:57:32.820 INFO:tasks.ceph:Archiving mon data... 2023-10-12T01:57:32.822 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/data/c2.mon.a.tgz 2023-10-12T01:57:32.822 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-12T01:57:32.841 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.hS3Ur6kUDK 2023-10-12T01:57:32.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.hS3Ur6kUDK 2023-10-12T01:57:33.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.hS3Ur6kUDK 2023-10-12T01:57:33.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 has been restored 2023-10-12T01:57:33.114 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.hS3Ur6kUDK is 360KB 2023-10-12T01:57:33.210 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.hS3Ur6kUDK 2023-10-12T01:57:33.228 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-12T01:57:33.228 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T01:57:33.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T01:57:33.294 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-12T01:57:33.304 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-10-12T01:57:33.657 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-12T01:57:33.657 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-12T01:57:33.658 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-10-12T01:57:33.658 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:33.702 INFO:tasks.ceph.c1.osd.0:Stopped 2023-10-12T01:57:33.703 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-10-12T01:57:33.703 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:33.745 INFO:tasks.ceph.c1.osd.1:Stopped 2023-10-12T01:57:33.746 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-10-12T01:57:33.746 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:33.794 INFO:tasks.ceph.c1.osd.2:Stopped 2023-10-12T01:57:33.795 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-12T01:57:33.795 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-10-12T01:57:33.796 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:33.819 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-10-12T01:57:33.819 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-12T01:57:33.820 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-10-12T01:57:33.820 INFO:teuthology.orchestra.run:waiting for 300 2023-10-12T01:57:33.878 INFO:tasks.ceph.c1.mon.a:Stopped 2023-10-12T01:57:33.879 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-12T01:57:33.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-12T01:57:33.948 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T01:57:33.948 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-10-12T01:57:34.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T01:57:34.407 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-10-12T01:57:34.508 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T01:57:34.508 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-10-12T01:57:34.582 INFO:tasks.ceph:Archiving mon data... 2023-10-12T01:57:34.582 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/data/c1.mon.a.tgz 2023-10-12T01:57:34.583 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-10-12T01:57:34.601 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.v99KPTU2Mo 2023-10-12T01:57:34.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.v99KPTU2Mo 2023-10-12T01:57:34.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.v99KPTU2Mo 2023-10-12T01:57:34.863 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.v99KPTU2Mo is 405KB 2023-10-12T01:57:34.960 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.v99KPTU2Mo 2023-10-12T01:57:34.977 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-12T01:57:34.978 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T01:57:35.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-12T01:57:35.053 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-12T01:57:35.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-12T01:57:35.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-12T01:57:35.112 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-12T01:57:35.129 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-12T01:57:35.204 INFO:tasks.ceph:Archiving crash dumps... 2023-10-12T01:57:35.206 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi112/crash 2023-10-12T01:57:35.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-12T01:57:35.235 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi118/crash 2023-10-12T01:57:35.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-12T01:57:35.277 INFO:tasks.ceph:Compressing logs... 2023-10-12T01:57:35.277 DEBUG:teuthology.orchestra.run.smithi112:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T01:57:35.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T02:09:19.134 INFO:tasks.ceph:Archiving logs... 2023-10-12T02:09:19.135 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi112/log 2023-10-12T02:09:19.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T02:09:35.446 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi118/log 2023-10-12T02:09:35.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T02:10:55.766 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-12T02:10:55.777 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-12T02:10:55.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T02:10:55.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T02:10:55.880 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-12T02:10:55.890 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-12T02:10:55.891 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T02:10:55.894 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T02:10:55.911 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-10-12T02:10:55.912 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2023-10-12T02:10:55.917 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T02:10:55.917 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-10-12T02:10:55.917 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 -184us[ -184us] +/- 91ms 2023-10-12T02:10:55.918 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 +1074us[+1076us] +/- 65ms 2023-10-12T02:10:55.918 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 61 -899us[ -897us] +/- 61ms 2023-10-12T02:10:55.919 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T02:10:55.951 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T02:10:55.952 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2023-10-12T02:10:55.952 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 -300us[ -300us] +/- 90ms 2023-10-12T02:10:55.953 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 7 377 124 +979us[ +947us] +/- 64ms 2023-10-12T02:10:55.953 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -1045us[-1045us] +/- 61ms 2023-10-12T02:10:55.953 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T02:10:55.954 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-12T02:10:55.964 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-12T02:10:55.964 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-12T02:10:56.018 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T02:10:56.120 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T02:10:56.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T02:10:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T02:10:56.289 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T02:10:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T02:10:56.323 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-12T02:10:56.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-12T02:10:56.344 INFO:teuthology.task.internal:Duration was 7567.335197 seconds 2023-10-12T02:10:56.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-12T02:10:56.355 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-12T02:10:56.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T02:10:56.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T02:10:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T02:10:56.437 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T02:10:56.877 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-12T02:10:56.877 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2023-10-12T02:10:56.878 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T02:10:56.902 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2023-10-12T02:10:56.903 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T02:10:56.928 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-12T02:10:56.928 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T02:10:56.946 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T02:10:57.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-12T02:10:57.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T02:10:57.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T02:10:57.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-12T02:10:57.135 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-12T02:10:57.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T02:10:57.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T02:10:57.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-12T02:10:57.261 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T02:10:57.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T02:10:57.288 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2023-10-12T02:10:57.305 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2023-10-12T02:10:57.336 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T02:10:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T02:10:57.394 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T02:10:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T02:10:57.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-12T02:10:57.424 INFO:teuthology.task.internal:Transferring archived files... 2023-10-12T02:10:57.425 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi112 2023-10-12T02:10:57.425 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T02:10:57.473 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157/remote/smithi118 2023-10-12T02:10:57.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T02:10:57.512 INFO:teuthology.task.internal:Removing archive directory... 2023-10-12T02:10:57.513 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T02:10:57.515 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T02:10:57.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-12T02:10:57.577 INFO:teuthology.task.internal:Not uploading archives. 2023-10-12T02:10:57.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-12T02:10:57.590 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-12T02:10:57.590 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T02:10:57.593 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T02:10:57.608 INFO:teuthology.orchestra.run.smithi112.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 02:10 /home/ubuntu/cephtest 2023-10-12T02:10:57.623 INFO:teuthology.orchestra.run.smithi118.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 02:10 /home/ubuntu/cephtest 2023-10-12T02:10:57.624 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-12T02:10:57.636 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-12T02:10:57.678 INFO:teuthology.nuke:Checking targets against current locks 2023-10-12T02:10:57.704 DEBUG:teuthology.nuke:shortname: smithi112 2023-10-12T02:10:57.705 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T02:10:57.730 DEBUG:teuthology.nuke:shortname: smithi118 2023-10-12T02:10:57.731 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T02:10:57.735 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 23:55:30.444168', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2JLp2+vfzHVyxL6Acrw4wk1jHYH8MuiL4DWD9tM4S7ijSMYlbEw5+h08dafiwWGRp9Fogc9JG7tebz2ws0vSE='} 2023-10-12T02:10:57.762 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_18:32:33-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7422157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 23:55:30.442643', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZJXmqjfBSt/G/tZUHbg1vA/GOsb80hVRDsm9AHTjNE85QX13Lf8BegOL5Bji68NC+JX6KnRItmilTlMA9TtVA='} 2023-10-12T02:10:57.790 INFO:teuthology.orchestra.console.smithi112:Power off 2023-10-12T02:10:57.792 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T02:10:57.817 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2023-10-12T02:10:57.818 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:10:57.836 INFO:teuthology.orchestra.console.smithi118:Power off 2023-10-12T02:10:57.837 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T02:10:57.849 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-10-12T02:10:57.862 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2023-10-12T02:10:57.863 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:10:57.888 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-12T02:11:01.851 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:11:01.889 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:11:01.963 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-10-12T02:11:02.001 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-12T02:11:05.967 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:11:06.002 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T02:11:06.080 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2023-10-12T02:11:06.115 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2023-10-12T02:11:06.182 INFO:teuthology.orchestra.console.smithi112:Power off completed 2023-10-12T02:11:06.216 INFO:teuthology.orchestra.console.smithi118:Power off completed 2023-10-12T02:11:06.320 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 7567.335196971893 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c98563a7a5fa4bb4875205fddcabd2a4 status: fail success: false 2023-10-12T02:11:06.321 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T02:11:06.404 INFO:teuthology.run:FAIL